Thursday, June 25, 2015

Trust No One Architectures

A Trust No One Architecture is one where each organizational unit minimizes accidental risk by owning as much of their processes as possible.  Companies end up with a Trust No One architecture where each sub-organization is most likely to meet its' goals if it controls as much of its development, technical and operational processes as possible. Each division / operational unit acts as an independent entity with loose coupling at the edges and just enough cooperation to meet the company goals. 

I recently attended a talk of a Departmental Information Officer for a large bank who said their software process accelerated and their business deliverables came in earlier when they pulled architecture, operations and infrastructure back from the corporate level to the department level. The bank traded costs, standards and duplicate work for time to market and agility. This was in strong opposition to the previous attempt at minimizing risk by centralizing functions.

Trust No One Architectures may be analogous to the agile concept of a cross functional team that owns all aspects of a deliverable. The team owns requirement, development, production, customer satisfaction, security and testing. The organizational team takes ownership of the project's success.  Scale that up to the 100-300 person organization and you end up with a self-contained mini company that attempts to minimize issues around external dependencies.

Tool for self-preservation or sign of organizational dis-function?

I always thought of Trust No One Architectures as a sign of dis-function.  I've come to realize it is really the only way many teams have any hope of meeting their business and technical needs without creating a system of unreasonable complexity and instability. Modern software continues to grow in essential complexity  because of ever expanding business requirements and ever increasing complex interactions.  

Concepts like Enterprise Architecture, Enterprise Services and Centralized Operations cyclically rise up and fall as they crash upon the rocks or organizational realities. Some claim SOA is a technology issue.  Public press tends to mostly describe the failures of SOA as organizational failures. SOA architectures are hard because they are technically hard.  They are brutally hard because they do not align with organizational structures.

Architecture Follows Organization

Developers and organizational anthropologists have known that software product form follows the structure of the organization that creates it.  Conway said this as far back as 1968.  The organization creates the accountability and incentive boundaries. 

Current software  shiny objects include distributed applications and micro services.  People rant about how services Amazon Services are the company's secret sauce.  This only works if people make it completely mandatory in a no if's and's or but's kind of way.  This is can only happen if companies organize around this.  A forward looking distributed architecture cannot be back ported into an existing organizational structure where teams attempt to minimize risk by owning as much as possible.

Orignally written 6/25/2015

Monday, June 1, 2015

Protect RabbitMQ data by encrypting the Mnesia database on Windows Server

RabbitMQ is one of the many caching and messaging tools that uses local disk persistent storage or as a backing store for in memory data.  These systems normally put data to disk in some format that is optimized for speed and not for security. Ex: RabbitMQ, ActiveMQ, Coherence, Gemfire, MongoDB.

This can cause issues when trying to comply with policies around protecting Personally Identifiable Information , making systems Payment Card Industry Data Security Standard (PCI DSS) compliant or when implementing S/Ox controls.

RabbitMQ Installation

We assume that you are running RabbitMQ under the local system account.  Users who run RabbitMQ under different accounts or in different locations must change certain commands or settings.  The RabbitMQ team has a good set of documentation on their web site.

Directory Encryption.

We'll use Encrypted File System (EFS), available with Microsoft operating systems, to encrypt the directories and files that contain the disk based information. EFS directories will be come unreadable by anyone other than the Local Service account.  This means you have to decide if you want to encrypt data directories, configuration directors or just individual files as they will be come unusable by anyone other than the service account.

Portions of this document are based on the following web postings.

General Process

  1. Stop RabbitMQ
  2. Download PsExec which is part of the awesome Sysinternals suite.  It is essentially a Windows version of remote commands and sudo.  
  3. Run the psexec command to encrypt the directory acting as the Local System account.
  4. Restart RabbitMQ

Enabling Encryption

  1. Download PsExec and unpack the zip file. The files can be run from the un-zipped archive without installation. Remember the path you put them on. I unzipped PSTools into \tools.
  2. Open an administrative command prompt.
  3. Stop the RabbitMQ service using the the services control panel. Find services via windows search or run services.msc from an administrative command prompt.
  4. Used Psexec to open a command prompt that is owned by the Local System account
    • <app-path>\psexec -sid cmd.exe
  5. Change focus to the new command prompt window.
    • Verify this command worked by typing whoami in the new command prompt window. It should say "nt authority\system"
  6. CD to the directory where your mnesia database is located. 
    • The default location is the AppData directory of the user id that installed RabbitMQ.  
    • On my machine it was in C:\Users\<userid>\AppData\Roaming\RabbitMQ\db\<clusternode>-mnesia
  7. Use the cipher command to encrypt the node's mnesia directory
    • cipher /e /s:rabbit@<machinename>-mnesia  
    • Ex:   cipher /e /s:rabbit@WIN8-MACBOOK-15-mnesia
    • You should see messages listing the files and directories that are encrypted
    • The directory will show in green in the file explorer on windows 8
  8. Start the RabbitMQ  service.
  9. Verify the service is working and accepting messages.


Move the rabbitmq data directory outside of the installer's AppData\Roaming\RabbitMQ folder.
  1. Open a rabbitmq command prompt
  2. rabbitmq-service.bat remove
  3. set RABBITMQ_BASE=<some_folder>
  4. rabbitmq-service.bat install
  5. rabbitmq-service.bat start
Note: I had to edit rabbitmq-defaults.bat to get this all configured correctly.  I'm not sure why

Reversing the process

You can revert to an un-encrypted mnesia database by using the cipher command with /d in place of /e

Performance Impact

My simple test showed no measurable performance difference when posting persistent messages to a local RabbitMQ server.  My test program publishes
  • 6800-6900 persistent 1500 byte messages per second with EFS enabled
  • 6800-6900 persistent 1500 byte messages per second with EFS disabled
using a local SSD in a quad core Macbook 15" mid 2012 running Windows 8. My test program is probably throttled somewhere else since that ran the CPUs at 15% and the disks even lower.

I was unable to meaningfully measure the the true CPU impact of this change.  

Operational Impacts

EFS is very easy to use in a situation where you don't expect to move files across systems outside of the applications using the data.  Backups and other system recovery tools may be rendered useless. 

The cipher command is run as the local system account. This means the directory is encrypted and owned by that account irrespective of whose AppData directory the database is actually installed in. I recommend moving your rabbit configuration , log files and mnesia database to some other location out side of some user account home directory.

Encryption certificates and recover keys may need to be retained or managed to facilitate data recovery or migration. Microsoft documents some of the key management issues in a TechNet article. The cipher command can be used to manage certificates and recovery keys.

Created 5/31/2015

Recommendations added 8/26/2015