The Right Stuff

A couple of days ago Hadi Hariri posted his set of MSpec (Machine.Specifications) templates for ReSharper. ReSharper’s templating system helps you type less repeated code. On top of that, ReSharper templates are much richer when compared to what’s built into Visual Studio. Plus, you edit them with a decent editor instead of hacking XML files.

Like Hadi, I also created a couple of templates specific to MSpec over the course of the last year or so and found them often to reduce the amount of text I have to write. ReSharper Templates are divided into three categories, with at least one MSpec template in each.

Legend

• foo denotes an editable part of the template
• | denotes where the cursor will be put upon expansion

File Template

Basically, this is just a new C# file with a single MSpec context in it.

using System;

using Machine.Specifications;

namespace ClassLibrary1
{
  [Subject(typeof(Type))]
  public class When_Context
  {
    Establish context = () => { | };

    Because of = () => { };

    It should_ = () => { };
  }
}

Live Templates (a.k.a. Snippets)

Live Templates provide expansion of keyword-like identifiers. For example cw (Tab) will expand to Console.WriteLine();

Surround Templates

Surround Templates are useful when you want to wrap a block of code with other code, for example, an if statement (this is one that’s built-in).

Unit testing frameworks almost always have means to assert that are particular test should fail with a specific exception, for example by marking the test method with the ExpectedExceptionAttribute.

The MSpec way of handling/expecting exceptions is to surround the code in Because with Catch.Exception:

public class When_a_negative_amount_is_deducted
{
  static Exception Exception;
  static Account Account;

  Establish context =
    () => { Account = new Account(); };

  Because of =
    () => { Exception = Catch.Exception(() => Account.Deduct(-1)); };

  It should_fail =
    () => Exception.ShouldNotBeNull();
}

There’s a surround template named Catch.Exception that we can make wrap the call to Account.Deduct:

1. Create the context with just the Account field, Establish and the Because. Select the highlighted code.

   public class When_a_negative_amount_is_deducted
   {
     // Account field and Establish cut for brevity.
   
     Because of =
       () => { Account.Deduct(-1); };
   }

2. Press the shortcut for ReSharper | Edit | Surround With Template, select "Catch.Exception" from the list of available templates.

   public class When_a_negative_amount_is_deducted
   {
     // Account field and Establish cut for brevity.
   
     Because of =
       () => { Exception = Catch.Exception(() => { Account.Deduct(-1); }); };
   }

3. Navigate out of the Because field, for example by pressing the (End) key. Type fail (a Live Template, see above) and press (Tab).

   public class When_a_negative_amount_is_deducted
   {
     // Account field and Establish cut for brevity.
   
     Because of =
       () => { Exception = Catch.Exception(() => { Account.Deduct(-1); }); };
     fail(Tab)
   }

4. Marvel at the amount of code you didn't have to write.

   public class When_a_negative_amount_is_deducted
   {
     // Account field and Establish cut for brevity.
   
     Because of =
       () => { Exception = Catch.Exception(() => { Account.Deduct(-1); }); };
   
     It should_fail =
       () => Exception.ShouldNotBeNull();
   
     static Exception Exception;
   }

Download

About two years ago I quit the pain of CruiseControl.NET’s XML hell and started using JetBrains TeamCity for Continuous Integration. While being a bit biased here, I have to admit that every JetBrains product I looked at is absolutely killer and continues to provide productivity on a daily basis.

I’ve been a fan of Continuous Integration ever since. I figured the next step in improving our practice was not only to automate building/compiling/testing the application, but also deploy it either by clicking a button or based on a schedule. For example, updates to this blog’s theme and the .NET Open Space web sites are automated by clicking the “Run” button on my local TeamCity instance.

Compare that button click to what we are forced to do manually for some projects at work. Every time we roll out a new version someone will:

• Build the deployment package with TeamCity.
• Download the deployment package, which is usually a ZIP containing the application and database migrations.
• RDP into the production server.
• Upload the deployment package.
• Shut down the web application, Windows services, etc.
• Overwrite the binaries and configuration files with the current versions from the deployment package.
• Sometimes we have to match up and edit configuration files by hand.
• Upgrade the database by executing *.sql files containing migrations in SQL Server Management Studio.
• Restart web application and Windows services, etc.
• Hope fervently that everything works.

I believe you can imagine that the manual process outlined has a lot of rope to hang yourself with. An inexperienced developer might simply miss a step. On top of that, implicit knowledge of which files need to be edited increases the bus factor. From a developer and business perspective you don’t want to deal with such risks. Deployment should be well documented, automated and easy to do.

Deployment Over Network Shares Or SSH

When I first looked into how I could do Continuous Deployment there were not many free products available on the Windows platform. In a corporate environment you could push your application to a Windows network share and configure the web application through scripts running within a domain account’s security context.

A different story is deployment over an internet connection. You would want to have a secure channel like a SSH connection to copy files remotely and execute scripts on the server. This solution requires SSH on the server and tools from the Putty suite (i.e. psftp) to make the connection. I had such a setup in place for this blog and the .NET Open Space web sites, but it was rather brittle: psftp doesn’t provide synchronization, integration with Windows services like IIS is not optimal and you’re somewhat limited in what you can do on the server.

MSDeploy

Last year, Microsoft released MSDeploy 1.0 which was updated to version 1.1 last week. MSDeploy is targeted to help with application deployment and server synchronization. In this article, I will focus on the deployment aspects exclusively. Considering the requirements for deployment, MSDeploy had everything I asked for. MSDeploy either

• runs as the Web Deployment Agent Service providing administrators unrestricted access to the remote machine through NTLM authentication, or
• runs as the Web Deployment Handler together with the IIS Management Service to let any user run a specified set of operations remotely.

Both types of connections can be secured using HTTPS, which is great and, in my opinion, a must-have.

I won’t go into the details of how MSDeploy can be set up because these are well documented. What I want to talk about what concepts we employ to deploy applications.

The Deployment Workflow

With about three months of experience with MSDeploy under our belts, we divide deployments into four phases:

1. Initial, minimal manual preparation on the target server
2. Operations to perform in preparation for the update
3. Updating binaries
4. Operations to perform after the update has finished

The initial setup to be done in phase 1 is a one-time activity that only occurs if we decide to provision a new server. This involves actions like installing IIS, SQL Server and MSDeploy on the target machine such that we can access it remotely. In phase 1 we also create web applications in IIS.

Further, we put deployments into two categories: Initial deployments and upgrade deployments. These only differ in the operations executed before (phase 2) and after (phase 4) the application files have been copied (phase 3). For example, before we can update binaries on a machine that is running a Windows service, we first have to stop that service in phase 2. After updating the binaries, that service has to be restarted in phase 4.

Over the last couple of weeks, a set of operations have been identified that we likely execute in phase 2 and 4.

It’s no coincidence that the operations read like PowerShell Verb-Noun cmdlets. In fact, we run operations with PowerShell on the server side.

The deployment directory that will be mirrored between the build server and the production machine looks like the one depicted in the image to the right.

The root directory contains a PowerShell script that implements the operations above as PowerShell functions. These might call other scripts inside the deployment directory. For example, we invoke Tarantino (created by Eric Hexter and company) to have our database migrations done.

$scriptPath = Split-Path -parent $MyInvocation.MyCommand.Definition

# Change into the deployment root directory.
Set-Location $scriptPath

function Create-Database()
{
    & ".\SQL\create-database.cmd" /do_not_ask_for_permission_to_delete_database
}

function Import-SampleData()
{
    & ".\SQL\import-sample-data.cmd"
}

function Upgrade-Database()
{
    & ".\SQL\update-database.cmd"
}

function Install-Service()
{
    & ".\Reporting\deploy.ps1" Install-Service
    & ".\Reporting\deploy.ps1" Run-Service
}

function Uninstall-Service()
{
    & ".\Reporting\deploy.ps1" Uninstall-Service
}

function Set-WebAppOffline()
{
    Copy-Item -Path "Web\App_Offline.htm.deploy" -Destination "Web\App_Offline.htm" -Force
}

function Set-WebAppOnline()
{
    Remove-Item -Path "Web\App_Offline.htm" -Force
}

# Runs all command line arguments as functions.
$args | ForEach-Object { & $_ }

# Hack, MSDeploy would run PowerShell endlessly.
Get-Process -Name "powershell" | Stop-Process

The last line is actually a hack, because PowerShell 2.0 hangs after the script has finished.

Rake And Configatron

As you might remember from last week’s blog post we use Rake and YAML in our build scripts. Rake and YAML (with Configatron) allow us to

• build the application,
• generate configuration files for the target machine, thus eliminating the need to make edits, and
• formulate MSDeploy calls in a legible and comprehensible way.

Regarding the last point, please consider the following MSDeploy command line that synchronizes a local directory with a remote directory (think phase 3). PowerShell operations will to be performed before (-preSync, phase 2) and after the sync operation (-postSyncOnSuccess, phase 4).

"tools/MSDeploy/msdeploy.exe" -verb:sync -postSyncOnSuccess:runCommand="powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command C:/Crimson/deploy.ps1 Create-Database Import-SampleData Install-Service Set-WebAppOnline ",waitInterval=60000 -allowUntrusted -skip:objectName=filePath,skipAction=Delete,absolutePath=App_Offline\.htm$ -skip:objectName=filePath,skipAction=Delete,absolutePath=\\Logs\\.*\.txt$ -skip:objectName=dirPath,skipAction=Delete,absolutePath=\\Logs.*$ -preSync:runCommand="powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command C:/Crimson/deploy.ps1 Set-WebAppOffline Uninstall-Service ",waitInterval=60000 -usechecksum -source:dirPath="build/for-deployment" -dest:wmsvc=BLUEPRINT-X86,username=deployer,password=deployer,dirPath=C:/Crimson

The command line is convoluted and overly complex, isn’t it? Now please consider the following Rake snippet that was used to generate the command line above.

remote = Dictionary[]
    
if configatron.deployment.connection.exists?(:wmsvc) and configatron.deployment.connection.wmsvc
    remote[:wmsvc] = configatron.deployment.connection.address
    remote[:username] = configatron.deployment.connection.user
    remote[:password] = configatron.deployment.connection.password
else
    remote[:computerName] = configatron.deployment.connection.address
end

preSyncCommand = "exit"
postSyncCommand = "exit"

if configatron.deployment.operations.before_deployment.any?
    preSyncCommand = "\"powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command #{"deploy.ps1".in(configatron.deployment.location)} #{configatron.deployment.operations.before_deployment.join(" ")} \""
end

if configatron.deployment.operations.after_deployment.any?
    postSyncCommand = "\"powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command #{"deploy.ps1".in(configatron.deployment.location)} #{configatron.deployment.operations.after_deployment.join(" ")} \""
end

MSDeploy.run \
    :tool => configatron.tools.msdeploy,
    :log_file => configatron.deployment.logfile,
    :verb => :sync,
    :allowUntrusted => true,
    :source => Dictionary[:dirPath, configatron.dir.for_deployment.to_absolute.escape],
    :dest => remote.merge({
        :dirPath => configatron.deployment.location
        }),
    :usechecksum => true,
    :skip =>[
        Dictionary[
            :objectName, "filePath",
            :skipAction, "Delete",
            :absolutePath, "App_Offline\\.htm$"
        ],
        Dictionary[
            :objectName, "filePath",
            :skipAction, "Delete",
            :absolutePath, "\\\\Logs\\\\.*\\.txt$"
        ],
        Dictionary[
            :objectName, "dirPath",
            :skipAction, "Delete",
            :absolutePath, "\\\\Logs.*$"
        ]
    ],
    :preSync => Dictionary[
        :runCommand, preSyncCommand,
        :waitInterval, 60000
    ],
    :postSyncOnSuccess => Dictionary[
        :runCommand, postSyncCommand,
        :waitInterval, 60000
    ]

It’s a small Rake helper class that transforms a Hash into a MSDeploy command line. That helper also includes console redirection that sends deployment output both to the screen and to a log file. The log file is also used to find errors that may occur during deployment (see below).

For your convenience, these are the relevant parts of the configuration, expressed in YAML and parsed with Configatron.

some_config:
  deployment:
    location: C:/Crimson
    operations:
      before_deployment: [Set-WebAppOffline, Uninstall-Service]
      after_deployment: [Create-Database, Import-SampleData, Install-Service, Set-WebAppOnline]
    connection:
      wmsvc: true
      address: BLUEPRINT-X86
      user: deployer
      password: deployer

What I Haven’t Talked About

What’s missing? An idea that got me interested was to partition the application into roles like database server, reporting server, web application server, etc. We mostly do single-server deployments, so I haven’t built that yet (YAGNI). Eric Hexter talks about application roles in a recent blog entry.

Another aspect where MSDeploy unfortunately doesn’t shine is error handling. Since we run important operations using the runCommand provider (used by -preSync and -postSyncOnSuccess) we would want to fail when something bad happens. Unfortunately MSDeploy, to this day, ignores errorlevels that indicate errors. So we’re back console redirection and string parsing. This functionality is already in my MSDeploy helper for Rake, so you can rely on it to a certain degree. Manually scanning log files for errors, at least for the first couple of automated deployments is recommended, though.

Since we’re leveraging PowerShell on the server, why should we have to build the PowerShell script handling operations ourselves? I can imagine deploying the PowerShell-based PSake build tool and a PSake build script containing operations turned build targets. This will allow for common build script usage scenarios like task inspection (administrators would want that), having task dependencies, error handling and so on.

Wrapping Up

In this rather long post, I hope I could provide you with information how MSDeploy can be used to deploy your applications automatically. For us, over the last couple of weeks, MSDeploy in combination with our Rakefiles has helped us tremendously deploying an application that’s currently under development: The pain of delivering current versions to the customer has gotten a breeze.

We’ve been using Rake for quite a while at work. Sometime last year I sat down and converted our ~30 KB NAnt build scripts to Rake, a light-weight Ruby build framework with low friction and no XML. Since then I have written a bunch of Rake tasks to support our builds (we use TeamCity).

I started a bit out of the blue, because frameworks like Albacore didn’t exist back then and other .NET-specific task collections didn’t fit our needs or simply were inconvenient to use.

Without prior Ruby experience it was also a great opportunity to learn Ruby and give the language and design concepts a spin. I have to admit, I like the fluent style of Ruby, it’s almost like the language tries to stay out of your way.

YAML

Soon after I started building the first Rake script I needed to configure the build for different environments. Like: in production, we have to use another database server. You want to externalize such information into a configuration file. Having database connection strings hard coded in your application’s App.config will make tailoring the application for deployment tedious and error-prone. I’ve been there, and I don’t recommend it!

I came across YAML which is an intuitive notation for configuration files (amongst others):

development:
  database:
    server: (local)
    name: Indigo

qa:
  database:
    server: DB
    name: Indigo_QA

production:
  database:
    server: DB
    name: Indigo_Production

Is that legible? I think so!

We use the configatron Ruby Gem to read such files and dereference configuration information in the build script.

configatron.configure_from_yaml 'properties.yml', :hash => 'production'

puts configatron.database.server
# => 'DB'

puts configatron.database.name
# => 'Indigo_Production'

YAML’s “Inheritance”

Another useful aspect of YAML is that it supports a simple form of inheritance by merging hashes.

qa: &customer_config
  database:
    server: DB
    name: Indigo_QA

production:
  <<: *customer_config
  database:
    name: Indigo_Production

Unfortunately this kind of inheritance has some subtleties as it wouldn’t work as you would expect. I read the snippet above like the production configuration inherits all values from qa and overwrites the database.name. Let's see:

configatron.configure_from_yaml 'properties.yml', :hash => 'production'

puts configatron.database.server.nil?
# => true
# Huh? That should be "DB".

puts configatron.database.name
# => 'Indigo_Production'

Actually, there is an article describing the problem with merging hashes in YAML files  that I found after our build broke in interesting ways after loading an incomplete configuration. The proposed solution is either to duplicate all configuration information between qa and production, or to use more anchors (&foo) and merge references (<<: *foo). I think both clutters a YAML file unnecessarily.

Custom Inheritance

After I identifying why composition doesn’t work as one would expect let’s see what we can do about it.

I went with solution based on a convention that inheritance should be defined using a default_to configuration entry.

qa:
  database:
    server: DB
    name: Indigo_QA

production:
  default_to: qa
  database:
    name: Indigo_Production

The default_to entry in the production section refers to another section that the configuration will be inherited from. You could also build inheritance chains like production → qa → default and additionally use ordinary transparent YAML hash merges.

Instead of initializing configatron from the YAML file, we’ll preprocess the deserialized YAML (basically, a Hash), evaluate the configuration inheritance chain and then pass the Hash to configatron:

yaml = Configuration.load_yaml 'properties.yml', :hash => 'production', :inherit => :default_to
configatron.configure_from_hash yaml

puts configatron.database.server.nil?
# => false

puts configatron.database.server
# => 'DB'

puts configatron.database.name
# => 'Indigo_Production'

The code for the Configuration class that accounts for evaluating the inheritance chain is up on GitHub.

Over the last years I’ve been contributing to several Open Source software projects, just to name the most recent:

• Fluent NHibernate
• Machine.Specifications
• FubuMVC
• Rhino Mocks

All of the above projects follow their own style how to lay out the source code using indents. It seems like everybody has a different opinion you would have to cater for, for example:

• Indenting is done with tabs
• 4-space indents
• 2-space indents

Often times these conventions are implicit, you have to read the source code to see the actual style the authors follow. It is encouraged to apply these guidelines to your patches ensure they will be accepted.

I often switch between developing for projects, so before writing a single line of code I have to hit Visual Studio’s Tools | Options | Text Editor Options dialog and change the indent settings to match the project’s conventions.

This has become very tedious, additionally, I often forget to adjust the indent settings before writing code. (Perhaps I forget it because it’s so annoying.)

To scratch that itch I sat down and wrote some Visual Studio macros that apply the most commonly used settings:

Imports System
Imports EnvDTE
Imports EnvDTE80
Imports EnvDTE90
Imports System.Diagnostics

Public Module Fonts
    Sub TwoSpaces()
        Dim textEditor As Properties

        textEditor = DTE.Properties("TextEditor", "AllLanguages")
        textEditor.Item("IndentStyle").Value = vsIndentStyle.vsIndentStyleSmart
        textEditor.Item("TabSize").Value = 4
        textEditor.Item("IndentSize").Value = 2
        textEditor.Item("InsertTabs").Value = False
    End Sub

    Sub FourSpaces()
        Dim textEditor As Properties

        textEditor = DTE.Properties("TextEditor", "AllLanguages")
        textEditor.Item("IndentStyle").Value = vsIndentStyle.vsIndentStyleSmart
        textEditor.Item("TabSize").Value = 4
        textEditor.Item("IndentSize").Value = 4
        textEditor.Item("InsertTabs").Value = False
    End Sub

    Sub OneTab()
        Dim textEditor As Properties

        textEditor = DTE.Properties("TextEditor", "AllLanguages")
        textEditor.Item("IndentStyle").Value = vsIndentStyle.vsIndentStyleSmart
        textEditor.Item("TabSize").Value = 4
        textEditor.Item("IndentSize").Value = 4
        textEditor.Item("InsertTabs").Value = True
    End Sub

    Public Sub NormalFonts()
        SetFontSize(10)
    End Sub

    Public Sub LargeFonts()
        SetFontSize(14)
    End Sub

    Sub SetFontSize(ByVal size As Int32)
        Dim textEditor As Properties

        textEditor = DTE.Properties("FontsAndColors", "TextEditor")
        textEditor.Item("FontSize").Value = size
    End Sub
End Module

These macros are associated with toolbar buttons:

The first two buttons are associated to the LargeFonts and NormalFonts macros that set the editor font size. I like to invoke these when doing presentations. No more fiddling with Tools | Options to ensure your audience is able to read the code on the wall.

The last three buttons should be self-explaining, they’re to quickly set tabbed, two-space and four-space indents, respectively.

After listening to the Hanselminutes episode on Microsoft Solver Foundation (MSF) I decided it’s time to give it a shot today. Solver Foundation seems to be a solution to a set of constrained problems I sometimes face:

• Sharing costs and calculating minimal money transfers after trips with my friends, where each friend spent some money.
• Giving out questions to attendees of our User Group “Boot Camps”: Speakers prepare ~20 questions, ranging from easy to moderate levels. We assign each attendee an easy question and one to chew a bit upon. Further, every question should be given out to two attendees, so in case someone doesn’t make it to the meeting we’re still able to cover the question.

Something I don’t remember Scott Hanselman and his guest talking about is that Solver Foundation comes with an Excel Add-In that is supposed to make creating models easy easier, no code needed. Along with the “Solver Foundation for Excel Primer” document that is installed along with the binaries I figured Excel would be a good way to start looking into Solver Foundation.

After the MSI ran, I started Excel but didn’t find the Solver Foundation tab that’s advertised in the primer. The COM Add-Ins dialog said something about that the Add-In could not be loaded. Nice! Luckily the Event Viewer was more helpful in terms of error messages where I found this beauty of an exception:

Microsoft.VisualStudio.Tools.Applications.Runtime.CannotCreateCustomizationDomainException:
Customization could not be loaded because the application domain could not be created.
---> System.IO.FileLoadException: Could not load file or assembly 'MicrosoftSolverFoundationForExcel, Version=1.0.6.4890, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies.
The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

My first guess was that an old MSF assembly was referenced and I decided to go with an assembly binding redirect for excel.exe. Didn’t help. The next step was to get into the innards of VSTO deployment. What I found in the MicrosoftSolverFoundationForExcel.vsto and MicrosoftSolverFoundationForExcel.dll.manifest files wasn’t surpising: Several references to old versions of MSF. None of which were deployed by the MSI installer, so Excel trying to load such dependencies failed.

<assemblyIdentity name="MicrosoftSolverFoundationForExcel" version="1.0.8.6048"…
<assemblyIdentity name="MicrosoftSolverFoundationForExcel" version="1.0.6.4890"…

I updated both references to the match the installed version 2.0.2.8632, just to find myself faced with another error saying that the manifest’s digital signature is broken.

Now was time to contact my friend Lars Keller who is an expert in VSTO development. Lars told me that I would have to re-sign the .vsto and .manifest files to make the signature reflect my changes. The Office Development with Visual Studio blog has the full details.

1. I had to create a certificate that can be used for code signing:

   makecert -r -pe -n "CN=Your Name" -b 01/01/2010 -e 01/01/2099 -eku 1.3.6.1.5.5.7.3.3 -ss My

2. Export the certificate as a PFX file using certmgr.msc
3. Create a backup copy of the MSF Excel Add-In .manifest and .vsto files
4. Open a Visual Studio Command prompt and navigate to the manifest's location
5. Make edits to the manifest file correcting the assembly versions of MicrosoftSolverFoundationForExcel to 2.0.2.8632
6. Update the digital signatures for both the manifest and the VSTO file:

   mage.exe -update MicrosoftSolverFoundationForExcel.dll.manifest -CertFile <your-cert.pfx> -Password <cert-export-password>
   mage.exe -update MicrosoftSolverFoundationForExcel.vsto -appmanifest MicrosoftSolverFoundationForExcel.dll.manifest -CertFile <your-cert.pfx> -Password <cert-export-password>

7. Restart Excel, the Solver Foundation tab should be on the ribbon

(Tested with Office 2010 beta.)