Contact

admin

About Me · Send mail to the author(s) E-mail · Twitter

At GROSSWEBER we practice what we preach. We offer trainings for modern software technologies like Behavior Driven Development, Clean Code and Git. Our staff is fluent in a variety of languages, including English.

Feed Icon

Tags

Open Source Projects

Archives

Blogs of friends

Now playing [?]

  1. Marilyn Manson – Slutgarden
  2. Marilyn Manson – Ka-Boom Ka-Boom
  3. Marilyn Manson – (s)AINT
  4. Marilyn Manson – Use Your Fist and Not Your Mouth
  5. Marilyn Manson – Doll-Dagga Buzz-Buzz Ziggety-Zag
Audioscrobbler/Last.fm

ClustrMap

How We Practice Continuous Integration And Deployment With MSDeploy

Posted in Build | Deployment | PowerShell at Saturday, February 06, 2010 6:35 PM W. Europe Standard Time

About two years ago I quit the pain of CruiseControl.NET’s XML hell and started using JetBrains TeamCity for Continuous Integration. While being a bit biased here, I have to admit that every JetBrains product I looked at is absolutely killer and continues to provide productivity on a daily basis.

I’ve been a fan of Continuous Integration ever since. I figured the next step in improving our practice was not only to automate building/compiling/testing the application, but also deploy it either by clicking a button or based on a schedule. For example, updates to this blog’s theme and the .NET Open Space web sites are automated by clicking the “Run” button on my local TeamCity instance.

Deployment Build Configurations in TeamCity

Compare that button click to what we are forced to do manually for some projects at work. Every time we roll out a new version someone will:

  • Build the deployment package with TeamCity.
  • Download the deployment package, which is usually a ZIP containing the application and database migrations.
  • RDP into the production server.
  • Upload the deployment package.
  • Shut down the web application, Windows services, etc.
  • Overwrite the binaries and configuration files with the current versions from the deployment package.
  • Sometimes we have to match up and edit configuration files by hand.
  • Upgrade the database by executing *.sql files containing migrations in SQL Server Management Studio.
  • Restart web application and Windows services, etc.
  • Hope fervently that everything works.

I believe you can imagine that the manual process outlined has a lot of rope to hang yourself with. An inexperienced developer might simply miss a step. On top of that, implicit knowledge of which files need to be edited increases the bus factor. From a developer and business perspective you don’t want to deal with such risks. Deployment should be well documented, automated and easy to do.

Deployment Over Network Shares Or SSH

When I first looked into how I could do Continuous Deployment there were not many free products available on the Windows platform. In a corporate environment you could push your application to a Windows network share and configure the web application through scripts running within a domain account’s security context.

A different story is deployment over an internet connection. You would want to have a secure channel like a SSH connection to copy files remotely and execute scripts on the server. This solution requires SSH on the server and tools from the Putty suite (i.e. psftp) to make the connection. I had such a setup in place for this blog and the .NET Open Space web sites, but it was rather brittle: psftp doesn’t provide synchronization, integration with Windows services like IIS is not optimal and you’re somewhat limited in what you can do on the server.

MSDeploy

Last year, Microsoft released MSDeploy 1.0 which was updated to version 1.1 last week. MSDeploy is targeted to help with application deployment and server synchronization. In this article, I will focus on the deployment aspects exclusively. Considering the requirements for deployment, MSDeploy had everything I asked for. MSDeploy either

  • runs as the Web Deployment Agent Service providing administrators unrestricted access to the remote machine through NTLM authentication, or
  • runs as the Web Deployment Handler together with the IIS Management Service to let any user run a specified set of operations remotely.

Both types of connections can be secured using HTTPS, which is great and, in my opinion, a must-have.

I won’t go into the details of how MSDeploy can be set up because these are well documented. What I want to talk about what concepts we employ to deploy applications.

The Deployment Workflow

With about three months of experience with MSDeploy under our belts, we divide deployments into four phases:

  1. Initial, minimal manual preparation on the target server
  2. Operations to perform in preparation for the update
  3. Updating binaries
  4. Operations to perform after the update has finished

The initial setup to be done in phase 1 is a one-time activity that only occurs if we decide to provision a new server. This involves actions like installing IIS, SQL Server and MSDeploy on the target machine such that we can access it remotely. In phase 1 we also create web applications in IIS.

Further, we put deployments into two categories: Initial deployments and upgrade deployments. These only differ in the operations executed before (phase 2) and after (phase 4) the application files have been copied (phase 3). For example, before we can update binaries on a machine that is running a Windows service, we first have to stop that service in phase 2. After updating the binaries, that service has to be restarted in phase 4.

Over the last couple of weeks, a set of operations have been identified that we likely execute in phase 2 and 4.

Operation Description During Initial Deployment During Upgrade Before Or After Deployment
Set-WebAppOffline Shuts down a web application by recycling the Application Pool and creating App_Offline.htm No Yes Before
Set-WebAppOnline Deletes App_Offline.htm No Yes After
Create-Database Creates the initial database Yes No After
Update-Database Run migrations on an existing database No Yes After
Import-SampleData Imports sample data to an existing database for QA instances Yes No After
Install-Service Installs a Windows service, for example one that runs nightly reports Yes Yes After
Uninstall-Service Stops and uninstalls a Windows service No Yes Before

It’s no coincidence that the operations read like PowerShell Verb-Noun cmdlets. In fact, we run operations with PowerShell on the server side.

The deployment directory that will be mirrored between the build server and the production machine looks like the one depicted in the image to the right.

The root directory contains a PowerShell script that implements the operations above as PowerShell functions. These might call other scripts inside the deployment directory. For example, we invoke Tarantino (created by Eric Hexter and company) to have our database migrations done.

 

$scriptPath = Split-Path -parent $MyInvocation.MyCommand.Definition

# Change into the deployment root directory.
Set-Location $scriptPath

function Create-Database()
{
    & ".\SQL\create-database.cmd" /do_not_ask_for_permission_to_delete_database
}

function Import-SampleData()
{
    & ".\SQL\import-sample-data.cmd"
}

function Upgrade-Database()
{
    & ".\SQL\update-database.cmd"
}

function Install-Service()
{
    & ".\Reporting\deploy.ps1" Install-Service
    & ".\Reporting\deploy.ps1" Run-Service
}

function Uninstall-Service()
{
    & ".\Reporting\deploy.ps1" Uninstall-Service
}

function Set-WebAppOffline()
{
    Copy-Item -Path "Web\App_Offline.htm.deploy" -Destination "Web\App_Offline.htm" -Force
}

function Set-WebAppOnline()
{
    Remove-Item -Path "Web\App_Offline.htm" -Force
}

# Runs all command line arguments as functions.
$args | ForEach-Object { & $_ }

# Hack, MSDeploy would run PowerShell endlessly.
Get-Process -Name "powershell" | Stop-Process

The last line is actually a hack, because PowerShell 2.0 hangs after the script has finished.

Rake And Configatron

As you might remember from last week’s blog post we use Rake and YAML in our build scripts. Rake and YAML (with Configatron) allow us to

  • build the application,
  • generate configuration files for the target machine, thus eliminating the need to make edits, and
  • formulate MSDeploy calls in a legible and comprehensible way.

Regarding the last point, please consider the following MSDeploy command line that synchronizes a local directory with a remote directory (think phase 3). PowerShell operations will to be performed before (-preSync, phase 2) and after the sync operation (-postSyncOnSuccess, phase 4).

"tools/MSDeploy/msdeploy.exe" -verb:sync -postSyncOnSuccess:runCommand="powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command C:/Crimson/deploy.ps1 Create-Database Import-SampleData Install-Service Set-WebAppOnline ",waitInterval=60000 -allowUntrusted -skip:objectName=filePath,skipAction=Delete,absolutePath=App_Offline\.htm$ -skip:objectName=filePath,skipAction=Delete,absolutePath=\\Logs\\.*\.txt$ -skip:objectName=dirPath,skipAction=Delete,absolutePath=\\Logs.*$ -preSync:runCommand="powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command C:/Crimson/deploy.ps1 Set-WebAppOffline Uninstall-Service ",waitInterval=60000 -usechecksum -source:dirPath="build/for-deployment" -dest:wmsvc=BLUEPRINT-X86,username=deployer,password=deployer,dirPath=C:/Crimson

The command line is convoluted and overly complex, isn’t it? Now please consider the following Rake snippet that was used to generate the command line above.

remote = Dictionary[]
    
if configatron.deployment.connection.exists?(:wmsvc) and configatron.deployment.connection.wmsvc
    remote[:wmsvc] = configatron.deployment.connection.address
    remote[:username] = configatron.deployment.connection.user
    remote[:password] = configatron.deployment.connection.password
else
    remote[:computerName] = configatron.deployment.connection.address
end

preSyncCommand = "exit"
postSyncCommand = "exit"

if configatron.deployment.operations.before_deployment.any?
    preSyncCommand = "\"powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command #{"deploy.ps1".in(configatron.deployment.location)} #{configatron.deployment.operations.before_deployment.join(" ")} \""
end

if configatron.deployment.operations.after_deployment.any?
    postSyncCommand = "\"powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command #{"deploy.ps1".in(configatron.deployment.location)} #{configatron.deployment.operations.after_deployment.join(" ")} \""
end

MSDeploy.run \
    :tool => configatron.tools.msdeploy,
    :log_file => configatron.deployment.logfile,
    :verb => :sync,
    :allowUntrusted => true,
    :source => Dictionary[:dirPath, configatron.dir.for_deployment.to_absolute.escape],
    :dest => remote.merge({
        :dirPath => configatron.deployment.location
        }),
    :usechecksum => true,
    :skip =>[
        Dictionary[
            :objectName, "filePath",
            :skipAction, "Delete",
            :absolutePath, "App_Offline\\.htm$"
        ],
        Dictionary[
            :objectName, "filePath",
            :skipAction, "Delete",
            :absolutePath, "\\\\Logs\\\\.*\\.txt$"
        ],
        Dictionary[
            :objectName, "dirPath",
            :skipAction, "Delete",
            :absolutePath, "\\\\Logs.*$"
        ]
    ],
    :preSync => Dictionary[
        :runCommand, preSyncCommand,
        :waitInterval, 60000
    ],
    :postSyncOnSuccess => Dictionary[
        :runCommand, postSyncCommand,
        :waitInterval, 60000
    ]

It’s a small Rake helper class that transforms a Hash into a MSDeploy command line. That helper also includes console redirection that sends deployment output both to the screen and to a log file. The log file is also used to find errors that may occur during deployment (see below).

For your convenience, these are the relevant parts of the configuration, expressed in YAML and parsed with Configatron.

some_config:
  deployment:
    location: C:/Crimson
    operations:
      before_deployment: [Set-WebAppOffline, Uninstall-Service]
      after_deployment: [Create-Database, Import-SampleData, Install-Service, Set-WebAppOnline]
    connection:
      wmsvc: true
      address: BLUEPRINT-X86
      user: deployer
      password: deployer

What I Haven’t Talked About

What’s missing? An idea that got me interested was to partition the application into roles like database server, reporting server, web application server, etc. We mostly do single-server deployments, so I haven’t built that yet (YAGNI). Eric Hexter talks about application roles in a recent blog entry.

Another aspect where MSDeploy unfortunately doesn’t shine is error handling. Since we run important operations using the runCommand provider (used by -preSync and -postSyncOnSuccess) we would want to fail when something bad happens. Unfortunately MSDeploy, to this day, ignores errorlevels that indicate errors. So we’re back to console redirection and string parsing. This functionality is already in my MSDeploy helper for Rake, so you can rely on it to a certain degree. Manually scanning log files for errors, at least for the first couple of automated deployments is recommended, though.

Since we’re leveraging PowerShell on the server, why should we have to build the PowerShell script handling operations ourselves? I can imagine deploying the PowerShell-based PSake build tool and a PSake build script containing operations turned build targets. This will allow for common build script usage scenarios like task inspection (administrators would want that), having task dependencies, error handling and so on.

Wrapping Up

In this rather long post, I hope I could provide you with information how MSDeploy can be used to deploy your applications automatically. For us, over the last couple of weeks, MSDeploy in combination with our Rakefiles has helped us tremendously deploying an application that’s currently under development: The pain of delivering current versions to the customer has gotten a breeze.

Saturday, February 06, 2010 7:48:03 PM (W. Europe Standard Time, UTC+01:00)
Excellent post Alex.
Wednesday, March 17, 2010 10:30:34 AM (W. Europe Standard Time, UTC+01:00)
Great stuff! Thanks for putting it out there.
Thursday, April 29, 2010 8:03:49 PM (W. Europe Daylight Time, UTC+02:00)
This is great stuff Alex!
I am implementing something similar but can't get my powershell (2.0) script being run during preSync to force kill the process. I tried the hack you have but it just times out.
Was there something else you did to exit the ps process in addition to the hack in your script?
Thursday, April 29, 2010 8:25:22 PM (W. Europe Daylight Time, UTC+02:00)
Hey Mike,

No, that's just it: We kill all PowerShell processes when we're done. We're also using Posh 2.0. Perhaps -Force helps? An alternate solution might be to use taskkill.exe or pskill.exe from Sysinternals.
Thursday, April 29, 2010 10:26:02 PM (W. Europe Daylight Time, UTC+02:00)
I tried both those, but no joy. I can run other commands in preSync so it is definitely a ps issue. I wish they wouldn't have made msdeploy such a pain.
All comments require the approval of the site owner before being displayed.
(will show your gravatar icon)
 
[Captcha]Enter the code shown (prevents robots):

Live Comment Preview