THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

Jamie Thomson

This is the blog of Jamie Thomson, a data mangler in London working for Dunnhumby

Implementing a build and deploy pipeline for Cronacle

In my new role I am working on a system that makes heavy use of Redwood Software’s Cronacle. If you hit that link you’ll find a lot of marketing mumbo jumbo about Cronacle such as:

  • Cronacle gives you greater process consistency and quality
  • speed up your mission critical IT and business processes by as much as 90%

Read into that what you will, to me Cronacle is a scheduler and orchestration tool. That might come across as slightly belittling but its not intended to be, in my (limited) experience Cronacle does a couple of things and does them very well (certainly in comparison to the tool I’m more familiar with, SSIS, that’s a comparison I hope to explain more fully in a future blog post).

In the aforementioned post New year, new beginnings I said

My aim is to instil my beliefs about continuous integration, unit testing, failing fast, on-demand provisioning into my new team

which, to use a buzzword du jour, might well have been written as “My aim is to instil a devops culture into my team”. First step has been to integrate Cronacle into our build/deploy pipeline and in this post I’d like to describe how one goes about doing that.

Scene setting

We are using Cronacle v9. We are developing Job Chains and Job Definitions (aka Process Chains and Process Definitions) and it is those objects that we want to:

  1. store in a version control system (aka source control)
  2. compose into a single deployable artefact as part of a Continuous Integration (CI) build
  3. deploy to our various dev, test and prod environments in a repeatable fashion

Storing in VCS

The first pre-requisite to implementing a successful devops culture (in my opinion) is to use a version control system (VCS). I’m not fussy about which VCS one is using, as long as one uses one of them. We are using Subversion. My only stipulation is that no code in our build and deployment pipeline relies on us using Subversion because I don’t want to be tightly coupled to a particular technology.

Typically one stores human-readable files in a VCS and I was keen that we did the same for our Cronacle Job Chains and Job Definitions. Cronacle enables one to export objects:

SNAGHTMLa54c3a2

when selecting that option a binary file with a .car extension is downloaded:

image

.car file is shorthand for “Cronacle Jar”. Jar files are (as I understand them) used in the Java ecosystem as a mechanism for distributing stuff, they’re basically just .zip files with a different extension. Hence, get hold of some compression software that understands the .zip format (I recommend 7zip*), unzip your .car file and in the unzipped folder you’ll find a JobDefinition folder containing an XML file that defines the object that was exported:

image

That XML file is the human-readable file that we check in to our VCS.

Building

We use TeamCity on which to run our CI builds (its my first time using TeamCity in anger and I have become an unabashed fan) in combination with an internally-built build automation framework called PSP.Build that is built atop PowerShell and Psake. PSP.Build is loosely based on Lloyd Holman’s OneBuild and follows the principles that Lloyd sets out at Run the same build process everywhere. We also use Pester for unit testing our Powershell code - I’m loving me some Pester.

Building for Cronacle means reconstituting those XML files that we have in Subversion into a deployable .car file. The easiest way to do that is simply to zip them up for which, again, we are using 7zip. In addition there are some other files that need to be bundled into the .car file, one can see those files in the aforementioned .car file that is used for exporting from Cronacle.

image

You can simply take a copy those files, they don’t need changing. (We bundle them inside PSP.Build so that we always have them available to our build process.)

So in summary, in order to build a .car file you need three things:

  • XML files that define the Cronacle objects
  • A zip utility
  • com and META-INF folders

Once you have them it just takes a bit of PowerShell to bundle them up. Here’s a direct copy of our code that does that:

Remove-CarFiles -sourcePath $sourcePath #Simply deletes all the .car files it finds underneath $sourcePath

Get-ChildItem -path "$basePath" -Recurse | Where-Object {$_.Name -eq "CronacleCarIncludes"} |

       Get-ChildItem | Copy-Item -Destination "$sourcePath\$nameofFolderToBeZippedUp" -Recurse -Force #fetches "com" & "META-INF" folders

 

$zipperExe = Get-ChildItem -Path $basePath -Include "7za.exe" -Recurse |

       Select-Object -First 1 | foreach {$_.FullName} #find 7zip, which is bundled in our build framework, PSP.Build

if ($zipperExe -eq $null) {throw "7za.exe could not be found in $basepath"}

 

"Creating .car for sourcePath: $sourcePath\$nameofFolderToBeZippedUp" | Write-Verbose

Push-Location "$sourcePath\$nameofFolderToBeZippedUp"

& $zipperExe a -tzip "$nameofFolderToBeZippedUp.car" * #zip it all up!

Pop-Location 

[I’m slightly reluctant to show my PowerShell code in public as it nearly always get criticised, so be gentle OK Smile]

Deploying

So, the output from building all our Cronacle objects is a .car file. How do we deploy that .car file to our Cronacle cluster? Its actually quite simply, we execute our .car file using java.exe and pass in a few parameters. The Cronacle documentation informs us of the generic form of the command and gives an example:

image

(This documentation does not appear online anywhere so I can’t link to it I‘m afraid)

The smarts to say what to do with all the Cronacle object definitions is within that com folder that was mentioned above, hence its imperative that that com folder is included in the .car file.

Again, we wrote a load of PowerShell code to automate the process of building up the command-line and executing it, here’s that code if you’re interested:

function Invoke-DeployCronacleCarFile () {

    [CmdletBinding()]

    Param(

        [Parameter(Mandatory=$true)][System.IO.FileInfo]$carFilePath,

        [Parameter(Mandatory=$true)][System.IO.FileInfo]$javaExeFilePath,

        [Parameter(Mandatory=$true)][string]$server,

        [Parameter(Mandatory=$false)][string]$username,

        [Parameter(Mandatory=$false)][string]$password,

        [Parameter(Mandatory=$false)][string]$importRuleset

    )

    try {

        if ($carFilePath.Extension -ne ".car") {throw "File: '$carFilePath' is invalid. File must have a .car extension."}

        if (-not (Test-Path $carFilePath)) {throw "File: '$carFilePath' does not exist."}

       

        $cmd =  "$javaExeFilePath"

        $arguments = '-jar',"$carFilePath", '-server', "$server", '-txt','-ruleset',"$importRuleset"

 

        if (-not ($username -eq $null -or $username -eq "")){$arguments += '-username', "$username"}

        "cmd: $cmd" | Write-Verbose

        "arguments: $arguments" | Write-Verbose

        if (-not ($password -eq $null -or $password -eq "")){$arguments += "-password","$password"}

        Invoke-CommandLine -cmd $cmd -arguments $arguments

    }

    catch {

        throw

    }

}

function Invoke-CommandLine ([string]$cmd, [string[]]$arguments, [string]$errorMessage = "Error executing command: " + $cmd) {

    & $cmd @arguments #| write-host

  if ($LastExitCode -ne 0) {

    throw $errorMessage

  }

}

That may look slightly daunting but there’s not too much going on here. It takes the path to a .car file, the path to java.exe, server, username, password and an ImportRuleSet**, concatenates things together into a bunch of arguments, then sends it to Invoke-CommandLine to be executed.

Summary

That’s pretty much it. The nuts and bolts of this is relatively simple, you’re simply zipping your definitions up and deploying that zip file using java.exe.

Hope this helps.

@Jamiet 

* We bundle 7zip inside PSP.Build so we don’t have to rely on the actor who kicks off the build (which would be TeamCity or one of our team members) having it installed - I’m pretty fierce about reducing external dependencies for our build/deploy pipeline. If its a pre-requisite then it gets bundled into PSP.Build.

** An ImportRuleSet defines environment-specific settings. A discussion of ImportRuleSets is outside the scope of this article however if you’ve come this far you’ll probably already know whether you need to be using them or not.

Published Tuesday, March 10, 2015 9:47 AM by jamiet
Filed under:

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

No Comments

Leave a Comment

(required) 
(required) 
Submit

This Blog

Syndication

Privacy Statement