THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

Jamie Thomson

This is the blog of Jamie Thomson, a data mangler in London working for Dunnhumby

Archiving Azure Automation logs to Azure BLOB Storage

For the past few months at work I’ve been diving headlong into a service on Microsoft Azure called Azure Automation which is, to all intents and purposes, a service for hosting and running Powershell scripts (properly termed “runbooks”). I’ve got a lot to say and a lot to share about Azure Automation and this is the first such post.

Each time Azure Automation runs a runbook (a running instance of a runbook is termed a job) it stores a log of that job however that log isn’t readily available as a log file on a file system as one might be used to. There are two ways to access the log, either via the Azure Management Portal at http://manage.windowsazure.com or programatically using Powershell cmdlets. Both are handy however I wanted the logs available as a file in Azure BLOB Storage so that they could be easily viewed and archived (in the future we’ll be building Hive tables over the top of the logs – remind me to blog about that later once its done). I have written a runbook called Move-AzureAutomationRunbookLogs that will archive logs for a given runbook and for a given time period into a given Azure BLOB Storage account and I’ve made it available on the Azure Automation gallery:

image

https://gallery.technet.microsoft.com/scriptcenter/Archive-Azure-Automation-898a1aa8

It does pretty much what it says on the tin so if you’re interested hit the link and check it out. For those that are simply interested in reading the code I’ve provided a screenshot of it below too (suggestions for how to copy-paste Powershell code into Live Writer so that it keeps its formatting are welcome!!). There’s a screenshot of the output too if that’s the sort of thing that floats your boat.

After this archiving runbook has run you’ll find two folders under the root folder that you specified:

image

and inside each of those is a folder for each runbook whose logs you are collecting (the folder contains a file per job). The “job” folder contains a short summary of each job however the “joboutput” folder contains the good stuff – the actual output from the run.

One last point, if you’re going to use this runbook I highly recommend doing so in conjunction with a technique outlined by Eamon O’Reilly at Monitoring Azure Services and External Systems with Azure Automation. Eamon’s post describes a technique for running operations such as these on a schedule and its something that I‘ve found to be very useful indeed.

Hope this is useful!

@Jamiet

image

image

Published Tuesday, November 25, 2014 11:09 PM by jamiet
Filed under: ,

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

David said:

Hi, I am getting the number of messages returned from Get-AzureAutomationJobOutput by using this code. However there are only ever 100 messages returned, whereas if I look in the job history I can see 143 pages with 25 messages in each one.

How can I get all messages, not just the first 4 pages(100 messages)?

$theJobOutput = Get-AzureAutomationJobOutput -AutomationAccountName $using:AutomationAccountName -Stream Any -Id $_.Id -Verbose

               Write-Verbose "There are $($theJobOutput.Count) messages in job output" -verbose

               $theJobOutput | `

                   Select JobStreamId,

                       Type,

                       @{Name='Text';Expression={$_.Text -replace @("`n",';')}},

                       Time,

                       AutomationAccountName,

                       Id                                                    | `

April 23, 2015 3:06 AM
 

David said:

Turns out if you don't specify the -StartTime flag, it gets you the first 100 messages. If you want more than that then you need to specify a later -StartTime to get the later set of 100 messages.

April 23, 2015 5:52 PM

Leave a Comment

(required) 
(required) 
Submit

This Blog

Syndication

Privacy Statement