DevOps is getting a lot of attention in SQL Server and Developer communities these days. Two friends have blogged about the outage and accompanying data loss experienced by GitLab:
I’m sure there are other posts, these are the two I read recently. Like Brent and Mike, my first impression on hearing about the incident was: I admire their transparency. It’s on-going. GitLab posted about the incident and they were, in my opinion, incredibly honest with themselves. That’s integrity. And it’s worth its weight in californium.
Brent’s post translated the episode into SQL Server-speak, which is awesome. Mike’s post, also awesome, used the GitLab incident to reinforce some DevOps best practices.
Mike emphasizes the balance needed between the Dev side of DevOps and the Ops side. I’m going to kick that up a notch.
If you are not balancing Development and Operations efforts, you are not practicing DevOps.
What is DevOps?
I (and likely Brent and Mike) have seen a lot of organizations struggle with this question. I’ve helped companies implement DevOps (Enterprise Data & Analytics helps organizations implement DevOps – learn more here). In my experience, companies that struggle with DevOps most often misunderstand the difference between DevOps and Agile methodologies. When well-intentioned, good-hearted, intelligent people confuse DevOps and Agile, they try to implement something like Scrum in Operations. In my humble opinion and based on experience, this either fails outright or morphs (in order to survive) into something other than Agile Operations. (Note: One can label their operations activities “Agile.” That doesn’t mean they’re practicing agile.)
Operations is its own practice. Even in DevOps. Can we use some of the same tools and best practices designed for managing and monitoring software development in operations? Goodness yes. But that doesn’t make Operations agile.
“What Does DevOps Mean in Operations, Then, Andy?”
Automation. Scripted and documented, idempotent (re-executable without harming state), repeatable automation.
DevOps documentation includes but is not limited to vendor and internal team documentation, wikis, knowledge bases, how-to guides, and run books. Run books
may should include practices, policies, and procedures. Additional Operations documentation will cover Outage Response Management which will include on-call schedules and escalation policy and procedures.
Most Operations teams use Service Desk ticketing software such as Jira Service Desk or ZenDesk. This too is part of Operations documentation.
Creating scripts and/or using tools to manage previously-manual or menial tasks – that’s automation. Automation impacts both Development and Operations in the DevOps enterprise. Regarding data-related technology, tools like Business Intelligence Markup Language (Biml) save SSIS package development time and improve quality. For managing SSIS, DevOps enterprises should take a look at the DILM Suite – a collection of tools and utilities that surface SSIS-related configurations, generate scripts, and automate deployments between Development, Test, QA, and Production instances of SSIS Catalogs.
Creating and maintaining this level of Operations documentation costs time and money. It impacts time to market. But it is as necessary as unit testing software.
I’m a huge fan of Agile methodologies. I can produce witnesses – ask anyone who’s worked with me. I’ve led many Agile Data Warehouse and Agile Business Intelligence projects. I like Agile because it’s faster and cleaner than waterfall methodologies. I like Agile because it places decision-making power for development where it belongs – in the hands of the software developers (a concept Agile borrows from Kanban, the Japanese just-in-time scheduling system).
Creating and maintaining much of the Operations documentation is the responsibility of the development team. Your Scrum Board (or software development project plan) needs a marker for this documentation. More importantly, the project needs to be given the time to create (or update) this documentation. Granted, not all documentation needs to be complete before the software goes live. Run books describing execution, monitoring, and Operational responses, though? That information is vital at go-live.
How much does it cost to build this level of documentation? It depends.
One thing’s for sure: It’s cheaper than a single catastrophic failure.
You might like working with Enterprise Data & Analytics because we grok DevOps and SSIS.
Broken References in the SSIS Catalog
SSIS Academy: Using the SSIS Catalog Day 1 - Create the Catalog and Deploy
SSIS Academy: Using the SSIS Catalog Day 2 - Package Execution and Monitoring
SSIS Academy: Using the SSIS Catalog Day 3 - SSIS Configuration
SSIS Catalog Management
SSIS Lifecycle Management
IESSIS1: Immersion Event on Learning SQL Server Integration Services – April 2017, Chicago
IESSIS2: Immersion Event on Advanced Integration Services – Oct 2017, Chicago
From Zero to Biml - 19-22 Jun 2017, London