THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

Davide Mauri

A place for my thoughts and experiences on SQL Server, Business Intelligence and .NET

Changing the BI architecture. From Batch to Real-time, from Bulk Load to Message Processing

In a world of Microservices, CQRS and Event Sourcing is more and more common to have the requirement that the BI/BA solution you’re developing is able to deal with incoming information (more precisely, messages and events) in almost real time.

It’s actually a good exercise to try to understand how you can turn your “classic” batch-based solution into a message even if you’re still following the batch approach because this new approach will force you to figure out how to deal with incremental and concurrent update. Problems that can help you to renew and refactor your exiting ETL solution to make it ready for the future. I really believe in the idea of continuous improvement which means that every “x” months you have to totally review an entire process of the existing solution, in order to see how it can be improved (this can can mean: make it faster, or cheaper, or easier to maintain and so on).
 
It’s my personal opinion that if everything could be managed using event and messages, even the ETL process would be *much* more simpler and straightforward that what typically is today, and to start to go on that road, we need to stop to think in batches.

This approach is even more important in the cloud since it allows a greater efficiency (and favor usage of PaaS instead of IaaS) and helps to have a cheaper solution. In the workshop I’m going to deliver at SQL Nexus I’ll show that this, today, is something that can be easily done on Azure.

All of  this also perfectly fits in the Lambda Architecture, a generic architecture for building real-time business intelligence and business analytic solution.

If you’re intrigued by these ideas, or you’re simply facing the problem to move the existing BI solution in the cloud and/or making it less batch and more real-time, the “Reference Big Data Lambda Architecture in Azure” at SQL Nexus at the beginning of May is what you’re looking for.

Here’s the complete agenda. Near 7 hours of theory and a lot demos to show how well everything blend together and with practical information that allows you to start to use what you’ve learned right from the day after:

  • Introduction to Lambda Architecture
  • Speed Layer:
    • Event & IoT Hubs
    • Azure Stream Analytics
    • Azure Machine Learning
  • Batch Layer:
    • Azure Data Lake
    • Azure Data Factory
    • Azure Machine Learning
  • Serving Layer:
    • Azure Data Warehouse / or Azure SQL
    • Power BI

See you in Copenhagen!

PS

In case you’re wondering, everything is also possible on-prem, obviously with different technologies. Way less cool, but who cares, right? We’re here to do our job with the best solution for the customer, and even if it’s not the coolest one, it may well do it’s job anyway. Yeah, I’m talking of SSIS, pretty old right now, but still capable of impressive things. Especially if you use it along with Service Broker or RabbitMQ, in order to create a real-time ETL solution.

Published Monday, April 25, 2016 6:28 PM by Davide Mauri

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

Ian Pike said:

Davide

Good article and I'm particularly interested in finding out more in this area. Unfortunately can't make SQL Nexus as I'm at SQL Bits, but hopefully you are in the UK again and doing this as a session at some point.

Great if you ever make it up to the North West of England and Manchester.

Ian

April 26, 2016 4:47 AM

Leave a Comment

(required) 
(required) 
Submit

About Davide Mauri

Davide Mauri - MCP, MCAD, MCDBA, MCT, MVP on SQL Server - has worked with SQL Server since version 6.5, and his interests cover the whole platform, from the Relational Engine to Analysis Services, from architecture definition to performance tuning. He also has a strong knowledge of XML, .NET and the Object Oriented Design principles, which allows him to have the correct vision and experience to handle development of complex business intelligence solutions. Having worked as a Microsoft Certified Teacher for many years, Davide is able to pass all his knowledge to his co-workers, allowing his team to deliver high-quality solutions. He currently works as a Mentor for SolidQ and can be found speaking in many Italian and internationals events.

This Blog

Syndication

Privacy Statement