THE SQL Server Blog Spot on the Web

Welcome to - The SQL Server blog spot on the web Sign in | |
in Search

Davide Mauri

A place for my thoughts and experiences the Microsoft Data Platform

Changing the BI architecture. From Batch to Real-time, from Bulk Load to Message Processing

In a world of Microservices, CQRS and Event Sourcing is more and more common to have the requirement that the BI/BA solution you’re developing is able to deal with incoming information (more precisely, messages and events) in almost real time.

It’s actually a good exercise to try to understand how you can turn your “classic” batch-based solution into a message even if you’re still following the batch approach because this new approach will force you to figure out how to deal with incremental and concurrent update. Problems that can help you to renew and refactor your exiting ETL solution to make it ready for the future. I really believe in the idea of continuous improvement which means that every “x” months you have to totally review an entire process of the existing solution, in order to see how it can be improved (this can can mean: make it faster, or cheaper, or easier to maintain and so on).
It’s my personal opinion that if everything could be managed using event and messages, even the ETL process would be *much* more simpler and straightforward that what typically is today, and to start to go on that road, we need to stop to think in batches.

This approach is even more important in the cloud since it allows a greater efficiency (and favor usage of PaaS instead of IaaS) and helps to have a cheaper solution. In the workshop I’m going to deliver at SQL Nexus I’ll show that this, today, is something that can be easily done on Azure.

All of  this also perfectly fits in the Lambda Architecture, a generic architecture for building real-time business intelligence and business analytic solution.

If you’re intrigued by these ideas, or you’re simply facing the problem to move the existing BI solution in the cloud and/or making it less batch and more real-time, the “Reference Big Data Lambda Architecture in Azure” at SQL Nexus at the beginning of May is what you’re looking for.

Here’s the complete agenda. Near 7 hours of theory and a lot demos to show how well everything blend together and with practical information that allows you to start to use what you’ve learned right from the day after:

  • Introduction to Lambda Architecture
  • Speed Layer:
    • Event & IoT Hubs
    • Azure Stream Analytics
    • Azure Machine Learning
  • Batch Layer:
    • Azure Data Lake
    • Azure Data Factory
    • Azure Machine Learning
  • Serving Layer:
    • Azure Data Warehouse / or Azure SQL
    • Power BI

See you in Copenhagen!


In case you’re wondering, everything is also possible on-prem, obviously with different technologies. Way less cool, but who cares, right? We’re here to do our job with the best solution for the customer, and even if it’s not the coolest one, it may well do it’s job anyway. Yeah, I’m talking of SSIS, pretty old right now, but still capable of impressive things. Especially if you use it along with Service Broker or RabbitMQ, in order to create a real-time ETL solution.

Published Monday, April 25, 2016 6:28 PM by Davide Mauri

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS



Ian Pike said:


Good article and I'm particularly interested in finding out more in this area. Unfortunately can't make SQL Nexus as I'm at SQL Bits, but hopefully you are in the UK again and doing this as a session at some point.

Great if you ever make it up to the North West of England and Manchester.


April 26, 2016 4:47 AM

abx said:


May 31, 2018 11:06 PM

dongdong8 said:



June 29, 2018 3:30 AM

linying123 said:


July 16, 2018 8:40 PM

dongdong8 said:



July 23, 2018 11:43 PM

qqq said:

August 16, 2018 12:33 AM

chenjinyan said:


August 22, 2018 11:15 PM

shenyuhang said:


August 23, 2018 10:31 PM

kakakaoo said:

October 8, 2018 2:04 AM

chenjinyan said:



October 9, 2018 6:50 PM

chenqiuying said:


October 10, 2018 6:33 PM

kakakaoo said:


November 8, 2018 1:51 AM

Leave a Comment


About Davide Mauri

Director of Software Development & Cloud Infrastructure @ Sensoria, an innovative smart garments and wearable company. After more than 15 year playing with the Microsoft Data Platform, with a specific focus on High Performance databases, Business Intelligence, Data Science and Data Architectures, he's now applying all his skills to IoT, defining architectures to crunch numbers, create nice user experiences and provide meaningful insights, all leveraging Microsoft Azure cloud. MVP on Data Platform since 2006 he has a very strong background development and love both the ER model and OO principles. He is also a fan of Agile Methodology and Automation, which he tries to apply everywhere he can, to make sure that "people think, machines do".

This Blog


Privacy Statement