THE SQL Server Blog Spot on the Web

Welcome to - The SQL Server blog spot on the web Sign in | |
in Search

Davide Mauri

A place for my thoughts and experiences the Microsoft Data Platform

Engineering BI (Or “On my PASS seminar”, prelude)

I’ve always stated that creating a software solution is half and art and half an engineering process. Just like building an house, or a car, or any mechanical or electrical device, the art is that part of the work where someone has to create something new to satisfy the customer need.

Architects are the figures that plays the artistic role. Sure, they don’t re-invent everything from scratch every time: Architectural Design Pattern are here to demonstrate that ideas can be reused successfully. But, still, architects put patterns and new ideas together in order create something new.

Engineers are the more practical side of this role-playing game and they are more concerned with processes and implementation of the solution so that it can feasible in first place, but also be maintainable, predictable, replicable and cost-effective.

In the software industry we have Architects but my feeling is that we’re missing the Engineers. This means that a lot of what is being produced from the software industry is more an art than a science and this drives to the fact that the quality, and the maintainability of the final product is completely dependent on the people working on it.  In addition to that, since there are no standards to follow, each one will do as it wish. If you put several people working together you’ll obtain a cauldron of different styles that leads inevitably to higher maintenance cost and low stability and predictability. (I don’t know if you have the same feeling, but I would like to have a feedback on this, if you care to share.)

Of course I’m generalizing a lot here – I complete discussion on this matter would easily fill a book – and there are several exception to this, but I think I can say without any doubt that if I had to make a picture of my perception of the software industry I’d draw a conglomerate of craftsman workshops and not a factory district.

In the last years I struggled a bit with this lack of the “engineering” discipline: particularly I found an evidence of this situation in BI projects. While there’s at least two well-know architectural approach (Kimball and Inmon), with other options rising (like the Data-Vault), but still there are no rules and tools that can make the creation of a BI solution more standard.

In order to be able to add people to an existing project quickly, being sure that they would create something that anyone of the existing team can easily understand, that the infrastructure of all of our BI solution would be similar in order to make its development faster and easier to maintain, I’ve defined a sets of rules and tools that helps to make the work easier and higher in quality.

The idea is that the rules will define the standards, and once we have the rules, we can also create the tools to apply them quickly. This will shorten a lot the time need to build the solution (which is the part with the lowest added-value) and leave much more time to design it (where there is the “core” value of it).

I’ve got this idea watching “How it’s made” on Science Discovery channel, watching people building the awesome Airbus 380. If such complex machine can be assembled by several people everyone doing a small part of the work, we should be able to do it also with BI solution! We “only” need rules that will drive organization of work and tools to make work easier and faster for anyone. Of course this also leads to a solution that is flexible enough to accomodote quickly new customers requests or changes of direction.

Apply the idea to Business Intelligence and you’ll have and “Engineered BI Solution” which will be both sound and agile. I like to call it “Adaptive”. That’s what I’ve already presented on PASS European Summit 2010 in an specific session that had very good feedbacks and that’s what I’m going to show in my “Creating a BI Solution from A to Z” post-conference seminar (8 hours of deep dive, blending theory and practice, architecture and engineering, making it a unique session of its kind), in my next BI Virtual Chapter meeting (more on this in another post) and also in the next posts of this series.

If you’re new to BI or you’ll starting a new BI project in the next months, or you simply want to learn more on this approach, don’t miss the opportunity, and share your feedback!

Published Saturday, August 28, 2010 4:27 PM by Davide Mauri

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS



TrackBack said:

August 28, 2010 9:50 AM

DM Unseen said:

If you look at DWH design tools like wherescape-red, or genereic DWH managament tools like Kalido, BIReady or Quipu there are defnitly solutions going the right direction. Note that comparing DWH architectures (like Inmon or Kimball) is difficult because they are often on a different level. Note that Data Vault is a recipy for creating the central EDW for eg an inmon or Kimball DWH, while Kimball is a recepy for conforming your data-marts. Also note that according to inmon the Data Vault is a best practice for the central EDW inmoon style.

August 29, 2010 9:54 AM

Davide Mauri said:

@DM Unseen: The products you mention are surely interesting and I have to say you a big "Thank you" beacuse you helped me to discover also BIReady that I didn't know and that seems to be very very interesting.

My feeling, anyway, remains the same. We should have rules that we define and that we determine are the correct ones for us, despite having a product that implement and enforces them or not. Otherwise we are too much bound to the tool we choose to use. I'd like to have something like the PBM for SQL Server. We know the rules (eg: data and log files should be on different volumes) and we can apply them manually. If there is a tool that help us (PBM in this case) we can automate the check and in same cases enforce their application automatically. In this way our rules will drive the development and the implementation and not the opposite, where a tool drives to rules.

Thanks a lot for your feedback!

August 30, 2010 4:20 AM

DM Unseen said:


This is possible, it is actually called Informationb Architecture. Information Architecture deals with the rules/principles like  Auditability, Standardisation, Integration, flexibility & usability etc. We then use Data Architecture to choose modeling techniques like start schemas, normalized or Anchor Modeling(Data Vault) & repositories like staging, central EDW or Data Marts to implement the constraints/principles from the Information Architecture. We also choose a LOAD (ELT/ELT) architecture based on the principles to get to an integrated EDW Data Architecture that fulfills all our requirments. This ia of course not as easy as a BPM tool;)

Tools like BIReady are indeed complex apps because they assume that there is "One (EDW/Data) Architecture to rule them all". Note that while DWH management tools are quite flexible by nature, they still cannot be universally applied and need to be evaluated against the required information architecture.

August 31, 2010 10:54 AM

Davide Mauri said:

@DM Useen

do you have some link to share regarding IA and BI/DWH? AFAIK is mostly relatated to the web and not related to BI/DWH specifically.

September 1, 2010 9:30 AM

DM Unseen said:


For understanding Information Architecture Principles you could look eg at

For an intersting Data Architecture that incoorporates BI you could look at

September 2, 2010 4:31 AM

Davide Mauri said:


September 2, 2010 8:34 AM

Leave a Comment


About Davide Mauri

Director of Software Development & Cloud Infrastructure @ Sensoria, an innovative smart garments and wearable company. After more than 15 year playing with the Microsoft Data Platform, with a specific focus on High Performance databases, Business Intelligence, Data Science and Data Architectures, he's now applying all his skills to IoT, defining architectures to crunch numbers, create nice user experiences and provide meaningful insights, all leveraging Microsoft Azure cloud. MVP on Data Platform since 2006 he has a very strong background development and love both the ER model and OO principles. He is also a fan of Agile Methodology and Automation, which he tries to apply everywhere he can, to make sure that "people think, machines do".

This Blog


Privacy Statement