11/20/2018

The Modern Data Ecosystem is the New Kid in Town

What better than to create something out of nothing.  As kids we learn to build with blocks.  Such a rudimentary game, yet the underlying principles are timeless.

You've got different shapes and sizes.  You can stack high, wide, whatever you want.  These building blocks are the underlying foundation for today's shift in technology.

ETL is a block.  So are source systems.  Staging databases too.  And Data Warehouses.  Self contained units of substance.  You can build each block traditionally, on-premise.  Or you can take pieces and apply the same block in the cloud.

Source system on-premise.  ETL to the Cloud, just lift and shift (yes another overused term today).  There are lots of connectors to lots of data sources run from on-premise or Cloud.  Just pick up the block and move it to the Cloud.  Store in Data Lake, use bits or pieces as needed.

From there, the Staging block fits nicely, many options to choose from, same with Data Warehouse.  Merge with other data sets.  Options are limitless.

Each stage along the trail is simply a block, and the way technology is evolving, it doesn't matter where it lives.  Technology has broken each piece into re-usable patterns with no predefined home.

A good example is SQL Server Managed Instance, basically took the high end version and ported to the Cloud.  And of course, there are other options that run as a service.  There is a solution for almost every scenario.  Or take bits and pieces and apply the to your unique environment.

But not only that.  That's the easy part.  All the other blocks play nicely with the other parts.  Its a smorgasborg of various components that adapt and connect with other pieces of technology.  The vendors are staying a step (many steps) ahead of the developers, with high level architecture in mind.  The new Modern Data Warehouse has links to traditional Data Warehouse, yet the MDW has grown exponentially.

It really is mind blowing when you think back to the days of single reporting tool, against a single database, with a single select statement, running against a production database, slowing down the entire application for the entire company, is someone running a large query or month end in production?.

Those who work in data, and carved out a niche on a single specific technology, will need to revamp their game.  DBAs, Query Tuning, Report Developers.  The world of data is in completely new territory, and there's no turning back.  The Modern Data Ecosystem is the new kid in town.