Popular Posts

5/16/2018

The Tail Now Wags the Dog

The tail doesn't wag the dog.  I first heard this statement from a very successful entrepreneur, that leased us office space.  He was on the phone with someone, and it clearly indicated that they were not doing what they were told.

Reporting was traditionally an afterthought.  The designers of the front end application, were concerned with business logic, front end design, performance, scalability, architecture and state.  They were not bothered with the back end database architecture, such that, reporting would be easy and seamless.  Hence, the tail doesn't wag the dog.

When I entered the field of IT, reporting was almost non existent.  I volunteered to do the reports, as the IT did not have the bandwidth to do so.  It was then, I discovered the hidden gem of data.  Sure, my earnings were but a fraction of the high priced project managers, the army of front end developers, business analysts, domain experts, etc.  Reporting was always left to the end, if there was time.  To say that data and reporting was undervalued, is an understatement.

Except.  I would get calls from senior executives, that were salivating at the mouth, to get their hands on data, to make decision, manage and get a pulse of the business.  Data was where its at.  But nobody else saw it.

So why wasn't the database constructed in such a way, to allow easy reporting?  Because data was an afterthought, although a required ingredient, the DBAs at the time were more concerned with excluding everyone from the database, so that it performed well, no hiccups to the system, etc.  Report writers weren't allowed access to the database, except when needed.  It was a black box to the business for sure.

That was around 1996 or so.  I imagine people were building OLAP cubes at that time, along with data warehouses, although I was not aware of such things.  And the reason EDW became popular are the reasons just mentioned.  The business needed access to the data, via slicing and dicing of the data.  This developed into high priced developers that specialized in building models, constructing cubes and the complex language of MDX.  This allowed for the "single version of the truth" as it was coined.  Except, these system costs a lot, for hardware, software, licenses, developers time, extended project timelines, not able to ingest all the data, and inflexibility to add new data sources.  It worked, at giant costs.

That was the workaround for lack of upfront design to the database structure.  Because the OLTP database main function was to store data, in a Relational Database, to handle large transaction, hence the name Transaction Databases.  So here can see the main disconnect.


Since that time, data has exploded, everyone and their grandmother has jumped aboard the data train express, reaping job security, good wages and agility in their careers.  

Incoming Data Scientist have mastered the latest cutting edge tools using Statistics, Analysis and Algorithms, yet they may not be aware of the pain and frustration stemming from the early days of data and reporting.

Yet compare the salaries of Data Scientists to traditional report writers or ETL developers and you'll see quite a contrast.  Old style reporting tools are not glamorous, yet they are still the bread and butter of many organizations, to produce Operational Reporting and State and Federal reporting.  ETL is still the most difficult process of the life cycle to transform data, move it here and there.

The new concept of Data Lake has introduced some good solutions to complex problems.  It allows you to report on the data where it sits, which could be in the Cloud or On-Premise.  You can model your data when needed, yet the data doesn't have to be ported from a variety of sources, you can gather all the data from a single location, the data lake.  This concept also reduces costs, so ALL the data can be stored.  This data is used by multiple departments including Operational Reporting, Cubes, Data Scientist, State and Federal Reporting as well as Self Service reporting.

Data is no longer stored in the bowels of an organization, requiring approval from God him/her self, to get a glimpse of the all knowing data.  Now you just need to submit a RIT, get access and the riches of data can be yours as well.

In other words, "the tail now wags the dog".