2/23/2016

Hadoop Broke the Barriers of SQL and Data Warehouse

You may have read my post on Getting Bit by Hadoop.

The main theme was we were happy in our SQL world with 20 years of solid Data Warehouse technology.  Along came this new technology.  We gently placed it on the alter. And worshiped to it to solve all our problems.  After some years, the reality sunk in and it the hype dwarfed it's widespread adoption..

However, the fact that a new technology rose to fame is such short time is amazing.  The underlying concepts are still valid.  A mechanism to handle huge amounts of data, both structured and unstructured, in a distributed computing environment.

This opened the doors to an entire new industry.  And an entire new breed of developers and career opportunities.

It broke through the barriers and stranglehold of traditional SQL and Data Warehousing.  And made a name for itself.  Not an easy feat.

That set the foundation for new development.  As in Spark.  And a dozen other integrated frameworks bundled into the Apache Foundation.  Including security, streaming, machine learning, in-memory, data frames, integration with R and Python and Scala and other languages besides Java.

From the movie Moneyball, getting it in the teeth, the first one through the wall always gets bloody:



The bottom line, Hadoop was ahead of its time.  It paved the way for new technology, new ways to handle data and find insights.  And mostly, it put the software in the hands of everyday people.  Creating opportunity for the big guys as well as the small shops.

It's only going to grow from here.  Influx of developers.  Better adoption rates.  Ease of use.  Tighter integration.

Overall, I think the biggest contribution of Hadoop is to shine the light on the importance of data.  Data Driven companies are the future.  Integrating different data sources to derive insights is the new norm.  Building applications with the forethought of data, rather than afterthought, will propel business' into the future, as the other fall to the wayside.

Big Data has grown up.  Let's see where it goes from here.