5/23/2013

#BigData Is Here to Stay

Big data is here to stay.

And the amount of data will continue to grow.

Except not all of it will be in Traditional Relational Structured format.

So will relational databases survive?

Sure, for transactions perhaps.  And as a pass through to get unstructured data into reports, it will pass through Structured format.

And the amount of relational data is outpaced by data generated by alternative sources.

And relational data flows into Traditional Data Warehouses.

Which are difficult to maintain, change as well as expensive to produce and find talent to support.

Big Data is an extension of the Data Warehouse.  And Big Data is now accessible via SQL like languages like HIVE.

As well as SQL based languages such as Cloudera Hadoop, which by-pass Map-Reduce.

So you ingest as much data in any format into Hadoop, perhaps move that data into a NoSQL database such as HBase.  From there you can mount that data into structured format and run Ansi-SQL to query the data super fast.  On top of that you have software for visualization, graphs and maps as well as Dashboards and Reports.

From my perspective, the technology is moving closer to the Data Analysts. 

I see Data Scientist moving towards the technology, perhaps, but not as much due to the learning curve and shortage of talent.  Also, there's some difficulty in the exact definition as well as hiring manager expectations of what the position entails.

Granted there's a learning curve to become proficient in Big Data, but there was also a curve to learn Star Schema's and MDX. 

As technology improves, the average Data Analyst will know enough to tame Big Data in my opinion as the trend has already begun.

As with Traditional Reporting, Big Data poses challenges such as bad data in, bad data out, Data Cleansing, Data Governance and producing accurate reports.

And finally, we still need a simple way to interpret the data.

And then do something with that insight.

Big Data is here to stay.  As far as Traditional Relational Databases and Traditional Data Warehouses, time will tell.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.

Thoughts to Ponder