Big data is here to stay.
And the amount of data will continue to grow.
Except not all of it will be in Traditional Relational Structured format.
So will relational databases survive?
Sure, for transactions perhaps. And as a pass through to get unstructured data into reports, it will pass through Structured format.
And the amount of relational data is outpaced by data generated by alternative sources.
And relational data flows into Traditional Data Warehouses.
Which are difficult to maintain, change as well as expensive to produce and find talent to support.
Big Data is an extension of the Data Warehouse. And Big Data is now accessible via SQL like languages like HIVE.
As well as SQL based languages such as Cloudera Hadoop, which by-pass Map-Reduce.
So you ingest as much data in any format into Hadoop, perhaps move that data into a NoSQL database such as HBase. From there you can mount that data into structured format and run Ansi-SQL to query the data super fast. On top of that you have software for visualization, graphs and maps as well as Dashboards and Reports.
From my perspective, the technology is moving closer to the Data Analysts.
I see Data Scientist moving towards the technology, perhaps, but not as much due to the learning curve and shortage of talent. Also, there's some difficulty in the exact definition as well as hiring manager expectations of what the position entails.
Granted there's a learning curve to become proficient in Big Data, but there was also a curve to learn Star Schema's and MDX.
As technology improves, the average Data Analyst will know enough to tame Big Data in my opinion as the trend has already begun.
As with Traditional Reporting, Big Data poses challenges such as bad data in, bad data out, Data Cleansing, Data Governance and producing accurate reports.
And finally, we still need a simple way to interpret the data.
And then do something with that insight.
Big Data is here to stay. As far as Traditional Relational Databases and Traditional Data Warehouses, time will tell.
I signed up for the Hortonworks Certified Associate exam last Thursday. Figured if I sign up, I'd have to take the test. And if I tak...
Saw a post today on Twitter, " Microsoft releases CNTK, its open source deep learning toolkit, on GitHub " This is big news. Be...
It seems like open source applications are the mainstream today. So many new products delivered through Aache foundation. Some do this. S...