Perhaps we need to assess the situation for what it is.
With the rise of Big Data, there were claims that it would solve all the world's problems. Turn water into wine. Save all your data. Find insights. Cure any problem.
I marched to that drum as well. Put a lot of faith into it. I don't think it turned out like we thought.
It's complicated. It keeps changing. Finding insights is difficult. Inherent issues working with any data, regardless of size.
It's just another tool in the toolbox. What can it do that traditional SQL databases can't?
Handle huge volumes of data. Handle Unstructured and semi-structured data. Serve as a central repository across large cluster of commodity servers.
The ecosystem has grown internally, with different iterations of this and that, some proprietary, and open source Apache projects. It's become a mature ecosystem.
And fragmented. As many orgs dipped their toes into the water, only to get burned as newer technology depreciated their entire projects. A case of early adopters scorn.
And many options sprung forth to ease the pain. Fully functioning applications to do heavy lifting ETL, importing and extracting data, monitoring, security, batch processing, streaming. All good stuff. Fragmented none the less.
And finding qualified developers was a problem. Who are the main adopters of Big Data? DBA's could care less about writing T-SQL let alone Java Map Reduce jobs. SQL Developers most likely don't know Java either. System Admin's usually don't code. Nor do network admin's. Or Business Analysts or Project Managers. It's a new breed of developers. And they need to know architecture, security, coding, data, best practices, ETL, business process' and business rules, networks, hardware, servers, administration, and on and on. Who knows all this stuff. College grads don't have the business experience, yet. So the market didn't specify who the ideal developers were. More of a shotgun approach.
And the vendors lined up offering proprietary offerings, each a bit different, to make things easier.
I would have given my right arm to program in Hadoop a few years ago. Today, not so much. Sure it would be nice to get the real world experience. To be honest, I haven't been keeping up with all the latest stuff, too many twists and turns.
What technology am I focusing on now? Artificial Intelligence. Internet of Things. Quantum Computing. This stuff is exploding right now. However, even with all the excitement and hoopla, I'm taking more of a reserved approach this time. The claims running through the noise are similar to Big Data, will solve all the worlds problems, etc. Yes, it will change things, for the better we hope, but will also introduce new issues and new complexity and continue to create a technology gap based on limited quality resources. It will drive new markets, create new players and move things forward. Rather than rehashing legacy code with newer frameworks, AI, IoT and Quantum Computing could potentially be real game changers. And put us all out of work for good.
And there you have it~!
Here's a nice piece by industry leading analyst Howard Dresner, who coined the term "Business Intelligence", where he discusses the topic of Big Data, enjoy...