The Cloud has risen. It's come a long way, years ago people were hesitant to work off-site, house data elsewhere. Now its a given. There're just too many options available to build, integrate and support applications in the Cloud.
I've blogged before, about creating a dedicated position that is primarily responsible for keeping up to date on the latest offerings, and assisting the organization on new options.
There's really no way for a single developer to comprehend everything available, while doing their everyday job, and get experience in everything. Not possible, in this reality.
On the positive note, its interesting to see how each of the technologies is simply a piece of the puzzle. Did you ever play with Lego's as a kid, or as adult. Each of the pieces is designed in a way that it fits nicely with other pieces of different shapes and sizes. What they've done with the Cloud offerings, is creating units of technology that fit nicely with others.
You have data, move it to the Cloud. Store in a database, flat files, Data Warehouse, Blob Storage. The data can sit there nicely for a long time, as the costs to store data is not outrageous. When you need it, use it, in place, by mounting wrappers on top, or move it around as needed. Analyze it, interrogate, mash it with other data or display it for consumption.
And there are so many ways to integrate and build systems all contained within the Azure Portal.
A nice thing about Cloud offerings, you can do the work remote. The data is secure at rest or in transit.
Everything is backed up, and duplicated across clusters for Disaster Recovery. And there are multiple ways to accomplish the same tasks. For example, there may be a button to upload data built into the web site, or you could use Visual Studio to push or an API to automate or PowerShell to create repeatable scripts.
There are so many options, you have to bite off a chunk of technology, learn the nuances, and build out from there. Learn how to integrate other features into the system, to leverage pre-existing patterns.
With that said, you could actually pick a particular technology, become an expert, and base your entire career on that particular niche. Take U-SQL for example. Its a newer language, more than a year old now, what does it do? Well, it allows developers to access data, similar to existing technologies. Except, it works with both structured and non-structured data. So that's nice. And you can parse through data in files, file folders, Blob Storage, Azure Database, Azure Data Warehouse, HDInsight, Hadoop VM using External Tables.
Once you have access to the data, U-SQL allows you to manipulate that data, as in Extract, Transform and Load (ETL), similar to the Pig language of Apache Hadoop or SSIS in MSBI. And you can mount that data into Tables, or store back into raw CSV files in the Cloud, similar to Apache Hive in Hadoop.
And the U-SQL language is similar to Linq, which is a derivative of SQL, and it's wrapped in C#. So you have all the power behind the well established language to work with the data. And you can bring in Windows Functions and create Assemblies for re-usable code. And it integrates with Data Catalog.
It basically takes a bunch of existing technologies, bundles the together, to create a new and improved language which has tremendous flexibility. You can run the code locally, in Visual Studio, in Azure Portal, via API, or PowerShell, then store in source code repository.
The reason Hadoop took a while to pick up steam, was DBA's didn't write SQL, Java programmers didn't write SQL and SQL Developers typically didn't write Java. This tool, solves that problem, by allowing developers to use DBA skills, ETL, SQL Developer, Hadoop, Programming, and Cloud skills. I believe it has tremendous potential.
And its cutting edge technology, so there isn't a tremendous amount of examples on the web. So one could become expert by putting in the hours, and then have a fair amount of projects lined up, to leverage the expert knowledge, use existing code base, and knowledge of best practices surrounding this language.
That's one of the things about technology that I like, things always change, yet build on existing skills. Cloud technology has gained widespread adoption and road is wide open to carve new paths to solve problems, learn new things and add value to new and existing clients.
Here's a few URL's to get started with U-SQL: http://www.bloomconsultingbi.com/2018/06/url-references-for-u-sql-getting-started.html