9/23/2018

In With the New U-SQL

I worked on a solid Microsoft Business Intelligence project, went okay, then off to another.  Parallel Data Warehouse, for a while, then off to another.  Then a Hadoop - PolyBase - Microsoft BI - MDS - Tabular project, went okay, then off to another.  Then a quick Tableau project, went okay, then off to another.  Then a SQL against AWS Hadoop project, went okay, then off to another.

Where are we at now?  Building out an Archive project, using Microsoft SSIS, c# script component to automate, Azure Data Lake Store and Azure Data Lake Analytics components, U-SQL Databases.

You can see, the projects differ every time.  Get up to speed, become expert level, deliver, then change technology.

Traditional Data Warehousing never excited me, I was late to the party, will slowly lose luster and support, for variety of reasons.  I think the Cloud is the place to be.

U-SQL is a newer language that combines SQL, c#, Linq and ETL.  It runs as a job, can be kicked off from On Premise IDE Visual Studio or in the Cloud.  It's dynamic, works with Structured and Unstructured data.

I think the language will mature, the IDE install will be streamlined, the reporting on U-SQL Databases will be available to more Reporting tools and Intellisense will improve.

U-SQL reminds me of combination of Hive and Pig, in the familiar Visual Studio IDE we've known for years, executes in the Cloud Azure, stores past jobs both Success and Failed, run time, Error Messages.

Quite tasty.