Automated Solution to Repetitive Coding

What if you were tasked with creating an SSIS package.  To pull from a database, send the data to Azure Data Lake Store as CSV.  Then pick up that data, flow to U-SQL database table.

Well, you'd probably need a source ODBC connection to the database.  And an SSIS package component to flow to Azure Data Lake Store.  And another to flow to U-SQL db.

So, once you go through the effort to complete the task, let's say you had to build 500 similar SSIS packages.  Well, it if takes 2 hours per package, and you don't make any errors, you may get done this month or next.

Or.  You could automate.  As in, build out each package, line by line, using an SSIS package Script component.  Maybe use XML template for repeatable code snippets, pass in data as variables, and create the 500 SSIS packages, in under a minute.

Well, that's what the team's been working on.  And it's been quite tasty.  Building c# code certainly is fun, different from writing standard SQL reports.  One of the great things about coding, you certainly get a different perspective every so often as projects change.

I've been working with SSIS package when they were DTS packages, not DTSX.  And pulling data from databases for a very long time.  Except flowing to Azure is sort of new.  I was lucky enough to figure out the solution, there's some documentation on the web, blogs and such, but for the most part, you have to figure it out yourself.  Those are the juicy projects and this is my 2nd one in the past few years.  The other project being the Hortonworks Hadoop Enterprise Data Warehouse using Visual Studio components and Polybase and Master Data Services.

Being an expert in anything is daunting task, as things change, nobody knows everything.  It's when you approach a project with fresh eyes, you stay open minded to see different options, try this, try that, see what works, document as you go.  And perhaps you become expert for a moment, until the next projects begins.

Programming is still a descent job.  I work out of my house, past 5 years actually.  You begin the day before most people, and you end the day after most people.  There's no commute.  There's no off button.  Yet, it seems to work just fine.  If you can work fast, if you're responsive, if you can deliver solutions to tough problems, what more can you ask for.

And so it goes~!


Supporting Public Facing Java Applications

I once supported a web application, public facing, many users.  First task was to add Cookies & Captcha.  After that, a re-branding of the website to update with different color schemes for consistent look and feel.

It actually was written in Java using IBM VA Java, which I was hired to be a Senior Java Developer.  Luckily I learned it on my own years back, they were just looking for someone that knew that flavor of Java.

In fact, it hooked to the Mainframe on the back end.  So we had copybook frameworks which mapped to the back end, which connected to Mainframe COBOL and returned data, pieces at a time.

Then we migrated to use Web Services with XML, and then migrated the entire application to JDeveloper java.

Also, the application had hooks into the IVR system which I supported, to accept credit card payments over the phone.  Was interesting to test the application using break points and calling in the number and seeing the code stop on specific line of Java code.

Lastly, it also hooked into the Kiosks which we had a few around town, which also accepted payments, and it too was written in Java connecting to the Mainframe back end.

It was fun to program in Java, although we had internal teams moving to Struts and I didn't get to work on much of that.

I also supported other applications in Java, it was a high profile position as all were public facing.  In addition to being the Business Objects Admin for a while, introducing Crystal Reports to the ecosystem and also had a stint of Project Management.

Overall an excellent job.  Lasted more than 4 years.  Not too shabby~!


SSIS Automation using SSIS

Do you ever sit down and write code.  For hours.  Never get up.  Even to eat or use the rest room.

That's what I've been up to the past few days.  Writing c# code.  In SSIS script component.

It's quite tasty.  It's actually programming.  Within SSIS.  In the data space.  It's an SSIS package to create other SSIS packages.  Using custom code, XML templates.  Reading from source system.  And Excel file template.  It outputs ".DTSX" packages.  To automate the process.  Rather than write each package one at a time.  Automation is key.

Also writing U-SQL scripts.  Create, truncate and dropping tables in Azure Data Lake Analytics U-SQL database.  And pushing CSV files up to Azure Data Lake Store.

Sure is fun to be heads down in the code.  I'll be there in 5 minutes.  2 hours later, where are you?  I'll be there in 5 minutes.

And so it goes~!