Flood of Competing Open Source Products Built by IT Vendors

My first programming language was c++.  A solid object oriented language still in use.  Could use different IDE's, different operating systems.  Quite nice.

Then I got into Visual Basic 4, 5, & 6.  Proprietary language, easy to get up to speed for simple to complex programs.  Had ability to call the registry, build COM component into a DLL pool and build Active-X.  Yet it was proprietary.  And tons of jobs back then.  Except they killed it off.  Gave us VisualBasic.net.  New language, kept the same name.  A real turn off for us classic VB enthusiasts.

The thing to note is the proprietary language, IDE and it only ran on Windows.

Then I got into Java a decade later.  Nice language, multiple IDE's, multiple platforms, operating systems, web and client side, connect to Mainframe COBOL copybooks program from the web.  You had the choice of IDE's which was nice.

But then they opened it up to Multiple Frameworks.  The pages seemed to jump around and I had much difficulty trying to figure out how to step through the code.  You could see the pages changing in the IDE, but it was confusing for me anyway.  I got out of Java and into Business Intelligence full time.

So we can see a mix of proprietary languages that run on proprietary operating systems using proprietary IDE's to free languages supported on multiple IDE's with community backed frameworks evolving over time for specific purposes and certain benefit's and weakness'.

Where are we now?  Open Source Community projects released for public consumption.  Major IT vendors contributing to the source code, and exposing that source code for public domain.  Not vendor specific, platform specific, operating system specific, nobody really owns it, except for Vendors that package and sell service and maintenance.

Let's say there's a need to do x.  Multiple vendors build their version internally, flush it out, then release to public.  So we have a market flooded with competing free open source products each claiming to be better than the others.  That's great for flexibility, perhaps it's diluting the quality, who knows.  And for developers, how can one stay up to date on all varieties and flavors.  And what are the downstream costs for early buy in and want to upgrade or switch later on.  And how do companies find talent to support and build apps when things are so diverse and ever changing.

Should we go back to the proprietary lock in when things were simpler?  I doubt it.  But the rapid pace of newness is a water hose very difficult to drink from.  And so it goes~!

Thanks Boston, see you soon

Well tomorrow wraps up an 8 week project in Boston. I've traveled back and forth for 8 consecutive weeks.  From my driveway, Uber, to airport, to layover, to Boston Logan, to Uber to AirBnB, on Sunday's.  Flights were mostly 7am which leaves a bunch of time on Sunday to putter around the city, eat some great food, get ready for Monday.

I typically pack up mid day Friday, Uber to airport to layover to Tampa to Uber to home, getting in close to Midnight.  Hang out with the Wife, Pups and chores Saturday, back on Sunday.  I did stay over 2 weekends and stayed at an actual hotel 2 weeks, not AirBnB.  Travel time ranges 7 to 9 hours, plus the 40 billable hours and admin time, averaging 56 hours per week.

What did I do on the project?  Well, I created reports in DBVisualizer, against AWS Data Lake Hive tables.  Luckily I have a ton of Insurance knowledge at the business and data level as I use to balance the books for an Insurance company for 4 years.  Written Premium, Earned Premium, Inforce, Claims, Commissions, Tax reports.  And I got to work with a great team of report writers.

So what's next?  Well, they extended the contract 8 more weeks.  I plan to work from home next week as I have to take care of some things.  Will continue to write reports remotely and Skype for Business.  And then travel back to Boston.  And have a scheduled vacation in a few weeks.  Sure will be nice to hang out with the dogs, catch up on some stuff and spend time with the Mrs.  I'm a bit tired.

Thanks Boston, see you soon~!


Compare Option in DBVisualizer

I've been working with a tool DBVisualizer.  This tool allows development against a variety of databases.  I happen to be working against AWS Data Lake Hive tables.  After you configure the Connection, you can go to the tables tree view, expand, highlight all objects, right click, script objects, to a file or window.  I selected the "Create" button and ran it.  544 objects to create, runs for a while.  So we have a file containing all table objects.

From there, we connect to another environment.  Assuming I had access to the Production Environment, I would perform same steps, generate a second file.

From there, you can click the Tools dropdown, Compare option, select your 2 new files, and see the differences. 

Now to view the differences, green indicates new:

This is a handy feature when developing, as sometimes the objects do differ between environments, and it's no fun to deploy a report that's been fully validated, only to have it fail in production, with different objects.  Missing Views or Tables, fields renamed or missing.

And there you have it~!

Humming Along in Boston

Flying home today, completion of week #7 on latest project.  Working for an insurance company in downtown Boston.  Used frequent flyer points to have my wife stay with me for a week, which was a nice treat.

Boston is such a vibrant town, from Red Sox, to American History, bus tours, restaurants, we had a great time.  Since my wife grew up in these parts, she was able to visit longtime friends and family.  We should arrive home around midnight, some chores, quick Rowdies soccer game, and back at the airport for 6:40am flight Sunday.  And see the pupperonies!

I'm enjoying this project, basically writing SQL statements against AWS Data Lake Hive tables.  One interesting thing, I worked for an insurance company a while back, balanced the books for two Insurance companies, so I learned the business inside and out.  What's interesting is the current project utilizes the same lingo, so having that deep industry knowledge helps.  And many of the reports are similar, Claims, Coverages, Written vs Earned Premium, Commissions.

I've lived in Florida since about 1982 and worked there since about 1996.  It sure is refreshing to get new experience in other regions with other clients and other technologies.

A breath of fresh air~!


Hive SQL Date Functions Cheat Sheet

Since I've been working with Hive SQL lately, against AWS Data Lake, assembled a quick list of key Date functions to speed up development:

    from_unixtime(unix_timestamp(TO_DATE(CURRENT_DATE) , 'yyyy-MM-dd'), 'MM-dd-yyyy') AS TodaysDate
    from_unixtime(unix_timestamp(DATE_ADD(CURRENT_DATE,-(DAY(CURRENT_DATE)-1)), 'yyyy-MM-dd'),
    'MM-dd-yyyy') AS FirstDayThisMonth ,
    'yyyy-MM-dd'), 'MM-dd-yyyy') AS LastDayThisMonth,
    from_unixtime(unix_timestamp((DATE_ADD(CURRENT_DATE, -1-DAY (CURRENT_DATE))) , 'yyyy-MM-dd'),
    'MM-01-yyyy') AS FirstDayPriorMonth,
    from_unixtime(unix_timestamp(DATE_ADD(CURRENT_DATE,-(DAY(CURRENT_DATE)+1)) , 'yyyy-MM-dd'),
    'MM-dd-yyyy')                                                              AS LastDayPriorMonth,
    from_unixtime(unix_timestamp(TO_DATE(CURRENT_DATE) , 'yyyy-MM-dd'), '01-01-yyyy') AS
    from_unixtime(unix_timestamp(TO_DATE(CURRENT_DATE) , 'yyyy-MM-dd'), '12-31-yyyy') AS
    from_unixtime(unix_timestamp(date_sub(concat(from_unixtime(unix_timestamp(), YEAR(CURRENT_DATE)
    -1), '-01-01'), 0), 'yyyy-MM-dd'), 'MM-dd-yyyy') AS FirstDayPriorYear,
    from_unixtime(unix_timestamp(date_sub(concat(from_unixtime(unix_timestamp(), YEAR(CURRENT_DATE)
    -1), '-12-31'), 0), 'yyyy-MM-dd'), 'MM-dd-yyyy') AS LastDayPriorYear,
    DATE_ADD(CURRENT_DATE, -90)                         TodayMinus90Days,
    from_unixtime(unix_timestamp((DATE_ADD(CURRENT_DATE, -1-DAY(CURRENT_DATE))) , 'yyyy-MM-dd'),
    'MMM')                                                                      AS PriorMonth3Char,
    eis_app.CLAIMSSUMMARYINFO CSI limit 1













Thanks for Reading all these Years

Here's a blog post to mark the 300,000 page views for this site.  It says the first post was back in 2010 which could be accurate, as I've done a few social media purges over the years and got rid of a bunch of stuff.  It says there're 1208 blog post on the site currently,

I originally didn't want ads on the site.  Added them as it was the thing to do.  You hear of people making $5k a month on ad revenue.  Remove 4 zeros more like it.

I grew up behind a keyboard, not sure the exact year, but it was an IBM PC original, no hard drive, just 2 floppies, color chrome monitor, dot matrix Epson printer, no mouse, and PC Dos.  Graphics weren't a thing really, we had modem's 1200 baud connect to local BBS boards.  That was way before the internet revolution, PC in every home, Smart Phone on every belt clip.

I didn't major in computers, just a few courses here and there.  Got into IT in 1996 or so.  Report writer, SQL, programmer client server.

Then web.  Then dot net.  Then Java.  Some project management.  Then Supervisor.  Then BI guy.  Then consultant.

We all have different motivators.  Some go for money.  Some go for fame.  I like to solve the problems that others can't.  Money is okay as a driver, up to a point.  If you don't enjoy the daily grind, you can't succeed.

Programming is a continuous battle of micro problems to be solved every single day, and like gravity constantly pulling us back to earth and inertia slowing us down, we get ambushed by the flood of new technology.  And more often than not, you don't get to choose the technology of your liking, at least for my career anyway.  I've had a tough time breaking into cutting edge technology, not sure why.

As far as blogging, I still enjoy it.  It's no longer fresh, where I can pump out 8 to 10 posts in an hour.  When you write, it just flows.  You don't know what you're going to write about until its on the screen and you go back and read it for the first time.  I feel people get offended by some post and I've had to filter back a lot of content, which reduces the quality, no longer writing out of spontaneity.

When it's raw, that's the best stuff.  Political correctness has removed our sense of being human and quality has gone with it.  As we inch ever closer to our robot future selves, devoid of emotions, talking monotone.  If you look to television for content generation, all the good shows happened in the past.  Same with movies, music also.  The quality was just better then.  I don't watch TV anymore nor do I go to the movies.  And I don't read the papers or listen to the radio.

There seems to be a complete drought of fresh content, that stimulates the mind, with new thoughts and ideas.  Society is stale, like a fishbowl that hasn't been cleaned.  Some days I feel like stirring it up.  Other days I seek a new bowl.  Either way, I continue writing my stories, for nickels in revenue per month, without the slightest indication that anyone reads them.  As far as the 300k blog reads, perhaps, I do know that every Saturday night, around 2am, somebody in Russia scrapes the site and reads 200 plus pages in under an hour, so that inflates the numbers, no way to block it, so who cares.

To summarize, for those who do read the blog, I appreciate it.  And for my 12th grade English teacher who thought I wasn't paying attention in the back of the class, I make a deliberate effort to put a space between "a lot" and not "alot".  I was listening the entire time, so thanks!



User Groups

the Kleptomaniac's Group will be post-poned as the signs have mysteriously disappeared, again.

The Claustrophobia's Group was really crammed last night with so many members.

The Pessimist's Group was a complete waste of time as everyone predicted.

The Anti-Social User Group continues to not meet next month.

The Web Designers User Group's web site is down again, so meeting will be held next month.

The Caffeine Anonymous User Group will be held at the local coffee shop.

The Chronically Late User Group will be meeting at 7pm instead of 6pm, something came up.

Week 6

Today started week 6 of the current project in Boston, MA.  The work is fun, people are easy to work with, technology is good and it's humming along nicely.

I'm out of town 6 days a week, takes between 7 and 10 hour each way home and back.  Averaging 57 hours per week including timesheets, travel and expense reports.

The project is slotted for 8 weeks, so the end is in sight.  Of course, it could get extended.  If so, I'll probably find direct flights, arrive Monday return Thursday and try to work remote more often as much of my communication is via Skype for Business throughout the day.

And there you have it~!

Keep on Blogging

Blogging is fun.  If it weren't, I wouldn't blog.

Some topics are related to technology.  As it's basically a technical blog.

Some blogs are a bit out there.  As you may be aware.

I've been here almost half a century.  If you view the career / life timeline, it's a stead and steep incline for the past 26 years.  Things have only gotten better.

Do you actually think my view points in life, obscure and outlandish they may be, have any impact on the ability to get work done, fast and accurate, learn on the fly?

In fact, the stories illustrate a variety of life skills: the ability to think outside the box; attention to details; observation skills; storytelling; abstract thought; critical thinking; ability to weave desperate stories into a cohesive whole; variety of interest; life experiences; knowledge of variety of subjects; cutting edge technology; continuous learning, discipline.

Should I be concerned that the story topics are read by clients and workers, I don't think so.

Perhaps we should be more concerned with those who have no social media presence, who could care less about continuous learning, narrow minded, lazy.

For those who don't understand this style of writing, feel free to read elsewhere.  For those who enjoy the topics, thanks for reading.  Either way, I'll probably continue to blog as I still enjoy it.

And there you have it~!


USL Soccer in Tampa Bay - the Rowdies

I was real surprised my wife wanted to go see a soccer match.  Turns out, the Tampa Bay Rowdies are the team to see.  We went to a game last season, descent seats, and had a really good time.  The team supports family atmosphere, the stadium is really cool as it faces the marina, and the players are talented.

One game, we saw Joe Cole do an upside down backwards bicycle kick.  That was an amazing kick which doesn't happen very often. 

One game we headed down to the midline seats to see what they looked like and one of the executives saw us walking there and got one of the employees to give us a tour.  They let us sit in the fancier seats and when we called the following week, we upgraded our seats to the Midfield.

We attended every game last year and it was really a lot of fun for both of us.

We went to a game at USF last year which was a lot of fun.  We went to the Florida Cup, got to see some amazing international players.  This year, we went to the pre-season matches, one of which was held at University of Tampa.  We attended the fan appreciate party and got to meet a lot of the players.  This season the team started off with a bang and won their first two matches in a new division called the USL.

Explosive offence, solid defense, fast players, lots of shots on goal and all around hustle.  I'd have to say they look impressive.

The thing is, before I got into tennis I played soccer up north for the Town of Poughkeepsie traveling team.  We went to a lot of different cities playing tournaments and such.  You had to try out for the team and it was a lot of fun.  If I didn't have to have hernia surgery in 7th grade, I would have played on the school team.  Only played a year of soccer in Florida, but I did attend a game or two at the old Tampa Bay sombrero stadium when the original Rowdies played in Tampa.

Looking forward to a great season and quality time with the Mrs.

The Rowdies are, a Kick in the Grass!


Hadoop Project

Much to my surprise, the guy slotted for the Hadoop project left the company.  And so did the next in line.  I spoke with my wife, I wanted to work on Hadoop.  Sure I learned it 4 years prior, never got a shot at it.  I emailed the boss, sure enough, I got the project.
The first week, client offered to stand up the Hadoop cluster.  Except we learned Hadoop no longer supported on Windows operating system.  I was on vacation the 2nd week of the project.  I decided to take the laptop on vacation to the cabin and got Hadoop working on Microsoft Windows 10 Hyper-V, no easy feat.  I configured it to allow remote connection from the laptop.  After doing some research, turned out Visual Studio 2015 had 3 new components to work with Hadoop.  I played around with it and got flat files to flow t HDFS, send Pig Scripts which worked and pushed data to Hive tables.  That was all I needed to return on Week 3 of the project.  Upon return, I was brought in to assist in the troubleshooting and we reinstalled Hadoop one or two more times, finally got it stood up on Linux, Master Node with three Data Nodes.  I ported my Visual Studio 2015 source code to the server, connected with Git and the project was humming along nicely.  Data flowed from Excel, to text files, pushed from Shared Folder to HDFS, Pig script to message the data, pushed into Hive tables, then Hive ORC tables.  Then, figured out how to install Polybase on SQL Server 2016, allowing seamless flow of data from Hadoop to SQL Server using common T-SQL.  I architected the SSIS project.  Flowed the data into Master Data Service entities.  Ran the data through some custom logic to parse and clean the address.  Then called an SSIS component in c# to call an internal web service to Geo Code the Latitude and Longitude.  Then called another web service using c# to send in Lat & Lon to get Geo Tagging.  Then pushed data into SQL Server ODS tables.  However, around that time, the Hadoop cluster went down.  I troubleshot the server for hours and hours and hours, deep into the nights.  The thing about Hadoop, if it stops working, there are so many configuration files to go through and investigate.  If one file has one entry with an extra space, the entire thing stops working.  I looked through every folder and many files interrogating everything.  And took tedious notes along the way.  And searched many a website/blog in attempting to fix.   Permissions.  Config files.   User accounts.  Error logs.  Host files.  You name it.  I was told to hold off on the issue as I had to jump back on my actual assignment.  I couldn’t let it rest and would work on it after hours.  Then I saw something, in one of the webs, you could see the jobs: success, failed, killed, etc.  There it was.  After picking through the logs, I saw the job was still running.  And so were 150 other jobs.  Which I initiated during my development.  The server got backed up.  Found a site on how to kill the process.  And proceeded to execute a command to kill 1 process at a time, for 150+ processes.  Restarted  Ambari, bam!  I could execute Hive with no error on the command line.  Then flowed some Pig scripts through Visual Studio 2015 using the WebHCat server and sure enough, back in business.  Solved it~!
I like to solve difficult problems.  Especially the ones that others gave up on.  Those are the juicy problems that are not easy to find.  That took some meticulous troubleshooting for many, many hours.  I rolled off the Hadoop project before the Data Warehouse was completed but I created the SSIS files to handle the Dim and Fact tables as well as refresh the Tabular Model.   That is what you call a great project.

And there you have it~!

First Crystal Reports Job

After I got into IT full time, I soon left for greener pastures.  I found a job doing Crystal Reports full time.  This was 1998 or so, when nobody was doing reporting for a living.  I got hired to work for a small company.  First day on the job, boss pulls me into his office, closes the door.  I have to tell you, we are being purchased by a big company, couldn’t tell you during the interview process.  Who?  Kronos.
So I was soon writing Crystal Reports for hospitals across the country.  Timekeeping reports.  Tons of reports queued up, some modifications, enhancements.  We had account reps that went out to the hospitals, took care of their Oracle instances, etc. and loaded my reports.  It was a cool job.  I worked with a c++ programmer and a guy that wrote documentation, we were definitely unique bunch of programmers to say the least, with a “different” sense of humor that only we got.  Finally people I could relate too.
The boss asked if I could write a small application in Visual Basic 4 at the time.  It basically kept track of Bugs, a bug Tracker.  The best part, VB had a cool install package wizard and I added that so internal people could load on the machines, that went over well.  Although there were bugs for the bug tracker also.
I wanted to do more programming and not just reports, but they said I should take a COBOL class, with that I found a new job.
And that’s the story of working with Crystal Reports way before people saw the value of data and flooded the market with people and products and experts.  I still like writing reports, it’s an acquired taste perhaps.  Swimming in the numbers trying to get them to match.  A puzzle every single day of work.
And there you have it~!

Mountain Living