4/27/2017

Flood of Competing Open Source Products Built by IT Vendors

My first programming language was c++.  A solid object oriented language still in use.  Could use different IDE's, different operating systems.  Quite nice.


Then I got into Visual Basic 4, 5, & 6.  Proprietary language, easy to get up to speed for simple to complex programs.  Had ability to call the registry, build COM component into a DLL pool and build Active-X.  Yet it was proprietary.  And tons of jobs back then.  Except they killed it off.  Gave us VisualBasic.net.  New language, kept the same name.  A real turn off for us classic VB enthusiasts.


The thing to note is the proprietary language, IDE and it only ran on Windows.


Then I got into Java a decade later.  Nice language, multiple IDE's, multiple platforms, operating systems, web and client side, connect to Mainframe COBOL copybooks program from the web.  You had the choice of IDE's which was nice.

But then they opened it up to Multiple Frameworks.  The pages seemed to jump around and I had much difficulty trying to figure out how to step through the code.  You could see the pages changing in the IDE, but it was confusing for me anyway.  I got out of Java and into Business Intelligence full time.



So we can see a mix of proprietary languages that run on proprietary operating systems using proprietary IDE's to free languages supported on multiple IDE's with community backed frameworks evolving over time for specific purposes and certain benefit's and weakness'.


Where are we now?  Open Source Community projects released for public consumption.  Major IT vendors contributing to the source code, and exposing that source code for public domain.  Not vendor specific, platform specific, operating system specific, nobody really owns it, except for Vendors that package and sell service and maintenance.


Let's say there's a need to do x.  Multiple vendors build their version internally, flush it out, then release to public.  So we have a market flooded with competing free open source products each claiming to be better than the others.  That's great for flexibility, perhaps it's diluting the quality, who knows.  And for developers, how can one stay up to date on all varieties and flavors.  And what are the downstream costs for early buy in and want to upgrade or switch later on.  And how do companies find talent to support and build apps when things are so diverse and ever changing.


Should we go back to the proprietary lock in when things were simpler?  I doubt it.  But the rapid pace of newness is a water hose very difficult to drink from.  And so it goes~!
 

Thanks Boston, see you soon

Well tomorrow wraps up an 8 week project in Boston. I've traveled back and forth for 8 consecutive weeks.  From my driveway, Uber, to airport, to layover, to Boston Logan, to Uber to AirBnB, on Sunday's.  Flights were mostly 7am which leaves a bunch of time on Sunday to putter around the city, eat some great food, get ready for Monday.


I typically pack up mid day Friday, Uber to airport to layover to Tampa to Uber to home, getting in close to Midnight.  Hang out with the Wife, Pups and chores Saturday, back on Sunday.  I did stay over 2 weekends and stayed at an actual hotel 2 weeks, not AirBnB.  Travel time ranges 7 to 9 hours, plus the 40 billable hours and admin time, averaging 56 hours per week.


What did I do on the project?  Well, I created reports in DBVisualizer, against AWS Data Lake Hive tables.  Luckily I have a ton of Insurance knowledge at the business and data level as I use to balance the books for an Insurance company for 4 years.  Written Premium, Earned Premium, Inforce, Claims, Commissions, Tax reports.  And I got to work with a great team of report writers.


So what's next?  Well, they extended the contract 8 more weeks.  I plan to work from home next week as I have to take care of some things.  Will continue to write reports remotely and Skype for Business.  And then travel back to Boston.  And have a scheduled vacation in a few weeks.  Sure will be nice to hang out with the dogs, catch up on some stuff and spend time with the Mrs.  I'm a bit tired.


Thanks Boston, see you soon~!

4/21/2017

Compare Option in DBVisualizer

I've been working with a tool DBVisualizer.  This tool allows development against a variety of databases.  I happen to be working against AWS Data Lake Hive tables.  After you configure the Connection, you can go to the tables tree view, expand, highlight all objects, right click, script objects, to a file or window.  I selected the "Create" button and ran it.  544 objects to create, runs for a while.  So we have a file containing all table objects.


From there, we connect to another environment.  Assuming I had access to the Production Environment, I would perform same steps, generate a second file.


From there, you can click the Tools dropdown, Compare option, select your 2 new files, and see the differences. 




Now to view the differences, green indicates new:




This is a handy feature when developing, as sometimes the objects do differ between environments, and it's no fun to deploy a report that's been fully validated, only to have it fail in production, with different objects.  Missing Views or Tables, fields renamed or missing.


And there you have it~!

Humming Along in Boston

Flying home today, completion of week #7 on latest project.  Working for an insurance company in downtown Boston.  Used frequent flyer points to have my wife stay with me for a week, which was a nice treat.

Boston is such a vibrant town, from Red Sox, to American History, bus tours, restaurants, we had a great time.  Since my wife grew up in these parts, she was able to visit longtime friends and family.  We should arrive home around midnight, some chores, quick Rowdies soccer game, and back at the airport for 6:40am flight Sunday.  And see the pupperonies!

I'm enjoying this project, basically writing SQL statements against AWS Data Lake Hive tables.  One interesting thing, I worked for an insurance company a while back, balanced the books for two Insurance companies, so I learned the business inside and out.  What's interesting is the current project utilizes the same lingo, so having that deep industry knowledge helps.  And many of the reports are similar, Claims, Coverages, Written vs Earned Premium, Commissions.

I've lived in Florida since about 1982 and worked there since about 1996.  It sure is refreshing to get new experience in other regions with other clients and other technologies.

A breath of fresh air~!

4/20/2017

Hive SQL Date Functions Cheat Sheet


Since I've been working with Hive SQL lately, against AWS Data Lake, assembled a quick list of key Date functions to speed up development:





SELECT
    from_unixtime(unix_timestamp(TO_DATE(CURRENT_DATE) , 'yyyy-MM-dd'), 'MM-dd-yyyy') AS TodaysDate
    ,
    from_unixtime(unix_timestamp(DATE_ADD(CURRENT_DATE,-(DAY(CURRENT_DATE)-1)), 'yyyy-MM-dd'),
    'MM-dd-yyyy') AS FirstDayThisMonth ,
    from_unixtime(unix_timestamp(LAST_DAY(DATE_ADD(CURRENT_DATE,-(DAY(CURRENT_DATE)-1))),
    'yyyy-MM-dd'), 'MM-dd-yyyy') AS LastDayThisMonth,
    from_unixtime(unix_timestamp((DATE_ADD(CURRENT_DATE, -1-DAY (CURRENT_DATE))) , 'yyyy-MM-dd'),
    'MM-01-yyyy') AS FirstDayPriorMonth,
    from_unixtime(unix_timestamp(DATE_ADD(CURRENT_DATE,-(DAY(CURRENT_DATE)+1)) , 'yyyy-MM-dd'),
    'MM-dd-yyyy')                                                              AS LastDayPriorMonth,
    from_unixtime(unix_timestamp(TO_DATE(CURRENT_DATE) , 'yyyy-MM-dd'), '01-01-yyyy') AS
    FirstDayThisYear,
    from_unixtime(unix_timestamp(TO_DATE(CURRENT_DATE) , 'yyyy-MM-dd'), '12-31-yyyy') AS
    LastDayThisYear,
    from_unixtime(unix_timestamp(date_sub(concat(from_unixtime(unix_timestamp(), YEAR(CURRENT_DATE)
    -1), '-01-01'), 0), 'yyyy-MM-dd'), 'MM-dd-yyyy') AS FirstDayPriorYear,
    from_unixtime(unix_timestamp(date_sub(concat(from_unixtime(unix_timestamp(), YEAR(CURRENT_DATE)
    -1), '-12-31'), 0), 'yyyy-MM-dd'), 'MM-dd-yyyy') AS LastDayPriorYear,
    DATE_ADD(CURRENT_DATE, -90)                         TodayMinus90Days,
    from_unixtime(unix_timestamp((DATE_ADD(CURRENT_DATE, -1-DAY(CURRENT_DATE))) , 'yyyy-MM-dd'),
    'MMM')                                                                      AS PriorMonth3Char,
    DATEDIFF(TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP())), TO_DATE(CSI.LOSSDT)) <90 as="" br="">    CheckForXDaysAgoTrueFalse
FROM
    eis_app.CLAIMSSUMMARYINFO CSI limit 1

















4/19/2017

Time

Time

So
much
time

Tick
Tock

Not

So
much
time

Wasting
the
Hours
Away

Times
Up

Where
Did
all
the
Time
Go

Wasted
the
Hours
Away

4/12/2017

Thanks for Reading all these Years

Here's a blog post to mark the 300,000 page views for this site.  It says the first post was back in 2010 which could be accurate, as I've done a few social media purges over the years and got rid of a bunch of stuff.  It says there're 1208 blog post on the site currently,

I originally didn't want ads on the site.  Added them as it was the thing to do.  You hear of people making $5k a month on ad revenue.  Remove 4 zeros more like it.

I grew up behind a keyboard, not sure the exact year, but it was an IBM PC original, no hard drive, just 2 floppies, color chrome monitor, dot matrix Epson printer, no mouse, and PC Dos.  Graphics weren't a thing really, we had modem's 1200 baud connect to local BBS boards.  That was way before the internet revolution, PC in every home, Smart Phone on every belt clip.

I didn't major in computers, just a few courses here and there.  Got into IT in 1996 or so.  Report writer, SQL, programmer client server.

Then web.  Then dot net.  Then Java.  Some project management.  Then Supervisor.  Then BI guy.  Then consultant.


We all have different motivators.  Some go for money.  Some go for fame.  I like to solve the problems that others can't.  Money is okay as a driver, up to a point.  If you don't enjoy the daily grind, you can't succeed.

Programming is a continuous battle of micro problems to be solved every single day, and like gravity constantly pulling us back to earth and inertia slowing us down, we get ambushed by the flood of new technology.  And more often than not, you don't get to choose the technology of your liking, at least for my career anyway.  I've had a tough time breaking into cutting edge technology, not sure why.

As far as blogging, I still enjoy it.  It's no longer fresh, where I can pump out 8 to 10 posts in an hour.  When you write, it just flows.  You don't know what you're going to write about until its on the screen and you go back and read it for the first time.  I feel people get offended by some post and I've had to filter back a lot of content, which reduces the quality, no longer writing out of spontaneity.

When it's raw, that's the best stuff.  Political correctness has removed our sense of being human and quality has gone with it.  As we inch ever closer to our robot future selves, devoid of emotions, talking monotone.  If you look to television for content generation, all the good shows happened in the past.  Same with movies, music also.  The quality was just better then.  I don't watch TV anymore nor do I go to the movies.  And I don't read the papers or listen to the radio.

There seems to be a complete drought of fresh content, that stimulates the mind, with new thoughts and ideas.  Society is stale, like a fishbowl that hasn't been cleaned.  Some days I feel like stirring it up.  Other days I seek a new bowl.  Either way, I continue writing my stories, for nickels in revenue per month, without the slightest indication that anyone reads them.  As far as the 300k blog reads, perhaps, I do know that every Saturday night, around 2am, somebody in Russia scrapes the site and reads 200 plus pages in under an hour, so that inflates the numbers, no way to block it, so who cares.

To summarize, for those who do read the blog, I appreciate it.  And for my 12th grade English teacher who thought I wasn't paying attention in the back of the class, I make a deliberate effort to put a space between "a lot" and not "alot".  I was listening the entire time, so thanks!


~JB