5/23/2017

Mountain Living

3 dogs.
1 wife
Some food.
Guitar.
Couple books.
Box of cigars.
No computer. 
Vacation list.
Our cabin is very secluded.
BBQ food.  Hike in the morning. Then thrift stores. Lunch. Relax. Sleep.
No deadlines.  No meetings.  No pressure. 
Slower pace of life.  Mountain living.  Could get used to this.
The sky changes colors. As do the leaves.  I think God lives here.
Life is good~!

Georgia on my Mind






Relax.

5/19/2017

We Interrupt this Broadcast

I won't be here.  Will be somewhere else.  In a nice cabin nestled amongst the trees in the mountains.  I will not be thinking, working, thinking about working or anything of the sort.  The old noggin needs a rest.

“Close your mouth,
block off your senses,
blunt your sharpness,
untie your knots,
soften your glare,
settle your dust.
This is the primal identity.”
Lao Tzu, Tao Te Ching: A Literal Translation with an Introduction, Notes, and Commentary

And so it is...Be back soon...

5/17/2017

Life Requires Big Decisions

If you knew me growing up, you'd wonder why I didn't talk much.  And why I floated through school invisible.  Yet when I got on the tennis courts, I got to be who I was and not who I was told to be.

Yes, I solved the Rubik's cube in the 5th grade.  And began programming at age 13.  And yes I majored in Anthropology in college.  And yes I worked temp jobs after college to earn money, before my first "job".

And yes, I'm basically self taught programmer for 20 years.  When I was contacted by a large social media company a few years back, I didn't move forward with the technical interview, because we had no plans moving to Silicon Valley.

When I was contacted more recently by another big tech company to interview in Seattle, I had to decline as well.  Just not in a position to relocate at this point in time.

Would it be the coolest thing to work at a leading tech company working on cutting edge technology for a living, you betcha.  I've walked away from much bigger dreams in the past and had to deal with it.  These are minor comparatively.

You are not required to grow up, but sometimes you have to make decisions that affect you, and then live with the downstream consequences, and that is life.

Some decisions haunt you for decades, others you forget.  In the end, tomorrow is a new day.  That's all we can ask for.

Before I became someone else

I played tennis today, it was a lot of fun.  We both grew up in this area, both played tennis and soccer as kids, even went to the same college.  Although we didn't know each other back then.  My boss from a few years ago.

There's nothing else like it.  Running around on the clay courts, cool breeze, nice sky, bit windy, hitting the ball, running around like a fool chasing down every ball, rally until there's no energy left and keep pushing further.

I grew up on the tennis courts.  Back when I had tunnel vision and nothing else existed.  Tennis was my identity, I had no plans on growing up, entering the workforce, getting a job.  That is who I was before I became someone else.

I threw away all the trophies and ribbons and plaques.  Nothing to remind of the past.

Except when I get back on the courts.  That's who I am.  And nothing else matters.

5/15/2017

Cookie Cutter Patterns to Programming, Writing and Life

I know this one guy, he can hit the golf ball farther than anyone I've ever met.  I real treat to watch the ball fly endlessly into the fairway.  Only downside, the guy can't hit from the fairway, sand trap or the green.

In order to perform at maximum efficiency, he has to tee up the ball in the fairway, set it up nice and high, haul off and bam!

I've seen programmers like this also.  In order for the to work at their best, they have to position all environment conditions to their liking.  Whether that's working in isolation, taking specific tasks they excel at and dishing off non preferred tasks to others, aligning with management to ensure specific assignments, etc.  So long as the task aligns with their strength, they are awesome.

Except if they are given something to do they are not familiar with or haven't worked with before.  Or troubleshooting an unknown.  Good luck.

I've seen writers like this too.  Doesn't matter the introduction subject matter, they always end up aligning with their cookie cutter format.  If they excel at steps A, B, C, D, and E, doesn't matter where they start.  At some point in the story, they steer it towards A, B, C, D and E, Bam!

The true mysteries of life, the opportunities to grow, are when we dabble off the beaten track, steer away from our comfort zone, and into the unknown.  It is the deep end we must find ourselves, for it is there, that we are challenged to become better, smarter people.  And that is truly the goal of life.

And so it goes~!

Guilty Until Proven Innocent

We prefer to think that a developer has some cushion in regards to accountability.  Where a person gathers some report specifications, writes some SQL, validates, throws into Production.  He is awarded the highest rank, hoisted upon the shoulders and paraded around the office to receive accolades.  Wake up, dream's over.

In reality, the report writer is perhaps thrown into the deep end, learns the business, process', database on the fly perhaps without assistance or data dictionary.  He or she may struggle to assemble a report with or without the help from business, present to the business, bam!  Prime sucker ready for picking.

This report doesn't match.  Fix it!  We're waiting.  Guilty until proven innocent.

Turns out the root cause of the inaccurate report is that it doesn't match an existing report, causing someone else to look bad.  Or perhaps is uncovers a missing business rule, another land mine.  Or perhaps the business logic requires new features added upstream.  The report writer is sometimes a pawn for political maneuvering.  Perhaps the data required is unavailable or the DBA's refuse to provide access.  A series of possibilities.  The Report Writer takes it from every side. 

And they must prove without a shadow of a doubt, that the report presented in accurate, regardless of roadblocks, setbacks, lack of information or assistance.

I've been writing reports since the early 1990's.  Tons of people flocked to new found profession.  Root job description hasn't changed much.  Nobody trusts the reports.  And they never match.  And they always have a few more reports to work on.

And so it goes~!

5/12/2017

4 Myths

Let's cut to the chase.  Let's look at a few basic concepts that could change your perspective on life.

First, GPA's in school don't matter.  In life, can you do the work, great.  If not, they'll probably find out soon enough.  Don't try for perfect grades, it's not necessary.

Next, retirement.  Sure, great concept for our parents generation to spur the economy after WWII, but it doesn't really "work" now a days.  Let's say none of us will actually retire.  We'll just keep working forever.  With that said, why worry about saving x $ per month to enjoy retirement.  More like the pasture, where people who've built their entire personality on their specific job role, to wake up and realize they have to fill the time without the backing of reputation and authority garnered by humble employees.  Don't save for retirement, just getting others rich with advice and fees, and by the way, most investments can lose 100+ percent.

Also, posting opinions on social media, how's that working.  You think the unique post is going to change people's minds.  They've already formed their opinion.  Combine them both, won't buy a cup of coffee.  If people sought your opinion, perhaps they would ask, what's the rational behind your viewpoint, that's interesting, I hadn't thought of that.  Guess what, a lot of things are making a lot of people unhappy.  Can you change it?  Yes?  Then make the change.  If not, there's plenty of safety in disgruntled apathy.

Unicorn dreams.  How many kids want to grow up and become ballerinas or marine biologist.  How many people do you know who occupy that position today.  Might as well strive to become Santa Clause or the Easter Bunny.  I heard there may be an opening for tooth fairy, must have experience though.  And it's part time .  How many people dream of getting a mediocre job where they can't stand anything about it, yet need the paycheck or insurance.  I think my guidance counselor, if I had one actually, I don't think so, said you'll make a good dishwasher maybe.  I probably could have been, just need to focus and apply talents, anything's possible.

 Lastly, our great great grandparent had messy houses.  Dust everywhere, ring around the collars, you name it.  When did we declare that a spotless home is desirable and shows the signs of impeccability.  I don't know how to break the news, but life is messy.  No commercial promising new product to clean up the world exists long term.  Houses and cars get dirty.  So do lives.  That's the way it works.

So hopefully we can move forward with our lives with the knowledge presented.

And there you have it~!

Might be a True Story

Was talking with the Uber driver.  What is it you do?  Well, I type on the keyboard.  That's it?  I also close tickets.  They pay you for that?  Yeah.  Could I learn it?  Sure.  Would you teach me?  Well, I taught a course, but you may want to sign up for some free MOOC courses, I looked up a few on the cell phone, wrote them on a scratch paper for the driver.

You know you've been traveling a lot when the bank fraud department flags your transaction as unusual activity when you're back home.  This was a short week, flew up Monday direct flight, flew home late Thursday, arrive home midnight.  Oh two dozen tasks that need to be handled at home.  What a relief, home again.

Weather change.  Sure is hot in Florida.  Sure is not hot up north.  40's.  Brrr.  Food's good though.  People walk a lot up north.  In a hurry I guess.  They'll walk super fast right at you, if you don't get out of their way, run you over.  Or they'll stop directly in front, turn around, human road block.  They don't really acknowledge others when walking, everyone's busy looking at phone or heads down.  I never realized how much people stare, like lasers scanning your soul, eerie.  You nose itches, you look around, someone staring.  When did that become socially acceptable.  Hey, if you're going to stare, maybe pay me like $1 a minute or something.  We can schedule daily meeting, you can stare, just so long as we agree to the terms, otherwise, stare at someone else.  You cooperation is appreciated, weirdos.

How can we make your life more difficult.  With a smile.  That seems to be the new mantra.  I didn't get the email.  When did society become so "all for me, none for you".  There's quite a few homeless where I've been staying.  These people battle the cold, the environment.  When I first got there, I gave my some money, then a few more, then a few more, Enough, I'm out.  It never ends.  Some appear to be legitimately down on their luck.  Others look like professionals, loud, boisterous, can you spare some change, hey why are you clean shaven with nice clothes.  Rows of people lined up, not 100% sure their circumstances are the same as others sleeping in doorways with mountains of random belongings in bags, covered in their specimens.  When I check out of living space, walking with my luggage, I wonder if some people think I'm homeless.  I'm heading to work.  To type on the keyboard.  Maybe close out a few tickets.  And take Uber back to the airport.  So I can get flagged fraud charges on my purchases from home.  It's a strange story, but I think its true.  Who knows.

Duality is a Myth

Duality.  Left brain, right brain.  Creativity or logic.  One or the other.

Don’t let your kids grow up to artists, musicians, they’ll starve to death.  Better off with practical skills you can depend on, which have value and marketable.

I played violin from an early age.  Scales, repetition, memorize, perform, repeat.  I attended art class in school, my grandmother still has a picture hanging in her apartment of the Golden Gate Bridge from 1986.  I like to take pictures, write poetry, stories.  All in the genre of creativity.

I enjoy solving problems, ones that involve logic, look for the needle in the haystack, uncover layers of an onion until the solution appears, as if waiting patiently.  I look for patterns in everyday life, observant to the minute which people disregard.

If you look closely, art and logic aren’t much different.  In fact, they overlap.  There’s creativity in solving problems and there’s structure in art and music.  Separated by semantics.

Sometimes when working on a problem for a while, you step away, and for some reason, the solution appears like a lightning bolt from nowhere.  Giving the brain a break allows the underlying intuition to get busy and somehow find the solution without effort.

Sports are a hybrid of art and logic.  You practice the foundation skills, repetitive, until perform without thinking.  During play, your instincts take over, intuition plus logic results in creative output, in the zone, on fire, unbeatable.

Perhaps that’s the real goal, finding your unconscious intuition to perform at peak levels, regardless if it’s a musical instrument, sporting event, complex logic problem.  I tend to joke that it’s fun to gather steam, so much, that you can coast uphill, float into the stratosphere, where no boundaries exist.  Where there no rules, just fun.  The peak moments, rare occurrences that dissolve when you try to capture.  The moments where gravity doesn’t exist.

Perhaps today, for no particular reason, your thoughts will become energy through action.  It could be the best day of your life.

5/10/2017

Format Currency, Percentage, Date, and Mask SSN in Hive SQL

In Hive SQL, to format Currency, Percentage and Date, use the following SQL:


SELECT
    CONCAT('$',format_number(COALESCE(12345.6789,0),2))                 AS `Sample Currency Format`,
    CONCAT(format_number(COALESCE(98.6543,0),2),'%')                    AS `Sample Currency Format`,
    from_unixtime(unix_timestamp(TO_DATE(CURRENT_DATE) , 'yyyy-MM-dd'), '01-01-yyyy') AS
    `Sample Date Format`,
    CONCAT('****-**',SUBSTR("123456789", LENGTH("123456789")-2, 3)) AS MaskedSSN








You could also apply the Round function if needed.


Happy coding~!

5/04/2017

Back Home

My first week at home in a while, been making full use of it.  I've been taking care of chores, wake up early in the morning, try to tackle a few things before work.  Spending time with the pupperonies.  And hanging out with the Mrs.

Today I took a PTO day, walked to the park, trimmed the bushes in the back yard, played the guitar, took the dogs for a walk, and walked up to the tennis courts.

It's amazing how being on the courts brings back the energy, a lot of fun.  Back to work tomorrow finish out the week, then fly back to Boston next week.

And there you have it~!

4/27/2017

Flood of Competing Open Source Products Built by IT Vendors

My first programming language was c++.  A solid object oriented language still in use.  Could use different IDE's, different operating systems.  Quite nice.


Then I got into Visual Basic 4, 5, & 6.  Proprietary language, easy to get up to speed for simple to complex programs.  Had ability to call the registry, build COM component into a DLL pool and build Active-X.  Yet it was proprietary.  And tons of jobs back then.  Except they killed it off.  Gave us VisualBasic.net.  New language, kept the same name.  A real turn off for us classic VB enthusiasts.


The thing to note is the proprietary language, IDE and it only ran on Windows.


Then I got into Java a decade later.  Nice language, multiple IDE's, multiple platforms, operating systems, web and client side, connect to Mainframe COBOL copybooks program from the web.  You had the choice of IDE's which was nice.

But then they opened it up to Multiple Frameworks.  The pages seemed to jump around and I had much difficulty trying to figure out how to step through the code.  You could see the pages changing in the IDE, but it was confusing for me anyway.  I got out of Java and into Business Intelligence full time.



So we can see a mix of proprietary languages that run on proprietary operating systems using proprietary IDE's to free languages supported on multiple IDE's with community backed frameworks evolving over time for specific purposes and certain benefit's and weakness'.


Where are we now?  Open Source Community projects released for public consumption.  Major IT vendors contributing to the source code, and exposing that source code for public domain.  Not vendor specific, platform specific, operating system specific, nobody really owns it, except for Vendors that package and sell service and maintenance.


Let's say there's a need to do x.  Multiple vendors build their version internally, flush it out, then release to public.  So we have a market flooded with competing free open source products each claiming to be better than the others.  That's great for flexibility, perhaps it's diluting the quality, who knows.  And for developers, how can one stay up to date on all varieties and flavors.  And what are the downstream costs for early buy in and want to upgrade or switch later on.  And how do companies find talent to support and build apps when things are so diverse and ever changing.


Should we go back to the proprietary lock in when things were simpler?  I doubt it.  But the rapid pace of newness is a water hose very difficult to drink from.  And so it goes~!
 

Thanks Boston, see you soon

Well tomorrow wraps up an 8 week project in Boston. I've traveled back and forth for 8 consecutive weeks.  From my driveway, Uber, to airport, to layover, to Boston Logan, to Uber to AirBnB, on Sunday's.  Flights were mostly 7am which leaves a bunch of time on Sunday to putter around the city, eat some great food, get ready for Monday.


I typically pack up mid day Friday, Uber to airport to layover to Tampa to Uber to home, getting in close to Midnight.  Hang out with the Wife, Pups and chores Saturday, back on Sunday.  I did stay over 2 weekends and stayed at an actual hotel 2 weeks, not AirBnB.  Travel time ranges 7 to 9 hours, plus the 40 billable hours and admin time, averaging 56 hours per week.


What did I do on the project?  Well, I created reports in DBVisualizer, against AWS Data Lake Hive tables.  Luckily I have a ton of Insurance knowledge at the business and data level as I use to balance the books for an Insurance company for 4 years.  Written Premium, Earned Premium, Inforce, Claims, Commissions, Tax reports.  And I got to work with a great team of report writers.


So what's next?  Well, they extended the contract 8 more weeks.  I plan to work from home next week as I have to take care of some things.  Will continue to write reports remotely and Skype for Business.  And then travel back to Boston.  And have a scheduled vacation in a few weeks.  Sure will be nice to hang out with the dogs, catch up on some stuff and spend time with the Mrs.  I'm a bit tired.


Thanks Boston, see you soon~!

4/21/2017

Compare Option in DBVisualizer

I've been working with a tool DBVisualizer.  This tool allows development against a variety of databases.  I happen to be working against AWS Data Lake Hive tables.  After you configure the Connection, you can go to the tables tree view, expand, highlight all objects, right click, script objects, to a file or window.  I selected the "Create" button and ran it.  544 objects to create, runs for a while.  So we have a file containing all table objects.


From there, we connect to another environment.  Assuming I had access to the Production Environment, I would perform same steps, generate a second file.


From there, you can click the Tools dropdown, Compare option, select your 2 new files, and see the differences. 




Now to view the differences, green indicates new:




This is a handy feature when developing, as sometimes the objects do differ between environments, and it's no fun to deploy a report that's been fully validated, only to have it fail in production, with different objects.  Missing Views or Tables, fields renamed or missing.


And there you have it~!

Humming Along in Boston

Flying home today, completion of week #7 on latest project.  Working for an insurance company in downtown Boston.  Used frequent flyer points to have my wife stay with me for a week, which was a nice treat.

Boston is such a vibrant town, from Red Sox, to American History, bus tours, restaurants, we had a great time.  Since my wife grew up in these parts, she was able to visit longtime friends and family.  We should arrive home around midnight, some chores, quick Rowdies soccer game, and back at the airport for 6:40am flight Sunday.  And see the pupperonies!

I'm enjoying this project, basically writing SQL statements against AWS Data Lake Hive tables.  One interesting thing, I worked for an insurance company a while back, balanced the books for two Insurance companies, so I learned the business inside and out.  What's interesting is the current project utilizes the same lingo, so having that deep industry knowledge helps.  And many of the reports are similar, Claims, Coverages, Written vs Earned Premium, Commissions.

I've lived in Florida since about 1982 and worked there since about 1996.  It sure is refreshing to get new experience in other regions with other clients and other technologies.

A breath of fresh air~!

4/20/2017

Hive SQL Date Functions Cheat Sheet


Since I've been working with Hive SQL lately, against AWS Data Lake, assembled a quick list of key Date functions to speed up development:





SELECT
    from_unixtime(unix_timestamp(TO_DATE(CURRENT_DATE) , 'yyyy-MM-dd'), 'MM-dd-yyyy') AS TodaysDate
    ,
    from_unixtime(unix_timestamp(DATE_ADD(CURRENT_DATE,-(DAY(CURRENT_DATE)-1)), 'yyyy-MM-dd'),
    'MM-dd-yyyy') AS FirstDayThisMonth ,
    from_unixtime(unix_timestamp(LAST_DAY(DATE_ADD(CURRENT_DATE,-(DAY(CURRENT_DATE)-1))),
    'yyyy-MM-dd'), 'MM-dd-yyyy') AS LastDayThisMonth,
    from_unixtime(unix_timestamp((DATE_ADD(CURRENT_DATE, -1-DAY (CURRENT_DATE))) , 'yyyy-MM-dd'),
    'MM-01-yyyy') AS FirstDayPriorMonth,
    from_unixtime(unix_timestamp(DATE_ADD(CURRENT_DATE,-(DAY(CURRENT_DATE)+1)) , 'yyyy-MM-dd'),
    'MM-dd-yyyy')                                                              AS LastDayPriorMonth,
    from_unixtime(unix_timestamp(TO_DATE(CURRENT_DATE) , 'yyyy-MM-dd'), '01-01-yyyy') AS
    FirstDayThisYear,
    from_unixtime(unix_timestamp(TO_DATE(CURRENT_DATE) , 'yyyy-MM-dd'), '12-31-yyyy') AS
    LastDayThisYear,
    from_unixtime(unix_timestamp(date_sub(concat(from_unixtime(unix_timestamp(), YEAR(CURRENT_DATE)
    -1), '-01-01'), 0), 'yyyy-MM-dd'), 'MM-dd-yyyy') AS FirstDayPriorYear,
    from_unixtime(unix_timestamp(date_sub(concat(from_unixtime(unix_timestamp(), YEAR(CURRENT_DATE)
    -1), '-12-31'), 0), 'yyyy-MM-dd'), 'MM-dd-yyyy') AS LastDayPriorYear,
    DATE_ADD(CURRENT_DATE, -90)                         TodayMinus90Days,
    from_unixtime(unix_timestamp((DATE_ADD(CURRENT_DATE, -1-DAY(CURRENT_DATE))) , 'yyyy-MM-dd'),
    'MMM')                                                                      AS PriorMonth3Char,
    DATEDIFF(TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP())), TO_DATE(CSI.LOSSDT)) <90 as="" br="">    CheckForXDaysAgoTrueFalse
FROM
    eis_app.CLAIMSSUMMARYINFO CSI limit 1

















4/19/2017

Time

Time

So
much
time

Tick
Tock

Not

So
much
time

Wasting
the
Hours
Away

Times
Up

Where
Did
all
the
Time
Go

Wasted
the
Hours
Away

4/12/2017

Thanks for Reading all these Years

Here's a blog post to mark the 300,000 page views for this site.  It says the first post was back in 2010 which could be accurate, as I've done a few social media purges over the years and got rid of a bunch of stuff.  It says there're 1208 blog post on the site currently,

I originally didn't want ads on the site.  Added them as it was the thing to do.  You hear of people making $5k a month on ad revenue.  Remove 4 zeros more like it.

I grew up behind a keyboard, not sure the exact year, but it was an IBM PC original, no hard drive, just 2 floppies, color chrome monitor, dot matrix Epson printer, no mouse, and PC Dos.  Graphics weren't a thing really, we had modem's 1200 baud connect to local BBS boards.  That was way before the internet revolution, PC in every home, Smart Phone on every belt clip.

I didn't major in computers, just a few courses here and there.  Got into IT in 1996 or so.  Report writer, SQL, programmer client server.

Then web.  Then dot net.  Then Java.  Some project management.  Then Supervisor.  Then BI guy.  Then consultant.


We all have different motivators.  Some go for money.  Some go for fame.  I like to solve the problems that others can't.  Money is okay as a driver, up to a point.  If you don't enjoy the daily grind, you can't succeed.

Programming is a continuous battle of micro problems to be solved every single day, and like gravity constantly pulling us back to earth and inertia slowing us down, we get ambushed by the flood of new technology.  And more often than not, you don't get to choose the technology of your liking, at least for my career anyway.  I've had a tough time breaking into cutting edge technology, not sure why.

As far as blogging, I still enjoy it.  It's no longer fresh, where I can pump out 8 to 10 posts in an hour.  When you write, it just flows.  You don't know what you're going to write about until its on the screen and you go back and read it for the first time.  I feel people get offended by some post and I've had to filter back a lot of content, which reduces the quality, no longer writing out of spontaneity.

When it's raw, that's the best stuff.  Political correctness has removed our sense of being human and quality has gone with it.  As we inch ever closer to our robot future selves, devoid of emotions, talking monotone.  If you look to television for content generation, all the good shows happened in the past.  Same with movies, music also.  The quality was just better then.  I don't watch TV anymore nor do I go to the movies.  And I don't read the papers or listen to the radio.

There seems to be a complete drought of fresh content, that stimulates the mind, with new thoughts and ideas.  Society is stale, like a fishbowl that hasn't been cleaned.  Some days I feel like stirring it up.  Other days I seek a new bowl.  Either way, I continue writing my stories, for nickels in revenue per month, without the slightest indication that anyone reads them.  As far as the 300k blog reads, perhaps, I do know that every Saturday night, around 2am, somebody in Russia scrapes the site and reads 200 plus pages in under an hour, so that inflates the numbers, no way to block it, so who cares.

To summarize, for those who do read the blog, I appreciate it.  And for my 12th grade English teacher who thought I wasn't paying attention in the back of the class, I make a deliberate effort to put a space between "a lot" and not "alot".  I was listening the entire time, so thanks!


~JB

4/10/2017

User Groups

the Kleptomaniac's Group will be post-poned as the signs have mysteriously disappeared, again.

The Claustrophobia's Group was really crammed last night with so many members.

The Pessimist's Group was a complete waste of time as everyone predicted.

The Anti-Social User Group continues to not meet next month.

The Web Designers User Group's web site is down again, so meeting will be held next month.

The Caffeine Anonymous User Group will be held at the local coffee shop.

The Chronically Late User Group will be meeting at 7pm instead of 6pm, something came up.

Week 6

Today started week 6 of the current project in Boston, MA.  The work is fun, people are easy to work with, technology is good and it's humming along nicely.

I'm out of town 6 days a week, takes between 7 and 10 hour each way home and back.  Averaging 57 hours per week including timesheets, travel and expense reports.

The project is slotted for 8 weeks, so the end is in sight.  Of course, it could get extended.  If so, I'll probably find direct flights, arrive Monday return Thursday and try to work remote more often as much of my communication is via Skype for Business throughout the day.

And there you have it~!

Keep on Blogging

Blogging is fun.  If it weren't, I wouldn't blog.

Some topics are related to technology.  As it's basically a technical blog.

Some blogs are a bit out there.  As you may be aware.

I've been here almost half a century.  If you view the career / life timeline, it's a stead and steep incline for the past 26 years.  Things have only gotten better.

Do you actually think my view points in life, obscure and outlandish they may be, have any impact on the ability to get work done, fast and accurate, learn on the fly?

In fact, the stories illustrate a variety of life skills: the ability to think outside the box; attention to details; observation skills; storytelling; abstract thought; critical thinking; ability to weave desperate stories into a cohesive whole; variety of interest; life experiences; knowledge of variety of subjects; cutting edge technology; continuous learning, discipline.

Should I be concerned that the story topics are read by clients and workers, I don't think so.

Perhaps we should be more concerned with those who have no social media presence, who could care less about continuous learning, narrow minded, lazy.

For those who don't understand this style of writing, feel free to read elsewhere.  For those who enjoy the topics, thanks for reading.  Either way, I'll probably continue to blog as I still enjoy it.

And there you have it~!

4/03/2017

USL Soccer in Tampa Bay - the Rowdies

I was real surprised my wife wanted to go see a soccer match.  Turns out, the Tampa Bay Rowdies are the team to see.  We went to a game last season, descent seats, and had a really good time.  The team supports family atmosphere, the stadium is really cool as it faces the marina, and the players are talented.

One game, we saw Joe Cole do an upside down backwards bicycle kick.  That was an amazing kick which doesn't happen very often. 


One game we headed down to the midline seats to see what they looked like and one of the executives saw us walking there and got one of the employees to give us a tour.  They let us sit in the fancier seats and when we called the following week, we upgraded our seats to the Midfield.

We attended every game last year and it was really a lot of fun for both of us.

We went to a game at USF last year which was a lot of fun.  We went to the Florida Cup, got to see some amazing international players.  This year, we went to the pre-season matches, one of which was held at University of Tampa.  We attended the fan appreciate party and got to meet a lot of the players.  This season the team started off with a bang and won their first two matches in a new division called the USL.

Explosive offence, solid defense, fast players, lots of shots on goal and all around hustle.  I'd have to say they look impressive.

The thing is, before I got into tennis I played soccer up north for the Town of Poughkeepsie traveling team.  We went to a lot of different cities playing tournaments and such.  You had to try out for the team and it was a lot of fun.  If I didn't have to have hernia surgery in 7th grade, I would have played on the school team.  Only played a year of soccer in Florida, but I did attend a game or two at the old Tampa Bay sombrero stadium when the original Rowdies played in Tampa.

Looking forward to a great season and quality time with the Mrs.

The Rowdies are, a Kick in the Grass!

4/02/2017

Hadoop Project

Much to my surprise, the guy slotted for the Hadoop project left the company.  And so did the next in line.  I spoke with my wife, I wanted to work on Hadoop.  Sure I learned it 4 years prior, never got a shot at it.  I emailed the boss, sure enough, I got the project.
The first week, client offered to stand up the Hadoop cluster.  Except we learned Hadoop no longer supported on Windows operating system.  I was on vacation the 2nd week of the project.  I decided to take the laptop on vacation to the cabin and got Hadoop working on Microsoft Windows 10 Hyper-V, no easy feat.  I configured it to allow remote connection from the laptop.  After doing some research, turned out Visual Studio 2015 had 3 new components to work with Hadoop.  I played around with it and got flat files to flow t HDFS, send Pig Scripts which worked and pushed data to Hive tables.  That was all I needed to return on Week 3 of the project.  Upon return, I was brought in to assist in the troubleshooting and we reinstalled Hadoop one or two more times, finally got it stood up on Linux, Master Node with three Data Nodes.  I ported my Visual Studio 2015 source code to the server, connected with Git and the project was humming along nicely.  Data flowed from Excel, to text files, pushed from Shared Folder to HDFS, Pig script to message the data, pushed into Hive tables, then Hive ORC tables.  Then, figured out how to install Polybase on SQL Server 2016, allowing seamless flow of data from Hadoop to SQL Server using common T-SQL.  I architected the SSIS project.  Flowed the data into Master Data Service entities.  Ran the data through some custom logic to parse and clean the address.  Then called an SSIS component in c# to call an internal web service to Geo Code the Latitude and Longitude.  Then called another web service using c# to send in Lat & Lon to get Geo Tagging.  Then pushed data into SQL Server ODS tables.  However, around that time, the Hadoop cluster went down.  I troubleshot the server for hours and hours and hours, deep into the nights.  The thing about Hadoop, if it stops working, there are so many configuration files to go through and investigate.  If one file has one entry with an extra space, the entire thing stops working.  I looked through every folder and many files interrogating everything.  And took tedious notes along the way.  And searched many a website/blog in attempting to fix.   Permissions.  Config files.   User accounts.  Error logs.  Host files.  You name it.  I was told to hold off on the issue as I had to jump back on my actual assignment.  I couldn’t let it rest and would work on it after hours.  Then I saw something, in one of the webs, you could see the jobs: success, failed, killed, etc.  There it was.  After picking through the logs, I saw the job was still running.  And so were 150 other jobs.  Which I initiated during my development.  The server got backed up.  Found a site on how to kill the process.  And proceeded to execute a command to kill 1 process at a time, for 150+ processes.  Restarted  Ambari, bam!  I could execute Hive with no error on the command line.  Then flowed some Pig scripts through Visual Studio 2015 using the WebHCat server and sure enough, back in business.  Solved it~!
I like to solve difficult problems.  Especially the ones that others gave up on.  Those are the juicy problems that are not easy to find.  That took some meticulous troubleshooting for many, many hours.  I rolled off the Hadoop project before the Data Warehouse was completed but I created the SSIS files to handle the Dim and Fact tables as well as refresh the Tabular Model.   That is what you call a great project.

And there you have it~!

First Crystal Reports Job

After I got into IT full time, I soon left for greener pastures.  I found a job doing Crystal Reports full time.  This was 1998 or so, when nobody was doing reporting for a living.  I got hired to work for a small company.  First day on the job, boss pulls me into his office, closes the door.  I have to tell you, we are being purchased by a big company, couldn’t tell you during the interview process.  Who?  Kronos.
So I was soon writing Crystal Reports for hospitals across the country.  Timekeeping reports.  Tons of reports queued up, some modifications, enhancements.  We had account reps that went out to the hospitals, took care of their Oracle instances, etc. and loaded my reports.  It was a cool job.  I worked with a c++ programmer and a guy that wrote documentation, we were definitely unique bunch of programmers to say the least, with a “different” sense of humor that only we got.  Finally people I could relate too.
The boss asked if I could write a small application in Visual Basic 4 at the time.  It basically kept track of Bugs, a bug Tracker.  The best part, VB had a cool install package wizard and I added that so internal people could load on the machines, that went over well.  Although there were bugs for the bug tracker also.
I wanted to do more programming and not just reports, but they said I should take a COBOL class, with that I found a new job.
And that’s the story of working with Crystal Reports way before people saw the value of data and flooded the market with people and products and experts.  I still like writing reports, it’s an acquired taste perhaps.  Swimming in the numbers trying to get them to match.  A puzzle every single day of work.
And there you have it~!

3/26/2017

Will automation take people's jobs?

When you drive through a toll booth, there's nobody waiting to take your money, make change, small talk.  Just a bunch of empty booths.  With a sign indicating your charge will be though your license plate.  A mysterious picture scanning your plate, determine the charge, finding its way to your mailbox.  Or pre-purchased account on your windshield.

I drove to the ATM, took out some cash.  No teller necessary.  When I called the customer service,  a friendly voice instructing which buttons to push, authorize my credentials, perform the transaction.

Will automation take people's jobs?

That's a good question.

One possible theory, artificial intelligence will wipe out humanity, or put us in cages at the zoo, or will outperform humans to such a degree that we become obsolete.  P
eople speak of a new form of socialism or guaranteed basic income where people will enjoy luxury activities in their idle time.

Some question comes to mind, 


  • Do you think the banks will forgive people''s outstanding debt.
  • Who will pay for people's healthcare
  • Who will pay for people food.
  • Who will pay for people housing.
  • Why would people reproduce if their offspring have no chance of employment.
  • Will people enter debtors prison for unpaid debt.

I don't think we know at this point.  Let's examine some current trends.

  • We know many jobs have already been automated.
  • We know that prices are going up sharply for basic goods and services
  • We know that many incomes are flat or decreasing.
  • We know that job competition is stiff for low wage earners.
  • We know that some high paying jobs have been pushed offshore.
  • We know people are working more hours.
  • We know people are working well past retirement age.
  • We know average household debt is approximately $12,000.
  • We know many households are upside down on their mortgages and many have entered the foreclosure process.
  • We know that student loan debt has risen substantially and is longer candidate to write off during bankruptcy.
  • We know the cost of healthcare is rising and there's a penalty for not having any.

Let's take out our magic crystal ball and attempt to synthesize these observations.

Are the current trends benefiting society as a whole.

If we circle back to automation and artificial intelligence, will they have an impact on society.  


  • Will somebody step in and subsidize payments for a good chunk of society and if so, how will that get financed.
  • What will people do 24 hours a day with no chance of employment ever.  
  • How will the logistic take place to house and feed and care for all these unemployed people, forever.
  • With less money flowing through the economy, what impact will that have to business'.

This is a tricky sticky subject and I'm surely no expert in the field.  I'm just asking basic questions on possible scenarios and how they could play out.  And pointing out some trends and food for thought to see which way we're heading.

One thing to keep in mind, if the purpose of a business is to maximize profit and decrease costs, and automation and Artificial Intelligence are definite ways to accomplish those, what impact will that have on the economy, humanity and the planet.  Are there any safety nets in place or in the works to ensure a soft landing if automation was to cause ripple effects.  Will humanity continue it's trend of upward mobility, from the jungle, into the grasslands, onto farms, into cities.

Perhaps time will tell.

3/19/2017

The Data is Priceless

We hear the drum that data is the new oil.

IBM owns the weather channel.  Surely those weather points are valuable.

Microsoft owns LinkedIn.  Sure that data is valuable.  Just about every person that is employed is on LinkedIn.  With their complete work history, timelines, places, job descriptions.  How much could the data alone be worth.  Priceless.

What are some other data points that could be purchased?  That's what investors should focus on.  That data is worth more than diamonds, oil, land.

In my humble opinion.

3/18/2017

Quick Ideas for New Software Apps

I was thinking of some apps for the smartphone.

Like a diet app, when it determines your location is a bakery or fast food joint, it send an electronic zap to your phone.  Call it "Shock Diet".

Here's a handy feature, have a sensor in your mailbox, sends notification when mail is delivered.  How many times have you walked to the mailbox, opened the box, nothing. 5 x in a day.  That would save time and money.  But then again, surprised we still have snail mail.  Call it "MailboxAlert".

How about an app that can scan your food, tell you how many calories, how many sit ups you'll need to do to work it off, how many days it will cut short your life.  Call it "DietaryGenie".

How about an app so when you fly, it tells you what city/state/providence you are flying over at that exact moment.  Call it "FlyOverU".

How about an app that tells you where things are in a store.  Have you ever walked around a giant box store, wondering aimlessly, with no sales people in sight.  Call it "SalesPersonGhostTown".

How about an app that does your waiting on hold for you.  We are experience longer than usual wait times, your call should be handled in a week from Tuesday.  Call it "IfWeActuallyCared We'dHireSufficientStaff".

How about an app that rates the office staff of a place you visit.  You know the ones that could care less who you are or why you're there, that make you wait for long periods while the surf their smart phones.  Call it "TooLazyForARealJob".

How about an app that monitor's the free items at food places.  You know the people who take 25 sugar packs to stock up at home, or salt, napkins, creamers.  How about an app that limits these petty thieves.  Call it "Ain'tStealingHereNoMore".

That's all I can think of.  I should have those apps completed by lunchtime.

Thanks for reading~!

9 Things to a Happier Life

Get rid of cable.  First of all, it ain't cheap.  Second, it turns the mind to mush.  Third, think of all the better things you could be doing.  Get a part time job, read a book, walk in nature, fix the house, anything is better.

Get rid of your land line.  With all the negative aspects to smart phones, they really do assist in your day to day activities.  Of course you can talk on them, get voicemail, read emails, find directions, etc.  I don't know how we existed without them.  Why would you need a land line, other than to host a fax machine.  And get rid of you musket and covered wagons too.

Outsource the stuff you don't want to do.  Sure it may costs money, if you can free up your time to do things you enjoy, it's worth it.

Do the stuff you've been putting off.  Nobody's getting younger.  There's only so much time.  People get health issues.  Fact of life.  Don't wait until the end, there won't be time.

Enjoy the ride.  Life is a marathon.  There is no imaginary place off in the future, where everything magically becomes great.  That theme generates just about every marketing effort since the beginning of time.  That illusion will cost you.  Life may not be perfect, but it's all we have.  You don't have to stop and smell the roses, enjoy the treasures of life in the small details of everyday living.

These are the good old days.  Guess what, taxes will probably go up for the rest of our lives.  Aches and pains become more frequent.  People will not always be around.  Life has been a struggle going back to the days in the jungle.  Life is what it is.  It will never be perfect.  Best enjoy life and be thankful for what you have.  You may not always have it.

Do nice things.  It's easy to turn a blind eye.  Throw that rubbish on street.  Cut someone off in traffic.  Doesn't take much skill or effort to be self absorbed.  Give someone a compliment.  Do more than what's asked.  Show up early, stay late.  The world could use some kindness.

Get a dog.  Dogs are God's present to mankind.  They are loyal, caring and furry.  We grew up with cats.  Cats are great, but they aren't dogs.  Dogs are tremendous.  We have 3 spoiled dogs.

Get rid of fear.  Some people say there are only two forces in the Universe, Love and Fear.  Fear is a lower vibration.  We find it everywhere.  Television, news, you name it.  People are afraid of just about everything, including fear itself.  Fear is mostly an illusion.  Except when you get that feeling that a tiger is about to eat you while drinking from the watering hole.  That fear is real.  It's the false fear that weighs us down.  And turns us into not so nice people.  Fear keeps people afraid.  When people are afraid, they become docile and make you old before your time.  Stay young.  Don't fear fear.  Just an illusion.

And there you have it~!

Reunion in Boston

It's not everyday you have a family reunion.  Well, today was that day.

While in Boston, MA for business, my mother was in town as my 97 year old Grandmother wasn't feeling well.  She arranged to meet along with my Aunt and Uncle.  We met at a nice Italian restaurant and had a great reunion talking about the past and present.  Sure has been a long time.

Bit of a chilly night, yet warmer than last week.


Earlier in the day, had a chance to walk over to Harvard University.  Went to the Peabody Museum of Archaeology and Natural History, I majored in Anthropology and Archaelogy.  Enjoyed viewing the ancient Native American Indians, especially from the SE and NE of USA.  The old brick buildings were everywhere and the weather was quite nice.



Good times~!

3/15/2017

Tennis & Programming Similarities

When you've been playing tennis all your life, you can basically pick up a racquet at any time and play descent tennis.  Sure the foot work, timing and endurance are slower than expected, but the natural game is still there.

When I was a junior, I stopped all other sports to focus on tennis.  My coach Cid played #1 position for the local university.  Unorthodox, yet fierce.  Had a one hand backhand slice, the ball never rose.  Pin point accuracy.  And used the same grip for both forehand and backhand.  Serve wasn't spectacular, could place the ball anywhere on the court at any time with complete accuracy.  And never missed a shot due to unforced error.

That's the game he taught me.  Minus the one hand slice backhand.  Primarily because I throw left handed and played right hand, so the two handed backhand was better than the forehand.

Every lesson, had to do physical strength exercises.  And consistency.  Hit a forehand crosscourt, inside the singles lines, behind the service box.  25 times in a row without missing, or start over.  Then go down the line.  Then backhands, same thing.  Volleys.  Overheads.  Serves.  And running.

Those private lessons helped to compete with the local community players.  Our coach taught at Nick Bolletteri in Bradenton, Florida.  The best players in town practiced every night after school.

Some of those juniors went on to the pros.  The interesting thing, when you've played at a certain level, you can determine a players ability in a short time.  When I taught tennis for a while, you could pick up players strengths and weakness pretty quick.  Did they have talent.  Were they still getting better or have they plateaued.  Some people get to a certain level, and never get better.  I'm sure the same applies to life.  I think programming is similar.  You can size up a persons skills and depth of knowledge.  Are they keyword specialist, do they have history of proven success, do they avoid work at all costs, are they continuous learners or one hit wonders.

I've played tennis since the 6th grade and got started in computers in the 8th grade.  Quite a long time on both fronts.  I find the similarities are, performing the actions without thought.  In tennis, after you have the skills, you run on instinct.  Maybe true for programming too.  Who knows.

I suppose the major takeaway is this.  You practice and learn for a long time.  Without knowing it, at some point, you have the skills to teach others.  I still enjoy hitting the tennis ball.  And still enjoy solving complex problems.  Just trying to get better, and enjoy the ride.

3/14/2017

AWS Data Lake Hadoop Hive with DBVisualizer Project

About midway through the 2nd week of an 8 week project.  I'm working for a large insurance company located in Downtown Boston.  What technologies am I working on for this project?  I work on Operational Reports for the Actuarial department.  They have a source database, a team that gets the data into AWS Data Lake, Hadoop Hive tables.  We connect using an IDE called DBVisualizer and write custom SQL statements.  Also some Power BI and Tableau development. 

I spent some time researching Hive optimization techniques.  They have partitioning, bucketing, indexing, writing better SQL code, but they also have other options.  They recommend using Sort By rather than Order by, specify the order of your Group By fields, avoid nested Sub-Queries, use Between rather than <= and >=.  

Found a few good links I read:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.0.2/ds_Hive/optimize-joins.html

http://stackoverflow.com/questions/32370033/hive-join-optimization

https://www.justanalytics.com/blog/hive-tez-query-optimization

https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-optimize-hive-query

https://hortonworks.com/blog/5-ways-make-hive-queries-run-faster/

https://www.justanalytics.com/blog/hive-tez-sql-query-optimization-best-practices

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_performance_tuning/content/ch_query_optimization_hive.html

Basically its full life cycle report development.  Gather specs, map the fields, write the queries, validate the data with the Business, deploy to production, document, maintain and enhance.   I've worked for an Insurance company before, so I understand the basic concepts such as Inforce, Written Premium, Earned Premium, Claim Payments, etc. 

I do enjoy working in different regions with different clients, people, projects, challenges, scenery and weather.  I guess that's one good thing about consulting, never the same day twice.

And there you have it~!

3/02/2017

Great Podcast on Anthropology and Technology

I've been reading and listening to James Altucher for a long time.  Quite a fascinating person with interesting background and exemplary chess player.

I just listened to a fascinating podcast.  It talks about Anthropology and Technology.  Lots of intelligent discussion on real subjects that will shape society.

Ep. 216: Yuval Noah Harari – The Next Step in Our Evolution

Here's the link: http://www.jamesaltucher.com/2017/03/yuval-noah-harari/?utm_source=linkedin&utm_medium=syndicate&utm_term=noah-harari&utm_content=playbar&utm_campaign=noah-harari

Have a listen...

3/01/2017

Getting Started with Docker

Microsoft now offers SQL Server on Linux.  Now that's big news.  Here's a blog post from the team:  https://blogs.microsoft.com/blog/2016/03/07/announcing-sql-server-on-linux/#sm.00016g1jw81e4bdoku6pmahks7tll

I read this link that has a download available for Public Preview:

https://www.microsoft.com/en-us/sql-server/sql-server-vnext-including-Linux



The first step, is to install Dockers for Windows 10 using this URL:  https://docs.docker.com/docker-for-windows/install/


I clicked the Stable channel, downloaded file, ran the install.


Install complete!



Docker has started...


In the task menu, there is a whale, right click to see version and settings:


There are several settings on this page which is easy to use and is similar to the Hyper-V Settings I've used in the past.

From the Advanced tab, I set the Memory to 2816, clicked apply, Docker resets.  As a note, I originally select 4096 and it threw an error insufficient memory.


It sets a default sub-net address, sub-net mask and you can modify the DNS server if needed:


Following the steps from this post:  https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup-docker we open our trusty Command Prompt, we check the Docker Version to verify it installed correctly (you can also use Power Shell):


Still within Command Prompt, we initiate the Pull request:


Downloading bits:


Extracting:

 
 
Completed, typed in > Docker info
 


Per the instructions on the website, type in:

docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=' -p 1433:1433 -d microsoft/mssql-server-Linux

It create a VHDX file which can be opened in Hyper-V on Widows 10:



Looking at Hyper-V, it loaded the new server as MobyLinuxVM:


From within Hyper-V, click Connect:


The VM did not load, so uninstalled Docker (stable) and downloaded the beta version.  Then initiated another pull, this time using Power Shell:


I poked around on some of the Docker blog posts and learned quite a bit.  I will use Power Shell to work on Docker going forward.

In time, I'll go back and get SQL Server working on a Docker Hyper-V VM.  Seems like a cool way to download pre-built containers, distribute and maintain images.

Thanks for reading~!

Mountain Living