Merge the Relational Database with a #BigData Distributed File System

Relational Databases have been around for a long time.

And Hadoop is rising in popularity.

How about this, merge the two.

Put a files system within the Relational Database.

For example, you take SQL Server, add a Microsoft Distributed File System within it, where you can add files, and then access it's data, combined.

Having a file system within the database makes sense, why keep them separate.

A single place to store data, mashing up structured and non-structured data via SQL.

And what would be the next obvious evolution, right, move it to the cloud.

Why have two separate systems, why not leverage the existing Database with all it's ETL and Business Intelligence features and combine it with Big Data.  No need to let the two systems talk to each other, have just one system.

I'm not sure why Hadoop doesn't have much competition in the file system distributed architecture.

Seems like the vendors would hop on board and add their own merged Relational / File System databases.

Maybe it's just a matter of time.


Employee #1

I left the Country two years ago, to work for the School Board.

I left the School Board on June 25, 2012 to work for GFI Software.

And GFI Software spun our division off to a new company called ThreatTrack Security.

Where I've been working diligently to convert all the apps to our side.

And I continued to work my part time jobs.

One for my health care client, which hasn't had much work really since October 2012.

And another contract for Agile Bay.  I've been working on a project for a client to build a Data Warehouse with some Crystal Reports.

And after meeting the client earlier this week, I was offered a full time position with Agile Bay.

To work as a Senior Business Intelligence Developer.

Work from home.

So I accepted!!  And gave my 2 week notice on Wednesday.

Which means I'll be employee #1.

I've got the full support of my wife and I'm ready to dive into the world of consulting.

I think it's going to work out great!

#EmbeddedBI Tweet Chat

I was lucky to attend #BIWisdom weekly tweet chat yesterday.

The topic of discussion was Embedded Business Intelligence.

Basically, it has to do with displaying metrics within another application.

I tweeted:

There're many vendors to choose from depending on your business objectives and Embedded BI could provide value if you want to customize your web app using Widgets, custom code or a combination of the two.
Overall, I see embedded BI as a useful methodology to display data to the users.
The full life cycle of BI has many components: from storing the data; to ETL, moving data and applying business rules; to data cleansing; to presenting data to users.
As we know there's no "one size fits all" in Business Intelligence.
Choose wisely!


Always Be Improving

Today I've gotten a lot done.

Woke up, fed the dogs, let them out, did the litter boxes.

Disassembled my old desk, set up my new IKEA desk.

Then did 4 loads of laundry.

Did the monthly shopping at Costco, then picked up some food at the vet for the cat.

Then mowed the lawn, took care of the yard by trimming the bushes, then a quick dip in the pool.

And now to do some part time work.

Then off to dinner, food shopping and call it a night.

I'm taking a floating holiday on Tuesday, then working from home on Friday as Maddie goes into the vet for surgery, remove a cyst.

So the key lesson is always be moving, always be improving.

And so it goes.


Everyone is Really an Independent Consultant

It used to be standard procedure that Full Time employment was the way to go.

First of all, you got job security.

And benefits.

However, unless you've been sleep working for the past decade, the world has changed.

Job security is a thing of the past.

Your employer will send you to the curb at the first hiccup in the stock price.

Or when new management takes over and decides to clean house and bring in their buddy's from their last company.

In case you were not aware, we are all "Independent Consultants" now.

Your job is to provide the best product / service you can for each day that you go to work.

Because you may not be coming back tomorrow.

Your relationship with your employer may be full time, yet you should really treat it as a daily contract.  If you do good work today, there's a good chance you'll be invited back tomorrow.

Do a good job, try to add value and learn something new each day.

Isn't that basically what an Independent Consultant does.

The only real advantage to full time is the benefits.  Not the paid time off.  But the Health plan.

Because Health Insurance is the last major scam played on society.  Yet everyone needs it.

So spend some time rethinking your relationship with your full time employer.  Perhaps you're not as "secure" as you thought.  And you should make plans so that if they let you go, for whatever reason, you'll land on your feet.  And be better prepared next time around.

Just saying!

What's in a Title

I think my current title is Analytic Developer.

Notice the lack of "Senior".

Which is odd.

Because I've been a "Senior" developer for the past 10 years.

And my last job, I was Reporting Administrator for the School Board, which was a Supervisor position.

A lot of people spend a lot of time worrying about their specific job title.

So it has some *bling* on LinkedIn or your business card.

I think having the word "Principle" or "Scientist" sounds good in the world of technology.

Yet I'll have to settle for whatever the company chooses.

Thing is, I'll probably change the title on the resume at some point.

And here's the funny thing, now that the word "Senior" dropped off my job title, my salary went up 33%.

Funny but true.

And so it goes!

Real World of a BI Developer

Some organization fly by the seat of their pants.

They have a Business Intelligence programmer.

Who's job is to maintain production.

House the business rules.

Create new applications.

The BI developer must do all the specification gathering.

Learn the business rules from osmosis, because there is no documentation, anywhere.

Obviously, they must already know every technology the org supports, and learn the new stuff immediately, on the fly.

They must be their own project manager.

And manage their own priorities, as every task is top priority, so they must get all done regardless of time allowed.

They must QA their own work.

Then battle the Admin's to get the app deployed.

And that's when the fun starts.  Every manager from every corner chimes in shouting how the work is wrong, none of the numbers match, the programmer did not create the product the customer requested.

The entire company is now focused on blaming the BI developer.

And the BI developer must take time to defend their actions.

At the same time, fix all the work they just did, in production, with all eyes on him/her.

You think this stuff doesn't happen in the real world.

I'm guess it's not stated in any the BI training manuals.

And so it goes!

Choose a Language that will Go the Distance

Being a programmer you have to choose what technology you want to work in.

Many people select a Brand of Technology, like web development.

Then they select a Vendor, let's say Microsoft .net.

Then they choose a Language, let's say c#.

Then they choose a Sub-Language like JQuery and HTML5.

Perhaps because it's cutting edge.  Or because it pays a lot.  Or there's tons of free examples on the web.  Or because it's cool.

Except what happens next year, when the new flavor of the month comes along.

Take "Silverlight" programming for example.  Kind of passé if you think about it.

All that time and effort learning, perfecting, now nobody uses it.

So when choosing something to work in as your career, perhaps choose something that has longevity.

For me I choose SQL.  It hasn't change much since I started back in 1996.

Sure there've been new features added, but the core language hasn't changed much.

May not be as lucrative perhaps or glamorous, but you can have a solid career in it.

COBOL is the same thing, hasn't changed much, there's still jobs out there, which pay descent.

So think of your career as a marathon, and select something that will go the distance.

And so it goes!


Programmers Must Do More than Just Code

Programmers job is to program.

Write code, solve problems, add value.

Wouldn't it be nice if a programmer could survive based on their skills alone.

In reality, politics plays a huge role in the success of a programmer.

Are you connected to the right people in the organization?

Is business intelligence considered a vital part of the enterprise?

Do people understand the impact that BI brings?

Chances are, the biz is concerned with creating a quality product.

And Marketing is busy bringing in new customers.

And Sales department is frantically trying to meet their numbers.

The executives are busy running the business.

IT is busy keeping the infrastructure operating.

Help Desk / Support has too many incoming calls.

How does Business Intelligence get in front of the right people to make themselves known.

So they can impact the business, they can keep their internal customer's happy and meet their needs, by keeping up with technology, being cost effective and providing value?

Almost every problem that occurs in the org, has political impact. 

Some department is throwing blame and another is ducking for cover, people use these problems as grenades to push their agenda one way or the other.

Programmers are sometimes pawns in the game.

Perhaps Sales are low, because reports are not available, timely or accurate.

Perhaps Marketing is making plans behind the scenes to rid themselves of their dependency on BI.

Help Desk team goes out and buys their own software package to manage the call volume.

Executives aren't getting the data they need to base their decisions.

The corporate environment is a battleground.  And BI is sitting in open territory getting nailed from all angles.  Unless BI is ingrained in the infrastructure and supported from the higher ups, being a BI programmer is not so easy.  They don't get to sit and write code all day.  They are sending emails to get people to help, with the business rules, with getting the tools they need to do their jobs, to getting a seat at the table when decisions are made.

It would be nice if Programmers wrote code all day, to solve problems and add value.

Except in the real world, that's just not the case, in many places.

Programmers need to be skilled in technology, in the sales and marketing process, in understanding all facets of the org, to being sales people themselves by selling to internal customers, to being accountants to support their applications, to being lawyers so they don't get beat up from every department, to being psychologist to better understand their coworkers as well as being politicians.

Programming is not just about code, you have to be everything to everybody to prove your value in today's workforce.


What's causing the blank fields?

We had a bunch of new changes this week - 3 major systems got ported over or automated.

Which broke my reports and my cube.

I spent all day investigating.  Couldn't figure out what the heck was causing it.

My boss went home said give it a rest, try again Friday.

So I did what I usually do, never give up.

Why were the State / Country fields blank???

Got someone to give me the source code, partial, didn't help.

Asked everyone I could think of.

I sent an instant messenger to one of the processing managers.

She IMd back 2 hours later.

Asked her to stop by my desk, showed her the issue.

Said that's probably because we changed the procedure in Great Plains, we enter that info on another screen.

Bam!  Solution found.

Emailed the interested parties, my boss replied from home:

"Nice detective work Jon!
Nothing gets by you does it?!"
Nice way to roll into the 4th of July.


Swimming in the Data

What do you do?

Program computers.

What segment?

Business Intelligence.

Reporting, ETL, Analytics?

All of the above.


Reconciliation of numbers across different data sets.

How does that work?

You start off with a known set of numbers.  You have a second set of numbers which don't match.  Find out why.

So you would think to first find all the matches, then find the deltas, then figure out what's causing the differences.

Except what if you are off by $10, you would naturally think to look for a simple $10 plus or native transaction.

This is almost never the case.

You may have +500, -600, +200, -90 resulting in a -10.

I used to manage the month end for an Insurance company and I literally swam in the data all day everyday.

Most programmers would steer clear of this type of job, except I seem to enjoy it.

It's like solving a puzzle, just a matter of time before you find all the pieces and put them in the correct pattern so the solution presents itself.

I've been doing this at my full time job as I run the Month End and Sales Cube and a bunch of other reports / cubes.

Today I was able to track down some missing monies, which the super user was super happy.

Turns out some bad data was the culprit, so tomorrow I'll modify the code to handle the anomalies and the report should have the correct numbers.  Seems the Sales People would prefer the Commission reports be accurate, for some reason.

And there you have it.

#BigData the New Stealth Weapon

We've been talking about the benefits of Big Data.

How to harness the insight from that deluge of data.

It has risen in popularity and become mainstream.

And during it's path to popularity, some people questioned it's potential use.

If information fell into the wrong hands, it could be used as a weapon.

As in knowing everything about everyone all the time.

Kind of defeats what's formerly known as "privacy".

There's nothing you do that can't be tracked in some electronic format.

As you go through the motions of life, you are creating a digital log file.

So not only are your actions tracked, so are your inactions.

If you usually go grocery shopping on Saturday mornings, that is tracked.

So if you skip a week, that is also tracked because there's no entry for Credit Card at the local supermarket around 9am on Saturday.  So where were you?  What were you doing?  With whom?  Who did you call?  What for?  Why?

Don't worry, they already know all that.

Another example of how technology is being used against the population are Red Light cameras.  A pure money maker for local gov'ts.  They don't really help us in any way, they are working 24 / 7 days per week.  It gives them a reason to film our daily commutes.  How exactly are these devices helping?

Just saying that Big Data can be used to battle cancer, help ease our lives to make the world a better place.

Or it can put us in cages, track our every move and create a fascist 1984 society.

Perhaps time will tell.