5/31/2012

Programmer Styles

Not all programmers in the same space are equal.

There are some people who write superior code.

And some who write garbage.

And some in between.

I imagine we learn snippets of code, techniques, etc. that we keep in our tool belts.

And some we use more than others.

I've seen some amazing code in my day.

Stuff that I scratch my head and wonder what the heck is this code doing, just amazing!

And other code I wish I never looked at.

My code is in between, somewhat standard code throughout, some good stuff but nothing fancy.

Keep it simple stupid.

So the maintenance person behind me doesn't cuss me out!

Data Scientist is a Concept

There's all sorts of data.

You have customer information, perhaps gathered from a website, such as E-Bay.

In this case, web developers wrote a web app, containing business logic, which allows users to perform an action, which gets saved most likely to a Relational Database.

That info could be ETL to a Data Warehouse.

Either could have reports written against it for consumption by employees at E-Bay.

Another type of website could be Twitter.  Not much functionality, but more of a data gathering / input site.  Data gets stored in mega-databases and can be interrogated by a report developer.

Another type of data would be a call center Phone Switch which registers the flow of incoming calls, how they got routed, duration of each leg of the call, dropped percentages, etc.  Very little user info here as everything is automated (users don't even know this info is recorded).

Another type of data could be a non B2B site, such as a gov't site where users poke around the site looking for info, etc.  Their clicks could be stored to a database to track the flow of user for example.  This data could be mined to observer user behavior and to identify trends.

However, with all this data, how does one extract meaning information to provide value to decision makers.

Well, let's track the flow of structured data:

1. Internal Customer has need for website or info gathering.
2. The Business Analyst gathers the specs.
3. The Web developers / Vendors create a website.
4. The Database Admin or Developer creates a database to store info.
5. Someone requests reports.
6. Business Analyst gathers Report Specs.
7. Report Developer writes Reports.
8. QA verifies the Reports.
9. Change Management Pushes the report to production.
10. Someone reviews data looking for patterns.
11. Decision Maker person presents finding to Top Execs.
12. Someone makes a business decision.

So, we've tracked the data flow from design to decision.

Now, where is the Data Scientist in this scenario?  Is he/she writing the reports #7?  Or #10, looks for patterns.

I don't know you tell me.

In this case, I think the role of the Data Scientist is the accumulation of all the pieces of the puzzle.

Which may or may not be a single person.


How about Semi-Structured Data:

Almost every organization has emails.  And most of those emails are archived.  And someone could mine those emails for patterns, trends, aggregate data, etc.  This may or not require a Data Scientist.

Now how about Un-Structured Data:

Let's say your organization has a network.  And on that network you have files.  And those files are PDF, Word, Excel, Text, etc.  Those files can be consumed by Big Data Hadoop Open Source Software for example, in named pair values.  And that data can be parsed, usually not in real time, and a person could extract nuggets of info from all that un-structured data.

So to summarize, I don't necessarily see a definite need for a Data Scientist in every organization.  Sometimes, for predictive analytic or data mining, where statistical calculations are required, well then yes, a Data Scientist may be required.

That's what I think (at the moment).

Tally-hoe!

5/30/2012

Process

We've been bombarded with ticket requests.

"My report is not correct."  "Please add Bob User so he can access reports."

Well, first off, users are not providing any details.

Second, no examples to validate / reproduce the error.

Third, there's no middle person to validate the errors are legitimate.

Fourth, there are existing process' in place to add new users to reports which they are not using.

Fifth, these issues appear to be enterprise level wide.

Solution = Processes

I have asked my boss for the authority to implement new process across the board.

Meaning tickets must be validated prior to getting to the programmers.

So users don't waste the programmers time.

We will gladly work on the ticket / bug fix once it's validated.

And everyone lives happily ever after.

Quotas

Quotas were created to keep people performing at optimal levels.

When I was approving mortgages, we had a quota to approve or decline so many loans per day.

I volunteered to count the loans volume for all my teammates and create reports on production levels, because the IT department was not responsive to the department's needs.

We had to look at 100 loans per day.  And we could override a certain percentages a day.

Luckily the quota never varied, it was always 100.

However when I worked in the branch, I was required to bring in so many loans per month and collect so many past dues per month.

The quota's were designed so that you never reached them.

And if you had a good month and reached your quota, they soon raised it accordingly.

However, I was not on a commission, it was just part of the job.

That was my fault as I should have worked in a  position where I was rewarded for my hard work.

Now with the introduction of Business Intelligence and metrics for every action, we are really setting ourselves up for a fall.

Because now the people who can make the rules will apply rules to all aspects of every job.

And soon your every action will be measured, sliced, diced, compared and in the end, your pay will be based on these metrics.

So we are basically creating our own cages.

Perhaps that's an extreme way of looking at it, but its seems quite accurate.

I've seen some sci-fi movies that do exactly this.

And as we know, most sci-fi movies eventually become reality.

And so it goes!

Expert or Generalist

As I look back at my career, I wonder if I should have chosen to be an expert in a specific technology or a generalist who knows a bit of everything.

That is indeed the question.

Starting with Visual Basic (4,5,6) I programmed applications that connected to Oracle and had reports in Crystal Reports.

Then switched to classic ASP to Oracle to Access.

Then to Actuate Reports to Oracle and Vantive to Oracle.

Then to .net to SQL-Server.

Then to Java to Mainframe and Crystal Reports / Business Objects to Oracle.

Then to SSRS to SQL-Server.

Not exactly a straight line of technology.

So that has provided a good career, crossing technologies and industries.

However, I see some programmers who have specialized in 15 years of Oracle or 8 years of .net or whatever.

They would be considered experts in the specific field.

So once again, should I have taken a more traditional approach to career and learned one or two products really well instead of across the board?

I don't know.  It's a toss up I suppose.

I don't have any statistical data to prove one way or the other.

Although I've been in Business Intelligence my entire career, I've seen new programmers pass me with the latest push of Microsoft BI because the company's I worked for were behind the times.

I suppose I could catch up in a hurry if need be.

I like to think of it as a utility baseball player who can play any position on the team vs. a starting pitcher who plays ever 3rd game in the rotation.

Both serve a purpose.

Just a preference I suppose.

5/29/2012

Typing / Keyboards

When I was in the 9th grade, I had the opportunity to attend a typing class.

Back in the early 1980's the typewriters we used had keys which you really had to slam down to get ink on the paper.

So I learned to type at an early age.

And my father had some programs for the IBM PC to practice typing which I constantly tried to improve my wpm.

Along came the computers and I was lucky to already know how to type.

So when I program, I can really fly.

Now the keyboards vary depending on where you go.

I just got a new PC at home so I'm still getting used to the Home and End keys.

Then my work PC has changed 3 times this year each time with a new keyboard.

This tends to slow me down as I have to think about what I'm doing.

And my laptop at home, keyboard is way different.

Have to keep re-adjusting with each different keyboard.

ssrs Line 16 Error "children.0.style" is null or not an object

Today we got an error in one of our SSRS 2005 Reports.

The following error was reported by several users:

Line 16 Error "children.0.style" is null or not an object

It turns out that the Column property --> Layout --> Fixed Header was set to TRUE.

After changing the Fixed Header property value to FALSE, the problem went away.

Hope this helps!

5/29/2012

5/28/2012

#BloomConsultingBI Day Tripper

Today we headed north to the Webster Flea Market, in Webster Florida.

We had the day off from work for Memorial Day and wanted to test drive my new Honda Element.

Turns out the car runs like a champ, air works fine, cosmetically good as well as gas mileage.

It started pouring rain on the drive up and I was wondering if our day adventure was going to be a wash.

However, as soon as we drove up and paid our $2 parking, the guy said it hadn't rained yet, and it never did.

We purchase our obligatory Carmel popcorn bag, my wife got a necklace, we bought a few car magnets and a dog bowl for the SPCA.

As soon as we left the parking lot, it was raining.

We stopped at Subway for a quick bite to eat, then headed south and it really opened up rain.

Overall a good morning day trip, good to get out of town and test the car on the highway.

Now I'm ready for a nap, but first a bike ride to get some exercise.


Go To Guy

Everyone talks about team work.

Yet a lot of managers still need to have a 'Go To Guy / Girl' (GTG).

Someone who delivers when called upon under pressure.

Bottom of the 9th, need a few runs, you bring in the GTG, they usually produce in the clutch.

Top executive requests a report, needs it by end of the day.

Report Server stopped working.

Bug in one of the most visible reports.

Bring in the GTG.

I suppose you could bump up their salary.

Or you could create a new title called Principle or Architect or what have you.

But you really can't give preferential treatment, in a team environment.

You can, but it causes friction amongst the ranks.

I don't think its talked about too much but every manager needs someone to solve problems really fast.

A GTG should not be utilized all day every day however.

This would indicate lack of process' or bigger issues.

And will burn out your star player.

I've been the GTG in several roles over the years.

And I enjoy putting out fires.

It's a required role in today's IT world if you ask me.

5/25/2012

Data Enlightenment

Data Enlightenment, the actualization of the realization that the data in your organization has purpose to drive business decisions.

And how do you get there?

Good question.

Ask 20 Data Experts, you'll get 20.5 expert answers.

You see, we have evolved into our current state of affairs, through trial and error.

And with the advent of new technology, we can now harness bigger volumes of data in shorter periods of time.

However, the general philosophy hasn't changed much.

Access data sets, in a readable report for consumption by someone in charge, who has authority to make business decisions from the interpretation of data into information.

Data can be small, from an Access database.  To a mid size relational database such as Oracle, SQL-Server, Sybase, etc.  To large data sets like Netezza or Hadoop.

There are also new concerns around data governance and quality data, as we would like to have some data reliability and common understanding of the data we are analyzing.

The delivery has gotten more complex with Dashboard roll up aggregated reports, drill-able to the more granular levels.

To Key Performance Indicators and slice & dicing the data through data warehouse data sets.

To delivery options such as tablets and mobile BI solutions.

To viewing reports generated in the Cloud.

So obviously reporting and business intelligence has gotten more complex.

Yet its really simple, "base decisions from the data".

And there you have it.

DBA vs. Developer Role

When you talk about Databases, you obviously think of Database Administrator (DBA).

Back in the day, the DBA did backups, restores, added users, tuned, wrote SQL queries, Stored Procedures, Packages, PL/SQL.

Then one day, the role of Database Developer sprung up.

And that role allowed people other than the DBA to write SQL and reports and ETL and a variety of other things.

And the DBA and Developer were at slight odds with each other.

How dare the Developer trample through my database and cause havoc, adding new objects, writing inefficient queries and such.

And they have lived in peaceful existence ever since.

Enter stage left, the Cloud.

You can now, with a credit card, open an account with a hosted vendor to host your database in the Cloud.

And what does that mean, the role of the DBA is not as necessary as once was.

The Cloud creates the database, maintains it, backups and restores, anything you could think of.

Essentially, the DBA role is is now in competition with out sourcing.

And the role of the developer is skyrocketing.

I'm not here to say the the DBA role is completely gone, I'm saying there are now alternatives to high priced full time roles.

I don't see any role in IT that can't be outsourced at this point.

And so it goes!


Oranges vs. Data

Oranges used to grow everywhere in Florida when I moved here in the early 1980's.

Most side roads had Orange farms that ran for miles.

I imagine that Oranges that grew on the trees, were sold for lots of money.

And I imagine they squeezed the Oranges into Juice.

However, with the Real Estate boom few years back, lots of those Orange trees are now housing developments.

And so we accumulate data, kind of grows on trees.

And the role of the Business Intelligence analyst is to squeeze that raw data into Juice / Information.

You see, not all the data has value and gets discarded, the peel.

However, the concentrated juice / info is very valuable commodity.

And can be leveraged in many ways.

So here's to the Oranges & Data!

I'll drink to that!

Oracle User Group Meeting Tampa

I attended the oracle users group meeting in Tampa last night.

Hats off to the presenter Jeff Smith @thatjeffsmith.

I thought his presentation was great.

Good depth of knowledge, good content and had good audience interaction.

With his dry sense of humor he had the place in stitches multiple times.

I overheard the guy sitting next to me say, "That is awesome" a few times when learning some quick tips and tricks.

He mentioned Twitter at the beginning and the importance of getting connected but the room seemed apprehensive.

There were some good Star Trek references added in as well.

It's been yeas since I developed in Oracle however it was good to see the improvements to SQL Developer.

I thought the building was nice, the old IBM building where my credit union used to be, the food was good too.

I bumped into an ex co-worker and we chatted for a while and I saw another person too but didn't get a chance to say hi.

K-Force sponsored the event and I chatted with Scott, one of the recruiters for a bit.  It seems there are quite a few jobs for Oracle DBA including relocation assistance.

I apologize to Jeff for not getting the chance to say hi in person, the room was quite full and I had to get back home.

I felt the IQ quotient in the room to be quite strong, always a good sign, even if for a product I rarely use nowadays.

5/24/2012

Programming = 25%

You probably think programmers spend 100% of their day writing code.

Perhaps some shops do it that way.

I'd say the reality is more like 25%.

There are so many non-associated tasks that need to get done.

Tickets, time sheets, spec gathering, documentation, updates to software, training, meetings, etc.

Programming is actually the easy part.

That is when you get a block of time allocated for concentration.

The average day is a jumble of interruptions.

And when people are looking to hire someone, all they focus on is skill sets.

What have you already done?

Well, here's what I've done and here's how my skills are adaptable to whatever technology you are looking for.

And here's how I handle the non-programming aspects of a job.

You see, programmers are not machines or robots.

There are skills associated with the position that are non technical.

And just because you have a pool of similar candidates with same years of skills and same technologies, they are not all equal.

Actual "programming"  is perhaps 25% of the daily routine I would guesstimate.

Assuming the position is not a consultant. :)

Full time employees have a lot of duties that are non programming related.

That's all I'm saying!

Creative Work Places

People say that in order to rise up, you must surround yourself with people who are smarter, better, faster.

However, I have not always played by that mantra.

In tennis growing up, I tended to play weaker players, and I watched them become better players.

Didn't do much for my game though.

I suppose if I played better players, I would have excelled faster.

Then when I got in the business world, same thing.

I surrounded myself with people on the fringe.

The people who had very little chance of upward mobility.

And perhaps that is my downfall.

Perhaps I should have surrounded myself with people in better circles.

Maybe some of their 'mojo' would have rubbed off on me.

I remember going to lunch one day with 5 other developers.

When layoffs occurred all 5 developers were let go and I stayed on.

I like starting new jobs, get to learn new environments, people, systems, data, business', etc.

After the newness wears off, the challenges few and far between and I find myself in a environment that is less than optimal, I get my traveling on and find a new place to work.

If you have read any of my blogs for a while, you'll see that I crave new things, new ways of looking at the world, new challenges, as I get bored of repetition and stagnation.

Perhaps the business world views this creativity as negative.

But it shouldn't.

There should be opportunities for people who seek fresh challenges and infinite growth.

I wonder if and where these type of places exist.

Places of Employment

Jobs come and go.

Some good, some not so good.

Sometimes we find ourselves in a particularly bad job, for one reason or another.

And we make sporadic jumps to another job.

Only to find out it's not so great either.

And sometimes we make a career of jumping from one bad place of employment to the next.

So you have to sit down and wonder, is it me or is it them?

Well, that's a tough one I suppose.

Probably different for each person / each job.

I can say from my own personal experience that I do good work.

I'm fast, accurate, attention to details, hard working, take ownership, problem solver, always improving, etc.

You would think those are some good traits to have.

Except the working world is not perfect you see.

And some places you are actually looked down upon for being an exceptional worker.

Because there are politics, underground networks of people who do nothing, resentment, jealousy, etc.

I'd really like to get off the tread mill of bad places to work.

I feel I bring a lot of talent to an organization.

It would be nice to finally find a place to call home, where my work is appreciated, I'm given some freedom to explore, I'm compensated sufficiently, where I can grow to the next level and find some real challenge.

I'm getting tired of this game of impressing everyone at the start, getting frustrated with crappy environments, only to move on to the next place.

So is it me, or is it them?

5/23/2012

Reporting vs Business Intelligence

This is the way I see "Reporting vs Business Intelligence".

Suppose someone was repeatedly smashing your thumb with a hammer.

You could create a report on it, the time, date, repetition, frequency, strength, variations.

That would tell you everything you need to know about the occurances.

Business Intelligence would be to get that report into the hands of a person, who in turn, stops the hammer from smashing your finger.

Because the report led to an actionable event, for the good in this case.

The evaluation of data, over time, consolidated into a report, distributed to a person of influence, where an action was taken, to improve the system = Business Intelligence.

That's my take on it.

Data Science vs. Shaman

With the rise of Big Data, comes the importance of Data Science.

Predicting future events based on analysis of past data trends.

It seems you can only predict within the subset of data you are working with.

If you have data for the NE and SE regions of the US, you can't predict the results of the NE US.

So that would infer learning the unknowns amongst the unknowns is really more talk than actuality.

Sure you could extrapolate patterns and predict future events, but the probability would probably not be that great.

It seems to me that Data Science role has been around a lot longer than people think.

Back in the day of tribal society's, they had Shamans, who provided service to the tribal leaders.

They would forecast the rains, the food, health, they were healers who delved into unknown worlds and returned with nuggets of wisdom to help their tribe.

Similar to the modern Data Science.

However, if  you ask me, I'd prefer the wisdom from the Shaman if I had my choice.

Science alone is good, however, without Intuition, it's kind of making a decision in a vacuum.

I think of the story of 4 people describing what they think is in the dark room based on physical attributes such as touch, taste, feel, sight, etc.

The thing in the middle of the room is an elephant, except everyone describes something different.

Moral of the story, Data Science is a good thing, mixed with intuition, business experience, and should be taken with some grains of salt, as there are plenty of failure points between the data and business decision.

After significant evaluation of prior trends mashed with multiple sets of data, I would give about a 50/50 chance of being correct on this tidbit.

5/22/2012

Good Working Environment

What would you consider a good working environment?

How about a place where you have the tools to do your job?

And the flexibility to get your job done.

And a place where creativity can flourish.

And you ideas are valued.

And customers are friendly and helpful.

And workflow is steady with some ebb & flow.

And technology is current.

And you get the support you need.

And the team environment is helpful.

And there are some really smart people working there.

And a fun atmosphere.

And casual, except when customers are around.

And plenty of eateries and coffee.

And places to walk during breaks / lunch.

And a nice building with ample parking.

And centrally located.

And good management.

And a good product.

And where the upper management believes in data metrics.

And good compensations and benefits package.

And good vacation / time off policy.

And insurance too.

I think all those items listed above would classify as a good working environment.

5/21/2012

Electronic Medical Records

I once went to the doctor.

Doc asked, you feeling okay?

Yes.

You sleeping okay?

Yes.

You eating okay?

Yes.

You exercising?

Yes.

He said you look fine to me.

And that was that.

However, the one thing I notice when going to a doctor is they refer you for specific things.

They don't take the holistic approach where they service the entire body.

They segment everything into specialty's.

I'm not sure that's the best way.

Because the doctors don't compare notes between themselves.

So each doc is treating in a vacuum.

So maybe, when the medical industry gets everything electronic, and they can create a comprehensive picture of each patient, complete history, interacting and collaborating with other people, perhaps that would be a good thing.

Seems health has become a business too.

For profit.

And those who have the money can afford treatment.

I suppose the same thing could be said for eduction as well.

If you can afford it, the system works.

If you can't afford it, oh well.

5/20/2012

Virtual Matrix to Merge IT and Biz

IT is slow, bureaucratic, controlling and basically unresponsive to change or user requests.

The 'Users' are demanding to see their data NOW.

They need to view the data, in readable formats, to slice and dice the data, to manage their business.

So along comes Business Intelligence.

High priced consultants who create monstrous applications at high costs, slow implementations, that have to be maintained by someone.  They don't handle changes quickly, they don't incorporate data mining, predictive analytics, or mashing of data from external sources easily.

So what does the Biz unit do, they find people who can do this faster and cheaper.

They bring in a hired gun rouge developer to satisfy their needs.

They don't have time to waste on the lords of the IT department to get around to doing it.

And I agree.

IT no longer owns the data.  The business owns the processes and business logic.

So the age old question, how do you merge the IT and Business, so they work in synergy?

I say you solve this problem by creating a new unit, a 'Virtual Matrix'.

The CEO establishes a CBIO or Chief Business Intelligence Officer or Chief Analytic Officer.

Direction must start at the very top or will not gain traction.

He / She organizes a virtual team pulling in people from various departments from within the org.

IT people, Finance people, Marketing, Sales, tap into every department if need be.

Gather these people to identify the process'.

Find a software solution that fits your org needs.

Contract portions out to IT consulting if lacking talent internally.

Research and discover the problems you would like to solve.

Have a project manager organize the project.

Establish a 'Data Governance' board to oversee data quality / change issues.

Incorporate change process'.

Identify ALL data sources.

Brainstorm on the big picture.

Start off small, one step at a time.

And build an infrastructure that WILL satisfy your lack of insight needs.

Build a road map.

Do you need data mining?  Predictive analytics?  Need to mash data from external sources?  Work with unstructured data?  Need up to the minute data?

In order to give your Org a competitive advantage.

What problem are you trying to solve?

By having tentacles in every department, you will leave no stone unturned.

You can add or subtract people as needed to / from the Matrix.

The people on the team are most likely on your payroll already.

So set aside a chunk of budget specifically for Data / BI and adjust as needed.

You won't be using 100% allocation of these FTE, just a portion, so each team member could charge their time to this new department / team.

Sure you will need a real good PM or two to monitor status / progress.

I see it as a new challenge, to finally bridge the gap between IT and the Biz, with buy in from the top people down to the front line people.

It will change the infrastructure of business going forward.

It will drive sales, knowledge,  teamwork.

A win-win proposition.

Could be the new model of the future!

IT Lost Control of the Data

IT Lost Control of the Data.

I was in a Tweet Chat last week and someone sent this tweet to me.

And I agreed of course, but I had some time to think about it.

IT has been the gatekeeper for all the data going back to the 1970s on the mainframe and before.

Two places I've worked at, the mainframe is still the primary pathway to the data.

So IT controls who sees what when.

And while working for both orgs I've been key in getting that data out to the business users for consumption.

During one meeting, the customer complained they want their data, and they want it now.

So I opened MS Access, connected to the data source, within 30 seconds they could see all their data in every table.

Their jaws dropped.

So then they wanted Crystal Reports to access their data, except they didn't know how to program.

So I gave them some demo's and they were on their own.

Early days Self Service BI adoption.

And they were some happy customers after that.

Now who's to say they couldn't upload that Access database to the web.  Or on a thumb drive, and someone gets a hold of that.

Years worth of private data now exposed into the public with no way to reverse.

So yes, IT has lost control of the Data, for better or worse.

Pandora's box has been opened.

And the Genie refuses to get back inside.

Users need to see their data.  And IT should not be the preventers of this.

And now, with Power Pivot, you can store millions of rows of data, a tremendous advantage over Access.

It's all about liberating the data from the tyrannical IT departments or something like that.

However, IT isn't going away.  Someone needs to store that data, on a database, on a server, on a network, with email accounts, etc.

I suppose the Cloud could handle a lot of that and have it outsourced so I guess anything is possible now a days.

5/18/2012

Bigfoot Sightings

We've all heard about the infamous 'Bigfoot'.

A big creature who wanders in the forest, leaving giant footprints, perhaps a remnant of the caveman people of long ago.

Except no one's ever taken an actual picture or seen it up close.

So having never seen this giant creature, we all have our opinions of what he looks like, where he lives, etc.

I see the same thing with Big Data (BD).

Not everyone has seen it live and up close, we all have a general idea of what it is and we describe it like we know the answer.

Yet the true definition is still unclear.

We hear story's that BD will change everything.

It will predict the future, finally open up huge volumes of data to they layman, mash private and public data, will drive the demand for data scientist and save wild bunny's from predators too!

Sure it could do some of these things and more I imagine.

Yes, I'm on board too!

Let's mine that data into gold!

Embrace the change!

5/17/2012

Customer Feedback

Today I created a list of issues we've encountered over the past year surrounding our Reporting Project for a particular internal customer.

Our task was to convert about 35 reports from a legacy ASP web app to SSRS.

The list contained flaws in the way the users were entering data.

And some bad business rules inherent in the new PHP app.

And the translation of business requirements into reports.

However, the feedback that caught me off guard was the fact that there have been 3 developers on this project.

I hadn't thought about it.

But it's true.

The reports got off to a slow start.  Slow and inaccurate.

So about 6-8 months into the project I took over the reports in flight.

I had to learn the guts of the business.

And then I transferred the project to a contractor.

And we had to clean up the bad code.

And re-learn the business.

And get the reports to match each other.

After finding all the issues stated above.

It has been a long and windy journey.

And I feel we are heading down the home stretch.

Today I updated the project plan and we don't have that much left to do.

Sure some new reports were added to the list, which we still need to get too.

But the vast majority of reports are matching up and looking good.

Yippie!


Social Media Gap

Social Media is a new phenomena.

You can now post what you're eating and where you're at.

At least that's the uninformed person's mantra.

They have not discovered the inner world of social networks.

Find some tweeters to follow who have similar interests.

Keep informed about the latest trends.

Find out who the key players are in your field.

And I believe there is a "Social Media Gap" dividing those informed and those who are not.

I like reading the newspaper each morning, so I can re-read what I read the day prior.

All my news is from Twitter.

I found out about Steve Jobs death minutes after it was reported.

Up to the date information is the future.

And those on the sidelines bragging how they don't want to broadcast their private info, are being left on shore while the ship of the future is pulling out to sea.

5/16/2012

Would You Hire Yourself?

As you glance across the work landscape, some people stand out and some people don't.

Because some people are consistently good workers, who have passion, get along with others, etc.

You know if you worked for another company and this person interviewed, you'd like them on your team.

So what about you?

Do you have what it takes to be hired by someone like yourself?

Or are you lazy and afraid to admit it?

Do you run from challenges or learning something new?

Do you feel most comfortable doing the same thing to avoid risk?

Do you have the qualities that you look for in other people?

5/15/2012

My College Degree is Worthless

My College Degree is Worthless.

I registered, attended and passed the required curriculum to be awarded a Bachelor of Arts degree.

Dedicated 4 years of my life and 10's of thousands of dollars.

For a piece of paper.

I was working a summer job Summer of '87 and skipped the student orientation.

I was dropped off on campus by my parents and left alone to survive.

With nothing more than a dorm room and food expense money.

I never met with a dean or school official to determine the appropriate major.

Or to discuss my career options.

Or find out my strengths or weakness.

Or explain the process of graduation and finding employment.

Or anything.

The Schools purpose, it seems to me, is to take money in, for participation of classes, and funnel you through to graduation with the least amount of contact possible.

It's a degree mill factory.

I learned almost nothing in college.

I learned to drink water for a hangover.

That sports generate more revenue than academia related stuff.

That kids who come from money will always have money.

That power games that occur at the student level, as dishonest and despicable occur in life as well.

I learned that the geeks, no matter how hard they try, rarely get the cheerleaders.

I learned that the jocks get the cheerleaders.

People segregate themselves with their own kind, if given the choice.

I majored in Anthropology.

Do I regret it?

Sure, I could have gotten a degree in Computer Science.

Could anyone, at the time, from the University, steered me in that direction?

Sure.  Am I bitter?  Not really.

Things didn't turn out that bad.

I'm just saying my college days, aside from receiving a diploma, could have been better spent learning something to prepare me for life.

Reporting aka. QA Department

We've been working on a set of reports for a while now.

And for some reason the numbers have not been matching up.

For one reason or another, there is bad data.

Dates outside of acceptable ranges, fields not set correctly, etc.

So the Reporting Team is tasked with troubleshooting all these bugs.

And we funnel them up the chain so the front end application can be modified.

And would you know it, the PHP app was written by a contractor, who rushed the job, provide absolutely no documentation on the app or database schema or anything, and they are no longer here.

What a freakin' mess!

So Reporting, aka. QA department, is fixing the bad business logic as we go.

This is just plain wrong!

Just a basic report for a typical user.

I had a co-worker, a team lead in IT, come to me today asking for some data.

First, that shows that our department has gained some traction in the enterprise as a place to find data.

So after listening to the request, all the closed tickets in the system going back as far as possible, for the auditors, I located a similar report, customized it, working on the web in less than 20 minutes.

Now how does that relate to BI?

Must know where the data is, how to get it, table structure, business rules, gather specs, set expectations, deliver fast accurate product, available on the web, downloadable in Excel, with hyperlinks back to the ticketing system for more details, added # of days aged to report even though they didn't ask for it, etc.

This is basically what I do for a living.

No Big Data involved, no predictive analytics, no data scientist involved, no model forecasting.

Just a basic report for a typical user.

White bread.

5/14/2012

Artificial Intelligence is the future

During my chat yesterday on Mother's day with my father, he was surprised that I was a team lead.

He was also surprised that I have 2 part time jobs.

He never heard of business intelligence.

However, he said I should get into Artificial Intelligence.

Having worked for IBM his entire career in Poughkeepsie, Tampa and Atlanta, he has some insight.

Everyone is chit chatting about Business Intelligence.  Big Data.

They are all tip-toeing around Artificial Intelligence.

Isn't that the main goal here.

Putting the information in front of business users to base decisions is the current mantra.

What happens after that?  Let the computer read the data, base its decisions, and self correct on the fly, totally automated.

Why involve error proned humans?

What do you think this IBM Watson is all about.

Then add that AI to robots and humans will become either obsolete or will not be able to compete with drones.

Let's look down the road a little further, passed the hoopla surrounding Big Data.

Artificial Intelligence is the future.

And so is Artificial Business Intelligence...

http://www.bloomconsultingbi.com/2013/01/ai-and-robotics.html

http://www.bloomconsultingbi.com/2012/06/artificial-intelligence.html

Business Intelligence Manager Role

What does it take to be a Business Intelligence Manager?

Well, I would say one must know the complete life cycle of data.

From the gathering of data into the OLTP system, database design, performance issues, triggers, replication, log shipping, backups, SQL knowledge, etc.

Then you would have to know about Extract, Transform and Load or ETL.  And how to apply business rules, cleanse data, synchronize data, backups, etc.

Then you would have to know about Data Warehousing.  Facts and Dimensions.  Star and Snowflake schemas.  Non Relational architecture.

Then you would have to know about Reporting.  How to most efficiently extract data into useful information for customer to manage the business and look for trends.

Obviously for each of the items described above, there are numerous vendors lined up for your business and you have to know the pros - cons between each.

Then you would have to know about Project Management and phases of projects and milestones and meeting with the user community and costs associated.

Then you would have to know about actual management of people / developers / DBAs and approving vacation time, sick days, promotions, hire and fires, raises, performance reviews, ability to have dialog with senior executives and customers.

And most likely you would have some years of coding experience behind you to be an effective BI Manager.

So there you have it!

Advanced Normal Forms

I recently browsed upon some advanced high level reasoning.

4nf & 5nf.

4th Normal Form and 5th Normal Form.

It basically has to do with relationships between data elements at a higher degree than most typically relationships.

Here's the link I referenced to get a brief understanding:

http://en.wikipedia.org/wiki/Fourth_normal_form

http://courses.cs.vt.edu/~cs4604/Fall08/lectures/lecture16.pdf

I don't pretend to be an expert on this subject, however, I'm surprised I never had much interaction with it prior.

If you can make sense of it in simple terms, please post a comment or link

Thanks!

5/13/2012

Generalist

We've had the handyman over this week to fix a few things around the house.

Steve repaired the wall where the roof leaked, patched up 7 or 8 spots that needed patching, replaced the light in the kitchen with a fan/light.

Stained the wood on a new futon.

Today he's adding a gate / fence to give the dog's a place to run on the side of the house.

After that he's going to replace the top piece of the chimney.

He also does plumbing and a variety of other things.

And that's basically the way I program.

My skill set is all over the map, a generalist.

Just like Steve.

Whatever you need fixing.

5/10/2012

Universal Database

I like the idea of having a Universal Database.

Basically, an organization, who decides to opt-in, can expose their data set universally, similar to a Web-Service.

Like a Web-Service, the location is uniquely identified based on a NAMESPACE.

The parameters, input values and return values are all documented in a WSDL.

So you could have a DATABASE WSDL which describes the data, tables, relationship diagrams, etc.

There are credentials required for authorization.

The WS returns XML using SOAP,

The data would be maintained by the HOST site and can Dynamically change so as not to be Static.

Or it could be Static data.

So potentially, you could issue a SQL statement, JOIN multiple tables from multiple Universal Databases, providing you know the location, security, what information is available and what field to JOIN on.

People could post there own data in the Cloud or they could host it locally.

People could donate their data, similar to OPEN SOURCE projects, this would be an OPEN DATA PROJECT.

For Earth consumption.

To better mankind!

Possible?

BI = 52 Card Pickup

When I was a kid I learned the game of 52 card pick up.

Someone holds your deck of cards, and flings them in the air.

They you get to pick up all 52 cards.

And re-arrange them in some type of order.

So is Business Intelligence.

Putting the 'order' back into data.

To form a comprehensive pattern.

From CHAOS to ORDER.

Simply put.


Report Slow Down

Ever since we implemented a new procedure for requesting a new report, things are starting to slow down.

One person requested a report, didn't specify any details, we asked them to complete the new form, weeks went by, this morning they closed out their request.

Another customer who frequently requests reports via the phone called the other day, I asked her to fill out a form, haven't heard a word since.

Perhaps the culture is in shock.

Having to think through their request in advanced.

Not dumping their work on the over loaded IT report staff.

Profound.

Except now perhaps my reputation of bending over backwards to satisfy every request at lightning speed may be in jeopardy.

I guess I can learn to live with that.

Being understaffed, we no longer have the time to do the 'thinking' for the user.

Just enter a ticket, complete the form, and we'll get to your reports as soon as we can in the order they are received, with exceptions.

Now, should I implement the new new process of having every report go through a throurough Quality Assurance review?

Perhaps they will run me out of town...

5/09/2012

Traditional Reporting

We've been trying to get the contractor hired on full time.

Because he's fast, accurate and always asks for more tasks.

So today he brought in some cheese cake with frosting that said 'day 1'.

He finally got hired.

So I went to my boss' office.

To ask how much time is allocated for reports.

He said he is only to work on 'Grant' reports since that's who's paying his salary.

So I looked at my list of reports in the queue and 2 of 23 reports are for grants.

So tomorrow I have to let all my customer's know that he is no longer available to work on their reports.

This is why traditional reporting fails.

Customer must wait in a queue for requesting data.

IT is the bottleneck.

So if you would like a report, submit a ticket, a new report request form, and pull up a chair.

It's going to be a while.

5/08/2012

Business Intelligence Purpose = WHO, WHAT, WHERE, WHEN, WHY, WHAT IF

Traditional reports tell us WHO did WHAT WHEN, HOW MANY and HOW OFTEN.

They slice the data horizontally and vertically.

Advanced analytics helps us discover WHY.

Why did this person have more sales volume.

Why did this product flop and this one succeed.

Also included in the WHY scenario is the predictive analysis WHAT IF.

What if I lowered the price of widets by $0.02?

What if the snow fall increases next year or rain decreases?

Report Writers or Business Intelligence Developers are in the 'discovery' business.

We travel the world of data / uncharted waters searching for treasures.

The sponsors of our voyage are the business users / CEOs / etc.

Gaining insight can give your organization a competitive advantage.

You have to know where you came from, where you are and where you are going.

You have to minimize costs, maximize resources and provide a valuable product or service.

That's all there is too it.

WHO, WHAT, WHERE, WHEN, WHY, WHAT IF


The goal of Business Intelligence to answer these questions.

Data is Data is Data

We have to classify everything.

Put into buckets.

For clarification.

We have 7 oceans on this here planet.

Except the ocean doesn't know it's divided into segments.

The water that flows from Australia  finds its way to Greenland eventually and Hawaii.

That water evaporates, ends up in the Clouds.

Then falls as rain to form freshwater streams, lakes and ponds.

No real distinction, water is water is water.

And data is data is data.

Small Data, Big Data, Desperate Data, Structured Data, Unstructured Data, Spatial Data, Financial Data, HealthCare Data, you name it.

Put it into buckets.

It's just a bunch of data.

And data is energy.

Stored Energy.

Potential Energy.

Waiting to be released.

5/04/2012

Helpful SQL Search Queries

Here's a few queries I run often:

1. This query will search through all Stored Procedures on a SQL-Server Database and return the Sproc name and code.  Very helpful to doing global searches on a particular string/field.



SELECT ROUTINE_NAME, ROUTINE_DEFINITION
FROM INFORMATION_SCHEMA.ROUTINES
WHERE ROUTINE_DEFINITION LIKE '%TEXT_TO_SEARCH_FOR%'
AND ROUTINE_TYPE='PROCEDURE'

2. This query will search through all Reports stored in SQL-Server Reporting Services for particular Keyword.  Very useful!



USE [ReportServer]
SELECT
        CONVERT(varchar(MAX)
      , CONVERT(varbinary(MAX), c.[content]))
      , c.[Path]
      , c.[Name]
FROM [ReportServer].[dbo].[catalog] c
WHERE path not like '/Data Sources/%'
AND c.[content] IS NOT NULL
AND CONVERT(varchar(MAX), CONVERT(varbinary(MAX), c.[content]))
LIKE '%TEXT_TO_SEARCH_FOR%'



5/03/2012

Data Warehouse Exposes 'Bigger' Issues

We are in the process of building a new centralized Data Warehouse.

That's easy, right.

Just write some code to move the data into the staging tables which will end up in the Star Schema.

Piece of cake.

Um, Houston we have a problem.

When inserting data into the staging table, we are seeing things that don't make sense.

With some investigation, asking the business users their process, we identified incorrect practices.

Which have been occurring since forever.

Um, Houston we have another problem.

Let's ask the business.

Okay, another process done incorrectly for decades.

Wonder why the auditors never caught that one.

Hmmm.  Wait a minute.

So what we've got here is the exposure of some bad business practices.

And it exposes the lack of documented process.

And it exposes the culture of silos.

And it exposes the lack of oversight.

And it exposes too many people who have job security because nobody knows what they are doing.

I sure love Data Warehouses.

For the fact that it seeps into every department and uncovers shaky practices.

And once the DW is operational, I can report on all this data.

One source of the truth.  Dashboards.  Scorecards.  KPIs.  Advanced Analystics. Possibly some Self Service BI.

Progress!

Reporting Crisis


We have two reports.

The grand totals don't match.

So the business user complained.

So the Reporting Services team identified the records causing grief.

And sent the list to the business.

They forwarded the list to the programmers.

The programmers say its not an issue, the queries are wrong.

Hmmm.

So much for group effort.

Okay, we'll take ownership of the problem.

The contractor and I did some root cause analysis on how this problem is occurring.

So we dissected the problem from a coding perspective.

And applied the logic to the business rules.

And there's a flaw in the business rules.

Which is not handled by the front end PHP application.

So we sent our analysis to the user to answer 'why' the numbers don't match.

And provided a few options on how, from a reporting perspective, we can overcome the issue.

And offered possible solutions to the Programming team.

We don't care one way or the other which number is correct, that's up to the business.

The business owns the reports, so we take their lead.

We can only point out the flaw in logic, recommend solutions and ways to plug the hole in the front end app.

That's all we can do.

People think reporting is simple, easy, anyone can do it.

There's really more to reporting than people understand.

We are part QA, part firemen, part psychologist, part mentor, part Project Manager, part negotiator, part a lot of things.

Almost like the glue that keeps everything running.

So this problem will get solved shortly.

And we'll go back to pumping out reports and making the customer's happy by providing value.

It's good to be counted on for having practical sound reasoning in times of crisis.

Reporting is more than putting numbers on a report.

We may not get recognition for what we do, but without us there would be a lot more chaos.

5/02/2012

Data & Self Service BI

When I got into IT in 1995, the big thing was Visual Basic.

Get the data into the Database.

Then report on it.

Fast forward, getting the data into the database seems to be the easy part.

We accumulate data from many different sources.

And we have different naming conventions and meanings across each of the DBs.

This causes great confusion when trying to roll up the data and present an Enterprise View for the executives.

Even at the level of small data, mashing data seems to be an issue.

Throw on top of that bad data, modified data, deleted data, out-dated data, bad database design.

Expand this issue exponentially for big data, I'm sure the same problems exist, just amplified.

You see, everyone is running like a chicken with their head cut off.

No-one has time to sit down and think through what they want, how they want it, how do we fix the problems, how to integrate the data in such a way to architect an Enterprise Level Data Warehouse, a single source of the truth.

There's simply no time.  Everyone is busy putting out fires every minute of every day.

And when not plugging holes in the ship, they are on to creating the next system with a new set of problems.

There are plenty of opportunities surrounding data, both big and small.

Users just want their data, they have deadlines.  They don't want excuses.

Which puts all the weight on the already time / resource strapped IT department.

When people say they want to put the power in the hands of the users, Self Service BI, I'm not sure what they mean.

Give them a tool that sits on top of the data, to query in real time with drill down and graphs and charts and KPI's.

To me that's similar to saying give people the ability to drive from NY to CA.

First the road needs to be built, the infrastructure to be put in place with traffic lights, stop signs, rest stops, gas stations, restaurants, etc.

The user stomps their feet and says 'are we there yet?'  I have to be in California by end of the month.

Sure, IT can build the infrastructure if sufficient knowledge, time and resources.

And IT gets to maintain the infrastructure, the network, the web servers, hardware, databases, applications, etc.

A lot of effort needs to be applied on the setup in order to achieve self service.

In the meantime, IT carries the weight / blame for inability to provide to their users.

Perhaps IT will always be blamed by the business, no matter what obstacles we have to overcome and what miracles are expected of us.

5/01/2012

Business Intelligence Required Skills

I decided to compile a short list of my skills, required to be competitive, in today's world, as an IT  developer.

SQL Queries
Views
Stored Procedures
Functions
Joins
Tables
Indexes
Query Tuning
Temp Tables
Pivot Tables
Unions
Select
Insert
Update
Delete
Databases
SQL-Server 2000
SQL-Server 2005
SQL-Server 2008
SQL-Server 2008 r2
SQL-Server 2012
Oracle
PL/SQL
Packages
Active Directory
SSIS
SSRS
SSRS Viewer (.net)
SSAS
Crystal Reports
Business Objects
Universes
Actuate
JReports
IIS
JavaScript
Java
JSP
J2EE
Applets
Jar Files
Classes
ODBC
OLE DB
TNSNames.ora
SQL Developer
Web Services
XML
WSDL
Web Servers
Security
Log Files
Certificates
Software Installation
Troubleshooting
Power Point
Power Pivot
DAX
Tabular Models
Data Governance 
Master Data Services
Data Quality Services
Performance Point
Power View
ColumnStore Indexes
SharePoint
SSRS-SharePoint Integration
c#.net
Visual Basic.net
ASP.net
PHP
Legacy Visual Basic
Legacy ASP
Big Data
Data Mining
Predictive Analysis
Hadoop
NoSQL
MySQL
Toad
Erwin
Data Modeling
Primary Keys
Foreign Keys
ITIL
Ticketing System
Configuration Management
Change Management
Incident Management
Asset Management
Release Management
MS Excel
MS Word
MS Access
Vantive (Peoplesoft)
Infra
Clarity
TechHelp Footprints (BMG)
CA Ticket System
Business Analyst
QA Methodology
Mentoring
Project Management Tools
OpenProj
PM Life Cycles
SDLC
Meeting Skills
Spec Gathering Skills
Process Creation
Organized
Good Time Management
Follow Through
Last 10% of Project
Attention to Detail
Fast Paced
Multi Tasker
Ability to Shift Priorities

Get Sh#t Done!