Data Scientist vs Astrologers

Business Intelligence is my life.

No, really, it's my job.

And that's great.

Because data exist everywhere.

And if you have the right tools and access, you too can work with data.

You can get data 'into' the database.

Or you can get data 'out' of the database.

You can aggregate the data.

Or show the details.

You can create Dashboards and Key Performance Indicators.

Or you can create cool visualizations.

Any way you slice it, the data is there waiting for you to do something with it.

Now if you spend enough time in school, get your PhD, and know your way around statistics, then you may become a 'Data Scientist'.

Perhaps that definition is to extreme.  Maybe a data scientist works with Big Data.  Hmmm.  Maybe not.

Perhaps a Data Scientist is someone who's a master at working with data, who understands and comprehends the business, the processes, the industry as well as the data and the tools necessary to process it.  One who can derive meaning from it.  Perhaps someone gifted in the art and science of turning data into information.

Hmmm..  That sounds more appropriate.  Because I really don't have time to go back and get a PhD at this point in my career.  And maybe the decade and a half experience I have isn't wasted time.

I'd go a step further.  What do you think an Oracle does, one who interprets Tarot cards or Astrology.  They view the patterns of the stars, how they are aligned, the relationship to other objects, and can produce probabilities based on that.

Now some people say that's hogwash, their results are so vague and can be interpreted the same for everyone.  And they are a bunch of con artist taking people's money for telling the future.

I'll be honest, I went to see a psychic and they said I'd meet someone named 'Crystal'.  This is no lie, I met 3 people named Crystal and I was programming in Crystal Reports at the time.  Coincidence?

Some people don't believe in electricity because they can't see it, so go ahead and touch that light socket and let me know if you still don't believe in electricity.

So what I'm basically saying is this, are Data Scientist just a new derivative of the ancient Astrology readers from 2000 years ago, who read the intestines of animals to predict the future?

Sure makes you think about if from a different perspective.

And what level does 'Intuition' play in the game of Data.  Do you not think the decision makers use practical business experience and gut instinct applied against the known facts, the result of the data insight, just in case the data has false positives?

I'd say intuition still plays a huge role, regardless of what the industry says.

Facts are great, if they're correct.  But I wouldn't bet the entire farm on them.

And that's what I think.

Stuck in the Mud

I learned the Structured Query Language in 1995.

And although the language has changed over time, the basic foundation hasn't changed much.

So in a sense, I have not had to re-learn a new language over time.

Which means I've been stuck in the mud, theoretically.

I've been coasting along riding the wave of a single language for 18+ years.

Granted, I used to program in languages such as Visual Basic, ASP, .net, Java, however the number of times I've had to learn a new language has been scarce.

And I've paid the price, for technology did not stand still.

The world of programming has splintered and grown in every direction.

And that is why I'm thankful for finding an online training website, www.PluralSight.com, to help get back up to speed.

As of today, I've watched close to 25 courses, in just over a weeks time.

And that's only scratching the surface of topics to learn.

The courses are great because they have an instructor, who speaks slow and clear, who types examples as they go, and you can go back and re-watch specific portions of code which you missed the first time.

I did not subscribe to the source code option however, which is okay for me, because I'm trying to get surface level knowledge of a variety of topics and not so much 'deep dives'.

So I'm furiously trying to catch up to the current technologies as things have changed over the years.

And if you think about it, things are in constant motion so the concept of learning never ends.

And so it goes!


Classic VB vs .net

Back when I programmed in classic Visual Basic I thought it was the greatest language ever.

Because you could program at any level.

From entry level business user writing code to advanced level programmers using classes, DLLs and COM+.

Downside was not object oriented, no built in error handling and code could become spaghetti really fast.

And then we were asked to migrate to .net.

When I say asked, we were basically forced.

As in no more support for VB6.

And programmers needed to learn object oriented coding really fast.

Although it's still possible to code in VB.net not using Object Oriented methodology.

Due to lazy programming styles.

However, the improvements between classic VB and let's say the 4.0 .net framework are completely different.

The only similarity between the two would be the name: Visual Basic.

We now have object oriented code, built in error handling via the garbage collector, the IDE environment is much cleaner, you can combined languages easily, the coding style is easier and way more powerful.

The downside is having to learn the new library's.

Now for me, I veered direction in 2006 and learned Java.

That was actually a good decision because c# is basically a rip off of the Java language.

So it looks like we can have our cake and eat it too.

.net is an incredible programming ecosystem.

Downside, it runs on Microsoft Windows only and IIS.

That's the price for using .net.

So if you ask me today am I disappointed the classic VB went away, well, the improvements outweigh the downsides.

And guess what, there are still company's that have VB6 in production today as well as ASP classic.

And that's a fact!


And now for some .net

With all the new learning I've been doing, I decided to go old school and see about .net again.

I programmed in .net in 2003-6 and a bit after that, except I wanted to see if I could still do it.

So first I loaded up Visual Studio 2008 r2 and downloaded some sample code from http://www.planetsourcecode.com.

Except the program I downloaded required AJAX.

So I downloaded AJAX and got it's sample web to run okay.

However, there was a mixup between the 3.5 and 4.0 dot net framework.

So I decided to create a simple CONSOLE application.

Which I got to run.

So then I decided to connect to my SQL-Server 2008 r2 database.

Which after some tinkering, success, I was able to produce the results of a SQL table to the console and loop through all the records.

Took some work to get the configuration settings from the app.config but its working.

Imports System

Imports System.Configuration
Imports System.Data.SqlClient

Public Class vbMyClass
    ' Shared variables

    Dim connectionSting = ConfigurationManager.ConnectionStrings("ConsoleApplication1.My.MySettings.ReportServerConnectionString").ConnectionString
    Dim con As SqlConnection = New SqlConnection(connectionSting)
    Dim cmd As SqlCommand

    Public Sub ShowData()
        cmd = New SqlCommand("Select * FROM Catalog", con)
        If con.State = ConnectionState.Closed Then con.Open()
        Dim sdr As SqlDataReader = cmd.ExecuteReader()
        While sdr.Read = True
            Console.WriteLine(sdr.Item("ItemID").ToString & " " & sdr.Item("Path").ToString)
        End While
    End Sub
End Class

This part of .net hasn't change much to be honest.

What has changed is the JSON, jQuery and MVC.

A co-worker gave me some work to do in MVC a few months ago, except it really didn't make much sense, same as today.

I'd like to build up to that eventually, but for now, I'd like to get .net back on my resume.

I currently support a few web services in production which I've had to modify recently.

One's a REST service and the other is an actual SOAP web service, both ODATA.

So sure I feel good connecting .net to SQL, the building block for developing webs and services and web services and client server apps.

Fun stuff to do on a day off from work.  I'd prefer to have the latest version of Visual Studio installed, maybe down the road.

That's it for now, JB

Year in Review 2012

The year 2012 is coming to a close.

I'd say it was a stellar year from several angles.

New Job: I got out of the Public sector into Private working for an International Software company doing Advanced BI.  My salary finally matches my skill set, better late than never!

Part Time Job: I put in a lot of hours this year, 538 hours to be exact.

New Skills: I've gotten hands on experience with SharePoint, Tabular Model, Power Pivot, c#, Team Foundation Services.

Public Speaking: I spoke at the local Microsoft Business Intelligence User Group meeting, got last minute notice but I enjoyed it and conquered my fear --> View Link

Training: Signed up for Plural Sight training site to keep my skills sharp.

Stuff: New Pool, new used car, new FIOS 150 speed

Vacations: Went to Blue Ridge Georgia in the Spring Day2 Ft. Mountain and Fall Day1 Day2 Day3.

Fitness: I started back at the gym in November and also got back out no the tennis courts around the same time Back on the Courts Get Back in the Ring

Health: I no longer smoke cigars, drink diet cokes and increase my fruit and vegetable intake.

Changes: We fired out CPA, pool maintenance guy, dentist and I switched from the credit union (1986) to a bank.

Overall a successful year!


Recent Training Courses

I've been viewing some training video's the past week.

And let me tell you, there's a lot to learn.

I've watched 15+ courses and that's not the tip of the iceberg.

From AWS to SQL Azure, Biz Talk, SSIS, OData, Big Data, SharePoint for Power Pivot, Power Pivot,  Javascript, jQuery, NodeJS, IronMountainRuby and some others as well.

Each course ranges from 30 minutes to 7 hours so  it does take effort.

It gives you a good understanding of each technology.

I feel smarter already.

If you care about your IT career, you should check out the site. http://www.pluralsight.com

Cloud is the Future of IT

In reviewing some of the latest Cloud solutions from Microsoft Azure and AWS I'm amazed at the opportunity.

It seems the Cloud is open for business.

In the fact that you can create web sites in a matter of minutes.

Same with Databases.  And Mobile solutions.  And Business Intelligence.  And VPNs.  And Networks.  And Pipes between the Cloud and On-Premise.

Just amazing.

And all of this is available via a web management console.

What does this mean for us?

Developers now have new opportunities for development, deployments, getting a site up and running fast.

What does this mean for system administrators?

I see potential to ease the workload from on-premise work to Cloud work.

What does this mean for Cost Savings?

I'm not 100% sure of the pricing models, but think of it this way.  Renting vs Buying. 

How would you like to provision a new server within minutes without having to purchase a new server, memory, apply software and patches, setting up permissions and users.  Now all this can be done in the Cloud.

And for Entrepreneurs?  Imagine setting up a website and database and as your company grows, you can increase the number of web servers or database size based on demand, and then scale back if necessary.  Let's say you run a 3 day sale, bump up the specs for 3 days then scale back, a built in elasticity if you will.

Personally, I see this as a niche business for someone like me to be the go-between for small to mid size company's to get up and running quickly where I would provide the skills, time and knowledge which the company's don't have.

Kind of a service oriented approach where end users would come to my company Bloom Consulting and I would set up their accounts and provision servers, databases, VPNs, and even develop some of the sites for them.

Not a bad business model if you ask me.

And then once up and running, business can hire remote workers to connect to the Cloud and work remote.  They can save on office space as well as hire contractors for specific projects on an as needed basis.

I am convinced that the Cloud is the future of IT.

Quality Reigns

There's quality and then there's the knock off brand.

And I've typically gone with the quality choices.

Recently, after some bad experiences at the local supermarket, we decided to try one of the alternative ones.

The place was always a ghost town.

The customer service was minimal.

And the quality of meats, fruits and cold cuts was nowhere's near what we were accustomed too.

And one day the stock guy turned quickly and nearly knocked my wife over, said it was an accident, um, I didn't think so.

And recently they stopped giving us coupons at time of checkout.

And the final straw was when they didn't know how to use our $50 gift card at checkout, we had enough.

So our next food shopping venture will be back at our old stomping grounds.

Where they have a store every mile or so with fresh quality food with Buy One Get One free deals all the time.

So we tried an alternative to quality with dismal satisfaction.

And so it goes!


Unlimited Training

Throughout my career, I've been limited to new technology.

And I saw people, some not as smart, excel faster.

Because they had access to technology.

Sometimes immediately after it was released.

I tend to think it's because they have access to software, for example MSDN.

But that's not the entire aspect, they must also have hardware to load and test with.

And also the ability to figure this stuff out on their own.

I can look back at my career and see all the stumbling blocks.

Where some new technology would be released.

Sometimes I purposely said "I'm going to hold off on that for now".

But not always.

So I suppose that's why I've been learning as much as I can as fast as I can.

To make up for lost time.

Because if I really think about it, the happiest I've ever been in the world of IT is when I'm learning something new.

In 1995-6, learned Crystal Reports, Visual Basic and Oracle.

In 1998 learning the internet with Microsoft ASP.

Actuate in 1999.

Not sure what year I learned .net, maybe 2002 or 3.

I even quit a job because they wouldn't allow me to learn Java.

Except I got a Sr. Java position in 2006 without writing one line of code prior.

And then Microsoft BI after that.

And this latest, I've learned more in 3 days than the sum of everything prior.

I do enjoy curiosity and putting the pieces together to see the big picture.

I've always liked to learn, especially the advanced stuff.

I read the dictionary as a child.

And I was in advanced math in the 2nd grade for a while.

And I was friends with an Indian boy in early grade school, he brought his almanac over to play one day, just for fun.

And I solved the Rubic's cube in the 6th grade.

I think being in the IT field provides a training ground to never stop learning.

And now that I've been exposed to unlimited training, the sky's the limit.

And so it goes!


Always Be Learning

I started out doing reports in Crystal.

At the same time, I learned Oracle and PL/SQL.

At the same time I learned Visual Basic 4,5&6.

So at the base of my IT career, there was a mixture of Programming or getting the data into the database, and Reporting or getting data out of the database.

And that has been the trend for the duration of my time in IT.

Except in 2006 I went to the other side, from Microsoft to Java.

And in doing so, I shot myself in the foot.

As my .net skills were becoming obsolete as my Java skills were learning 1999 technology of J2EE with the exception of doing Web Services in Java.

Then I became Supervisor or Team Lead so I was only writing T-SQL and SSRS with some SSIS, and no actual programming in .net or Java.

And that did not help my coding career, as my current job is to program and write reports.

So that's why I've been learning so much lately, to get back in the game, to get my coding skills sharpened, as to become marketable again in the full life cycle of programming.

Because we are all free agents or Independent Contractors here, even if we are employed full time.

Because they can give us the boot at any time as we all know.

So I'd like to get my c# skills up to speed, throw in some jQuery, OData and JSON and a little bit of AJAX.

Also, with HTML5 gaining traction, my JavaScript skills need a tune up as I'm predicting that will overtake c# in the days to come.

Add some Big Data experience to get some variety and I should be good to go.

Always be learning.

Online Technology Training

I've been watching training video's the past few days from Plural Sight.

After signing up for a year membership, they have tons of cutting edge technology video's to view.

I started off with Power Pivot for SharePoint, then Power Pivot, next Big Data, HTML 5  jQuery  NoSQL, NodeJS, OData, Advanced Reporting Services, Power Shell, SQL Azure.

The video content is great, the delivery mechanism is too, either HTML 5 or Silver Light visible on the web or IPhone.

For now, I'm just getting familiar with all the technology, from a high level view.

So I can understand what the technology does to get the big picture.

I'll have to drill down to the detail level at some point to figure out the inner workings of specific technologies.

Right now there's just too much to learn.

I counted about 400 courses online as of today.

Technology training, anytime, anywhere, as much as you can consume.

Not too shabby!


New Age of Training

When I first entered the profession of IT in 1995, my training was rather haphazard.

We had a consultant in town from St. Louis from our recent acquisition.

He and I sat in a room together for almost the entire day and I picked his brain for everything he knew about Crystal Reports.

Reason being, there were no books or training available anywhere.

I happen to go to Atlanta, Ga one weekend and found a Crystal Reports for Dummies book which my boss reimbursed.

However, that's all I needed to get started in Reporting.

Over the next 15+ years, I found it difficult to learn new software.

Reading the books didn't quite do it for me.

Especially when trying to learn SSAS Cubes back in 2001, I was completely lost.

My friend Joaquin was good at learning new technology like Microsoft ASP, and he would stop over my house and in a few hours I picked up enough information to be proficient.

And I've purchase books over the years and scanned the internet for specific questions.

And lately there's been some video's on You Tube.

Well, a co-worker of mine suggested an online training site called PluralSite about 5 months ago.

And I noted it but had no intention of spending money to get training.

Until recently, from a Twitter feed, Stacia Misner had a twitter post referring to PluralSite.com so I did some investigation.

Turns out I purchased a years membership for $300, less than $1 per day, which I can include in my tax write off for my business Bloom Consulting.

After completing the questionnaire and username / password, and paying by credit card, every course became available immediately.

So I watched the first video on PowerPivot for Sharepoint Administration and learned about 5 to 10 new things, which I emailed the SharePoint admin to inform of new discoveries.

And then I watched the next PowerPivot for Excel 2010 video, on my IPhone no less, connected through my home router as to not waste cell phone data package.

So I'm really impressed with the training video's and plan to watch at least one per week going forward.

There's a ton of valuable topics to choose from as I specialize in Business Intelligence but want to stay current on topics like ASP.net, c#, Big Data, NoSQL, SharePoint, programming for Cell Phones, etc.

There's no shortage of topics to choose from.

This should really help me stay current and learn some cutting edge languages.

And as we all know, being a computer programmer, our skill-set can add value to our career and keep us ahead of the curve.

Glad I finally found a good training site.

You should check it out!

What Advice Were You Given?

What career advice were you given by your parents growing up?

My father worked for IBM.

Every day he went to work in a suit and tie with white shirt.

He told me his job was to sharpen pencils.

That's the truth.

Because he always had several number two pencils in his shirt pocket.

And I never did know as a child what he did for a career.

It wasn't until decades later, after I was in the IT career path, that he gave me good advice.

He said "learn the business, how things integrate, get to know the decision makers, and solve their biggest problems."

And that's worked out great for me, although I instinctively knew that already.

You see, I avoided the career of IT on purpose while in college.

Except working in the workforce for a few years, I was naturally drawn to IT, specifically data and reporting.

And I realized that it was a natural fit.

Did the Anthropology degree help me to become successful, it got me a 4 year degree from a major university, that's about it.

I have been self taught in the world of programming for the most part, with a few courses: Fortran; c++.

However, I'd like to thank my father for letting me learn the IBM PC in 1982 in PC DOS, BASIC(a) and for the genes of thinking in code.


Be the CEO of your Career and Life

Most of us work for a living.

And many of us have full time jobs.

So we give our allegiance to the corporate office and trust they have our best interest at heart.

Work hard, keep your head down, earn a salary and hope for the best.

However, due to the economy and the trends of modern day work life, we should really consider ourselves as Independent Contractors who happen to work full time.

You really need to take control of your career and life.

And by that, you need to become the CEO of your life.

You need to be held accountable for your career.

You need to find the training necessary to stay current in technology.

You need to have alternative plans in case you are downsized or right sized or outsourced.

You need to have a financial cushion to hold you over for a few months.

You need to network with people in the industry.

You need to have a twitter account and stay current with market trends.

You need to keep in touch with recruiters and take the pulse of the market.

Because you are responsible for your career.

And by that you are the CEO of yourself.

You must sell, market, research and development, accounting, finance and budget, maintenance, the whole kit and caboodle.

You may have a full time job, but you are still an Independent Consultant if you ask me.

So accept your new role and take responsibility for you career and life.

And there you have it!


Why Do You Blog?

Q: Yo, JB, what's up?

Q: Everyday, you write these blog posts.

Q: About work, your personal life, things in general.

Q: Why do you do it?

JB: Self expression.

JB: A platform where free association rules on various topics.

JB: Where creativity blossoms, where the truth is revealed.

JB: And topics can be thought out in a public space.

JB: Some blogs revolve around the world of Business Intelligence.  Others about the IT industry.  And what it's like to work in the programming profession.

Q: Right on.  So you must be making tons of money off this site,

JB: Sure, if you mean no money whatsoever.  I haven't earned a dime off this.  However, it helps to build an online brand if you will.  We've had over 27,000 views on this blog which has been up and running for a few years now.  Surely somebody is interested in what I've got to say.

Q: So do you consider yourself a professional blogger?

JB: Well, I do enjoy typing away at the keyboard, armed with only an idea or concept.  And within a few minutes, that idea has taken form and is sent off into the world of the internet.  I usually write the blog from start to finish in under 5 minutes, with very little concept of the final product or even blog title.  It's basically like a tornado of thoughts captured in space and given birth to a new entry.

Q: Who would you say is your typically reader?

JB: Perhaps prison inmates with a lot of free time on their hands.  Perhaps the FBI, just kidding of course.  And perhaps some people in the IT industry at large.  My style is unique and the topics vary greatly in short periods of time so the reader is always guessing what type of article will be next.  I would have to say that if weren't for blogging, I would not be a blogger today.

And there you have it!


1, 3, 5, 10 Year Goals

1 Year Goal - I'd like to learn some newer technology, specifically:

  • Big Data
  • c#
  • Self Service BI
  • Cloud Technology
  • Mobile Technology

3 Year Goal - I'd like to become more senior level / BI Architect / Data Scientist.

5 Year Goal - I'd like to increase my salary by 20%.

- 10 Year Goal - I'd like to have my house paid off in 10 years or less.

BI - Microsoft Style

Business Intelligence

Traditional Reports 
-Microsoft SSRS

-Microsoft SSIS
-movement of data

-Tabular Model
-Microsoft SSAS

Self Service 
-Microsoft Power Pivot
-Microsoft PowerView
-Microsoft SharePoint
-Microsoft Excel

Mobile BI
-Microsoft Phone

Cloud BI
-Microsoft Azure DB
-Microsoft Azure SSRS

Embedded BI

Big Data
-Microsoft Azure Hadoop
-SQL Mashing
-Complex Data

Plagurism - Idea Theft

For those people who are creative, new ideas are everyday occurrences.

For those people who are not creative, new ideas are hard to find.

Being hard to find does not mean impossible to find.

Simply take someone else's ideas.

And there you have it.

I tend to think outside the box.

And see patterns in things that others do not.

And I'm fairly vocal about my finds, as they are sent off into the world.

Kind of a freebie if you will.

Sometimes I see people taking my ideas and not giving proper credit.

It's interesting to witness firsthand.

However, it's really plagurism.


It just means that I now have to monitor my conversations with specific people.

That's all.

And maybe one day, I will get the due credit for my unique perspective and problem solving techniques.

Time will tell.



I've been rocking and rolling lately on the new job.

And learning new things.

And today I had my headphones on and they apparently had a big meeting I didn't get to attend.

Somewhat towards the mid afternoon, they called us into another meeting abruptly.

And it was announced my boss got a promotion, which is well deserved.

So who my new boss will be I do not know.

And there were some people in other dept's let go.

Overall I like change.  However, the news kind of threw me for a loop.

So we'll have to see what develops going forward.

I was able to score some free online video training at work and watched a c#.asp video today, learned some new things.  They have hundreds of videos to watch, except I'm thinking to purchase the PluralSight video's as I've heard good things about them.

And tomorrow is the beginning of the Winter solstice and the Mayan trigger date for a calendar renewal and the universe will be perfectly aligned around Saturday so think good thoughts, as they will be amplified into the universe.

And I did get a chance to visit the Native American Indian spiritual mound this am before work which was nice.

So there you have it, change change and more change.

And so it goes!

Oh yeah, We'll Need Some Reports

Back when I started in IT, reporting was about the lowliest occupations one could encounter.

People spent millions of dollars on new enterprise applications.

And then one day someone asks the high priced project manager, "What about reports?"

"Oh yeah, we'll need some reports."

"Let's get the Reporting guy in here to dump some requests at the last minute.  No we're not going to include them in the database design, which would make their life easier.  Let them figure it out."

That's the view from my perspective.

Now all the whoopla is Business Intelligence, Data Scientist, Reporting is now the hottest occupation in the world.


Except I've been doing this for about 15 years and way back when we were just lowly report writers.

I knew it way back when, the top managers would contact me to get some report so they could manage their business.  I saw the writing on the wall.  Data was the gold.  Not building some application.

The report writers time has arrived.

We are no longer an afterthought of some high priced project.

We drive the business now.

And so it goes!

Reasons for IT Talent Shortage

Is there a justifying talent shortage in the IT industry.

I would have to say there is.

Partly because the number of jobs has increased.

So the Demand exceeds the Supply.

Classic economics.

And why is that?

Partly because of the pace at which technology changes.

If I, as a programmer, decide to learn a new technology, chances are, it will be replaced within a short period of time, by some newer technology.

So I must decide, strategically, what I think will be a valuable, marketable, sustaining skill in the future, and go in that direction.

However, even with this approach, the number of languages required to 'keep up' is overwhelming to say the least.  

Here's my recent blog post on what I plan to learn:

So you can never really get ahead in the IT industry.

You may get to use the cutting edge technology for a brief period, and shortly thereafter something will surpass it.

And that may intimidate many future IT professionals.

Let's say they want to become an architect or engineering, they learn the trade and are set for life.  Perhaps get some industry publications from time to time to learn the latest trends but they don't have to relearn everything each 6 months.

Musicians are not required to learn completely new instruments every 6 months, they learn one or two and make a career out of it.

And like I've said before, programming is just one side to the job, learning business rules, project life cycles, industry jargon like PCI, sarbane oxley, etc. are required to be learned on the job.

Another reason is the geek factor.

Most programmers are smart and competing with people every day is another reason people may choose to stay away from IT.

Being an IT profession is not easy, and I believe that constant change, steep learning curve and competition to be the main reason for the shortage of IT programmers today.

And there you have it!


Keep Up the Great Work

Well, we'd like to go over your performance review for 2012.

Although you rarely show up to work before noon, you seem to be very social and cordial, as you settle in to your desk around 1pm.

Granted you take a 2 hour lunch, so when you get back around 3pm or so, we understand many people feel its necessary to surf the web for a while.

So the 15 to 20 minutes of work consisting of emails sifting, when you're not drunk, you seem to have a good understanding of the components that make up your job description.

We completely understand that you have to leave early on occasion to jump start the traffic rush, although we may have seen you work until 5pm from time to time, which is commendable.

And many many people find it more comfortable to work in a casual environment, by removing their shoes, socks, pants, etc. at their work desk, which opens up the doors of creativity.

Overall, we think you're doing a great job.

Keep up the great work!

If only!


Data Dairy Farmer

So Milk comes from a cow.

And the farmer milks the milk.

To sell to a supermarket.

To be consumed by the customer.

So who owns the Milk?

It depends.

So data is stored in a Database.

Which is pulled by a Business Intelligence professional, or Self Service BI user.

Which gets into a report, dispersed for management.

So who owns the Data?

It depends.

The cow may produce the Milk, as a Database Administrator stores the Data.

But they don't own the product.

They release it into the world.

For consumption.

Milk feeds the people.

Data feeds the users.

I'd say data growth is taking off.

So choose your new career, be a Data Dairy Farmer, and Milk that Data.

And so it goes!

Self Service BI Do It Yourself Projects

Back in the day, people hired contractors to perform jobs they could not do themselves.

I'm talking about home projects, new roofs, new cabinets, new bathrooms, etc.

And homeowners were at the mercy the professionals, for pricing, quality, duration, etc.

Then along came a company called Home Depot, where the average person could purchase supplies to do the work themselves at substantial savings and convenience.

With Lowes cropping up, Ace getting larger and the home maintenance revolution was born.

So now take data.

Long ago, the data was housed in a large castle by IT, surrounded by a mote of crocodiles and man eating beasts, where no customer was allowed entry, ever.

And the business tired of having to wait, for inferior products and having to bow down to the mighty IT wizards.

And along came Self Service Business Intelligence.  The ability for end users, who are somewhat savvy with the data, to pull their own data turned into reports for consumption on their terms.

And a new revolution in data was born.

I don't think we can put the genie back in the box as users get a taste of what they've longed for.

So how will that impact IT.  Well, they will have to relinquish their throw, and make room for the new kid on the block.

As prices come down, technology increases and becomes more available, and the competition is already racing forward with their own initiatives, it only makes sense that SSBI will be the future, sooner than later.

So hop on the bandwagon, ride this puppy for the next decade and reap the rewards.

Self Service Business Intelligence is the new Self Service Home Improvement revolution!

And so it goes!


Structure vs Unstructured

When people in BI think of Structured vs Unstructured, they first think of data.

But what I'm thinking of is the work environment.

Some developers like to know what they are working on way in advance.

By that, they want to know who the customer is, what the assignment is, and each step broken out into line items of a project plan or ticketing system.

That is what we call a Structured project.

And then there's unstructured.

Customer is undefined, business requirements don't really exist, database sources are scattered about, how to link them is anyone's guess, the business rules are not documented, anywhere.

And your job is to make things happen.

Some people prefer the first methodology, and head for the door once they figure out that's what they're up against.  They prefer to be hand held each step of the way.

And other people prefer the freedom to discover what the end result will look like.  This is what I see as the primary difference between the Business Intelligence developer (1st scenario) and the Data Scientist (2nd).

The Data Scientist has the experience to bring to the table their understanding of technology, the business and the analytics.  They may have worked through the ranks up from a report developer to BI person to Data Scientist, or they may have had the educational background to land at the top right off the bat.

So which do you prefer, structured or unstructured.  Once you know that, then you can head in the direction best suited for you.

And there you have it!


Are Full Time BI Pros More Valuable Than Consultants?

To be a report writer, you really need to know the technology.
To be a business intelligence person, you need to know the technology and the business.

And if you are a contractor or consultant, you spend your days working for a client, then move on.

All that business knowledge down the drain.

So what about the full time business intelligence developers.

They stay put at their jobs.

And over time they learn the business inside and out.

So they become the domain knowledge experts.

And typically the business rules don't change that drastically over time.

So their knowledge becomes an asset to the organization.

You can hire a BI consultant any day of the week.

Because they know the technology.

They don't know the business, most likely.

And information is power.

And the people who are on staff, the ones soaking up the business rules, they are the go-to people now a days.

Because that info is more valuable than consultants ability to produce reports.

So perhaps full time BI is the way to go?

Who knows.  Things go in cycles.  Time will tell.

Self Service Business Intelligence Summary

Self Service Business Intelligence (SSBI) - provide mechanism for users to run their own reports on demand from anywhere.

Sure sounds easy.

IT just builds them the road, the pavement, the sewer lines, the traffic lights, the on and off ramps, set up an entire infrastructure to allow the user to drive.

"Boy this driving sure is easy, just press the gas and off you go.  I don't see why we had to wait this long for Self Service."

That's because IT is providing the sandbox environment to allow you to focus on "your" business.

And not have to worry about the complex data preparation, cleansing, mashing, governance, backups, permissions, etc.

IT is still moving the heavy blocks to allow the business the comfort and ease of Self Service.

And IT has access to every department's data, not just yours.

So they can see the bigger picture.

And isn't that the name of the game.

Include as much data as possible to enhance your view of the ecosystem to provide analysis which spawns insights, converted into action by leaders.

And there you have it!


IT is the Corporate Dumping Ground for Any Problem

In the Enterprise of many organizations, it seems the IT department is the goto place for many questions.

Many business divisions, when tasked with a problem, simple relate the problem in terms of technology, and send their request over.

And IT gets a lot of problems thrown their way.

Wrapped in technology.

So IT must solve the majority of the organizations problems whether they are technology related or not.

Sales department needs more leads, so they send a request down to IT to find a technology solutions for producing more leads.

Marketing department needs better tracking of money spent and return on investment.

Etc. etc.

Basically what I'm saying is that IT department gets legetimate requests for providing solutions via technology, however they are also "dumped" on in many orgs by many departments for doing the other departments job.

IT is the Google for the enterprise on solving problems, whether or not it's an IT related issue.

Just wrap it in technology and throw it over the fence.

Then demand time frames for expected results, regardless if the IT department has sufficient resources or talent on hand.

And so it goes!


Quick Meeting with the Boss

Most people tend to shy away from their Managers.

I went to talk to mine today, find out how things are going, if he's happy with my work.

I said I've been working on the data mash query for a month, now I'm knee deep on another project this week, haven't produced much in 5 weeks, but I'm working hard.

He understood and said he'll try to mix up the Research and Development projects with a few quick wins.

It's kind of different for me to work on longer projects as I'm used to juggling 10 things at a time, picking them off one at a time, feels like progress.

We still need to look at the work I did and validate it and see what else we can add.

And there's no shortage of work so we'll see what happens.

Overall, my favorite IT job ever.

Except my stints doing archaeology and tennis teaching were still better!

Business Rules

What happens when you develop a report?

You first outline what question you are trying to accomplish.

You get this by asking the user.

What's next?

You translate the business requirements into working code?

And what if you don't know where to get the data from or how the business rules operate?

You are at the mercy of asking someone else.

And there lies the problem.

Because some people don't necessarily give you what you're asking for.

And they give you the wrong information.

Or incomplete information.

Or don't take the problem you are trying to solve deep enough.

And then you go back to your desk and create a report, which is wrong.

And then it's your problem for creating a piece of crap.

But wait, you say such and such person gave you the wrong information.

They deny it or say they didn't understand the question or didn't have enough time to think it through.

I'm telling you that the report developer is at the mercy of the business domain expert to provide fast accurate business rules which can then be translated into workable code.

And what if you're business rules are not documented.

Well, then, you do have a problem.

That's the way I see it.

When a report developer is dependent on other people to supply what he needs, namely business rules, he is at their mercy for their time and for receiving correct information, because to be honest, the business user doesn't give a crap about you or your project.

To them you are cutting into their precious time and they're trying to get rid of you as fast as they can.

So go ahead and blame IT for all the problems in Reporting and I'll tell you why it's not 100% true.

So when you have a question, choose from the following:

A. Go Away!
B. Incomplete Answer
C. Wrong Answer
D. I don't know
E. Go ask someone else

Team of 'The Reporting Guys'

I've always considered my style that of the 'one eyed man in the land of the blind'.

By that, I got by for most of my career for knowing some of the ins and outs of technical jargon that many people did not know or understand.

Which has provided a good stable career in the field of Information Technology.

However, working for a real softward company now, I see that although my skills are descent, there are some people that far exceed good.

Not only do they know what I know, they know a whole lot more, other stuff.

They may know more technology.  More industry jargon.  More business rules.  More Accounting skills.  More political savy.

And some of that on much deeper levels.

Which is not to say that I'm a dunce or anything.

Which just means I continually have to learn the business in addition to the data as well as they systems, oh yeah, and the customers within each of the departments as well.

However, this is a good learning opportunity.

I used to be able to look at the SQL and the report and make the necessary modifications by myself.

Now I'm dependent on other's to provide the necessary business rules and where to find the data and how things changed over time.

I've always been 'the reporting guy'.

Now I'm part of a Business Intelligence team.

Working alongside a half dozen other taltented programmers and report writers.

I'm no longer a one man show, a one eyed man in the land of the blind.

However, I think my skills can keep up with the pack.

I'm not out to be the alpha, just contribute to the team and make a descent living.

And there you have it!

Appreciate the Feedback

For some reason lately the number of visitors have increased.

And I got some feedback, one in particular.

On a Power Pivot post from 12/2011, I stated some information that was not correct.

I suggested that you could create a composite key join in Power Pivot.

However this is not accurate, it must be done through DAX and not your traditional composite key join.

So for that I apologize.

Darn thing is, the post has been out there for about a year and nobody picked up on it, or let me know about it.

So I'm glad that we've got some more readers.

And I appreciate the feedback, helps me learn!

The Enterprise is the Database

A long time ago, we had VSAM files on the mainframe.

Then Relational Databases.

Now we have In-Memory and Columnar Indexes and Hadoop and such.

Except Hadoop exists off in some far away place.

While DBA's work with traditional databases and connect to Hadoop via connectors.

I wonder why we don't have a Hybrid Solution.

A mix of databases and large data sets.

So you can wind your way through one to the other without separation.

One unit of data, some of it structured and some of it unstructured.

Why do they have exists as different entity's.

I say the database should store data both traditionally structured, semi-structured and non structured.

Why should they exist in silos?

At some point they should be merged together, don't you think?

And become transparent to the report developer or data scientist.

And then go a step further.

Why should a database even exist at all?

Why couldn't you send a query out to every desktop in your organization and pull the necessary data right off the machines, why do they need to be in a cluster.

The entire enterprise should be the cluster.

And that is why you would need unique identifiers so you can establish a relationship to where you found this data and when it was pulled.

I see too many limitations on the traditional database, open it up, the data is sitting there waiting to be modeled and reported on.

Remove the boundaries of legacy systems.  The enterprise is the database.

Maybe it will be one day.


Life of a Report Monkey

In 1993, I answered phones at Sears Credit Central.

And I was asked to come in early and count the number of phone calls for the prior day, the number of approvals and declines, and enter the totals into Lotus 1-2-3.

My first data job.

Then at NationsCredit we took faxes, had temps data entry, it was chaos, I suggested to number the faxes as they arrived so we could track them, my first process modification to smooth business flow.

In 1995, working for NationsBank, I approved loans for the bank, again, they asked me to count up the approvals and declines.  For this, I was allowed to take a programming class at the college, I chose c++, got an A, they reimbursed me, I begged the IT manager to hire me, which he did, and I learned Crystal Reports version 5, my first real programming job.  ( I also did Visual Basic and Oracle).

Then at Kronos, I created reports for Hospital Time Sheets, and learned what a Left Outer Join was.

Then at Florida Power, I did Visual Basic / Oracle PL/SQL complex Access reports.

Then at Paymentech, I programmed Actuate reports.

Then did a bunch of Crystal Reports at Surfside Software, Z-Tel (Scorecards, MDX & Reports).

At West Point Underwriters, I balanced the books for two Insurance company's for close to 4 years, which burned me out.

Then at Bamboo Software, I did more Crystal Reports, then Health Integrated, more Crystal Reports, then the County, Java and more Crystal Reports, then I learned SSRS on a temp job and did SSRS for the School Board.

Which leads me to my present job, advanced BI using Tabular Model and SSRS.

So as you can see from my 20+ year career, I've been working with data regularly.

SQL language hasn't changed much.  Nor Crystal Reports.  What has changed is the number of BI professionals out there now.  When I started you couldn't find a reporting guy anywhere, they were non existent.  I was a lone wolf when it came to reports, as all programmers wanted the cool stuff.

Well guess what, data is cool.  And that's the life of a Report Monkey!


Microsoft's Advantage

Some people say that Microsoft is past it's prime.

I disagree.

They have the market cornered with their Office package.

They have an entire following for their Visual Studio platform with plenty of languages to choose from.

And they have a solid database to add to it's collection, available on premise, in the Cloud and in Parallel Storage racks.

They have their foot well established in the BI world, with SSRS, they have a solid ETL tool called SSIS, they have Traditional and Self Service BI through SharePoint, they have hooks into Big Data and of course the free yet powerful tool called Power Pivot for the Excel jockeys.

And they have everything available in the Cloud, called Azure.

Oh yeah, they have a product called the Operating System, you may have heard of it, now available in Windows 8.

They have Mobile technology as well.

And everything integrates tightly.

Some people say they are free falling, I don't agree.

Apparently they still have lots of smart people working there and I'm along for the ride.

They have all the bases covered, although sometimes slow out of the gate on new technology, namely the Internet, Reporting and now Big Data.

Don't count them out, they're still a player in the world of IT, in my humble opinion!

Beginners Guide - Microsoft Hadoop On Azure

From the Microsoft BI User Group meeting last week held in Tampa, they gave us a link to begin  Microsoft Hadoop on Azure website:


This is separate from Windows for Azure (free 90 day trial)


There's a site for Getting Started, however I don't have the necessary files installed on my laptop, in addition, we were told the 1 x tutorial sandbox is good for 5 days and I didn't want to waste the valuable opportunity without doing some basic research first...


Clicking on Hello World...

It seems there are some prerequisites for getting started, from the downloaded Word Doc above (HDInsight Jumpstart):

Software prerequisites for the jump start scenarios

Each example has the prerequisites needed to run the scenarios. Here is the complete list of software requirements which are needed to perform the exercises so that you can work through the examples with minimum disruptions.
Windows Azure trial account for SQL Database service and Azure Storage - http://www.windowsazure.com/en-us/pricing/free-trial/
Windows Azure HDInsight Service trial account – http://www.hadooponazure.com.
Client utilities for Microsoft’s Apache Hadoop-based services including the HiveODBCSetup msi - https://www.hadooponazure.com/Dashboard.ashx?r=%2FHome%2FDownloads
SQL Server 2012 Analysis Services installed with a Tabular model database instance on a server or virtual machine - http://www.microsoft.com/sqlserver/en/us/get-sql-server/try-it.aspx.
Microsoft JDBC Driver 4.0 for SQL Server – used for Sqoop scenarios with a local instance of a SQL Server database - http://www.microsoft.com/en-us/download/details.aspx?id=11774 (not required for the jump start examples)
Microsoft SQL Server Connector for Apache Hadoop – used for Sqoop scenarios with a local instance of a SQL Server database - http://www.microsoft.com/en-us/download/details.aspx?id=27584 (not required for the jump start examples)
Microsoft HDInsight Server – link coming soon – stay tuned at http://social.technet.microsoft.com/wiki/contents/articles/6218.getting-started-with-hadoop-based-services-for-windows-en-us.aspx. You can also use the Remote Desktop tile on your Windows Azure HDInsight Service dashboard to simulate performing these actions on Windows Server.

Note regarding code samples

This paper includes example command line commands that include punctuation that may have been altered within this Word document. Examples include the minus (-) sign and double quotes (“) characters. If you get an error attempting to copy and paste, check issues with the punctuation characters.

Need to Learn Tabular Model and Dax

Multi Dimensional Cubes have been around for a long time.

And I bought a book in the year 2000 to try and understand them.

Holy toledo talk about complicated.

And with no playground to experiment I eventually moved on to other stuff to learn.

Although I never lost the interest in learning.

And then I started to attend SQL Saturdays and conferences and it seems that the Cubes have an entire world of their own.

And they are in high demand, and pay well.

Except I never learned them.

So I investigated a bit and built my own cubes on my home environment and it seemed pretty straight forward.

Until I got to the MDX portion, complicated.

And my next hurdle was I couldn't load SharePoint to test the new functionality.

So I post poned my learning again.

Until I got my current job where I got immersed in Tabular Model.

And it all became clear.

I don't need to learn MDX, I have to learn DAX.

And I've already created 2 production tabular models and I maintain a corporate wide cube by adding permissions, setting people up with Power Pivot and SharePoint.

So by luck, I managed to bypass having to learn Multi Dimensional Cubes and went straight into Tabular Model.

Which is not backwards compatible.

There is a huge debate going on right now, will SSAS be supported in the future or will it become legacy, and what to do with all the billions of dollars invested in that technology and support.

I'm happy with knowing just Tabular so I'm moving into the future with no complaints or regrets.

Sure I was disappointed when Microsoft abandoned Visual Basic 6 for .net with no backwards compatibility, that's life.

And so it goes!


Need for a Chief Data Officer

There's a lot of talk lately about software BI vendors abandoning the Traditional reporting aspect.

They fling comments like IT is too slow, too domineering, their reports are wrong, they take too long and the reports never match.  They are uncooperative, bureaucratic ticketing systems, prioritizing system that makes no sense, unable to fulfill the needs of their customers.

And I would agree with many of these issues, as I know how the system works.

So the business has volunteered to take all the responsibility from IT and do their own reporting as they think they can do a better job and they have a corporate credit card to spend on Cloud technology and consultants.

And chances are they'll do a worse job than IT because they may know the business better, but they don't have the big picture and how all the data integrates and flows through all the systems.

I do think Self Service is needed for business users to get their own data, however, it should be architected and structured approach in tandem with IT, not against.

Because as you know, first problem they run into they'll be heading over to the IT department to figure it out.

Because IT has gone to school to learn this stuff, because they attend seminars and continue education to keep up with technology, as their main source of income and profession, not something they do in between meeting their sales volume or meeting with potential clients.

Has IT blundered the world of reporting in the past, sure.
Can the business do a better job by themselves, I doubt it.
Can they work together as a join effort to strengthen the bonds between them, you bet.

However, due to power plays and positioning and budgets, I think their should be a third team that integrates the two, as I've mentioned on the #BIWisdom tweet chat, their needs to be a department such as the Chief Data Officer, who matrix team members from IT and all divisions of the organization.

They can charge back the time spent working for the CDO so the theres no increase in staff and budget should not spike suddenly.

The Chief Data Officer would be responsible for the integrity of the data using Data Governance, can work with IT to integrate data sources, can work with the business for documenting business processes, can work with external people to investigate current market conditions and trends and how other company's are doing things, they would be responsible for the direction of the Business Intelligence software within the org.  They would report directly to the CEO, they could hire external consultants when IT skills can't be found in-house.

So is the world of Traditional Reporting going away, highly unlikely.
Is the business unit going to take 100% ownership of BI, highly unlikely.
Is there an opportunity to bridge the gap with a newly formed department, called the Chief Data Officer?  You betcha!

Continue reading this debate here: http://cwebbbi.wordpress.com/2012/12/05/why-corporate-bi-and-self-service-bi-are-both-necessary/


Big Data a Must Know Now

Listening to a Big Data presentation the other day, it's becoming more apparent that Big Data is seeping into the world of data.

Which I believe can not be ignored.

For any BI profession, they need to be aware of this growing trend.

Like me, you may not have access to Hadoop or unstructured data, however, you should at least be aware of the trends, the different plays, the architecture, the infrastructure, how it adapts to current software vendors and how the market is focused.

Just another technology to add to your wish list of skills to learn.

I know I plan on increasing my awareness, to stay in the BI game.

And there you have it!

Who's your Boss?

Who owns you?

Is it God, your boss, your wife/husband, your kids, your church, your civic organization, the gov't?

Because we all report to somebody.

One way or another.

We are not actually free beings on this planet.

Yes, we are truely free spirits, encapsulated in human form, for this lifetime, until we get recycled in the pool of the universe.

Except here in this life we are obligated to meeting certain expectations from external forces.

Noone is immune from it.

So the choice is yours, to find out who you report to.

Same with vices, choose wisely.

And there you have it!

IT is from Mars, Biz is from Venus

For some reason IT and the Business Units do not always see things the same way.

And the reason could be that IT is from Mars.

Concrete thoughts, thinking in mostly logic.

The business thinks in theories, in dollars and cents, in sales.

And there lies the problem.

Two different ways to see the same problem.

And the solutions are differnt too.

Business users have some vague idea of what they want.

IT wants specifics, don't beat around the bush, what is it you want.

Well, once we see what you deliver, then we'll think about it and change everything.

And when communicating to a group of mixed IT and Business folks, you really have to walk the line as to not shun either side.

You can't get too low level in the details, and you can't put too much fluff in there either.

Now the Business wants to take over a lot of IT functions.

And IT wants to take over much of the processes and such.

The grass is always greener on the other side.

Except getting the two groups to see eye to eye is not always easy.

And the main reason is IT thinks in right brain and Business thinks in left brain.

And rarely do they meet on the same wave length.

And there you have it!


Challenges of Working with Data

While working with data can be fun as well as reveal nuggets of insight, it ain't always easy.

For me, pulling data across multiple databases has really caused some inefficient queries.

They are slow and you really have to think like a chess player before starting an hour long process, think of all the downstream affects.

And there's no AAA map to help guide you from data sets to data sets, you're basically on your own.

That goes for both business rules as well as data structures.

Because the people that could offer assistance just don't have the time to help you out.  They've got their full time job to take care of.

And the thing you're trying to do, sometimes they don't tell you until you've got something to show them, then that gives them an idea of potentials and they change the specs.

What I'm talking about here is not your traditional reporting, nor is it common business intelligence practice.

It's basically data science work minus the big data.

It has tons of data, just not related data.

And the goal I've been trying to accomplish is to mash data sets that don't typically mesh.

And in order to do that, I've had to use fuzzy logic.

I've been working on this project for over a month now and I'm zoning in on some results.

I gave my boss an overview today and he said to narrow down the scope.

I was trying to figure out the entire project, account for every record in the database, which I just about figured out.

He wants a subset of the data, so we can QA it, and that's it.

So that's okay, I should have a sample file shortly as well as counts and percentages of the data he wants.

This has been a tough project.

Except if we can find insight, it will be well worth the effort.

Because I had to overcome lots of challenges along the way.

As I assume most data people do as well.

And there you have it!


User Group Meeting - Tampa BI

Tonight I had the opportunity to attend the local chapter of the Microsoft Business Intelligence Users Group in Tampa, hosted by Jose Chinchilla.

The presentation tonight was on:

Topic: Big Data for BI
Speaker: Robert Skoglund
Robert has deep technical knowledge of the Microsoft products required for delivering enterprise-class Business Intelligence & Data Warehousing solutions. Currently, he is the World-wide Business Intelligence Technical Community Lead internally for Microsoft, as well as the Technical Lead for Microsoft Enterprise Services Big Data Strategy. Robert has been delivering Business Intelligence and Data Warehousing solutions since 1996 using Microsoft technologies

Good event, good topic, good presentation.  Learned a bunch on Big Data Microsoft style, Hadoop and the new role of Data Scientist.

Food was provided by Vaco, got a chance to talk with Brady tonight.

Also got to chat with some new people about Business Intelligence.

Got to keep the SQL BI learning going and the Users Groups are a key player.

Thanks to everyone who hosted the event and to Robert for a great presentation!

Crossing 25,000

With any luck, today we shall cross over the 25,000 visits to the BloomConsultingBI.com blog.

Thank you everyone for your continued support.

We don't claim to be expert at much, but we like to write a lot of story's.

For your enjoyment, as well as mine.



Some More Fuzzy Logic

I've been playing with some Fuzzy Logic code.

Basically, you tell it to match on DomainName (from the user's email address) plus their Country and it looks for matches based on that.

Then the Fuzzy part, you tell it the Company name to search for and it looks for matches based on similarities based on percentages.


@SimilarThreshold FLOAT

-- Jaro-Winkler returns a value between 0 and 1, the closest to 1

-- the more similar it is. This variable allow us to ignore matches

-- with a lower score.


@SimilarThreshold = 0.825;

So you tell it to match on 0.825 and it only returns data equal or above that range for matching percentages.

I've been running the code for 375k rows and it runs for about a half hour give or take.

After it renders results, I can identify the percentage of matches found, and adjust the percentage accordingly.

Actually, I'd prefer to keep the percentage higher than lower, to make this code air tight.

However, once I get the figures in place, then I can begin testing lower percents as well as do some Fuzzy logic for other fields, like IP Address, Address, etc.

This really opens up a lot of doors when trying to match desperate data source accross servers that otherwise would be able to mash.

Fun stuff for a Friday!


Face to Face Contact Obsolete

Life is a little bit like poker.

Guard your hand, protect your cards, be aggressive when bluffing.

If you ever read this blog, you'll see that there's a huge amount of transparency.

Which is the way things are headed in real life.

Secrecy is a thing of the past.

However, some people still use cold war tactics.

They hide their true motives, they cloak their agenda's, they pretend to help while they steal you blind.

It's getting hard to tell the good guys from the bad guys, to be honest.

The dividing line has diminished, everyone's out for themselves.

Passive aggressive is the new norm.

People are superficial and there's no real substance.

Conversations tiptoe around the weather, television shows and sports.

Has the world of internet overridden true community?

With people's short attention spans and unquestionable thirst for newness and excitement, do Humans stand a chance in the new digital world?

Has face to face contact become obsolete?

As people stare at their smart phone in replace for conversation.

Personally, I think everyone's out for themselves one way or another.


Some Links on New BI Technologies

I was watching Chris Webb's video from the Nordic Pass Session: Chris Webb - The best Microsoft BI tools you've never heard of

and discovered a new technology called "Open Data Protocol" or "OData".

You can view it's site here: http://www.odata.org/introduction

I imagine lots of this open data can be consumed via Power Pivot.

Chris explains the protocol similar to an "ODBC" for the internet which makes sense.

He also talks about a new product called Microsoft Codename "Data Explorer".

Website is here: http://www.microsoft.com/en-us/sqlazurelabs/labs/dataexplorer.aspx

He equates it to SSIS for Excel.

You can connect to a variety of data sources and apply a series of repeatable steps.

Next topic was Node XL where you can import data from a variety of sources.

And another link mentioned was NodeXLGraphGallery which has some good graphs.

Another free download is LayerScape developed by Microsoft Research.

It takes data from Excel and allows data powered by World Wide Telescope.

So to summarize, Business Intelligence is growing really fast.

I'm glad we have organizations like PASS to help disseminate information easily.  You can watch some great video clips from the Nordic site here.

And thanks to Chris for the good info tidbits!



Fuzzy Logic

Today I'm going to work with Fuzzy Logic.

What that does is looks for matches of data based on a threshold.

So you can specify what percentage is an acceptable match.

I've done this before on a contract through Bloom Consulting, where the logic resided within an SSIS component.

However, today I'll be working with the Fuzzy Logic within T-SQL function is called Jaro-Winkler.

The goal is to find similar matches of data, so we can plug some missing values in some tables.

And by doing that we will essentially close the loop on some otherwise disperate data sources.

Wish me lucK, I'm going in!


Never Give Up!

I know a guy who was diagnosed with a terrible disease at an early age, around 21.

He could have given up, sat on the sidelines of life, chosen the easy path.

Except he didn't.

He chose to accept it and deal with it and live with it.

Although he knew deep in the back of his mind that it was not true.

Although nobody questioned the diagnosis.

Nobody bothered to offer assistance.

Or help in any way.

He was left alone do deal with life with his disability.

He tried to see other doctors to get another evaluation.

None of them bother to listened or help.

Primary care physician didn't help.

Had to pay extra for Life Insurance for extra risk.

Meanwhile, my friend never gave up, kept pushing forward.

Until one day, he called a doctor and scheduled an appointment.

The doctor listened to his story from start to finish.

And what he said was amazing.

After twenty years of living with a debilitating disease, the doctor said it was highly unlikely to be the case and was confident it was not true.

I don't think people can realize what a struggle it was for my friend.

Everybody gave up on him, nobody bothered to help.

And after twenty years, my friend found the truth.

Which he knew all along.

So here's to my friend and his new found life.

Sometimes hope does pay off.

And miracles do happen.

Never give up!


ROI Riddle

How much business was generated based on a specific campaign?

Good question.  How do you know that?

You'd have to have data for all your campaign info.

As well as you Lead info.

And determine which Leads were Qualified.

And of the Qualified Leads, which ones converted into sales?

So what percentage converted, sliced by product, region, etc.

It can be done, just takes time.

And I have time.

So I meet with people on the Analytics side.

On the Business side.

And the Technology side.

To document the processes.

And look for the gaps / holes.

And find solutions.

To produce a workable high level analysis from start to finish.

Every touch point a customer stops along the way.

Once the architecture is in place, it's just a matter of consolidating the data sources.

And creating a Tabular Model for the big wigs to slice and dice.

And that's how it's done.

It won't be long now until I have something to show.

And the riddle of ROI will be solved.


BI= Business + Technology + Analytics

Business Intelligency is comprised of Three Items:

How does the business work?
How do Apps work?
How do Processes work?

What Servers, Databases, Tables, Fields are available?
How does Data get Joined between sources?
What gaps are there in the Data?

What Questions / Hypothesis are to be answered?
How do we find Insights from the data / information?

Traditional Reporting consisted of Technology plus some business knowleged.
Business Intelligence added more of the Processes / Business lingo to the equation.
Data Scientist encapuslates more of the three at deeper levels.


Business Intelligence Is More than Just Data

When people think of Business Intelligence they think of different things.

Some people will rattle off the top 10 vendors in the BI space.

Other will differentiate the key functionality between Cloud BI, Mobile BI, Big Data, Traditional BI, In Memory BI, ETL, Data Governance, etc.

Still others will get into the nitty gritty aspects of how the functionality works.  How to best optimize queries, how to mash data, how to get Unstructured Data onto a report.

And yet still others will talk about the Business Rules for an organization.  How data flows between departments, between systems, between databases, and how it all integrated into a cohesive whole.

Business Intelligence is not just data into information to produce actionable tasks.

It's about Collaberation.  Discussions about technology.  About Business Processes.  About Analytics.

It's about communication.  Discovery.  Detective work.  Finding the truth.

It blends art and science, faith and facts and micro and macro.

What else could you ask for?


Using Data to Answer Hypothesis - Step by Step

In order to solve a problem you must first state your hypothesis.

Once you determine the questions to be answered, you must ask yourself some basic questions.

Who, What, Where, When, How and maybe even Why.

So if you are tasked to find out your customer base for example, here's how I'd approach it.
  • Who: Who is your Customer?
  • What: What Products did they purchase?  How many units?
  • Where: Where do the customer's reside?  (Region, Territory, Country, State, City)
  • When: When did they purchase products?  How long interval between first touch and closed sale?
  • How: How did the customer find our product?  (Campaign, Google Search Word, Advertisements)
Those questions would help to answer the "Dimensions" part of the equations, so you can slice and dice.

The "Measures" or numeric values would be:
  • Sum(Purchase Amount)
  • Average(Purchase Amount)
  • Count(Customers) 
Next step would be to identify the Data Sources, where does my data reside.  What fields would help to answer my questions?  What information is just noise and can be disregarded?

Next step would be to ETL, Extract, Transform and Load your data, assuming your data has already been cleansed and data governed.

Next step would be to create a workable Query or Queries to pull the data into Dimensions and Measures, with the final resting home to be a Cube for slicing and dicing.

That's how I would approach it anyway, as my expertise is from the Business Intelligence side.

If you were a Data Scientist, I think you would follow the same steps, except you may be pulling from Big Data or UnStructured Data.

And you may be applying Models for Statistical or Mathematical predictions and calculations.

In the end you should either have answered your question / hypothesis, or discover you need to modify your approach, find more data, revise your models, etc.

And document along the way, your assumptions and findings, so you can later reproduce the experiment.

That's the way I see it anyway.

Yearly Review

So this year we're trying something different.

We've given the responsibility of evaluation to the brightest minds.

Our pets.

Here's what they had to say.

Mickey the Cat: Jon has done well to provide a stable environment.  He provides boxes frequently for me to sit in.  He plays with me in the morning.  Although he is not the actual person who feeds me, he does a descent job scooping the litter box.  Rooms for improvement, I could use some more time on the porch to watch the lizards.

Charlie the Cat: Over the past year, Jon is an excellent provider, although he tends to spend the majority of time with the canines.  Keep the treats coming and there won't be any issues.

Sunshine the Cat: I usually hide upstairs until it's feeding time so I don't have any complaints except the two dogs tend to sniff me a lot.

Maddie the Dog: Overall Jon is outstanding at what he does, which is to let us out in the morning to do our business.  He is generous with the treats, and he's usually good for a walk in the evening.  I like to go on car rides and find he is very accommodating on that front.  I am impressed with the new swimming pool, however, I feel that the swims should be daily, not occasionally so there's room for improvement.

Chloe the Dog: I would say I'm treated fairly, there's been talk of a Union, however, for the record I'm not a part of that.  Jon has the capacity to think like a dog, so he understands what we want, which is either some petting, some treats, a walk or swim or car ride or more treats.  Jon shows potential and I feel with the right attitude, he will go far.

Overall Rating: 4 out 5 Paws

Signed: The Pets


Assembling Data Sources Into Single Query

I've been putting together the mother of all queries.

Joining data from 2 separate data sources, then combining that with SalesForce, then combining that with a final data source.

Let me say there's a lot of data.

And let me say the query is pulling from multiple servers and multiple databases.

And the queries take a long time to run.

So I've been piecing them together, slowly.

Mostly in Temp tables, with indexes.

So today I moved some of the data closer to the source, and now the query runs a lot faster.

So I can actually see the data results after a few minutes opposed to hours.

Which means tomorrow I should be able to dissect the relevant information to answer some questions we posed as hypothesis.

My goal is to answer the questions, and document the methods in which the data was gathered.

As there will always be exceptions, assumptions and work a rounds.

However, in the end, we should have a global view of the data mashed with multiple data sources to then find insight.

And give the uppers an ability to manage the business in a way they were never able before.

And put the data into a Tabular Model cube for interrogation.

That's really what it's all about.