8/30/2015

Empower the Users through Self Service Reporting

In 2007 or so, I worked a job for the County.  We supported many agencies.  In doing so, we wrote and maintained the applications and data.

I was assigned a new project to help out with reports.  During our first meeting, the client mentioned their data was silo'd behind our walls and they never had access to their own data, except for what they downloaded off the internal application.

The conference room had a computer, so during the meeting I logged on, opened Microsoft Access, connected to the development instance of their data stored in Oracle, and proceed to download their entire database in about a minute.

Opened a copy of Crystal Reports, pointed to the Access database, used the Wizard to create a quick report.  Within a minute, we had a report displayed on  the screen from the projector.

The Agency could see how easy it was to view the data and build a report.  They couldn't believe it.  And asked us for access to the database and how they could purchase 5 licenses of Crystal Reports.  And some of my time to get them up to speed on the reporting tool.

It was great to be the one to open their eyes and see their reaction.  A hidden world opened up in just a few minutes.  Empowerment.  Self service.  No longer dependent on IT.

Fast forward to today, just about every organization is aware of the benefits of accessing their data.  Back then, being a one eyed person in the land of the blind made a huge impact.

8/29/2015

From Reporting to Hadoop to Machine Learning to Black Box Algorithms

Back in the day the DBA was the gate keeper to the data.  Very tight access.  They had the keys to the kingdom.

Programmers were granted access to the applications, reluctantly.  First client server, then web and now mobile.

And then there were the report writers or Business Intelligence people.  DBA never liked them much.  Always pulling reports, slowing down the database, causing locks.  Writing crappy code.

Enter Hadoop.  I didn't see many, if any, DBA's making the leap into Hadoop.  Since they don't like writing SQL, and they don't program Java, Python, Scala, Hive or Pig, why would they like Hadoop.

Report Writers - Business Intelligence - Data Warehouse people, are they getting into Hadoop?  Some.  They have an understanding of the data, although Unstructured and Semi-Structures is a bit foreign.  They can mount the data into SQL Tables in Hive and access from a variety of sources, including ODBC, OData feeds, Power BI, etc.  Just turn the data into Relational and they're off and running.  But they don't necessarily program Java, Python or Scala, from what I've seen.

So if the main players in the data space, the DBAs and Report Writers / BI / Data Warehouse don't have a clearly defined entry way into the world of Hadoop, that could explain why there was more hype than real life Hadoop growth.

From what I saw, the people working with Hadoop, were out of college, or start ups or major companies with financial resources to bring in top developers.  Perhaps now, more companies are jumping on the bandwagon.

It used to be, "What is Hadoop?"

Then, "What do I do with Hadoop?"

Now, Hadoop is another tool in the tool chest, for working with data.

The "shiny" newness wore off.

For companies entering the Hadoop space, I suppose they need a real world question to answer, a use case.  Then create a Hadoop sandbox, obtain a developer and admin, and start ingesting data.  Mash it around.  Look for insights.  Use those insights to run the business.

Reporting has always been past tense, numbers explained what happened.

The hot thing now is Machine Learning, algorithms and statistical analysis.  This allows forward thinking, predictive analysis, forecasting.  This space is hot right now.

Looking forward, algorithms will become a commodity.  Just pick one off the shelf, integrate into your app, and you're off and running.

There's already sites out there now to leverage existing black box algorithms.

You can't expect people with 20 years of experience in IT to magically produce a PhD degree, it's impossible.  The train left the station two decades ago.  Instead, bring the complexity down to a level that Data Professionals can work with it.

That's how I see it.  Time will tell.

8/28/2015

Selecting PowerBI as Next Business Intelligence Solution

Baseball is a fun all American game.  I played as a child in a few different leagues.  Didn't realize I threw left handed until a few years of playing.  Once I switched to throwing left handed, I got a lot better. My positions were mostly outfield and second base, which is odd for a left hand thrower.  I got on base quite often, and was on the all-star team one year.

However, I was also on the swim team.  I entered ever possible race every swim meet.  And I played on the traveling soccer team. And the traveling basketball team. I played in the bowling league. And started to play tennis tournaments. As well as the violin.  Its difficult to get really great at one thing when the energy is fragmented across multiple sports.  So when we moved to Tampa at age 12, my mother had some good advice.  Pick one sport and get really good.  I chose tennis.  And stopped all other sports.  I played #1 position on the high school tennis team in singles and doubles during 10th, 11th and 12th grade, and had scholarship offers for college.

In the world of programming, I work with databases, ETL Extract-Transform-Load, Reports, Data Modeling, Cubes, Dashboards, dot net programming, Hadoop, R statistical language and a wide assortment of other technologies which support the world of Business Intelligence.  And I study Artificial Intelligence, Machine Learning and Internet of Things on the side.

There's a similar pattern here, with sports and technology.  I tend to spread the energy too thin.  Perhaps now is a good time to get really good at a particular technology with the awareness that I have to let other technologies fall to the wayside.

So looking at the various technologies out there in the data space, the direction of the vendors, what the customers will be looking for down the pike, in order to maintain employment over time, I've decided to focus on the Microsoft PowerBI stack solution.  Here's why:
  • First, it revolves around data.  
  • Second, it's available in Excel, the PowerBI cloud and PowerBI Desktop.  
  • It connects to a variety of data sources.  
  • You can pull in existing Excel models, Power Query M code and Reports.  
  • You can get up and running relatively fast and at the same time, you can push the envelope of advanced coding so it's designed for beginners as well as advanced senior programmers.  
  • PowerBI models can be ported to SQL Server Analysis Server Tabular Models for enterprise access and ported to the Cloud for collaboration.  
  • The PowerBI Cloud solution has on-premise utilities to schedule auto refreshes. 
  • And you can connect using Active Directory.  
  • And the PowerQuery M language is similar to SSIS, allowing users to perform ETL Extract Transform and Load without having to know the SQL Language.  
  • Based on the number of releases to PowerBI, that kind of gives you the direction of where the product is heading.  
  • Which means it should be around for a while which means it's a good candidate to pursue going forward as a career choice.  
  • And so many organizations already have Excel and the free plug ins for PowerPivot, PowerQuery, PowerMap and Natural Query Language, it's kind of a no brainer.  
  • And the visualizations are really starting to take shape to compete with similar competitor product offerings.  
  • I need to learn more about the DAX language to see the latest changes.  
  • Another good selling point is shortened life cycle of a Business Intelligence life cycle.  
  • Connecting to a data source, transforming the ETL, pulling into a PowerPivot Model, creating PivotTable reports and dashboards, and uploading to the PowerBI could be done rather quickly.  
  • Trying to do the same thing in traditional BI tools would take a bit longer, which increases the cost of projects and extends the deliver time of final product.  
  • The industry is starting to prefer short bursts of iterative deliverables as opposed to half year projects.  
  • It fits nicely into the Agile / Scrum / Sprint methodology.  
  • And it's good for Self Service.  
  • I can see the R Statistical language making it's way into PowerBI.
  • I see a potential for training Power Users within orgs to build their own Ad-Hoc reports.  
  • Consultants could still build the Models based of existing Data Warehouses or mashing disparate data sets into a single model, and releasing it to the base users for discovery.  
  • That way, best practices and data governance rules could still be followed, along with release management to prevent silos data sprawls with different reports showing different results.
  • Lastly, PowerBI seems like a cost effective set of tools, with a free version as well as paying options with no long term contracts.

So that's a few reasons why I'm selecting Microsoft PowerBI solution as my next go-to product of choice for the next stage of my Business Intelligence career.  

I did complete a solution for a client a few months back, posted about it here: http://www.bloomconsultingbi.com/2015/03/power-query-odata-calls-to-tfs.html


And I was an early adopter back in 2013, presenting at the SQL Saturday BI Edition in Tampa: http://www.bloomconsultingbi.com/2013/11/sqlsat248-tampa-bi-big-data-edition.html
  Thanks for reading~!

8/27/2015

String of Bad Luck

Life is funny.  Things occur in waves.  Good luck.  Bad luck.

Lately, I've experienced a string of bad luck.

First, somebody shot my car.  Not sure when, where, who or why.  But I arrived at a client site about an hour from my house and noticed a hole in the car on  the passenger side.  Opened the trunk area, found a .38 bullet.  Hmm.

Then, I downsized my house, moved some junk to the curb, put a sign up, said "Free".  Also had some stuff I was keeping, set it near the front door walkway.  Headed out for an hour.  Upon returning, people took all the stuff near the curb as expected, but they also took stuff from my piles near the front door.  Wasn't expecting that.  Uhg.

Then, I was trimming bushes on the side of the house.  Turns out the dogs dug a hole, which I proceeded to fall into, hearing a snap.  I probably broke my ankle, yet hobbled for weeks, not visiting the doctor.

I sure hope that's the end of the bad luck streak.  Could use some good luck for a change.