1/24/2015

Observations on Life

If you don't follow sports, then its hard to understand people who take it so serious.  Uh, let me guess, some team won.  Amazing.  In a game of two opponents, isn't it probably that one team is likely to be victorious.

And what about entertainment, all the stars and performers.  If you don't follow them, then watching people who get all excited about it doesn't make much sense.  Some people base their lives around television shows.  Have to keep up with all the latest.  Watching television hypnotizes you into a potato.

And politics.  During election time, every person seems to supportive of their candidate.  But if you don't really care, then the people look a bit ridiculous.  Do you think the candidate gives a flying leap about you individually, or are they more concerned with getting elected, and you are simply the means to the end.

And religion.  We've got one guy who stands on the side of the road every night with a big sign, gets all the people to honk their horn and give the thumbs up.  I mean, that's great that you believe deeply in your religion but c'mon, I'm a non practicing Jew, I'm not sure I understand.  Is this normal behavior?

And retail shopping.  Every time you drive past the mall, there's cars as far as the eye can see.  People spending more than they earn on products that don't last to impress people who don't care.  You're going to be paying on that credit card for the next 13 years.

And 24 hour news.  Here's something that I'm not sure people understand.  Everything the newscasters say is pre-scripted.  And the guest are there for a reason, either to provoke an argument or reiterate whatever the day's mantra is.  There is no transmission of information, they are shaping your reality, either to the right or to the left, your choice.

And Conspiracy Theorist.  Don't you think that by now, with all the circulating evidence, that some of these theory's would have been proven correct by now.  Sure they're fun to follow along, kind of like a blend of soap opera and x-files, but nobody has been able to prove anything substantial as far as I know.  Sure the leaders my be hybrid alien race handed down since the days of the pyramids, and some ultra rich fraternity club may be ruling the world, but don't count on become a member any time soon.

The way I see it, the traits and characteristics of the "normal" people seem a bit odd.  A lot of clinging to groups for self acceptance and identity.  It's a lot more difficult to stand alone and be independent.

I think J. Krishamurti says it well here:  "Who Am I"

Solved the Rubik's Cube 4 x 4

I've written many times on this blog that I solved the Rubiks Cube in the 5th or 6th grade.  My brother showed me a move, when applied, did not affect the opposite side.  And since then I like to play with the cube to help me relax.

While in Germany last December, my wife bought me another Rubiks Cube, except this one has 4 x 4 pieces instead of 3 x 3.  So I've been tinkering with it from time to time.  The other day I was able to get 3 of the 6 sides to match.  Then 4.  And last night I had all but 2 pieces in the correct position.  In trying to solve it, I messed it up completely.  And had to work at getting it back to its 4 sided state.


The patterns are similar to the 3 x 3 cube, except not exactly.  There are some new moves required and I'm still learning them.  Except some of the moves are the same, and I got it to a point that I remembered from the other cube.  So I twisted and turned and when I aligned the colors, holy smokes, all 6 sides were now in the correct position.  I solved the 4 x 4 sided Rubik's cube last night.  Yay.

There's also a 2 x 2 cube which I solved while working for the County in the late 2000.  All have similar patterns but some have different patterns.

I like to work with the cube.  Solve problems.  Get into the zone, tune out and focus.  Kind of like programming.

And there you have it~!

1/23/2015

The Life of a Business Intelligence Consultant

Working as a Business Intelligence consultant takes many skills sets, both wide and deep.

Obviously you have to know databases.  How to install, configure, backup, security, redundancy, fail over, Cloud vs. On-Premise and costs.  Sort of the DBA side.



You must know how to build Tables, Views, Stored Procedures, Triggers, Referential Integrity.  And construct queries, optimize with Indexes.  The Data Modeling side.



And you must know reporting to pull that data into some usable format for easy consumption.  Which requires knowledge of installing the software, configuring, assigning users to run the services, keys, certificates, scheduling, formatting, network configuration and backups.

And you must know other reporting tools like Dashboards, KPI, Drill down.  Which requires knowledge of OLAP cubes, the MDX language and Role Based permissions.



You must know Extract - Transform - Load or ETL.  Pulling data from multiple data sources, staging them, applying business rules, creating Keys for Surrogate and Alternate (Business).  Which requires knowledge of all the available components to perform various tasks.

And you should have some idea of Hadoop, HDFS, Map/Reduce, Hive, HiveSQL, Pig.  And how to flow data into and out of Hadoop.  And an understanding of the internal configurations and mechanism in which the ecosystems runs.  And possibly the installation on multiple nodes.



And you need to know when to use what technology.  Is the Cloud a good option or On-Premise.  Can you leverage the existing Active Directory.  What type of security should you use for Authentication.

So that's the technology part.

You also need to have knowledge of the Consulting aspect.  Meeting clients.  Selling your services.  Project Discovery.  Creating Statements of Work (SOW).  Estimating projects.  Agile Methodology, Scrum Meetings, Waterfall Method. You need to account and track your time for Billable Hours.  As well as keeping the Client informed.  Moving code to Testing and Production.  Creating a Maintenance plan going forward.  And closing out a project.

And then there's the Business aspect.  What does your client actually do.  You need to get up to speed on the terminology.  And understand the dynamics of the staff as well as the hierarchy and chain of command and decision makers.  And how does the data flow through they system using what applications.  What functionality are you required to deliver to satisfy the needs of the client, stay in budget and meet your deadlines.  And how is that measured.  And how can you position yourself for repeat business.

So Consulting requires many skills across many domains: Technology, Consulting, Business.  There are many aspects to know and its a fast paced environment where you are only as good as your current project.  And you must deliver again and again.  And be constantly learning in the background to stay current.

So there you go, my synopsis of the life of a Business Intelligence Consultant.

Service Broker Resident Expert

A few weeks ago I was tasked with a Microsoft SQL Server - Service Broker application.  Granted I knew very little at the time.

So I did what I usually do when confronted with a new technology.  I immersed myself by reading all the articles I could find on the internet.  And used that information to create a working solution.  In addition, I took the time to create re-usable scripts for easy deployment.

We showed the scripts to the Architects of the Client and they agreed to move forward with the project.  However, the architecture may have to change from Point to Point to a hybrid of P2P and Hub and Spoke.

And then just yesterday, I find out that another project has a need for sending information to other locations and they chose to use Service Broker.  Which is good because the investment of learning a new technology is going to pay off.  As I will get a chance to implement Service Broker on another project.  And become the resident expert.

I think this process is what defined my career in taking on new projects, learning new technology, and leveraging for other projects.  Our careers are a series of building blocks, each brick laid one at a time, over time, to build a giant structure.  It's kink of like people who sew.  They don't build an entire quilt in one day.  They build a piece here, then here, and over time they stitch them all together into a giant blanket.

I don't think there's a lot of orgs using Service Broker, so learning a solid technology that's not super mainstream is alright with me.  It's a great product for exchanging messages at the database level.  I would like to create a utility for monitoring the traffic flow because it's not build into the architecture of the product, and to be honest, it really should be.

1/22/2015

Social Contract Diminishing

Society is made up of people.  Many people.  In order for society to function, we depend on rules.  Some of the roles are written and some of those rules are assumed.

How do we know what the rules are?  But the underlying values that make up a society.  A social contract if you will.

Basically, I agree to treat you with respect and fairness, in return for reciprocal treatment from you.  That glue holds us together to get along and live somewhat safe and predictable lives.

However, that social contract is dissolving at a fast rate.

Here's an example.  Stores sell things.  They sell things that we want or need.  And they provide those products are fair prices and relatively adequate quality.  In exchange for that product, we give the company money.  That's the established agreement.

Not anymore.  Now the company's and / or sales people, have deviated from the original model.  Now they sell smaller products, with lesser quality for increased prices.  And they don't care if you are satisfied with the transaction, you no longer have a say and no longer matter.  You value and role in the equation has diminished.

Now a days, it seems the average sales person could care less about you or if you buy the product, as long as they do the minimum amount of work as to not get fired.  I believe the new mantra of the workforce is "no problem", after the transaction.  Doesn't matter if they provide value, satisfied your request, or you got frustrated with their apathy and complete disregard with upholding the social contract.

Personally, I've watched this evaporation of the social contract first hand.  And I could name ten incidents at the top of my head, within the last month.

But this contract also applies to people.  Interactions between people have devolved gradually.  Have you ever been tailgated on the highway, some aggressive drive inches from you bumper.  Or cutting you off in traffic.  Or cutting in line in a store.  Or bumping into you without apology.  And so on.  People today disregard the agreement of courtesy and cooperation and it's every person for themselves.

To summarize, in order for society to get along, we follow rules, written and unwritten.  People have steered more towards the disregards for common courtesy in exchange for "what's best for me".  The social contract is the glue that holds us together, and it's rapidly deteriorating.  We are inching ever more closely to our ancestors who swung from vines in the Urban Jungle.

I think Axl Rose summed it up nicely, "Welcome to the jungle".  Can anyone relate to this observation I've seen over and over lately.  Let me know.  Ok, no problem.


3 Problem Types to Build Your Troubleshooting Skills

Growing up, I would tend to lose "stuff" all the time.  In order to locate the item, I would look everywhere.  I would look, relook, look again, and then repeat.  Then I would replay the day's events in my head to try and remember where I left the item.

By doing this, you activate your mind.  You stimulate certain areas which otherwise don't get used.  And the brain is a muscle, and if you exercise it, it will strengthen.

So this did three things.  First, it gets you to look at a problem from multiple angles, using systematic and chaotic approaches.

Second, it gets your brain to visualize, you see a picture in your mind of the day's events as you retrace your steps.

Third, it build focus and concentration, as you block out everything which does not directly help in locating the missing item. (What's interesting is what you find during your search that you weren't looking for.)

More often than not, by retracing ones steps, creating mental images of the lost item and relentless pursuit of your goal, you will eventually find that what you are looking for.  And I believe looking for misplaced items was good exercise for developing solid troubleshooting skills.

I call that the "Lost Item" problem.

Another problem type I call the "Tangled Knot" problem.  This requires great patience, trial and error and meticulous attention to detail.  At first the problem seems insurmountable, so much tangledness, where does one start?  At the beginning.  I would say this too is a valuable skill for troubleshooting problems.

Lastly, there's the "Onion" problem.  This is when you have to solve a problem, sometimes easy, but in order to do so, you must solve multiple problems to get there.

For example, I'd like to move the clothes from the washer to the dryer.  But before that can be done, you must move stuff out of the way.  Before that can be done, you must do such and such.  And before that, another road block.  This is similar to the tangled knot problem, except it's iterative layers of complication that must be addressed prior to solving the real issue.

So solving problems builds character, and a solid set of skills used for troubleshooting down the road.  Whatever doesn't kill you makes you stronger.  In having the mental strength and determination, you can solve just about any problem that presents itself.  Which is good, because in the world of Programming, we solve problems all day every day.

1/21/2015

SharePoint List Source and Destination In Action

For my latest assignment I was asked to pull and write data to SharePoint List into SQL Server.

First thing, I did a web search.  Found a free site containing components for SharePoint Lists:

http://sqlsrvintegrationsrv.codeplex.com/releases/view/17652

It appears to have the blessing from Microsoft so it's legit.  However, after installing in Visual Studio 2013, after installing the SQL Server Data Tools, the components did not appear in the SSIS Toolbar.  I tried installing the DLL into the GAC to no avail.

After researching, found this article, saying the Visual Studio 2013 works with SQL Server 2014:

https://msdn.microsoft.com/en-us/hh297027.aspx

So installed Visual Studio 2012 and SSDT, installed the SharePoint components and sure enough they appeared in the Data Flow of the SSIS toolbar:



So I created a new SharePoint connection, dropped the SharePoint List source component onto a Data Flow canvas, entered the SharePoint URL, name of the List, mapped the fields:



created a destination table in SQL Server, ran and presto, it ran first time with no errors. 


How often does that happen?

And the data appeared in the SQL Table:

 

And to do Add, Modifications and Deletes, you simply reverse the order of the components from SQL Server table to SharePoint.


To do a Modification, you change the type to Modification:

 and match the ID Key:

To Add, remove the link between the ID fields:
 
Run the SSIS Package and you'll see your new entries listed in the SharePoint page:

And to Delete, keep the ID fields matched, and change the Type to Deletion:

So that's an easy introduction to SharePoint components for Pull and Pushing data from SQL Server.

Hope you enjoyed~!


1/20/2015

Data Analysts Have Been Passed Over

Big Data Scientist.  Wow, the most sought after position going today.  Finding insights.  Mashing data.  Programming.  So cool.

Except, if you think about it, many of us have been doing this for 20 years under different titles.

Reporting, Data Warehouse, Business Intelligence, Programmer, Project Manager, Team Lead. 

Just add in Hadoop, unstructured data and Machine Learning and you've got Data Scientist.

Some of us have been swimming in the data, exposing it to find insight, the "go-to" guy for upper management. 

Only downside, we were the red headed step children of the IT world.  Back in the mid-1990's nobody wanted to do reports.

Data, ah, that gives me a headache just looking at it.

The only time people spoke of reports, was just before a large project was about to go live.  "Uh, ya know, we may need some reports with this multi million dollar software package."

Get data into the database, front end / middle tier development, that's where everyone wanted to be.

Always an afterthought.  Never the main course.

Well, the data analyst hasn't gotten the glory of the newly appointed Data Scientist, because the lack of PhD, Math and Statistics skills, along with Predictive Analytics and Machine Learning.

I'm just saying that the guts of the Data Science position have been around for a long time, back when nobody could give a hoot.  You can still make a great living working with data and programming, even if you don't have an advanced degree.