1/25/2013

Planes, Trains and Big Data

If you've ever seen the movie with John Candy called Planes, Trains and Automobiles, then maybe you can relate.

You see, I've been producing my own comedy of errors this week.

I've been attempting to get Pentaho working by connecting to a Hadoop cluster on Cloudera.

I ran into every possible road block that you can imagine.

I was able to connect to hadoop via the command line:

hadoop fs -ls hdfs://localhost:8020/

As well as the web page.

However, I could not connect via the Pentaho Spoon application on Linux.

So I customized several settings after researching all the articles on Google with people claiming to know the solution.

Well, as luck would have it, after applying one change bug fix, 3 more bugs cropped up.

Think spiraling downward into the infinity of the abyss.

Well, maybe not that dramatic.

What I did do was to learn many aspects of Linux Hadoop settings as well as Pentaho ecosystems.

I'm fairly intimate with these new products, we are on a first name basis.

And I never did Linux prior to this week minus a few ls commands.

So the Linux guy responsible for Big Data initiated a conference call with one of the sales/technical people.

Very nice guy and very smart, however, it's still broken

And I put in 5 additional hours since the conference call.

I basically started over, a clean slate, by installing a new VM with Cloudera, downloading the Spoon application.

No luck, different errors now.

Son now I've tried just about every combination there is to try.

I had to modify xml config files through VI, edit properties files, move jar files, download and install jar files, try "localhost" "localhost.localdomain" "127.0.0.1" "192.168.186.129".

I've read hundreds of potential bug fixes on Google.

I've probably tried 1000 different combinations in the past few days.

And you now what, I'm a heck of a lot smarter now that before I started.

Sure the project isn't working 100% yet, but I did get about 75% there.

And maybe the problem is some undocumented fix or typo or something trivial.

I'll try again next week.

Regardless, I feel good about this weeks accomplishments.

Although I may watch the movie Planes, Trains and Automobiles this weekend just to relive my work week.

However, I do see Big Data on the horizon and all the research I'm doing now should pay dividends down the road!

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.