Cutting Edge Technology

If you've been following some of the latest developments in the technology space, there's a lot of new development going on.

Today I listened to a podcast:

Some of the topics to follow are:

Deep Learning
predict stock market
crack the gambling mystery
understand DNA
visual recognition
speech translation
self driving cars

Wearable Tech
monitor blood pressure
track running patterns

home robots
workforce robots

Hybrid Cyborgs
implants into humans
self sustaining energy sources

Molecular Nanotechnology
manipulation of the small (atoms)

3-D Printers
print on demand

Internet of Things (Everything)
connected radio frequency sensors

Energy Sources
longer lasting batteries

Electronic Currency
pay by cell phones

Virtual (Augmented) Reality
hologram applications
inner worlds connected to external reality
upload your consciousness to virtual reality

What is Transhumanism?

Artificial Intelligence
Narrow AI
Strong AI

Human Longevity
Will you live for 600 years
Will you have the same spouse for 500 years?

Sex with Robots


Solved the Rubik's Cube 4 x 4

I've written many times on this blog that I solved the Rubiks Cube in the 5th or 6th grade.  My brother showed me a move, when applied, did not affect the opposite side.  And since then I like to play with the cube to help me relax.

While in Germany last December, my wife bought me another Rubiks Cube, except this one has 4 x 4 pieces instead of 3 x 3.  So I've been tinkering with it from time to time.  The other day I was able to get 3 of the 6 sides to match.  Then 4.  And last night I had all but 2 pieces in the correct position.  In trying to solve it, I messed it up completely.  And had to work at getting it back to its 4 sided state.

The patterns are similar to the 3 x 3 cube, except not exactly.  There are some new moves required and I'm still learning them.  Except some of the moves are the same, and I got it to a point that I remembered from the other cube.  So I twisted and turned and when I aligned the colors, holy smokes, all 6 sides were now in the correct position.  I solved the 4 x 4 sided Rubik's cube last night.  Yay.

There's also a 2 x 2 cube which I solved while working for the County in the late 2000.  All have similar patterns but some have different patterns.

I like to work with the cube.  Solve problems.  Get into the zone, tune out and focus.  Kind of like programming.

And there you have it~!


The Life of a Business Intelligence Consultant

Working as a Business Intelligence consultant takes many skills sets, both wide and deep.

Obviously you have to know databases.  How to install, configure, backup, security, redundancy, fail over, Cloud vs. On-Premise and costs.  Sort of the DBA side.

You must know how to build Tables, Views, Stored Procedures, Triggers, Referential Integrity.  And construct queries, optimize with Indexes.  The Data Modeling side.

And you must know reporting to pull that data into some usable format for easy consumption.  Which requires knowledge of installing the software, configuring, assigning users to run the services, keys, certificates, scheduling, formatting, network configuration and backups.

And you must know other reporting tools like Dashboards, KPI, Drill down.  Which requires knowledge of OLAP cubes, the MDX language and Role Based permissions.

You must know Extract - Transform - Load or ETL.  Pulling data from multiple data sources, staging them, applying business rules, creating Keys for Surrogate and Alternate (Business).  Which requires knowledge of all the available components to perform various tasks.

And you should have some idea of Hadoop, HDFS, Map/Reduce, Hive, HiveSQL, Pig.  And how to flow data into and out of Hadoop.  And an understanding of the internal configurations and mechanism in which the ecosystems runs.  And possibly the installation on multiple nodes.

And you need to know when to use what technology.  Is the Cloud a good option or On-Premise.  Can you leverage the existing Active Directory.  What type of security should you use for Authentication.

So that's the technology part.

You also need to have knowledge of the Consulting aspect.  Meeting clients.  Selling your services.  Project Discovery.  Creating Statements of Work (SOW).  Estimating projects.  Agile Methodology, Scrum Meetings, Waterfall Method. You need to account and track your time for Billable Hours.  As well as keeping the Client informed.  Moving code to Testing and Production.  Creating a Maintenance plan going forward.  And closing out a project.

And then there's the Business aspect.  What does your client actually do.  You need to get up to speed on the terminology.  And understand the dynamics of the staff as well as the hierarchy and chain of command and decision makers.  And how does the data flow through they system using what applications.  What functionality are you required to deliver to satisfy the needs of the client, stay in budget and meet your deadlines.  And how is that measured.  And how can you position yourself for repeat business.

So Consulting requires many skills across many domains: Technology, Consulting, Business.  There are many aspects to know and its a fast paced environment where you are only as good as your current project.  And you must deliver again and again.  And be constantly learning in the background to stay current.

So there you go, my synopsis of the life of a Business Intelligence Consultant.

Service Broker Resident Expert

A few weeks ago I was tasked with a Microsoft SQL Server - Service Broker application.  Granted I knew very little at the time.

So I did what I usually do when confronted with a new technology.  I immersed myself by reading all the articles I could find on the internet.  And used that information to create a working solution.  In addition, I took the time to create re-usable scripts for easy deployment.

We showed the scripts to the Architects of the Client and they agreed to move forward with the project.  However, the architecture may have to change from Point to Point to a hybrid of P2P and Hub and Spoke.

And then just yesterday, I find out that another project has a need for sending information to other locations and they chose to use Service Broker.  Which is good because the investment of learning a new technology is going to pay off.  As I will get a chance to implement Service Broker on another project.  And become the resident expert.

I think this process is what defined my career in taking on new projects, learning new technology, and leveraging for other projects.  Our careers are a series of building blocks, each brick laid one at a time, over time, to build a giant structure.  It's kink of like people who sew.  They don't build an entire quilt in one day.  They build a piece here, then here, and over time they stitch them all together into a giant blanket.

I don't think there's a lot of orgs using Service Broker, so learning a solid technology that's not super mainstream is alright with me.  It's a great product for exchanging messages at the database level.  I would like to create a utility for monitoring the traffic flow because it's not build into the architecture of the product, and to be honest, it really should be.


3 Problem Types to Build Your Troubleshooting Skills

Growing up, I would tend to lose "stuff" all the time.  In order to locate the item, I would look everywhere.  I would look, relook, look again, and then repeat.  Then I would replay the day's events in my head to try and remember where I left the item.

By doing this, you activate your mind.  You stimulate certain areas which otherwise don't get used.  And the brain is a muscle, and if you exercise it, it will strengthen.

So this did three things.  First, it gets you to look at a problem from multiple angles, using systematic and chaotic approaches.

Second, it gets your brain to visualize, you see a picture in your mind of the day's events as you retrace your steps.

Third, it build focus and concentration, as you block out everything which does not directly help in locating the missing item. (What's interesting is what you find during your search that you weren't looking for.)

More often than not, by retracing ones steps, creating mental images of the lost item and relentless pursuit of your goal, you will eventually find that what you are looking for.  And I believe looking for misplaced items was good exercise for developing solid troubleshooting skills.

I call that the "Lost Item" problem.

Another problem type I call the "Tangled Knot" problem.  This requires great patience, trial and error and meticulous attention to detail.  At first the problem seems insurmountable, so much tangledness, where does one start?  At the beginning.  I would say this too is a valuable skill for troubleshooting problems.

Lastly, there's the "Onion" problem.  This is when you have to solve a problem, sometimes easy, but in order to do so, you must solve multiple problems to get there.

For example, I'd like to move the clothes from the washer to the dryer.  But before that can be done, you must move stuff out of the way.  Before that can be done, you must do such and such.  And before that, another road block.  This is similar to the tangled knot problem, except it's iterative layers of complication that must be addressed prior to solving the real issue.

So solving problems builds character, and a solid set of skills used for troubleshooting down the road.  Whatever doesn't kill you makes you stronger.  In having the mental strength and determination, you can solve just about any problem that presents itself.  Which is good, because in the world of Programming, we solve problems all day every day.


SharePoint List Source and Destination In Action

For my latest assignment I was asked to pull and write data to SharePoint List into SQL Server.

First thing, I did a web search.  Found a free site containing components for SharePoint Lists:


It appears to have the blessing from Microsoft so it's legit.  However, after installing in Visual Studio 2013, after installing the SQL Server Data Tools, the components did not appear in the SSIS Toolbar.  I tried installing the DLL into the GAC to no avail.

After researching, found this article, saying the Visual Studio 2013 works with SQL Server 2014:


So installed Visual Studio 2012 and SSDT, installed the SharePoint components and sure enough they appeared in the Data Flow of the SSIS toolbar:

So I created a new SharePoint connection, dropped the SharePoint List source component onto a Data Flow canvas, entered the SharePoint URL, name of the List, mapped the fields:

created a destination table in SQL Server, ran and presto, it ran first time with no errors. 

How often does that happen?

And the data appeared in the SQL Table:


And to do Add, Modifications and Deletes, you simply reverse the order of the components from SQL Server table to SharePoint.

To do a Modification, you change the type to Modification:

 and match the ID Key:

To Add, remove the link between the ID fields:
Run the SSIS Package and you'll see your new entries listed in the SharePoint page:

And to Delete, keep the ID fields matched, and change the Type to Deletion:

So that's an easy introduction to SharePoint components for Pull and Pushing data from SQL Server.

Hope you enjoyed~!


Frozen Remote Desktop Solution

I'm working for a client, they set me up with 3 VMs to work on.  The VMs are located on another domain.  I connect via Microsoft Remote Desktop.  So in order to get files there, I have to copy them.

Some of the files are over a gig.  And today, while copying a large file, the server froze up.  Wasn't sure how to get access back, tried everything.

So instead, I logged onto another VM which was operational, went to Server Manager, added the frozen VM to the list of Servers, and pressed the "Restart" option.

In about 5 minutes the frozen server was back online and I resumed the copy.

It's a neat trick if you ever run into similar situation.

And there you go~!


My Advice on Data Science Job

Recently a grad student contacted me on advice for a Data Science job.  Here was my response:

Thanks for connecting and reaching out.

Data Scientist is a good position in today's workforce and I admire your goals.

I agree that getting an intern position is beneficial to obtain real world experience prior to graduation. As I've seen graduate students who were well qualified not able to find positions.

I recommend getting involved in Social Media, follow the best and brightest, there are many out there. That way you get knowledge of the market trends, who the key players are and contribute to the conversations.

Likewise, starting a blog is good to build brand recognition as are speaking at events.

I would not get too focused on specific technologies at this point as they are ever changing. However, having knowledge in data, both relational and unstructured, along with solid programming skills are key components to the Data Scientist position.

Also, see if your professors can assist in lining up some "free" work just to get experience. Because once you get your first job out of school, it's your experience they look at, not the degree.

So good luck, keep in touch,


Intro to PowerBI Designer for Desktop

Saw a great posting today on LinkedIn http://www.jamesserra.com/archive/2015/01/power-bi-designer/

The Blog author, James Serra, does a great job explaining the new Microsoft feature called PowerBI Designer, for the Desktop.

So whenever there's something new, I like to download, test drive and blog about it.

I began by downloading the application here: http://www.microsoft.com/en-us/download/details.aspx?id=45331


Next, Next, Next:




Data Sources, whoah nilly, quite a few:

Selected an Excel doc with a PowerPoint Model embedded:

Selected a table to import:
Data Metrics now available on right hand side of designer pane:

With a few clicks, we have 2 Graphs/Charts:
Lots of Chart types to choose from the Menu bar:

With one click, changed to a Tree Map:
 When you Save as, new file type:

About screen, this is Preview edition:

I didn't see a way to save directly to PowerBI in the Cloud.  Likewise, no ability to select a Data Source from PowerBI.  Maybe its there and I didn't see it. 

However, I did see HDFS, Windows Azure and HDInsight, Analysis Services, SharePoint lists as well as our trusted ODBC.

So there you go, PowerBI Designer for Desktop quick intro.  Seems like a winner~!