12/13/2015

How do we get to true Artificial General Intelligence

I think reality as we know it is in for one major disruption.  I think the new reality will combine many new technologies.



Including Big Data, Artificial Intelligence, Quantum Computers, Massive Parallel Query Execution in real time and especially Virtual Reality.

The thing is, in order to create Artificial General Intelligence, the computer must simulate a human being.  And the best way to accomplish this is to bring the human to the virtual reality, home of the code/algorithms.

Google has already simulated a neural network artificial intelligence to self learn the game "Breakout".  In just a few short days, it surpassed the human's best score and plays to almost perfection.  Quite a break through.




So if a computer can become master of it's domain, why not allow it live in it's natural domain, in the land of code.

Then mix artificial intelligence with massive data sets, that cross domains.  Right now, neural networks are limited to one specific domain to become experts.  With the ability to query massive data sets, we open the door to new possibilities.

And then throw in Quantum Computers, which use an alternative method for computation.  Instead of using binary combinations of zero's and one's, it leverages a newer science called Quantum Physics, which behaves in bizarre scenarios.  In this world, you can have a state in which the binary can be both zero and one at the same time.  This allows for more scenarios to be calculated in the same time interval, thus, expanded the computation power, which is exactly what's needed for the next leap in AI.

Another thing, if we had this virtual reality, maybe we could upload our memories, thus, allowing humans to live for ever, long after the body deteriorates.  And perhaps remove some of the bad memories.

I think we are approaching the Perfect Storm of computation power, in a good way.  In order to get to Artificial General Intelligence or "Strong AI", we'll need to leverage and combine Neural Networks, massive data sets, parallel query capability, quantum computers and virtual reality.  Perhaps then, finally, we'll make some real progress.

Stay tuned~!

12/12/2015

Blending Humanity with Technology

Artificial Intelligence is all well and good, but how does that benefit me specifically.  What I'm talking about is having the ability to increase my mental capacity.  As in increasing my memory, ability to learn new subjects, perhaps super strength or better eyesight.  Kind of like the 6 million dollar man.

What I'm talking about is sort of a hybrid solution, by increasing the performance of humans by integrating with technology.  

Sports players have been taking performance enhancers for years.  Because when you're paid to perform, it makes sense to do what it takes to become a better performer.

So if humans have the ability to remember better, or learn faster, why not.  If I could download a program into the memory, to allow me to drive a tank instantly, that sure could become useful depending on the situation, Matrix style.

Altering the human body to become better.  It's easy to envision, in the not so distant future, a hybrid solution of humanity and technology.


I suppose you could go a step further, and discuss genetic engineering.  As in paying for body parts or genetically alterations.  Perhaps this topic boards on ethical concerns, as some people may or may not be able to afford such luxuries.  And what about good old human imperfections, been around for 100,000 years, we've gotten used to them.  I don't really have an opinion on this one.

So we talked about merging humans with technology and genetic engineering, both coming to a theater near you.  At some point.

12/10/2015

New Azure Cortana Analytics Course on the Way

I got into IT through the back door, without a formal degree in computer science, although I learned on an original IBM PC Dos in 1983.

My early career consisted of a variety of jobs working with code, reports and data.  Strangely, most of the work consisted of maintenance coding, in other words, supporting other people's code by applying tweaks and enhancements and troubleshooting production issues.

A few years ago, my boss got me to speak in front of a user group, and then SQL Saturday's and of course I've been blogging for a decade.

Microsoft Azure is a fairly new technology, few years old.  I've had an account for a while and tinker from time to time.  And more recently, Cortana Analytics showed up on the scene, as a data platform in the cloud, with a ton of new products and features, all tightly integrated.

I was asked to create an online course, I chose the Microsoft Azure platform, then included Cortana Analytics.  I created the outline, expanded into each topic, then created the videos and the demos, about 4 hours in total.

Here's a view of the course, to be released mid January 2016:



I really enjoyed the process of creating an online course.  Growing up you wouldn't catch me in front of the class giving a presentation, but over the years, things changed.  Teaching is kind of cool.  Especially discussing advanced topics.


I guess you can teach an old dog new tricks.

Here's a post to some of the information discussed in the course...

http://www.bloomconsultingbi.com/2015/11/what-is-cortana-analytics-suite-from.html

12/08/2015

Controversial Quantum Machine Bought by NASA and Google Shows Promise | MIT Technology Review

Controversial Quantum Machine Bought by NASA and Google Shows Promise | MIT Technology Review

Where is the World of Data Heading?

First we had reports.  Then we had Business Intelligence and Data Warehousing.  Then we had Hadoop and Big Data.

What's next?  An entire data platform.  Including relational databases, flat files, non structured data, large data, fast data, small data.  Any data.

Where?  In the cloud.  Or on-premise.  Or hybrid.

We're talking streaming data.  Events from senors and devices and applications.  Apply a layer of machine learning.  In real time.  Analyzed by domain experts.   To make decisions and alter processes.  To save money and reduce costs.

At some point, humans will be removed from the equation.  For a true artificial intelligence.

For now, it's about collecting, aggregating and deciphering the data.

Will we still have reports?  Business Intelligence?  Yes.  However, it seems the world of data is going to the next level.  As in data platforms in the cloud, accessible anywhere, anytime.  We merely architect the system, set it up and let it run.

That's what I see.

11/27/2015

We Need to Automate Everything, Artificial Intelligence to the Rescue

Artificial Intelligence is a tough nut to crack.  The goal is simulate human behavior.

So let's say we train a neural network to learn behaviors, traits and characteristics of human behavior.

For example, we teach the machine the process to dispense coffee at a coffee store.  We train the model, test and score and evaluate the results.

And we find some startling results.  Although the process of taking a customers order shouldn't vary, we find the results are all over the map.

Some customers are not satisfied with their drinks, some are oblivious and some are very satisfied.  But why is that?  If the process never varies, in theory, we should see similar consistent results every time.

So if we dive a little deeper, we find some interesting things.  First, there are many variables.  Some are known, others not known.

For example, client A got decaf coffee when he ordered caffeinated coffee.  Let's review the tape.  Well, bartista 1 took the order, processed the order and delivered the order.  According to the tapes, they performed their job as expected.  Let's ask bartista A a few questions.

Did you perform your job as expected?  Yes, the order was taken in timely manor, processed the order as expected, and delivered within timely manor.  Yes, but you gave decaf when they ordered caffeinated.  Oh really?  I'm sorry, my bad.

This could be chalked up to many factors, some obvious.  Over worked.  Not paying attention.  Read the order incorrectly.  Or it could be something else.

Let's look for the non obvious.  Perhaps the client looked like his/her ex-wife/husband, get back at her/him.  Perhaps the bartista is late on their rent, drank too much last night, not focused on their job.  

Perhaps they are unknowingly upset with themselves for their life situation, they should have studied more in school, now in a dead end job.  We'll make sure everybody is as miserable as me.  How about a nice hole in the bottom of the cup, you'll enjoy the coffee dripping when you get to your car.  How about we forget to include your donuts with your order.  

Or the opposite, how about we give this person exceptional service?  Because they look like me or they have similar tastes or interests.  

Let's apply this rule to everything, everywhere:  let's discriminate our behavior and services to our clients, based on internal bias, prejudices, beliefs and opinions.

Let's vary the outcome based on slight deviations from the actual process.

First off, many of these incidents don't get reported.  Second, nobody investigates why.  Third, the person may not even be aware of their errors.  So nothing gets corrected.

This scenario could be described as a black box.  And we typically don't know what happens in a black box.  Magic.

Yet if the process is identical and the results vary greatly, we must account for the variations of end results.  We need to explore the black box a little deeper.

And when we look closer, we see people performing the job descriptions as expected, yet with a bit of undocumented wiggle room.  Which result in varying results and outcomes.

Loose lips sink ships.  And varying from the process skews results.  And the results are based on undocumented internal bias.

Once we identify the patterns, the trick is to correct them.  How?  

Implementing guidelines, such that, there is no wiggle room for error.  Remove the barriers for deviation.  By following the prescribed pattern of actions, you can consistently provide results within expected statistical ranges.


Bartista: Listen, we know you are paying the same amount for your coffee, but I don't care for you, so I'm going to vary your order just slightly so that you get sub standard service or product.

Personally, I find this scenario to be widespread and prevalent in our society.  You can apply this formula to just about everything everywhere.  Yet nobody knows it exists or aware of it.  A silent, invisible signal affecting the outcome of everything.

I find it to be the source of chaos, confusion, corruption, and a vehicle to spread negativity into the world.

 When duplicating human behavior, we need to dive into the behaviors and processes to investigate.  We have to account for the variations in results.  And look to the reasons behind them.  I believe those deviations stem from bias and discriminating factors within the system, undocumented and unaccounted for, yet present.

We need to remove these variations from the equation.  Through process.  Documenting, Workflows, Business Rules, Implement through technology and algorithms.  To prevent unknowns from the equation.  For consistent patterns and results.

Then monitor results and modify to self correct on the fly.     

By streamlining processes, we can ensure quality service across the board.  This should reduce costs from having to correct problems downstream and customer retention.

If we matched human behavior 100%, we'd need a function to simulate stupidity, biases, prejudices, hatred, vengeance, revenge, evil, etc.  

We need to automate everything.  Remove the human factor which is tainted, biased, prejudiced and inconsistent.  I think this is the key to fixing a lot of systemic problems in every aspect of everything on the planet.  

Automate.  Remove bias.  Streamline.  Measure.  Repeat.

And the vehicle to automate this is called Artificial Intelligence.

11/23/2015

The Artificial Intelligence Gap

In the world of Artificial Intelligence, we have this image of computers rising up, dethroning humanity as the alpha species on the planet.

They call it the Singularity.  The moment computers wake up.  Become conscious.  Also known as Strong AI.

It could happen.  But probably not for a while.

So where are we at today?  We have gotten pretty good at identifying patterns in the data through a technique called Supervised Learning.  In that we can read mounds of data, where the data points are sufficiently labeled, and we can associate mappings between source and outcome in a paired value scenario.  And this is used to base predictions about future behavior.

We also have narrow or weak AI, which is basically Siri or Cortana.  Personal assistants that can return information based on voice recognition commands, kind of like Star Trek.

The part that is most difficult is the Unsupervised Learning.  Throwing data at an algorithm and having it deduce value and insights without human intervention or labels attached to the data.

There has been considerable progress in AI machines learning, by teaching themselves, in a simulated environment.  However, the AI systems are basically experts at a single domain, not multiple domains.  Likewise, the environmental variables within their world are limited to some degree.

So we have advanced the field of Artificial Intelligence a lot since the 1950s.  And the AI Winter and lack of funding of the 1980s.  And in the 2000s we've witnessed cheaper hardware, better processing power, advanced algorithms and better storage capacity.  As well as having advanced AI software in the hands of mere mortals, as in not PhD academia.

Will computers rise to the point that they displace humans.  Not yet.  But I wouldn't count them out just yet.

New Azure Virtual Machine Preconfigured for Data Science

Microsoft Azure is a platform to do just about anything.  They have so many software tools to choose from, it's an entire ecosystem on the web.

Since I've been programming in Microsoft since Visual Basic 4.0, I'm a definite fan.

Today there was a blog post: http://blogs.technet.com/b/machinelearning/archive/2015/11/23/announcing-the-availability-of-the-microsoft-data-science-virtual-machine.aspx introducing a new product, a Machine Learning pre-configured Virtual Machine.

So naturally I logged on, and proceeded to create the new VM:










Deploying:



Now we start the Virtual Machine:



So we're up and running.  And it appears, this new Virtual Machine has lots of Data Science goodies to get started with.

Including Azure Storage application to move files up to Azure Storage, Power BI Desktop for reporting and visualizations as well as connecting to different data sources and massaging the data.

It's got Revolution R application.  SQL Server 2014 limited edition.  Visual Studio 2015.  Python.  And links to Azure.  And Power Shell.

So if you need a server in a hurry, this is a great option.  It's funny though, I already have every application installed on my laptop.

So there you have it~!

11/20/2015

How to Solve the Rubik's Cube

If a tree falls in the woods, and nobody's around to hear it, did the tree actually fall?

Good question.  Maybe it did.  Maybe it didn't.

When I was in the 5th or 6th grade, I solved the Rubik's cube.  Some people knew, some didn't.

When I moved to Florida, nobody knew I could solve it.  It basically never happened.  In fact, they placed me in remedial math, with students who couldn't add or subtract.

That was a real treat.  In fact, I went the next 5 years without speaking basically or raising my hand to contribute in class, although I was on the honor roll every quarter.

Not one teacher made an effort to know a thing about me.  And I didn't offer any information to the contrary.

I would say, if a kid can solve the Rubik's cube and nobody's around to see it, did it ever happen.

Well, yes actually.  Here's a video I created on how to solve the cube:



So if any of my teachers from middle or high school are still around, have a watch.  This would probably be the first time you realize what my voice sounds like.

And so it goes~!

PS. as a follow up, I purchased a copy of Camtasia Studio on Cyber Monday 30% off.  Good stuff~

11/16/2015

Internet of Things through Azure Streaming Data and Azure Event Hubs

After getting carried away by the Big Data hype, I was more than skeptical about the Internet of Things.

Sensors in every piece of equipment.  Sure, but how does that affect me?  I only work with the data.   I don't build washing machines or have access to sensors.

True, but from a data professional perspective, the data is where it at.  And where is the data, it's going to be everywhere.  From everything.

Little pulses of information.  Sent continuously.  From just about anything.  How will the data get transmitted?  Well, it sounds like radio frequencies connected to a hub that routes the messages to the central hub.

Could be millions of incoming messages per second.  And there are tools to assist with the streaming of these messages.  Actually called Streaming Analytics, in Azure, from Microsoft.  An analytic platform to watch the data in flight.  Look for anomalies, send alerts, etc.

And how will these messages find there way?  Service Hub.  Another Azure product.  Similar to Service Broker, which is a messaging queue like application baked into SQL Server.  Doesn't require source or destination to be in same network, guaranteed delivery, in the order in which it was sent.  Event Hub is built on a concept called "partitions".  This was designed to handle issues of blocking from Service Broker.

The data professional will set up the infrastructure on the Azure side and then, once the data starts flowing in, there's lots to identify, in the Streaming data.  The incoming messages may not be verbose, it's like being eaten by little mosquito's,  tiny, yet when combined with a swarm of others, potent.

Now I see the potential impact of the "Internet of Things".  Monitoring millions of devices in real time 24/7.  Processing each message and detecting anomalies, then basing some kind of action downstream.

So perhaps Big Data wasn't all hype after all.  Because these little messages of streaming data from IoT will surely amplify our data, exponentially.

And there you have it~!

11/15/2015

What is Cortana Analytics Suite from Microsoft Azure?

Microsoft Azure has many tools for developers.  One of the new suites to appear is the Cortana Analytics suite.  It's comprised of multiple technologies, some you may already know.

Cortana Analytics is a combination of:
  • Azure SQL Database
  • Azure HDInsight
  • Azure Machine Learning
  • Azure Streaming Analytics
  • Azure Event Hub
  • Azure Data Lake Analytics
  • Azure Data Factor
  • Azure Data Catalog
  • PowerBI
Many of you are probably familiar with Azure SQL Database.  An Online version of SQL Server, either run by Azure entirely or where you maintain and support a version running in a VM.

Azure HDInsight is the Microsoft Cloud version of Hadoop.  It complies with all the Apache Foundation Hadoop features and scales nicely.

Azure Machine Learning is a sandbox to build experiments using Machine Learning algorithms.

Azure Streaming Analytics allows developers to watch data in flight, typically working with Azure Event Hub.

Azure Data Lake Analytics is a new product currently in Preview.  It allows big data developers a new platform to store and query big data across any language.

Azure Data Factory is a process of automating data movement in the cloud

Azure Data Catalog is a solution in the Cloud to keep track of your data sources, their metadata, configure who has access to which data sources and allow search, tag and big picture of data ecosystem.

PowerBI is a tool for bringing in data, mashing with other data sources, adding business rules, staging the data, creating reports and dashboards and collaborating on the data to derive insights.  There are three flavors, traditional Excel Add-Ins, Power BI Desktop and PowerBI.Com.

So there you have it, Cortana Analytics Suite.  An ecosystem based in Azure cloud to work with data from just about any angle.  To derive insights and run your business.


10/30/2015

The Information Race - Data is the Virtual Asset with Unlimited Supply

At one time, you could get really rich by purchasing assets that were scarce.  Land for instance.  However, just about every piece of land on the planet has been surveyed, assessed for possible extraction of resources, plotted and purchased.  There's very little untouched land on the planet.  The resource has been used up.

So without any land to purchase, what other resources are available to exploit, plunder and profit?

Information.

What is information?  It's an accumulation of data.  And where's the data?  Everywhere.  Every company is collection information minute by minute.  And those with the financial backing, leverage and people to transform data into information, will rise to the top.  Because information is the biggest asset going right now.  The quantities are unlimited potentially, unlike land, that plateau's at a certain point.

And the information is an asset.  And processing that information is power.  So those who have the financial backing to process data into information will be the next overlords of the planet.  Not land owners.

You can already start to see some of the big players purchasing huge volumes of data.  The "Information Race" has begun.  The "Arms Race" was yesterday's news.

Data is the Virtual Asset with Unlimited Supply.

10/23/2015

Technology Inflation

Many years ago, things were quite simple.  Simple in the fact that there wasn't much complexity.

For example, the average weekend warrior could put his car up on a block in his driveway, do his or her own oil change, swap out the brake pads, even put in a new transmissions.

Now, even the most skilled car fanatics can't work on their own cars.  Because the complexity that's been added.  Everything is based on sensors and electronic gizmos.  If you want to diagnose an issue, you have to take it to the dealer, hook up a device, and it spits out the specific error.  No way to duplicate this at home.

Apply the same process to computers.  Back in the 1960's punch cards to us seem primitive, but the coders back then knew exactly what they were doing.  Because back then, they were in the details, as in writing to memory and low level functions, complete control of everything.

Now we have 4th generation languages that do us the favor of generating source code and sort of dumbing down the process.  We as programmers are a few layers away from the memory and raw source code because it runs in a black box.

And that black box is great for knocking out tons of code relatively fast, but on the other hand, we don't know what exactly it's doing under the hood.  In addition, the number of dependent items have dramatically increase, exponentially.

We have to deal with networks, security, connections to web services and databases and external calls across the web and other apps that depend on our process'.

Technology has gone through the process of complexity inflation.  It was supposed to simplify things.  Make our lives easier.  In hindsight, it's done the opposite.

That's just my two cents.  With inflation, that equals about a buck seventy five.

Again, thanks for reading~!

10/22/2015

Latest Thoughts on Artificial Intelligence

Artificial Intelligence is making headway in the world today.  The group called Deep Mind at Google has built a system that can become master game players through the use of learning algorithms.

They simply start the AI system, supply it a stream of incoming data, in the form of pixels, like an old Atari game.  The AI system, is first confronted with a series of events.  It soon learns the rules of the game.  And then becomes a master at the game.  Learning strategy, optimal methods to achieve goals faster and can outperform most humans.

It uses a simple set of rules.  Observe a set of environmental behaviors, learn, determine optimal outcomes based on specific possible outcomes, and continue the cycle.

Observe + Learn + Decide + Action then repeat.

Some of the advancements recently have been to add read/write memory to Recurrent Neural Networks.  What that means is the traditional Neural Networks could learn over time through algorithms, they now have the ability to remember prior knowledge over time, retained in memory, for future use.

This is known as the "Neural Turing Machine" in which an Artificial Neural Network can learn new algorithms by example.  For instance, you show the system a routine to sort numbers, and than ask it to perform the function and it knows how simply by watching it previously, actually bypassing the need and time to train the model.

Here's a tweet I posted earlier today: "#AI is a self learning technique by observing, determining possible outcomes, selecting best alternative, then observing... #AGI holy grail."

And another: ""process that converts unstructured data or information into useful actionable knowledge"=AGI, tech is neutral, requires ethical constraints"

Most of this information was lifted from a YouTube video from the Deep Mind founder Demis Hassabis, you can watch the video here



I think the best part of the video is where he differentiates between true Artificial General Intelligence and Narrow Artificial Intelligence.  Siri, Cortana and Big Blue are considered Narrow.  What Deep Mind is focused on is AGI.  A subject that's been around a very long time.  With lots of smart people working towards the goal.  Although it seems to be a hard nut to crack.

I recommend watching the full video to get insight into the happenings at Google Artificial Intelligence team.  I followed Deep Mind before they were acquired by Google and they have a tremendous pool of talent in the space of AI.

The interesting thing about this field, is although it's deeply rooted in technology, they depend on outside science for assistance, like psychologist, neural research for brain activity to behavioral of humans and a bunch more.

This field is pretty exciting.  Although most of my knowledge is self taught, I'm not an expert on the underlying technology that makes it work.  I do understand the base concepts of what a Neural Network is, a Recurrent Neural Network (has memory) to Models and some of the different methods for training them.

It seems some of the latest challenges are applying domain knowledge across multiple domains without the need to train the models for long periods of time.  And perhaps adding memory to the model is one way to accomplish this.

Another factor is the majority of the work thus far has been in a virtual environment.  They have not adapted the AI technology to robots or external environments, where the number of potential conflicting input streams is potentially unknown.  

How would a robot handle a barking dog, or a plane flying overhead or the mailman driving by, all unknown events that don't happen consistently over time.  So the AGI system wouldn't have prior knowledge so it would take to to learn and adapt.  Unless you could simply plug in an algorithm to the system so it understands and adapts immediately.

When we think of ethical concerns, Mr. Hassabis indicates that the technology is neutral or independent.  It depends on how the software is used.

An analogy I think about it the speedometer in a car.  Sure they could set a control to cap the maximum speed of a car.  Because technically, the speed limit is typically 70 MPH in the United States, so why would a car have a need to exceed that limit?  If the driver exceeds the limit, he or she is breaking the  law technically.  Why don't the car manufacturer's place a restriction on the vehicle to prevent us from speeding?  Perhaps that's an ethical question. 

Could the car send a signal to the patrol station, which electronically generates a citation sent in an email, where an ACH withdraws occurs the following business day, removing a $100 fine from your checking account?  I'm sure that could be done.  But it's not.  Why is that.  

How would you create the algorithm to select which drivers to give a citation, or would you apply a blanket rule that everyone exceeding 80 MPH would get a citation, if that were the case, why not place a cap on the accelerator of the car not to exceed 80 MPH.  They could do that today without an AGI system.

I suppose there's some flexibility baked into the system.  Perhaps you need to accelerate to 75 to overtake a car in the slow lane, multiple possible reasons.  But the level of enforcement is by human interpretation and perhaps not applied evenly.  In fact, I'm not sure how they know to pull over certain speeders and not others.  A mystery.  But it could be automated.  But would we want it to be?

What level of automation do we want for our future AGI systems?  Who gets to decide what is deemed ethical and what isn't.  How do you handle the exceptions and the gray areas?

Stay tuned to find out.

10/21/2015

Intro to Quantum Computers

Computers run on 0s and 1s.  Bits and Bytes.  On or Off.  Binary logic.

If you can understand an IF STATEMENT:

IF this logic is true THEN do this
Else IF this logic is true THEN do that
ELSE do something else

That right there is the gist of programming.  Sure there's a few other details to learn but that's the bread and butter.

What if you modified the logic a bit.

IF this logic is TRUE and the same logic is FALSE then do something

Huh?  You can't do that.  Something can only be this or that.  Can't be both.

Well, using this new concept where something can be 0, 1 or both, is called Qubits.

Well.  That's the new logic underlying the new type of computer called the Quantum Computer.   Which borrowed the concept from Physics Quantum Mechanics.
Quote from Wikipedia: "Large-scale quantum computers will be able to solve certain problems much more quickly than any classical computers that use even the best currently known algorithms"
It's the next step in the revolutionary evolutionary framework of computer hardware.  Using Atoms.

And who's the leader in this technology: D-Wave.

And here's the timeline of the events leading up: Timeline of Quantum Computers.

And an article on How do Quantum Computers Work?

I see this technology assisting to advance the progress of Artificial Intelligence and Neural Networks.

Here's a YouTube video:







 We are living in interesting times....

9/30/2015

Microsoft OLAP Cubes 101

OLAP Cubes are still around.  I work with them every day.  For Microsoft, there are two flavors.  There's Multi-Dimensional Cubes and Tabular Model cubes.

Tabular is the newer flavor, it has some advantages.  Like easier to get a cube up and running in shorter time, it uses a new language called DAX, you can import models from Excel Power Pivot and I imagine the newer Power BI.  You can assign permissions to specific users and groups contained in Active Directory.  Basically you create Roles with specific permissions, then assign users to the Roles.  Create partitions.  And schedule refreshes from SQL Server SQL Agent to process the cube, for example, if you receive new data throughout the day, the cube will only display new data when refreshed.


Multi Dimension Cubes have been around before Tabular.  They are a bit complex, if you're coming from the traditional T-SQL world.  MDX is a beast of a language, I've worked with it for a few years and every time is a challenge.  Although some apps like SSRS have Wizards to help out.
 

 Once you have an OLAP Cube, you can build reports off it.

What type of reports can you build based off an OLAP cube?  Microsoft offers Power BI, Excel, SQL Server Reporting Services (SSRS), Performance Point and Power View.


The OLAP Cubes run in a different database than OLTP, called Analysis Services.  It contains the Cube, Dimension tables and Roles.  The IDE used to build cubes is typically the SQL Server Data Tools, also known as BIDS depending on the version, and you can use Visual Studio 2013 provide you install the Data Tools add-in.

Many times, you build a Cube based off a Data Warehouse.  A Data Warehouse has Dimension tables and Fact tables.  The dimension are typically things like, Person, Region, Warehouse, Date, (things) to slice the data with.  The Fact tables contain measures consisting of Sums, Average, Mins, Max, Avergages, they are almost always numbers.  And you can build hierarchies within the Dim tables, for example:


Country --> State --> County --> City
 

 That's how OLAP Cubes work.  Abbreviated edition.

9/29/2015

Effects of Artificial Intelligence

Here's a good article on Artificial Intelligence from distinguished Microsoft researchers.

http://cacm.acm.org/magazines/2015/10/192386-rise-of-concerns-about-ai/fulltext

When we were children, we had a fort in the back yard.  The kids would get together and hang out.  It was our little world tucked away from the bigger world.  In our world we could do what we liked.  A refuge for children to explore and grow.

As you get into the real world, you abide by the rules in school do what you're told.  The teachers and principle were the law, if they sent reports back to the home base, parents took action.  So although you were away from home for a good chunk of the day, somebody was still watching and monitoring your behavior.

Then we enter the workforce, the company and boss are there to make sure you performing up to expectations.  They don't call your parent, they let you go work somewhere else.

So you can see the progression of they system,  There's always someone watching you to ensure you maintain your behavior to align with acceptable norms.  If you don't comply, the school calls your parents, your employer fires you.

So the next possible iteration of the system is artificial intelligence.  An automated series of sensors that monitor every person on the planet 24 x 7.  It will scan your location, your actions, look for patterns and detect deviations.  

In addition to monitoring and surveillance, it can automate the enforcement of deviant behavior.  Once money becomes digital, they can simply freeze your account, your credit cards.  No need for electronic bracelets around your ankle, you have a cell phone.  

In addition, they can automate the police force, by militarizing robots to enforce rules, fight wars and monitor the citizens.  And lastly, these friendly robots could be doing your old job, while the majority sits home watching Jerry Springer and the Price is Right.  With no job, our disposable income won't be there, so fewer purchases, not sure how the capitalistic system of continuous growth can survive.  So what happens to the masses of unemployed people?

People suggest the Universal Wage, a social welfare to pay for basic needs.  Perhaps, although a system based on legacy workers in  a welfare system monitored continuously, reminds me of a modern day prison.

So there you go, a short story about the impact of artificial intelligence and potential effects.  If you ask me, I would like to  build a fort in the backyard, invite all the neighbors over, and talk about nothing, just like we did as children.

9/24/2015

What does a Data Scientist Actually Do?

When you hear of someone being a data scientist, you imagine the life similar to a major league baseball player.  Status, prestige, salary, pick of the jobs.

I guess what I'm wondering is once they get the job, with high salary and perks, what do they do all day?

Perhaps they report to some senior executive, work in a lab, have unlimited compute power at their disposal, with unlimited access and security to every piece of data in the org.

Or perhaps they report to an IT manager, have to sit in a cube, have to submit ticket request for access to data, perhaps wait days or weeks for permission, perhaps they aren't given enough computer power to do  their jobs, and maybe they're locked out of key pieces of data.

I guess it could vary depending on the org.

You can take the funniest comedian on the planet, walk up to them and say "be funny" and they may have difficulty being funny on demand.  Perhaps the same is true with data scientist, "where's my insights", you've been working on this project for a while, we'd expect our sales to have increased or reduced costs by now, are you sure you know what you're doing?

And what about project life cycle management.  Are data science projects scoped out like other projects, with agile sprints and continuous delivery?  Are there milestones and deliverable and phases to projects?  How does a data scientist log their hours, to specific projects?  How are project hours determined ahead of time?  And from who's bucket do the hours get charged to, Marketing, Sales, Finance, Overhead?

What if there mouse stops working, do  they enter a ticket into the Ticketing System?  Who creates the work environments, development, test and production?  Where is the source code saved to?  Who performs the database backups?  How does code get moved to production, especially if they work in an ITIL environment?

What if the data scientist gets embedded within a particular department?  Disconnected from the team of data scientist, do they still have periodic meetings with the team of data scientist to compare notes and leverage the knowledge pool of coding techniques and domain knowledge?  Is there a company Wiki for documenting the knowledge learned, or does the domain knowledge walk out the door when the data scientist leaves?

What if the data scientist is unable to discover a true insight?  How does a data scientist know which project to go after, easiest, move value, most exposure?  Who maintains the queue of experiments to work on, and who prioritizes the list?  Who does the quality assurance on the work, before moving to production?  Does the data scientist work with a business analyst or domain knowledge expert?

How many meetings per week does a data scientist attend, what are the meetings about, who attends the meetings?

What if the results of a data science project are completely wrong?  And management implements the recommended changes with bad downstream effects?  Can the data scientist be held liable?  Should Data Scientist purchase professional liability insurance in case they get sued?

These are just a few questions I  was thinking about regarding the sexiest job of the 21st century.  I still think the Data Scientist is the rock star of the workforce, because of the potential impact they have, the intelligence required to be successful and because they have tentacles into every department and every software application and potential external data sets.  The Data Scientist is the new "goto" person of any organization.  Data Scientist try to tame the wild beast of data explosion, rapid advances in technology and storage capacity and programming and algorythms and domain knowledge.  Beyond the hype, they still need to interact in a business with real people and real processes and keep pace with  rapid change as well as deliver results to an eager audience.

Quite a challenge.  I think it's worth it.

9/17/2015

[Video Blog] Neural Network Artificial Intelligence Microsoft AzureML Sample Experiment

In this video, we discuss Neural Network Artificial Microsoft Azure ML.  After logging on to the Microsoft AzureML platform in the Cloud, we review a sample experiment that deals with Neural Networks.  We see an example of a new language specifically for Neural Networks called net# and view some R code as well as the accuracy results of the experiment.




Here's a few reference links:

Neural Nets in Azure ML – Introduction to Net#

Multiclass Neural Network 

Train Model

Score Model

Evaluate Model

Sweep Parameters

Guide to Net# neural network specification language for Azure Machine Learning

TSV File Format

MNIST Link

My blog post on Neural Network Artificial Intelligence Microsoft AzureML Reference

Although I've been following Artificial Intelligence for a few years now, I'm still learning a lot.  Especially on the new Microsoft AzureM platform.

This video is intended to be an introduction level tutorial.

Thanks for reading~!

Neural Network Artificial Intelligence on Microsoft AzureML Reference

I've been interested in Artificial Intelligence for many years now.  Probably because it's a tough nut to crack.  And researches have been trying to figure it out since the 1950s.  It seems the technology has really made good progress recently.  Primarily due to the explosion of data, increased processing power and available technology to the layman programmers.

Here's a few links for getting started with Artificial Intelligence with Microsoft tools.


for the c# people, here's: Developing Neural Networks Using Visual Studio


And an article about Microsoft's offering of Machine Learning in Azure from 2014: Why Azure Machine Learning is a Game Changer

And my posts from 2013: My Latest Crush - Artificial Intelligence


[Video Blog] Azure Machine Learning Web Services Connect via Excel & Visual Studio

Here's a video for Azure Machine Learning Experiment.  In this video, we explain a sample Experiment, create a Predictive Experiment, save as a Web Service.  From there, you can access the Azure Machine Learning Web Service via Excel Add-In or Visual Studio c# (or Python or R).  Microsoft actually provides the sample Excel workbook (although you must have the ML Add-In already installed) as well as sample source code for your project.

Here's the video:



Here's a few URL's for reference:

Azure Machine Learning Excel Add-In Download
https://azuremlexcel.codeplex.com/


Data set in video
http://archive.ics.uci.edu/ml/machine-learning-databases/letter-recognition/letter-recognition.data

Thanks for watching~!

9/15/2015

[Video Blog] Microsoft Azure Upload Files to Storage Account Mount in HDInsight Hive



This video explains the process of uploading text files to Microsoft Azure Service Account using a Utility called Azure Storage Explorer.  Once ingested, we use an existing Azure HDInsight Hadoop cluster with Hive to mount the text file into an External Hive Table. 



The Video got cut off at the end, sorry for the abrupt ending...

Here's a few of the links from the video:
 

Use Hive and HiveQL with Hadoop in HDInsight to analyze a sample Apache log4j file
https://azure.microsoft.com/en-us/documentation/articles/hdinsight-use-hive/
 

Create and load data into Hive tables from Azure blob storage
https://azure.microsoft.com/en-us/documentation/articles/machine-learning-data-science-hive-tables/
 

Azure Storage Explorer
  
HDInsight: Hive Internal and External Tables Intro

Thanks for watching~!

Get Sh#t Done!