9/16/2018

Current State of Machine Learning and Artificial Intelligence

Machine Learning has three main features.  

For some, recommendation engines one websites are valuable tools, perhaps suggesting good movies similar to ones you've watched already, based on preferences

For some, clustering of homes on real estate sites are good tools.  What is my home value based on similar homes within the region.

And for others, classifying images is a good tool, perhaps recognizing your friends in a picture on social media, without having to enter the tags in manually.

Recommendation, Clustering and Classifying.  Three solid pieces of machine learning or types.  And a variety of algorithms to get to that goal.

There's also Neural Networks, that are useful for artificial intelligence, they have weights that get toggled true or false, depending on incoming variables, with multiple layers, which produce an end result weighted probability.

Some common themes are lots of data, to train the model, to learn from known data.  some data needs to be labeled ahead of time, which is "Supervised Learning".  The opposite is having the model learn on its own, with the luxury of "tagged" data ahead of time, called "Unsupervised Learning".

There're also models that learn from other models, to reinforce learned behavior, the event happened, this was the result, let me track that, for future reference.

Some Neural Nets are very large, with hundreds of layers deep, to get fairly precise results.  They do require more compute power and memory and take time to process.

There are a slew of languages and tools to use when working with machine learning and artificial intelligence, both on-premise, virtual machines and in the Cloud.

The fact we have bigger data sets, better compute power, more ram and machines chips that can crunch more data faster.  In the past, typically large institutions had access to these types of machines, so Universities or large computer organizations were the only place to work on this technology.  Now you can program this on a laptop in you living room if you desire.  So things have tricked down into the hands of the many.  Thus, faster progress, cleaner solutions, better results.

The results have gotten better, in image recognition, speech recognition, translation tools from language to language, some in real time and many more.  At this point machine learning is a useful tool to assist humans in everyday activities.  It has not gotten to the point where AI is replacing everyday jobs, in most cases.  But the tides will shift at some point, where people are competing with smart machines for everyday jobs.  And when robots are tasked with specific tasks, that too will automate some of the workforce.  And when smart machines can perform complex tasks in real time with minimal error, then we will take notice as computers do not necessarily require vacation time, or health insurance, or 401k matching contributions, and they work 24/7 without complaining or forming Unions.

So, we have come a long way, computers can crunch data and translate into meaningful information for consumption, to lower costs and automate menial tasks.  Where do we go from here?  More automation, easier to use tools, better integration across domains, access in everyday tasks, embedded IoT devices that assist in real time.  Artificial Intelligence should not be interpreted strictly as computers rising up, taking over man kind.  Surely, anything is possible, at this point in time, we are still telling the machines what to do.  And they do their specific tasks, in a sort of black box, and we do not necessarily know exactly how the results were derived.

With Moore's Law, things will only get faster, cheaper, easier to use and proliferate through everyday society.  That should keep us busy for a while, for sure.



9/10/2018

Put the Hat in the Ring

I went down pretty hard.  The bags of food balanced on the handlebars went everywhere.  The bike took a spill.  Injured wrist & shoulder.

Perhaps self sabotage.  Was flying out the next day.  To play a tennis tournament.  In Long Island.  National Indoor Clay, Under 35's.  I decided to go anyway.

Although won a lot of matches locally, national stage is different.  Most of these players knew each other.  As they played each other for years.  At the college level.  Pros.

I arrived at the club early.  Hit with a guy.  Decided to enter the doubles draw.  My opponent was driving in from DC, was late.  Tournament director let the match be delayed for hours.  Got a chance to watch some good matches.

My opponent arrived, went to the courts, indoor clay.  I got waxed.  Didn't get a single game.  My doubles partner probably thought about backing out.  As we had to play the number one seed.  individually both ranked in top 10, doubles, top 5.

Warmed up, each person won their serve.  The ball moved fast, quick points.  I served with full force, regardless of the pain.  Went to a tie breaker first set.  They squeaked it out.  Went on to win the 2nd set.

For me, still a victory.  To be on the court with top players in the country, gave a good fight.  No regrets.

We got a doubles ranking for that match, nationally.

Later that year, the National Outdoor Clay tournament in Daytona, FL, drove for this one.

Played a different guy from DC, I won that match.  Next match, one of the top seeds, same guy we played doubles in NY earlier that year.  Played a tough match, yet it wasn't enough.  He said after the match, he didn't remember the 2 handed backhand first time around, "your two hand backhand is a weapon".

Dropped into the consolation bracket.  Won the the next match.  Then played a guy from Boston.  We had similar game, get everything back, run down every ball, both had strong backhands.  I went up early, won the first set, he won the second.  It was real hot that day, took a break between sets.  By that time, lots of spectators watching, clapping, lots of long points.  After 2.5 hours, we shook hands, he went on to the next round.

I said, "I thought I was in pretty good shape, but you seemed to be in better shape."  He replied, "I told my coach I didn't want to lose due to fitness."  It worked.

Finished out the year with Florida ranking, National ranking both singles and doubles.  Also taught lessons around town, fill in for other teaching pros, group lessons, kids, moms, adults, private lessons.

After meeting my wife, decided to go back to work doing Crystal Reports consultant job, day after I accepted offer, got another offer to teach tennis full time at one of the clubs.  Had to decline.  And set down the rackets, again.

That was a good ride though.  Rose the ranks locally, singles and doubles leagues, local tournaments, and finally national tournaments and stint as teaching pro.  Teaching was a great opportunity and I enjoyed time on the courts.

That was 14 years ago.  Would be tough to find time to get back into tennis at that level.  Then again, you never know.

And so it goes~!

9/03/2018

Hop Aboard the Data Bandwagon

It does appear the Cloud is the place to be, certainly gained traction past few years.  Few key players to choose from.  Offer mostly the same services, just depends on what flavor you prefer, how well it integrates into current stack, and your developer pool staff skills and availability.

It seems Machine Learning is still hot along with Artificial Intelligence, although it bypassed many folks from the data pool skills set, requiring new languages like Python, R, Scala, Spark, Notebooks and algorithms.

Big Data didn't solve all our problems, but it's a nice addition to the data stack, has some good hooks to get data in and out, as well as reporting, although it won't be our traditional transaction database as expected.

Organizations still want their data, in readable format, in timely manor, with accuracy expected.  Although hiccups in the Extract Transform and Load still seem to play havoc on our daily reporting needs.

Data Science still seems hot, although just because you have talented DS, doesn't mean they can write standard SQL or knock out some traditional reports.  The two seem mutual exclusive and don't necessarily overlap, as new college grads jump straight into DS with little to no knowledge of traditional data ecosystem.

Data Engineer is a hot position, joining disparate data sets for others to work with.

Data Management has probably risen the fastest, as new GDPR rules necessitate solid data practices.  With that, having knowledge of where all the data resides, what it contains, how to access has pushed the Metadata into the spotlight, with Data Catalogs to handle such requests.

We are no longer at the point where knowing SQL or Access or Excel can guarantee you a data position.  Data skills have proliferated, grown, splintered, gone sideways and every which way.  The Data Ecosystem has exploded and many new comers have entered the arena, as developers, architects, software vendors and applications.

Throw in Blockchain, Agile Methodology, Streaming, IoT and Domain Knowledge, you can clearly have your hands full for the next decade or so.

Suffice to say, data is hot.  Anyone entering the workforce that's looking for a solid career, should look no further than data.  It's the bread and butter of every department in every organization.  There are some good companies foaming at the mouth, to get some talent in the door, to hit the ground running, and add value across the board.

So hop aboard the Data Bandwagon.



8/31/2018

Automated Solution to Repetitive Coding

What if you were tasked with creating an SSIS package.  To pull from a database, send the data to Azure Data Lake Store as CSV.  Then pick up that data, flow to U-SQL database table.

Well, you'd probably need a source ODBC connection to the database.  And an SSIS package component to flow to Azure Data Lake Store.  And another to flow to U-SQL db.

So, once you go through the effort to complete the task, let's say you had to build 500 similar SSIS packages.  Well, it if takes 2 hours per package, and you don't make any errors, you may get done this month or next.

Or.  You could automate.  As in, build out each package, line by line, using an SSIS package Script component.  Maybe use XML template for repeatable code snippets, pass in data as variables, and create the 500 SSIS packages, in under a minute.

Well, that's what the team's been working on.  And it's been quite tasty.  Building c# code certainly is fun, different from writing standard SQL reports.  One of the great things about coding, you certainly get a different perspective every so often as projects change.

I've been working with SSIS package when they were DTS packages, not DTSX.  And pulling data from databases for a very long time.  Except flowing to Azure is sort of new.  I was lucky enough to figure out the solution, there's some documentation on the web, blogs and such, but for the most part, you have to figure it out yourself.  Those are the juicy projects and this is my 2nd one in the past few years.  The other project being the Hortonworks Hadoop Enterprise Data Warehouse using Visual Studio components and Polybase and Master Data Services.

Being an expert in anything is daunting task, as things change, nobody knows everything.  It's when you approach a project with fresh eyes, you stay open minded to see different options, try this, try that, see what works, document as you go.  And perhaps you become expert for a moment, until the next projects begins.

Programming is still a descent job.  I work out of my house, past 5 years actually.  You begin the day before most people, and you end the day after most people.  There's no commute.  There's no off button.  Yet, it seems to work just fine.  If you can work fast, if you're responsive, if you can deliver solutions to tough problems, what more can you ask for.

And so it goes~!

8/30/2018

Supporting Public Facing Java Applications

I once supported a web application, public facing, many users.  First task was to add Cookies & Captcha.  After that, a re-branding of the website to update with different color schemes for consistent look and feel.

It actually was written in Java using IBM VA Java, which I was hired to be a Senior Java Developer.  Luckily I learned it on my own years back, they were just looking for someone that knew that flavor of Java.

In fact, it hooked to the Mainframe on the back end.  So we had copybook frameworks which mapped to the back end, which connected to Mainframe COBOL and returned data, pieces at a time.


Then we migrated to use Web Services with XML, and then migrated the entire application to JDeveloper java.


Also, the application had hooks into the IVR system which I supported, to accept credit card payments over the phone.  Was interesting to test the application using break points and calling in the number and seeing the code stop on specific line of Java code.

Lastly, it also hooked into the Kiosks which we had a few around town, which also accepted payments, and it too was written in Java connecting to the Mainframe back end.

It was fun to program in Java, although we had internal teams moving to Struts and I didn't get to work on much of that.

I also supported other applications in Java, it was a high profile position as all were public facing.  In addition to being the Business Objects Admin for a while, introducing Crystal Reports to the ecosystem and also had a stint of Project Management.

Overall an excellent job.  Lasted more than 4 years.  Not too shabby~!

8/16/2018

SSIS Automation using SSIS

Do you ever sit down and write code.  For hours.  Never get up.  Even to eat or use the rest room.

That's what I've been up to the past few days.  Writing c# code.  In SSIS script component.

It's quite tasty.  It's actually programming.  Within SSIS.  In the data space.  It's an SSIS package to create other SSIS packages.  Using custom code, XML templates.  Reading from source system.  And Excel file template.  It outputs ".DTSX" packages.  To automate the process.  Rather than write each package one at a time.  Automation is key.

Also writing U-SQL scripts.  Create, truncate and dropping tables in Azure Data Lake Analytics U-SQL database.  And pushing CSV files up to Azure Data Lake Store.

Sure is fun to be heads down in the code.  I'll be there in 5 minutes.  2 hours later, where are you?  I'll be there in 5 minutes.

And so it goes~!

8/13/2018

Time Stops for Worm Holes

Can you stop time?  Just open a slit in the space time continuum.  Drop down a worm hole.  Appear in a far off galaxy.  Hang out for a while.  Return home in time for dinner.

Populate Mars.  Maybe occupy some pre-built structures, from before they lost atmosphere, flew to Earth, built a race of working bi-pedal hominids.  History is fluid, most of the truth evaporated.

Atlantis.  Lost?  Shifted south.  As India shifted north.  Smashed, created some mountains.  Platonic shift?  Europia.  Popcorn please.  Fiction is so fascinating.

Big Bang.  Any witness'?  Got a few questions.  Infinity.  For how long?  Expand forever.  Where is the edge?  Been there a few times.  Took the worm hole.  Found the slit.  When time stopped.  Back for dinner.

Five senses?  Wow.  What about all the others?  Like doing math with no odd numbers, and excluding 7, 5, 1 and 9.  Sort of limiting.

Before arrival, sat around, with the guides, what would you like to accomplish this time, while in physical form?  Well, purgatory sounds like a good learning experience.  And poof.  Here we are.

Predict the future?  Can't predict the past.  Fiction.  Reality?  An aligned hallucination.  1. Work 2. Spend 3. Repeat

Unlearning is the first sign of intelligence.  Tabula Rasa.  Blank slate.  Dinner plate.  Time to go.  Famished.

7/24/2018

Azure Programming in U-SQL

Here's something I noticed recently.

I enjoy working in the Microsoft space.

When you work in Microsoft space of technology, things are fairly consistent.  Drop down menu's.  Command buttons.  Documentation.  It's fairly uniform across most applications.

When you compare that to open source or another vendor owned software, perhaps the fonts are different, or the screens are jumbled and difficult to read.

In other words, when you work with the same software apps by Microsoft, you intuitively know where things are on the screen, you know what to expect when you install an application, you have confidence the app is going to work as expected and most likely, it will integrate with other products of same vendor.

Now keep in mind, if you've ever tried to install Visual Studio for instance, you are aware of the number of dependencies and extra installs required just to get everything up and humming like dot net frameworks and such.  And there's a bunch of different places to download the software.  And try messing with the Gaac or Registry settings, no easy task.

All that is a side note.  What I"m talking about is ease of use, the common themes across products and its interaction with other apps.

We're not talking about deprecating good products, like Visual Studio 6, forcing developers to move to .net object oriented languages.

Speaking of new languages, I've been working with U-SQL for the past few months and I will say its a fantastic language.  You can develop the code in notepad and copy paste into the Azure portal, click submit and watch it run in real time, it shows the code, the errors, graphs, execution plans.  

Also, you can take that same code and run in Visual Studio then hop over to Azure portal and watch the job run in real time.  You can also see the Azure objects from Visual Studio, export data in VS or Azure, and watch jobs run in VS or view objects from Solutions Explorer.

It's amazing to write code in on a VM, execute the code to pull from on-premise data, push to the Cloud Azure Data Lake Storage Group, mount that data using U-SQL and send to U-SQL database table.  Hybrid Programming is the future.  When you execute a job, you can see the Estimated Cost / Charges on the web page.

This environment is so open and fluid and really opens up a lot of options, and it has the look and feel you get with similar vendor products.  I've been working with Microsoft products since 1996 professionally with Visual Studio 3,4,5,6, then ASP then .net, SQL Server, MSBI and now Azure.

Azure Cloud Programming has a lot of good features and opportunity.  Bite off a particular area to work in, I recommend U-SQL.  Good stuff.

6/28/2018

Attending the Real BI Conference at MIT in Boston

Wrapping up day 2 of the Real Business Intelligence Conference, See Beyond the Noise, held at MIT in Boston.  Overall, it was a great event and glad to attend.

https://www.realbusinessintelligence.com/

The nice thing about the conference, was the venue.  Just being at MIT, you sort of feel smarter.

Next, the quality of speakers was great, with a mix of topics ranging from Futurism to GDPR to Real World Data Project success stories.

Food, snacks, great.

I like the single auditorium, single track approach.  Lots of conference have a number of multiple tracks running parallel and having to choose between multi options.

The conference also had a community feel to it, as in getting to meet people from all over the world, have discussions of quality, on variety of topics around technology.

The two day approach was nice, as some conferences run 4 or 5 days, and very technical, by the end of the week, your brain is fried, don't remember much and actually skip some of the sessions to give the mind a rest.

This conference was concise, personal, quality speakers and content.

If you're interested in attending next year, the website is:  https://www.eventbrite.com/e/2019-real-business-intelligence-conference-tickets-47210074604

The most surprising topic was around GDPR, which is basically a world mandate, to protect EU user's data, the rules, and implications surrounding non compliance. Seems like a good opportunity for consulting firms to hold business' hands in getting compliant.

The main theme in almost every talk, "Ask better questions."  Overall, very happy to have attended the Real BI conference 2018 at MIT in Boston.  Even got a photo with Howard Dresner, who coined the term "Business Intelligence" term in 1989, his firm hosted the event

Thanks for reading~!