4/26/2018

Introducing a Simple Framework for Working with Data

This week I blogged about 5 new features to data.  It starts off simple, builds upon previous idea, to form the building blocks of Strong Artificial General Intelligence, a grandiose concept indeed:

Tag Data at Time of Inception - integrate a framework such that data gets tagged upon inception using XML tree like structure to capture meta-data for external use


Open Data Set Framework - standards applied to generic data sets for public or private consumption


Open Reporting Tools - generic report reader seamlessly ingest Open Data Sets - allow any user to work with data to find insights


Global Data Catalog - Cloud Based storage of Metadata for consumption by Open Data Set ingestion

Automate Machine Learning Artificial Intelligence Ingestion to dynamically scan Global Data Catalog for purposes of Unsupervised Machine Learning Ingestion, to automatically build and refresh Data Models in real time to answer specific questions in any domain


Programmers have built Frameworks for a variety of languages.  Frameworks serve the ecosystem by organizing concepts and techniques into re-usable patterns.  Not sure why the world of Data has steered clear for so long, I'm proposing a new foundation, a series of non threatening concepts, when combined, will produce results greater than each individual line item idea.


Remember to tell 'em who you heard this from first, before its gobbled up and re-distributed as someone else' idea.  Jon Bloom.


As always, thanks for reading~!



http://www.bloomconsultingbi.com/2018/04/apply-structure-to-data-sets-for.html

http://www.bloomconsultingbi.com/2018/04/self-describing-data-tagged-at-time-of.html


http://www.bloomconsultingbi.com/2018/04/to-reach-artificial-general.html


http://www.bloomconsultingbi.com/2018/04/self-describing-data-tagged-at-time-of.html




How to Survive the Rise of Automation, Intelligence and Robotics

The great chasm that divides society will be of knowledge and how that translates to marketable skills.   
 With the rise of automation, many manual tasks will be performed by Robots and / or Algorithms.  Reason being, human capital is not cheap, automation is. 
 Once a computer model is trained in specific domain, at expert level, it's speed, accuracy and documented audit trail would be no match for average people. 
 In order to survive the next economy, one must have knowledge and the ability to translate that into a necessary skill that's in demand. 
 A Data Scientist could train a machine learning model, by feeding it information about court cases, going back 500 years.  The Model would learn the logistics, the exceptions, the probability of outcomes over time, and be a source of information going forward, so long its updated over time and verified for accuracy. 
 That translates to reduced demand for those in the legal profession, like research.  Imagine having tons of valid info at your fingertips, in real time, scanning millions of court cases on the fly. 
 Now, ripple that to scenario to other professions and you see very fast the impact automation will have on society. 
 Throw in Robots, Self Driving Vehicles, Transportation and Logistics, Food Service, Education and many more industries will be severely impacted. 
 With fewer individuals able to earn gainful employment, less money flowing through economy, perhaps slow down in GDP, the stress and burden on society could increase as costs and consumer debt rises, the picture becomes a bit more bleak. 
 There's mention of Basic Income, yet if you begin to review what a global welfare system would look like, you see very quickly there are many holes.  As in who will finance a great chunk of society, would crime and black market increase, what would people do during idle time, will population increase or decrease, what chance will offspring have to become educated and find employment. 
 However.  Those that have quantifiable legitimate skills, that are in demand, would find work.  Perhaps in technology, or a service that requires on-site tasks, or something creative that requires humans specifically.  They will have pick of the litter, luxuries not available at lower rungs, as their skills will be demand. 
 Looking at things from this perspective, you would imagine any youngster frantically learning everything they can get their hands on, as their future could depend on such knowledge and skills, in order to stay afloat, down the road, when automation and robotics make their way into mainstream society. 
 And there you have it~! 

4/24/2018

To Reach Artificial General Intelligence We Must First Tag All the Data at Time of Creation

What are the basic steps a report writer performs to do their job? 
  
  1. Obtain requirements by mapping out required Fields, Aggregates, Filters 
  1. Write SQL Statement(s) using Tables, Views, Joins, Where Clauses, Group By, Having clauses 
  1. Validate Data 
  1. Push to Production 
  
What if we applied Self Service over this process. 
  
  1. Users specify requirements by mapping out required Fields, Aggregates, Filters 
  1. Table or View Joins were already created in background, User select fields, aggregates, filters, etc. 
  1. Data validated prior to model deployment, so in reality data should be accurate 
  1. Model uses Production data, can save off Self Service report, schedule to run on frequency 
  
 What if we applied Semi-Automated-Self-Service process to deliver reports. 
  
  1. All data elements, tables, views, fields, existing reports with report title, report use / function, existing fields, parameters, would all get stored into a Metadata repository similar to Data Dictionary or Data Catalog ahead of time. 
  1. User specify what problem they are trying to solve 
  1. System would pull specific fields from pool of available fields that correspond to answering the asked question 
  1. Report would self generate for user consumption 
  
What if we applied Weak Artificial Intelligence to deliver reports. 
  
  1. User specify what problem they are trying to solve 
  1. AI would process request, pull associated data to support answer to question 
  1. User receives instant response with high percentage probability correct answer 
  
What if we applied Strong Artificial Intelligence to deliver reports. 
  
  1. AI system would generate their own questions 
  1. AI system would know where to find their answer 
  1. AI system would solve their own problems unassisted by human intervention 
  
How do we get to Strong AI? 
 My guess, AI Systems require data which is labeled or tagged, to perform Unsupervised Machine Learning, to build and run Models, to derive fair amount of accuracy of probability.  Most of the world's data is not tagged.  It also doesn't mash well, out of the box, with other data sets.  For example, if you have a data set of financial transactions of specific customers, how do you join that data set to a data set of home values over time.  There aren't any pre-defined keys that you are aware of. 
 So if we tag the data at time of creation, sort of like a self referencingself documenting XML file associated with a data set or SQL Table, you basically create a WSDL of high level data structure, along with audit trail to track changes over time, along with any revisions or changes or updates or deletes to the record set, perhaps IP address of where data was born, time stamps, etc. 
 Any ingestion process could read this new self defining WSDL type file, determine what the data set consists of, fields names, field types, etc. such that it could automatically deduce the contents of the data, without having to ingest everything.  By doing so, the AI ingestion process, could read a global encyclopedia of archived data sets, continually added over time, and pull in any required data set for consumption, to add to the model, refresh, in order to derive an answer to a question, with high degree of accuracy based on probability. 
 What I'm saying is by tagging the data at creation time, with an externally consumable file, the AI ingestion system is empowered to pull in specific data sets it finds useful to support a model to answer questions.  This open data framework is flexible to support automation and would support the building blocks to Artificial General Intelligence at rudimentary levels with room to grow into full blown true Artificial General Intelligence (AGI). 
 Similar Post:  

Self Describing Data Tagged at Time of Creation