Funnel Approach to Receiving and Pulling Data

When I was working my first real programming job, we had a consultant who was working on a really cool product.

This guy knew his stuff.  Smarter than anyone I've ever worked with.  And he showed me his app. He coded it to work with multiple data sources on the back end, in Visual Basic 6.

He code each data flow to work in SQL Server, Oracle or Access.  Which meant three types of SQL.  And it was completely controlled with the flip of a switch in the config file.

I thought that was brilliant.  In 1998.

He also explained how DLLs worked.  So I asked him, we have have code.  So for code reuse, we create sub routines.  And perhaps functions with parameters to add flexibility.

And we take those subs and functions, combine them into a class.  And bundle those classes into a DLL, with getters and setters.

And then we move those DLLs to COM+ distributed so it can be accessed from client server apps or web apps.

Eventually, those COM+ got moved to web services, exposed on the web via SOAP calls and then OATH2 calls.

So we've moved that code up a level about 7 or 8 times now.  Which increases complexity yet makes it easier.  Solving the same problem with new techniques.

So I was thinking, why not have a data model based on a Funnel.  The funnel accepts new data.  Doesn't matter where it was derived from. Or the type, whether structured or unstructured. and the Funnel handles the request appropriately, sends it to where it needs to go, and then waits for the next request.

It could funnel the data to a database, any flavor, or it could push the data to another funnel and so on.  Or it could write the data to an HDFS, or a key/value pair database. all controlled through settings in the funnel software.

You present the data, the funnel takes care of the heavy lifting, ensures consistency and accuracy and everybody's happy.

I suppose the Funnel could work in the opposite direction as well.  You make a request, the Funnel has an internal understanding of where to  get the data, goes and finds it, and returns the data in some usable format to the requester.

I don't know all the details of how the Funnel approach would work.  But it seems like it would have  potential in scalability, accessibility and dumb down the process of sending and receiving data from a variety of programs.


No comments:

Post a Comment

Note: Only a member of this blog may post a comment.

Have You Seen My Double