A year ago, Blazent CEO Gary Oliver brokered a meeting that brought four like-minded technologists together. That meeting erupted into a 10-hour white board session that has in essence never stopped and that culminated last week with the establishment of the Office of the CTO.
I’ve been a musician all my life, and working with Mike, Nandu and Jack reminds me of some serious jam sessions I’ve had over the years with musicians who knew the difference between playing for yourself vs. seeking to bring out the best in yourself and your partners, to create something larger than the sum of its parts. The result, in both cases, was serious music!
Data Evolution: the step beyond ‘Data Quality’
Mike and I brought to the meeting the concept of a canonical flow generic enough to be easily expanded but initially designed to handle large amounts of streaming data coming from disparate data models. The process—which we came to call ‘data evolution’—consisted of breaking the data apart, transforming it into match values, identifying relationships, purifying all knowledge to a single reference and then preserving everything historically. And of course it all had to happen in near time so it could be moved directly to analytics.
Jack and Nandu had applied similar concepts in another paradigm and immediately dove in on what application stacks should be explored to handle this workload and ways to realize what was necessary within the flow. The timing was perfect: had the conversation occurred a few years earlier we might have never gotten past the first day. Advancements in cluster computing fueled by Spark, NoSQL databases, Data Structure Servers and Machine Learning libraries have, from a technological point of view, opened the door to solving complex problems that previously were unachievable. My guess is, that is exactly why we got along so well. Deep inside, all four of us have a great love for solving complex problems and this one was a monster.
The good news is we have come light years from our initial conversations and after hundreds of hours of white board sessions, countless stack changes, and pretty significant Alpha and Beta legwork we’ve achieved something rather significant. We’re looking forward to sharing these thoughts with you in coming blogs and articles and in upcoming product releases.