None who wander should be lost… Never in the data lake
Recently archeologists found evidence suggesting that our seafaring history goes back at least 130,000 years. How our ancestors of that period navigated the vast oceans is not yet known. Till GPS, accurate clocks and reliable weather forecasts became common place, exploration was difficult and risky.
Ancient explorers used a lot of ingenuity and learning of their time to find their way – position of stars, weather patterns, presence of certain flora/fauna and so on. For example Floki Vilgerdarson took caged ravens during his voyage from Norway to Iceland 1100 years back. When he suspected that land might be near he released one of the birds. If land was actually far-off, the raven will circle aimlessly. But, if it sensed land, it flew in that direction and Vilgerdarson’s boat would follow. With techniques like these, it is easy to imagine why exploration then was hazardous. Even during Europe’s age of discovery, despite more advanced tools like the compass, astrolabe and better maps, the failure rates were significant. For every Vasco Da Gama and Columbus, there were many more who perished in their quest for new lands, riches and glory. Not all who wandered were lost, but too many were indeed lost.
When it comes to data, many organizations today are in the age of exploration/discovery. Most are exploring vast swathes of data for the first time. And with the tools at their disposal, the journey is particularly perilous. Thankfully, unlike oceanic exploration lives are not lost. But, for many organizations loss of revenue/profit opportunities and competitiveness can be a death sentence. Some of the reasons for reaching erroneous conclusions are well known – data hygiene, not knowing the question that needs to be answered, data silos… to name a few. These are equivalent to the navigation problems from Vilgerdarson’s times. Most analysts today are cognizant of these and remedy them early.
In our experience most exploratory analysis today run aground because of two problems – (1) ignoring business relationships and rules between data points and (2) relying too much on summary measures. With JUNI, the AI powered Assistant from OpsVeda, we are looking to fix these and many more challenges for the data explorer. Through natural language interactions JUNI will help data analysts and executives explore customers, products, people, locations and transactions. More importantly it learns from past interactions and uses the learning to expand future conversations. A quick look at how JUNI is addressing the two major pitfalls mentioned earlier.
Ignoring business intricacies/relationships/rules between data points: The root cause here is business process complexity and the tedious work needed to get an accurate reflection of the same in the analysis. Fortunately JUNI learns these rules and relationships before he lands up at our customers. JUNI knows that it is unfair on part of the customer to cancel an order just before its shipping date. He also understands that the days left for shipment and the orders that were declined to accommodate this errant customer are couple of measures that can help evaluate the seriousness of the issue. Similarly, if his master asks him about due shipments, he understands that it is not just the order value that his boss needs to know. He will be ready with information like quantity short and exceptions that may prevent timely dispatch.
Can all of that be compiled? Oh Yes! But, is it a good use of executive or analyst time? Definitely not. Their time should be spent in understanding deeper business issues and engaging customers. Manually piecing together such information in 2018, is like chucking the GPS and using a sextant, chronometer and almanac to determine location.
Relying too much on summary measures: Anscombe illustrated the inadequacy of basic statistical measures almost 45 years ago. But, even today managers get carried away by summary measures too often. All of us have been subjected to countless presentations where conclusions were based on just the total and average measures. JUNI is trained to probe deeper. He examines the underlying drivers to give a comprehensive answer. To a question about growth last month, 8% would be a correct answer. But with JUNI you will get an answer like, “Overall growth was 8% but that is because of Walmart business growing at 50%. All other customers registered negative growth. Should we explore further…”
With JUNI our goal is to steer the conversation towards an area that is likely to trigger action. That means the system needs to understand a lot more than trends and aggregates. Feeding the system with intelligence about the business relationship between various entities, and preparing it to delve deeper into the underlying dynamics behind a KPI are primal to achieving that goal. For customers, it will be a welcome change from today’s shambolic data explorations, often disconnected from realities of business execution. With JUNI, you can dare to wander without sight of the shore. Rest assured that you will never be lost in the data lake, or should we say, the data swamp.
