Advances in Analytics

Recently, the training and research organization TDWI brought together an Expert Panel to discuss these Advances in Analytics. You can still view the conversation online here. I was happy to take part as an advisor to Lumenore, along with Fern Halper from TDWI and Jason Yeung of SAP.

The most visible advancements we discussed – those in the news – involve those chatbots, which use Large Language Models built over vast amounts of data and complex algorithms to generate coherent text and perform natural language processing tasks.  But as I said, they are not a replacement for business analytics. Meanwhile, other machine learning technologies are also improving and are promising for data analysis. Transfer learning has made significant strides, allowing models to use knowledge gained from one task to improve performance on a related task. Reinforcement learning (where an AI agent is motivated by “rewards”) has found applications in various fields, including robotics.

As Dr Halper described, improved explainability and bias detection have also been a focus, with explainable AI allowing humans to understand how an AI system makes decisions and bias detection becoming increasingly important as AI systems become more prevalent. These advancements are changing the way we approach data analysis and decision-making, making AI systems more powerful, efficient, and transparent.

Yet for many businesses, large and small, these developments remain ambitions rather than priorities. As TDWI’s research shows, the most pressing concerns of the analytic enterprise are very business-focussed: self-service, data literacy, and upskilling analysts. Top technical requirements are streaming analytics, automation and augmented tools and some machine learning. So among our panel, much of the discussion was about how we can achieve these priorities and how advances in analytics can help.

It’s all about that data

Those of us who have been working in analytics for many years, sometimes feel we are telling the same story again and again. But – one more time – the quality of analytics really does depend one the quality of data. However, to achieve real self-service analytics, what can we build to help those users who are business-smart but not data specialists. Some clear ideas came out of our conversation …

• Have a clearly defined data sourcing and ingestion strategy with smart connectivity and automation for consistency.
• Reduce data silos across the entire analysis process and ensure you have excellent lineage for data, so improving both data quality and governance.
• Ensure users are trained on the self-service analytics tools, but also choose tools where users can easily access relevant reports and insights from their own dashboards without needing additional support.
• Leverage technologies like Natural Language Processing to generate queries automatically from natural language instructions – reducing typos, mistakes and incoherent results.

When it comes to analysing this better quality data, the Conversational Intelligence that natural language enables will be critical for non-specialist users. But other advanced technologies have their role to play too.

Augmented analytics is very enabling for business users, where the system uses machine learning behind the scenes and a helpful user experience up-front to bring their attention to patterns and insights they may otherwise miss. Predictive analytics techniques, such as forecasting and scenario modelling help with that constant business question: what happens next, and what can I do about it?

These are advanced technologies, but they are already proving themselves in real use cases. Do listen to the webinar – it’s useful to hear a range of of perspectives  – and check out some of Lumenore’s advanced capabilities here.

Donald Farmer
Strategic Advisor, Lumenore 

Donald Farmer is an experienced strategy advisor, speaker, and writer with over 30 years of involvement in the data analytics space. His consulting has benefited a wide spectrum of clients including investors, enterprises and software vendors around the globe – both startups as well as renowned companies. Donald made his mark at Microsoft by leading teams building critically acclaimed groundbreaking tools for analytics during his 10-year stint there but he further stamped this position at Qlik as VP of Innovation & Design, when Forbes named it among their Top 10 Innovative Growth Companies. Donald also works as VP Research & Innovation at Nobody Studios – an enterprise speeding up entrepreneurship through a crowd-infused, high velocity approach to company creation. Donald also shares insights related to innovation and analytics strategies via Creative Differences newsletter alongside providing instructive tutorials and workshops.