Generative AI for Data Analyst

Generative AI is taking the world by storm, but how effective can it be for data analysts? Let’s explore!

Whether you’re a technology consumer or a software provider, there’s never been a technological innovation that has enthralled everyone — from a young child to teenagers, adults, and workforces in organizations of all sizes. Almost 50% of the world’s population have tried it, and a significant proportion aren’t even technology users — other than using their smartphones.

I was astonished when my teenager took to it faster than he did Snapchat or Instagram. Open AI’s ChatGPT was arguably the catalyst that got the world’s attention, with big companies such as Microsoft, Google, AWS, and many leading companies quickly jumping in and stoking the flames of excitement. Since then, its rapid infusion into all kinds of technology has been astounding.

As McKinsey’s annual global survey¹ points out, 2023 is the breakout year for AI. Mckinsey’s survey found one-third of respondents already use gen AI in at least one business function, and 40% of respondents plan to increase their overall investment in AI based on the potential of gen AI. Never before have we seen a new technology garner so much organizational attention, to the point where it’s now also a key topic on 28% of board agendas.

The Speed of Adoption in the Workplace

Most companies see generative AI as a breakthrough technology to boost worker productivity by augmenting human activities, not replacing them. Its potential to make workforces more productive is why organizations are making it a priority investment. Microsoft has its CoPilot for Office 365 in early access and sees it as a paradigm shift in how people will work.

It stands to make workforces work faster and eliminate routine tasks. Deeper research indicates gen AI’s most evident impact is in its application to initiatives that improve top-line growth — such as better personalized marketing, and more so, its substantial impact on boosting worker productivity — from automating otherwise human-intensive processes, creating content in seconds (even code), and giving people the power to work with even more data.

To quantify generative AI’s impact on productivity, MIT conducted a research study² on two groups. The study demonstrated that the group using generative AI tools improved the worker’s performance by as much as 40% compared with those not using it. It also showed it can reduce the need for highly specialized skill sets by enabling anyone, for instance, to generate basic code, analyze data, author a business process, and more.

But since the piece is about an analyst’s life and how it can boost analytics productivity, let’s look at a case involving working with data. Typically, to access data, a data steward with specialized skills would write an SQL statement to query data from a data source and then hand it off to the analyst to perform analytics.

Now, with gen AI, instead of relying on a data steward to query data, the analyst can enter a prompt that says create a customer list with ‘first name’ ‘last name’ ‘age’ ‘address’ and ‘monthly spend is greater than $200 a month’. It’s that simple; anyone can do this with the right AI-powered analytics tools.

Whether you’re an operations, finance, sales, marketing, or HR analyst, anyone can author a prompt to access the data they need using natural language and generate analytics-ready data sets. And it takes seconds.

Gen AI for Accessing Data, The First Step in Analytics

Given the power of gen AI for data access, it’s not surprising that data platform vendors have added generative AI into their products with good uptake.

Databricks recently announced LakehouseIQ, which enables users in any organization to use natural language to query data, search, and understand data.

Snowflake announced Document AI, which provides an LLM-based interface to understand and interpret PDF documents and convert data into data sets. Oracle infused gen AI into its Oracle Cloud infrastructure with a focus on use cases like fraud detection, where gen AI is used to sift through masses of financial data and generate new data to identify fraud patterns.

If you’re a data platform provider, adding gen AI capabilities to simplify access to data is a logical and required addition.

Gen AI for Interpreting Insights, The Last Step in Analytics

We’ve evolved from visual dashboards to business intelligence (BI) tools that add natural language generation (NLG) to help interpret insights. Several years ago, we saw an onslaught of these NLG-powered BI dashboards.

In 2021, Gartner advised companies sitting on unexploited unstructured data to use NLG-powered tools to extract differentiating insights. Usage was encouraged for intelligent document processing and insights interpretation — albeit using very structured syntax. Now that’s overturned with gen AI.

Fast forward two years, gen AI is enabling a leap forward in interpreting insights. Top BI vendors, like Tableau, provide gen AI-powered data visualization dashboards to provide business stakeholders an easier way to surface and interpret insights.

With gen AI, a dashboard can now tell you what to focus on by automatically generating content that says: “Of the 12 metrics, two are unusual for this week. The unusual insights are this week’s ‘Sales by Region’ and ‘Top 5 product sales’.” You then probe with the following prompt, and gen AI surfaces more discoveries on the unusual insights. No doubt, it is a step forward for business stakeholders consuming dashboards daily.

The Middle — Where is Gen AI In Solving the Hardest Part of Analytics?

Analytics processes first start with data access and end with insights to stakeholders. The infusion of generative AI into data platform solutions and BI tools helps to simplify these two ends of the process.

But when you consider how an analyst spends their day, the most significant time sink is the middle — the complex, repetitive tasks of prepping, transforming data, writing analytics logic, performing calculations, setting up reporting, and endless hours of analytic process iterations and maintaining dashboards.

While about 30% of analysts’ time is spent accessing data from data platforms or Excel files to create analytics-ready datasets, 70% is spent in painstaking cycles of assembling the analytics process, managing hand-offs, performing calculations, and maintaining insights.

Today, 60M+ business analysts in the workforce spend endless cycles doing manual, repetitive work from data prep, data blending, writing analytics logic, and wiring insights into dashboards, repetitively summarizing them in emails and static PowerPoint reports.

They use Excel as their tool or other desktop tools. The manual and tedious nature of the job is time-consuming and naturally results in errors that result in the business paying the price. One utility company revealed how data and analytics errors cost them $24 million in lost revenue.

How Specifically Can Generative AI Boost Core Analytics and Analyst Productivity?

It is not hard to imagine that the place where analysts spend the bulk of their time could benefit tremendously from generative AI-driven automation.

A recent Harvard Business Review article³ highlighted that finance, sales, and marketing analytical roles are amongst the jobs best positioned to use generative AI to improve analytics productivity by adopting gen AI.

From data extraction to data cleaning, data prep and blending, to analytics logic and delivery of insights, Generative AI-powered analytics solutions are proving to accelerate analytic results. Simply put, its application is throughout the analytics development process and is here today.

Gen AI’s Biggest Potential — Easing the Bulk of the Analytics Process

As a serial entrepreneur and now founder of Savant Labs, I believe in enabling greater analyst productivity with the combined power of automation and gen AI.

To that end, the opportunity ahead is a purpose-built analytics automation solution for data and business analysts that uses the power of the cloud, GPT conversational interfaces, and analytics automation to improve productivity by automating mundane tasks. Imagine a seamless, prompt-driven experience with drag-and-drop automation where you need it. No special skills are required, no writing code or scripts, and it is simpler than Excel, making it approachable to anyone who needs to work with data. It takes self-service analytics to a whole new level.

Unlike previous generation tools, the limitations that plagued legacy analytics platforms no longer impede modern analytics automation platforms. With modern-cloud native architecture at its core, many advantages make it better suited for the modern data stack.

Here’s how analysts can simplify and speed analytic outcomes:

AI-assisted Data Extraction

Gen AI prompts can extract data from various sources without requiring manual SQL statements or scripts. Not just that, it also makes recommendations and helps avoid common mistakes.

‍

AI-assisted Data Cleansing

‍Complex data cleansing can be simplified using gen AI prompts. Given data cleansing is otherwise a tedious, time-consuming process, analysts can now achieve results in record time on large data sets.

‍

‍

AI-assisted Data Preparation

‍Gen AI simplifies transforming and organizing data. By leveraging generative AI prompts, users can perform data prep tasks without the need to write complex expressions, understand syntax, or work with SQL. They can describe the task in natural language, and AI generates the code.

‍

‍

AI-assisted Analytics‍

Enables analysts to create complex analytics without manually authoring the logic, resulting in increased efficiency and reduced time spent on analytics workflow development.

‍

‍

AI-assisted Insight Delivery‍

Users can specify where and how to publish the generated insights or data outcomes, providing greater control and flexibility in sharing valuable information.

‍

The Future of Generative AI-powered Analytics

We envision a future state where analysts can build end-to-end workflows, opening the aperture for more users with varying skill levels to create analytics workflows.

Imagine a world where the analysts, using prompts, can create the end-to-end analytics workflow, including defining data sets and attributes that should be joined (data blending), the analytics logic described in the above examples, and destinations to publishing insights.

To automate the entire workflow creation with gen AI, new frameworks like LangChain show great promise. Let me elaborate on LangChain if you’re unfamiliar with it. LangChain enables LLM models to be chained together, where each model is trained to perform a specific task.

At Savant, we’ve built the underlying analytics automation infrastructure that leverages LangChain and an intuitive user interface (UI) that enables the entire workflow creation using gen AI. This is in addition to the discrete parts of the analytics process, such as data cleansing, data prep, data blending, and analytics logic described above.

I believe, as do many organizations and analytics leaders we speak to, that providing gen AI-assisted analytics for each discrete part of the analytics process while also providing gen AI to enable prompts to create the entire end-to-end workflow is essential to achieve analyst productivity boosts.

An AI-assisted analytics automation platform that provides both options truly delivers on the hope we’ve always had, which is that any analyst or business user should be able to pick up the tool and, in minutes, create their analytic processes.

Now, not only can 60M+ analysts simplify and ease their daily lives, but those of you who never dreamed of being able to perform analytics can now do so.

‍

[1] The State of AI in 2023, McKinsey, August 2023

[2] How Generative AI Can Boost Highly Skilled Workers’ Productivity, MIT Management, October 2023

[3] HBR — Boost your productivity with GenAI, June 2023