ChatGPT recently made a major play into quantitative analysis with its Data Analyst tool, which proudly boasts that with a simple upload of a data file or market research report, it can quickly and easily handle a bulk of the work that market researchers spend so much time on.
I was intrigued enough to try it out, and with my market research hat on, I chose a few common use cases to see how well it worked. In this article I’m sharing my findings – both the things that impressed me and some serious pitfalls.
What is ChatGPT Data Analyst? And Why Does it Matter for Market Researchers?
Let’s start with the basics.
By now, we all know that ChatGPT is a Large Language Model (LLM) that handles text-based queries pretty well. The Data Analyst plugin tool within ChatGPT enhances the model's capability to handle and analyze quantitative data. With this tool, ChatGPT can perform operations like reading and processing data from Excel files, conducting statistical analyses, generating charts and graphs, and even applying machine learning algorithms. This is particularly useful for users who need to analyze datasets, interpret statistical information, or visualize data trends. The Data Analyst tool effectively combines the AI's natural language processing abilities with the power of Python's data analysis libraries, making ChatGPT a versatile tool for both data analysis and conversational AI tasks.
I’ve seen plenty of tools that make strong claims about automating analysis, so I went ahead and put Data Analyst to the test!
How We Tested ChatGPT's Data Analyst
At the very beginning of 2024, the Intuify team published the results of a survey of U.S. Consumers and their intended spending habits and economic outlooks in the new year. The 2024 Consumer Financial Outlook Report was published as a white paper with quick facts and insights, as well as a full market research report for those who wanted to dive into the complete data set.
Data Analyst claims it can read and understand various computer file types commonly used in the field of market research. I decided to conduct two tests, one using two PDF files (the aforementioned 2024 Consumer Financial Outlook white paper and report), and the other using the actual raw CSV data from that same survey.
The PDF Test: Can Data Analyst Simplify a Report?
In the PDF instance, I uploaded both the summary and full report files to ChatGPT, with prompts stating that I’d like the system to read through and analyze the results. From there, I asked a few questions about the data, such as “What are top areas where consumers are cutting spending in 2024?,” which I expected the system to be able to read from the data. It did just fine in this regard.
I then asked the system to create infographics based on the data in the report, and this is where we encountered our first hurdle. From this prompt, Data Analyst was able to tell me several ideas for key infographic stories that could be found in the data. However, it did not actually produce the infographics. With a little coaxing through creative prompts, I was able to ask the system to generate actual charts, but it took some coercing. With the proper prompts, the system did, in fact, create accurate, readable, interesting charts based on the data found in the two reports I had given it. However, the charts themselves left something to be desired in terms of visual appeal. They really weren’t an improvement over what was already in the report.
Finally, I asked the system to perform a series of complex analyses: read through each of the two reports, and derive five actual stories found within the data by examining different data points and combining them into interesting narratives that reveal a connected series of insights. To my amazement, it did exactly that. The system looked through both reports, categorized several of the key individual charts and findings, and strung them together into five clear data-driven stories about consumer spending habits in 2024. I actually was able to share these stories with my fellow Intuify team members, and they in turn were able to create their own reports based on these derived stories.
The Raw Data Test: Can Data Analyst Build a Report from a Data File?
Next came the challenge I was most excited to test: can I repeat the results above, but this time asking the system to derive these results from the raw CSV data? To accomplish this, I uploaded the CSV file to Data Analyst and performed the same queries as before. Here I encountered some severe challenges.
If you’ve ever examined survey data in CSV format, you probably know that survey data files are often very “wide” – the data has many columns, as opposed to "tall" data which has more rows. I quickly learned that Data Analyst does not like wide data.
My first queries asked questions about different individual datapoints that I knew were in the data: What spending categories are individuals cutting costs on in 2024? What areas are consumers spending more on? What categories do consumers believe other consumers will be cutting in the new year? These were basic questions. In all three cases, Data Analyst answered: “this information is not included in your data set.”
That was an unfortunate answer, since I knew the data was included in the data set. So I asked Data Analyst to look at specific columns – several of which began with the letter “C” in the CSV file – that contained the data I was looking for, and this time Data Analyst was able to locate and, in fact, analyze the data. This identified a major problem of Data Analyst: unless specifically prompted to look at individual columns, ChatGPT’s Data Analyst is limited to only a few columns of data during its analysis phase.
I tried a few more prompts to confirm my findings, and ultimately came away with the wind taken out of my sails. In a perfect world, Data Analyst would be capable of understanding raw survey data in its purest format, but in its current iteration it’s just not capable of doing so.
The Pros of ChatGPT Data Analyst for Market Researchers
ChatGPT’s Data Analyst is fantastic at answering queries about report-based market research. Say, for instance, you were a market researcher who received – or maybe even created – a presentation deck with literally hundreds of datapoints (we’re talking everything from the individual results of each question in your survey, to the cross-tabulated results of gender/age/income/ethnicity splits), and you or your client need to know the summarized results of just one small group within the data. As long as that group has been reported on, Data Analyst is completely capable of deriving a written story, summary, or skeleton of a comprehensive infographic for whatever it is you’re trying to understand.
Even better, the tool can do an excellent job at deriving stories hidden within the data. When queried to do so, Data Analyst will efficiently gather all the datapoints you’ve given it, and present back comprehensive tales of connected market research insights. In an age in which storytelling is perceived to be the backbone of effective market research, this is an incredibly powerful tool.
Imagine getting hours of worktime back that were previously allocated toward ad-hoc analyses and ever-needed insight summarization. This is the benefit that Data Analyst delivers to market researchers today.
Want to take it a step further? With the right ChatGPT setup, a researcher can effectively load all of the charts and graphs they’ve developed into Data Analyst, and provide an actual chatbot to their stakeholders, who can then query the tool themselves and receive near-immediate feedback. The future looks very cool through that lens.
The Cons of ChatGPT Data Analyst for Market Researchers
Unfortunately, Data Analyst simply isn’t good at reading, understanding and analyzing raw survey data, one of the most common datatypes that market researchers encounter. I can’t stress enough, this is Data Analyst’s weakest area for market researchers. Without proper guidance, it literally cannot see columns of data, thus drastically limiting what it is capable of in terms of raw data analytics. For the moment, market researchers still need to pre-process data into tables and slides before they can use Data Analyst efectively.
Meanwhile, it is also limited in what it can derive from actual report data – specifically, it can only read what is included in a report. Its ability to read reports is excellent, but assuming a market researcher wants to be able to use the tool to perform ad hoc queries, the researcher must first have given ChatGPT literally all the answers to the questions that it will be asked. However, as market researchers already know, building a massive series of charts and graphs is not always simple and effortless, and thus researchers are left with the conundrum of deciding how much reporting they need to do themselves before giving the data to ChatGPT. How much is enough? That’s up to individual researchers to decide.
Finally, one very important challenge came about from the tests with Data Analyst: is your data really private?
I asked ChatGPT if it saves data, and this was its response:
“No, ChatGPT does not save or retain any personal data shared during the conversation after the session ends. Each conversation is designed to be independent, and OpenAI's policies are focused on user privacy and data security. The model doesn't have the ability to access or recall personal data shared in previous interactions. However, for training and quality improvement purposes, OpenAI may use conversation logs, but this is done in a way that is compliant with privacy standards and regulations.”
That’s a little deceitful. The truth is that using the Data Analyst plugin requires a user to enable plugins, and this is only possible if the user turns off the data protections that prevent conversation logging. With conversation logging enabled, the conversation is recorded and used to train OpenAI’s models. OpenAI’s claim that this is done in a way that is “compliant with privacy standards and regulations” does not provide nearly enough reassurance. Moreover, plugins are not yet available through the API, so there’s no good way to get access to Data Analyst securely.
The Verdict on ChatGPT’s Data Analyst Tool for Market Researchers
So, is ChatGPT’s Data Analyst the right tool for market researchers today? The short answer is, unfortunately, “not yet”.
While the tool is incredibly impressive in its ability to summarize report data, what market researchers really need is a way to conduct and report on the actual raw data that makes up the aforementioned reports in faster, more efficient way.
At the same time, ChatGPT’s inability to natively keep data private risks exposing confidential research results to the broader internet. Unless a researcher is working with publicly available data, the risk of data exposure is just too great.
However, before we completely dismiss Data Analyst, it’s critical to remember that ChatGPT is constantly changing and evolving, meaning many of the shortcomings this article has identified are likely to be updated throughout the year(s) to come. I personally am excited to experience a future in which AI can help translate market research data into easily accessible insights for stakeholders.
We’ll continue to keep you updated on new tools and tech as it comes out. Stay tuned for more!
About Matt Seltzer
Matt Seltzer is a seasoned Market Researcher with over 15 years of experience in diverse sectors. As Senior VP at Intuify, he leads in client management, team direction, and innovative research solutions. His background includes significant roles at Robert Half, YouTube, and the Las Vegas Convention and Visitors Authority. Matt holds a MA in I/O Psychology and a BS in Marketing. Recognized in the industry, he has received several awards and honors for his contributions.
About Intuify
Intuify, a leader in market research, excels in revealing deep consumer insights through innovative survey technologies and a cognitive science-based approach. The company specializes in a range of services including advanced analytics, marketing strategy, and qualitative research, catering to industries like finance, healthcare, retail, CPG, and technology. Intuify's unique methodologies, such as real-time voice capture and interactive, user-friendly survey designs, enable a deeper understanding of consumer behavior, aiding businesses in strategic decision-making and brand growth. To learn more, visit Intuify.com.