New Study Finds Writing by ChatGPT Makes Complaints More Convincing

A new study shows that consumer complaints written with the assistance of ChatGPT are more likely to receive relief from financial firms, providing some of the first evidence that AI can improve persuasiveness in human communication.

Justin Gluska

Updated December 1, 2023

The center of a grocery store where a man is complaining to automated robot, made with DALL-E 3

The center of a grocery store where a man is complaining to automated robot, made with DALL-E 3

Reading Time: 2 minutes

A new study published in arXiv provides some of the first empirical evidence that large language models like ChatGPT can be used to improve the persuasiveness of messages in human communication. The research focused specifically on whether consumers were using ChatGPT to write more effective complaints to financial companies.

Analyzing a database of over 780,000 consumer complaints filed with the Consumer Financial Protection Bureau (CFPB), the authors found that the number of complaints likely written with assistance from AI surged after the release of ChatGPT in late 2022. Using an AI detection tool called Winston AI, they identified which complaints had a high probability of being written with ChatGPT.

The study then compared the linguistic features between complaints deemed as "Likely-AI" versus those by "Likely-Humans." Key findings showed Likely-AI complaints were more coherent and polite, conveyed more negative sentiment, and were less readable.

Financial firms were over 5% more likely to provide relief for Likely-AI complaints, even after controlling for the complaint sub-issues and linguistic qualities.

To test whether using ChatGPT could directly cause an increase in complaint persuasiveness, the researchers conducted an online experiment. They had participants evaluate financial complaints either in their original form or after being edited by ChatGPT to enhance qualities like clarity, coherence, and professionalism.

The results revealed that hypothetical compensation likelihood was higher for ChatGPT-edited complaints compared to original ones.

This points in the direction that complaints edited by ChatGPT are more effective in achieving compensation than complaints in their original, unedited form.

This early research highlights ChatGPT's ability to improve the persuasiveness of human communication in real-world settings like consumer finance. The authors say that large language models could have transformative impacts on communication outcomes ranging far beyond just writing more compelling complaints.

However, more studies are still needed to fully understand the breadth of these potential influences on human behavior.

Want to Learn Even More?

If you enjoyed this article, subscribe to our free newsletter where we share tips & tricks on how to use tech & AI to grow and optimize your business, career, and life.


Written by Justin Gluska

Justin is the founder of Gold Penguin, a business technology blog that helps people start, grow, and scale their business using AI. The world is changing and he believes it's best to make use of the new technology that is starting to change the world. If it can help you make more money or save you time, he'll write about it!

Subscribe
Notify of
guest

0 Comments
Most Voted
Newest Oldest
Inline Feedbacks
View all comments