How Does AI Writing Detection Work And Is It Accurate?

So many tools and websites offer AI writing detection, but what exactly is it? Can you reliably and accurately predict artificially produced content?

Justin Gluska

Updated October 14, 2024

Reading Time: 9 minutes

From the release of ChatGPT to more nuanced writing tools like Jasper and Copy, AI is already one of the most powerful tools in a digital employee's arsenal and they’re poised to change the way we do work.

Though many including myself raise the question:

Can anything written with AI be detectable? The honest answer is not really — and even OpenAI will agree with this statement.

But how does AI writing detection work? AI detection is based on predicting patterns, and as large-language models continue to evolve, it's getting harder to separate these robotic pieces from human-written ones. Let’s talk more about this below.

Why Is AI Text Detection Even Important?

One of the biggest concerns going on is the importance of academic integrity. While AI tools offer numerous benefits, using AI-generated content in academic or professional settings without proper attribution can lead to severe consequences (and that's if one gets caught using them).

It's not that AI-produced content is necessarily bad, but shouldn't you have the option to determine what is and isn't AI writing? What's even crazier is that 65.8% of people believe AI content to be equal to or better than human-written content!

Sometimes academic researchers can't even detect the difference between the two.

The safety implications that could result from a world where ChatGPT-produced thesis papers are published (in fact, this has already happened) could certainly result in some dangerous things. Even if you can use AI to cite sources, you reach an ethical quandary.

Identifying AI text with a ChatGPT detector can help a person evaluate the efficacy of content up to a certain human standard. Unfortunately, it's getting harder with how incredible AI writing has gotten in just the last few months.

What Does AI Writing Detection Mean?

At its core, AI writing detection relies on reverse engineering language patterns to determine predictive text. This means that the machine breaks down a piece of text & then uses algorithms to detect patterns within those words.

If a pattern is easier to identify – it's more in tune with what an AI would write, increasing the odds it was written by AI.

The first letter in AI is actually really important – artificial. That's what separates humans from machines. AI writing detection is further based on noticing the differences in how words are arranged and used.

Machines write text based on the billions of data & patterns it was trained on, while natural human writing utilizes an aspect of creativity that can't easily be reproduced by one of these bots. More on that later.AI detection is also about context. As persuasive as it may seem, machines don't actually understand the meaning of words, but they can identify patterns that are commonly used really well. They can also pick up on repeated phrases & words, which is often a tell-tale sign of automation or copy-pasting.

Is AI Writing Detection Accurate?

No, not really. AI Writing detection tools are really just a prediction giving insight into where writing came from. If you suspect writing as being AI-written and run it through a detector that says 100% AI, it probably was written by an AI tool like ChatGPT.

If you see anything under 80% I wouldn't be too confident about accusing it as being AI. There's just too many unknown and unprovable variables to account for. Just look at what's happening to students across the world. 

We also tested eight of the most popular AI detectors in 2024, and guess what? None of them even came remotely close to a 100% accuracy. Take all the results you see with a grain of salt, as they really are just predictions.

But there have even been tools that can bypass AI detection. Tools like Undetectable AI and HideMyAI can get around detectors, showing low accuracy of even the strictest AI detection tools.

How Do AI Detectors Work

So now that we've gone over why detection is important and a little bit about what it means – we'll explain what goes on behind the scenes. 

Tools that "predict" AI content are largely based on analyzing the context to the left of the following word.

Imagine the sentence "The best part of my day is when I wake up for ___." In this example, work is the most commonly predicted word based on the 117 million data points the GPT-2 language model was trained on.

The AI model will think back to all of its training data, then identify and analyze patterns in the context of the word set. It might know, for example, that the word "day" is often used after the words "best" and "part". The algorithm will then calculate the likelihood of each word being the next predicted word, based on these contexts.

Based on training data, the word work had a 41% chance of occurring (the highest probability compared to other words), so it predicted it.

Example of AI prediction showing a 41% chance of the word work being predicted – tested with GLTR

Perplexity & Burstiness

Some other words you might see used when working with an AI detector are perplexity and burstiness.

Perplexity measures the unpredictability of a text. In simpler terms, it measures how often a piece of text might confuse or 'perplex' a reader. AI models strive for low perplexity, aiming for content that reads smoothly and logically.

Human writing often has higher perplexity, with more creative language choices and occasional errors.

Burstiness, meanwhile, evaluates the variation in sentence structure and length.

Temperature

An important thing to understand when working with AI-generated text is the concept of temperature. Temperature probability is a measure of the randomness of predictions. If the temperature is low, a model will probably output the most correct text, but it will be quite boring and monotonous as it has a smaller degree of variation.

If temperature probability is high, the generated text will be more diverse – but includes a higher chance of the model producing grammar mistakes or straight nonsense.

Consumer-facing AI text generation tools like Jasper and ChatGPT seem to err on the side of caution. Although ChatGPT responses produce larger variations than what Jasper does, they are still fairly predictable models.

If you're using an online tool to help write content, you're working with pre-trained models (which generally set conservative temperature probabilities to reduce mass errors)

Low vs High temperature variables for AI (low has little variability, high has high variability)

So after calculating this for a single sentence, keep going with the rest of your text. If a sampled piece of text consistently selects the most predictable word throughout paragraphs, you're almost certainly working with artificially generated text.

Think about it from a personal perspective – the best writers often make use of complex language and explain things in unpredictable, creative ways. Artificial writing doesn't.

As language models become more and more complex, predicting AI based on the context of words will become a lot harder. The more data in a set, the more variability in generations. But for now, you could follow this pattern to analyze large chunks of text.

It's extremely simple in concept: To what extent can an AI model predictively regenerate a given example of text?

Best Tools To Detect AI Writing

Besides using math, there are grammatical and syntactical ways you can help identify if something was written with AI, but you could do that just from reading.

So how do you determine what percentage chance the context of a word has? Well for starters, you could use a few online tools.

We wrote a larger article on how to detect AI content, but depending on what kind of writing you're checking you could use CopyLeaks (free) or Originality (paid).

If you want to check academic, industry, or professional content (especially in mass) – look into Copyleaks. It's free and is better than most other tools out there. We wrote a full review testing it out in much depth.

ChatGPT generating a sample paragraph
CopyLeaks AI detector returning a sample of ChatGPT text as being AI-generated

If you want to check for marketing or copywriting material that got forged using ChatGPT, I'd say Originality is a more in-depth and comprehensive choice.

Originality lets you check for AI, plagiarism, and gives you a percentage that it believes a block of text was written with AI. Check out our full review to see everything it can do.

Originality AI returning a sample of ChatGPT text as being AI-generated

Then, we have Sapling. I like this one as a replacement for Copyleaks because they have an AI likelihood score even if they’re free (something that Copyleaks removed for free users). I also find it to be really accurate for detecting machine-generated text.

Like I said earlier, we tested popular AI detection tools earlier this year, so I’d be remiss if I didn’t mention the one we found to be most accurate: Winston AI. It’s by far the most accurate at finding out if a piece of writing came from AI, even if it’s tweaked by AI humanizers like Undetectable AI. 

With an accuracy of more than 91% for true positive testing, it’s hard to fool Winston.

Detecting AI Writing Manually

Beyond automated tools, you could do a little more digging yourself. I won't lie – it's very hard at first. It's getting extremely difficult to detect since these generators are just able to write more and more like humans.

If you're looking at poorly-generated AI writing, these texts are often written in a monotonous tone, have predictable word choices, are repetitive, and lack bold, original statements.

AI-generated content might lean heavily on hedging phrases or display inconsistencies in voice and style.

The Future of AI Writing & Content Detection

Whether you like it or not, there's really no guaranteed way of determining if something was written with AI at this point in time. After ChatGPT went viral two years ago, tons of questions and concerns were raised about how this will impact the world: education, industry, and even literature.

The realm of AI isn't limited to text. AI-generated image and video generators are on the rise with Midjourney and Sora, making it super important to detect AI-created visuals, especially with the proliferation of 'deepfakes'.

Although a guest researcher at OpenAI revealed that they're developing a tool for "statistically watermarking the outputs of an AI text system." Whenever a system generates text, the tool would stamp an "unnoticeable secret signal" indicating where the text came from.

If other companies follow suit, we might mitigate some of the ethical dilemmas raised by this new technology. I'm just not sure if it's even mathematically possible to do.

Regardless, an unpredictable storm started and won't be over for a damn long time! Generative writing tools are only going to get more nuanced, more creative, and eventually more complex.

For now, it's best to use your intuition combined with detection tools if you're skeptical. The next few years are going to be very interesting and fun to see what's in store. How long until artificial intelligence can seamlessly integrate with our society? We honestly might already be there.

Want to Learn Even More?

If you enjoyed this article, subscribe to our free newsletter where we share tips & tricks on how to use tech & AI to grow and optimize your business, career, and life.


Written by Justin Gluska

Justin is the founder of Gold Penguin, a business technology blog that helps people start, grow, and scale their business using AI. The world is changing and he believes it's best to make use of the new technology that is starting to change the world. If it can help you make more money or save you time, he'll write about it!

Subscribe
Notify of
guest

1 Comment
Most Voted
Newest Oldest
Inline Feedbacks
View all comments