Gold Penguin Logo with Text

How To Check If Something Was Written with AI (ChatGPT)

With AI taking the world by storm, many have started to question the authenticity of what they're reading online. While AI writing detection tools aren't 100% accurate, there are a few combined methods you could use to help decipher the origin of what you're reading.
Updated December 22, 2023
An image of a cute robot writing on a piece of paper, made with DALL-E
An image of a cute robot writing on a piece of paper, made with DALL-E

Anyone who has spent a few minutes playing around with ChatGPT has most likely been amazed. But we're starting to see a lot more AI-written content – and it's getting harder and harder to tell which have actually been written by people like you and me.

Does it matter? Well if you're reading this, my guess is you are definitely concerned. I know I am. I work with writers every single day. I can't stand AI-generated writing when someone is supposed to write something on their own.

Some articles, essays, and reports have slipped through the cracks to publish robotic diarrhea... and it's quite concerning. I also won't sit here and say I hate ChatGPT, it's actually the opposite – I use it every day.

But anyone should still be able to check if something was created using the help of AI regardless of whether it was allowed or not.

With the release of GPT-4, it got even harder to decipher AI. The honest truth that you might not hear anywhere else is there is no one-size-fits-all approach to predicting AI, you cannot prove it. You can't "detect" AI, you can only predict it.

Anyone who claims otherwise is lying. This is not plagiarism, these are words on a screen. You can't reference words from a massive dataset because that's just not how it works.

Even the FTC put out an official statement about it. At the end of the day, you're basically trying to decipher between patterns of words that AI tends to write.

It's not always clear, and it's definitely not provable, but there are tools that help assist this process. After over a year of researching how to detect AI, here are my technical and non-technical methods you can use to check if something was written or generated using AI.

How To Tell If An Article Was Written With AI

Beyond the realm of Google, academics & other professionals have seen a huge surge in AI-generated content. So, whether you've come across content in an academic, professional, or casual setting, you might want a way to validate if another human wrote certain content.

Detecting AI-generated content requires multiple samples of writing, various tools, and still involves an aspect of predictability. Please do not rely on a single method of AI content detection to claim something was written with AI. It would be disingenuous to you and the person whose writing you are accusing.

I still find myself getting stumped depending on the complexity of the AI used, especially as AI gets better.

Use The CopyLeaks AI Detector

A free AI detector that's popped up with good reliability has been Copyleaks. The detector alerts you if it believes something is AI-written or human-generated and doesn't add any other fluff.

If you hover over sections of text that you think are suspicious (especially text highlighted in red), you'll be able to see a percentage breakdown. The tool supports GPT-4 in addition to older GPT versions and offers a basic & enhanced detection model.

They also have a free Chrome extension to check directly within your browser. It's free to check casual amounts of writing but requires a paid plan if you're looking to scan tons of documents quickly.

Copyleaks AI detector input box showing certain sentences in a paragraph as being written (or at least flagged as written) with AI

You'll be asked to create an account if you switch to the enhanced model. It doesn't seem to change anything on the surface or describe how anything is AI. You can check out their published report if you want to read about their accuracy studies.

Use Undetectable AI's Multi-Detection Tool

Undetectable AI is my next suggestion to help predict if something was written with AI. The tool works by checking content through a fine-tuned model that’s been trained off batched documents submitted to each of the AI detectors they feature (Originality, GPTZero, etc).

Behind the scenes, the tool assigns a likelihood based on its training to give a predictable result based on all the tested content. So when using Undetectable, the tool basically checks the likelihood of returning positive for AI-writing based on 8 different variations of detectors at once.

To use Undetectable's AI Checker, paste your sample of writing inside the input box & submit it for testing! You'll see results from popular detection tools like GPTZero, Writer, Crossplag, Copyleaks, Sapling, Content At Scale, Originality, and ZeroGPT.

Undetectable AI returning AI detected text based on ChatGPT writing. All the detectors: GPTZero, OpenAI, Writer, Crossplag, Copyleaks, Sapling, ContentAtScale, and ZeroGPT.

Originality.ai's Detector & Text Visualizer

If you want to go a step further than testing your article across various detection tools, you could use Originality AI to both check & visualize the writing progression. Originality is the harshest AI detection software I've ever used (take that as you wish).

The text visualizer feature is what sets it apart from many other AI writing detectors. If you are getting anything submitted to you through Google Docs, you can check the writing with Originality & then rebuild the article using their visualizer to see if it involved a lot of copy-pasting.

It looks something like this:

Originality.ai's Text Visualizer tool showing a writer copying & pasting writing into a document

Combine this with their writing detection tool and you'll have some really good intuition as to the origins of your suspected writing. In the example above, I actually gave a task to a writer I hired and they used AI to generate about half of it.

You can see it clearly when things get copied and pasted before getting tweaked.

Originality uses a combination of GPT-4 and other natural language models (all trained on a massive amount of data) to determine if submitted writing seems predictable.

With pricing starting at around 0.01 per 100 words, it's pretty reasonable if you're looking for a more professional, industry-level content detection checker. If you subscribe to their monthly plan you'll get a bigger discount. I've had good luck with it and will continue to use it when checking production-level copy.

You can visualize writing like before or can simply paste your text into the input box like all of the other tools. As a bonus feature, plagiarism also gets detected by default.

Remember, 5% AI doesn't mean 5% of the sample was written with AI. It means if you flipped 100 coins to predict whether something was written with AI, the detection tool would guess it was AI 5 out of those 100 times.

Teachers have been confusing these percentage values, and it's ended up getting students in trouble, which hasn't been too good to hear.

Screenshot of Originality AI showing pasted ChatGPT writing as being 100% AI writing (which is true)
Screenshot of Originality AI showing my own writing as being 0% AI writing (which is true)

Regarding plagiarism, it's also very impressive. Originality was able to find the exact blog I "copied" the content from and marked the text as being copied from a website (this one!!!). For what it's worth, combining AI detection with a plagiarism checker is an additional measure to be even more confident about the origins of written content.

Plagiarism line by line detection score using originality AI

Originality has been my go-to tool for anyone looking to bulk test writing.

They will also keep your scans saved in your account dashboard for easy access in the future.

Acceptable Detection Scores

According to the CEO of Originality, if the content is consistently ranking under 10%, it is almost certainly in the clear! Only when content rises close to 40 or 50% AI is when you should begin to get suspicious about its origins.

The longer sample you input increases the chance of detection being more reliable (larger sample sizes = more reliable detection) – and reliability doesn't mean accuracy!

Also, the more content you scan by the same writer, the better you will know when deciding if their writing is legitimate.

Just be careful as some results end up with false positives and false negatives. It is far better to review a series of articles and make a call on a writer/service compared to passing judgment on a single article or text snippet.

Another Way Is Using GPTZero

I like GPTZero because they seem to be one of the only AI detection companies that really cares about what they flag. While they can't promise 100% accurate detection, they only tend to mark something as AI if they're confident about it. You can read our full review if you want to learn more.

They focus more on academic and educational writing, with a goal of being used in the classroom. The tool is run by a team of talented ML & software engineers and built on 7 "components" of tech, likely making it the most accurate and reliable AI detection tool that is publicly available today. You can also upload files to it, which makes it even more efficient.

Content at Scale AI Detector (casual writing & free)

The team over at Content at Scale released a free AI detector that is also super quick and efficient. It can also test up to 25 thousand characters at a time (that's nearly 4000 words!)

To use the tool, paste the writing into the detection field and submit it for detection. In just a few seconds you'll see a human content score (indicating the likelihood that a human wrote a sample of text) and you'll also see a line-by-line breakdown highlighting what parts of your content have been flagged as suspicious or blatant AI.

A big part of how AI prediction works is by trying to recreate patterns. Patterns are great indicators because AI generators are literally trained to recognize them to produce what "fits" existing patterns the best. The more your text matches existing formats of writing, the higher the probability it was generated.

Below are two screenshots of a ChatGPT output compared to human writing. After testing, you'll also see a predictability, probability, and pattern score. These scores are a simplified explanation of what's going on behind the scenes. Human-produced writing is not very predictable because it doesn't always follow patterns. AI writing is the opposite, it only knows patterns.

Read these two excerpts and see if you could determine the difference in the writing. The first one seems very professional, but you can almost feel what the next sentence is going to be about. The human result is a lot more sporadic. It's still good writing – it's just got more creativity in it. Check out Content at Scale if you want a highly accurate, line by line explanation of what's going on

a mix of both human-generated and AI content detected with content at scale AI detector (50% likely)
likely human-generated content detected with content at scale AI detector (100% likely)

Sapling's AI Detector

I’ve never used an AI detector quite like Sapling AI. On a surface level, it looks simple and similar to GPTZero and Copyleaks. However, don’t let its looks fool you. This is definitely one of the strictest detectors I’ve ever used, and it’s only getting better with its latest update.

This tool works like every other detector in the market. Simply paste your text, wait for a few seconds, and you’ll get an output consisting of an AI likelihood percentage and highlights on the parts of the text that are likely to be non-human.

Not only can it detect ChatGPT and Claude content with precision, it can also pinpoint AI text that was transformed using bypass tools such as Undetectable AI and HideMyAI. 

This detector was built by former developers from Stanford, UCB, Google, and Meta, and it really shows in the robustness of its AI detection.

The Technical & Syntactical Signs

The next way to tell if an AI has generated a piece of content is to look at the technical aspects of the writing. This isn't as concrete & may seem obvious, but if you're having trouble with the previous tools or just want to break down further writing you've come across, you should look deep at the content. Here are a few things to look for:

1. Watch out for Transitional Words. ChatGPT loves to use transitional words. Every few lines it'll insert another one. Things like Furthermore, Additionally, Moreover, Consequently, and Hence are frequently written but don't always appear in human writing. We don't really "transition" our writing unless it's something more formal or professional.

2. Big vocabulary words are suspicious. Utilized, implemented, leveraged, elucidated, and ascertained are often overused, but what human talks like that in a general article they would write? Almost none.

In human conversations, simpler terms like used, explained and found are more common and relatable.

If you've tested content using one of the detection tools and if the content is creative & unique, I'd say it's in the clear. You need to look further into the technical content that comes off as confidently fishy.

3. Repetition of words and phrases: Another way to spot AI-generated content is by looking for repetition of words and phrases. This is the result of the AI trying to fill up space with relevant keywords (aka – it doesn't really know what it's talking about).

So, if you're reading an article and it feels like the same word is being used over and over again, there's a higher chance an AI wrote it. Some of the spammy AI-generation SEO tools love keyword-stuffing articles. Keyword stuffing is when you repeat a word or phrase so many times that it sounds unnatural.

Some articles have their target keyword in what feels like every other sentence. Once you spot it, you won't be able to focus on the article. It's also extremely off-putting for readers.

4. Lack of analysis: A third way to tell if an AI wrote an article is if it lacks complex analysis. This is because machines are good at collecting data, but they're not so good at turning it into something meaningful.

If you're reading an article and it feels like it's just a list of facts with no real insight or analysis, there's an even higher chance it was written with AI. With ChatGPT, we're nearing the point where AI is able to start to analyze writing, but I still find responses to be very "robotic."

People are starting to use AI to reply to tweets but don't realize how painfully cookie-cutter their responses are! You'll notice AI-generated writing is a lot better for static writing (like about history, facts, etc) compared to creative or analytical writing. The more information a topic has, the better AI can write & manipulate it.

5. Hallucination of Inaccurate data: This one is more common in AI-generated product descriptions but can also be found in blog posts and articles. THIS IS A HUGE INDICATOR! Since machines collect data from various sources, they sometimes make mistakes or simply aren't updated.

If a machine doesn't know something but is required to give an output, it'll predict numbers based on patterns (which aren't accurate). This happens all the time and is (in my opinion) the easiest predictor of AI.

So, if you're reading an article and you spot several discrepancies between the facts and the numbers, you can be very confident that what you just read was written using AI. If you come across spammy content, report it to Google. Save someone else the pain of having to waste their time reading something that is clearly inaccurate!

Verify The Sources & Author Credibility

This one might seem a bit unnecessary for a single blog, but it's still worth mentioning. If you're reading an article and the domain seems to be randomly associated with the content posted, that's your first red flag.

But more importantly, you should check the sources that are being used in the article (if any). If an author is using sources from questionable websites or simply declares things without any source, it's either the author isn't doing their research or could simply be automating a bunch of AI-generated content.

If you're trying to check an article on Google, click the menu and see all the information Google has on the site. Here's what that looks like for us:

Viewing history that Google has on Gold Penguin directly on their search

You can see we were indexed by Google about 2 years ago, but Google doesn't really know too much about us yet. Combine this with your own judgment to make your decision if something seems to be trustworthy.

Google showing when it first indexed Gold Penguin's website & that it can't find much information on the site

OpenAI Even Discontinued Their Official AI Detector

The company behind the madness themselves, OpenAI, released a tool a few months ago to help detect writing. Using the official tool, OpenAI had initially claimed only 26% of AI-written samples they tested were identified properly as AI.

With some doubt from the online marketing & writing community about the tool's accuracy, it seems like they were actually correct as OpenAI discontinued & removed their own AI detection tool from the website on July 20th, 2023:

As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy. We are working to incorporate feedback and are currently researching more effective provenance techniques for text, and have made a commitment to develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated.

https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text

My initial thoughts on the detection tool were it really looked like a coin toss. I tested many outputs from ChatGPT and got "unable to tell" and "unlikely written by AI." I never used the tool.

Gold Penguin's AI Detection Tool

A few weeks ago I got together with a development team and had them create us our very own AI detection tool. I was not happy using tools that over-detected a lot of writing. If it's THAT hard to decipher if something was written with AI or not – I'll just leave it as it is.

I didn't want anything to get detected when it wasn't, even if that meant I would let some actual AI get through. But that's fine, this technology can't accurately detect everything anyways.

The tool is free and, like every other tool, should only be taken with a grain of salt. It's great for letting you know if something is OBVIOUSLY AI, but for more intricate tools you should probably use another tool.

Gold Penguin's very own AI writing detection tool that won't overdetect content as being written with AI when it's not

What's Going To Happen Next?

It's not the easiest to tell if an AI wrote an article because you truthfully can't be sure. To make matters worse, AI just gets so much better each day. What is GPT-5 going to look like in a few months? I can't even imagine.

That being said, if you're questioning whether or not an article was written by an AI, your best bet is to use a combination of all of these tools combined with your own judgment. Test multiple papers by the same author for further reliability.

Make sure to remember to take the results you see with a grain of salt. Nothing you see is conclusive in any way, shape, or form since there's no concrete way to detect AI. Keep in mind that what you're working with leaves no watermark; you're just looking at words on a screen.

Hopefully these new tools benefit us by allowing skeptics to filter out AI-generated content across the internet, news, and within school systems across the world.

As AI becomes more sophisticated and the line between human and machine-generated content becomes increasingly blurry, it's only a matter of time until everything we reach the point where AI-generated content becomes indistinguishable!

Let's see what the next few months have in store for us all.

Want To Learn Even More?
If you enjoyed this article, subscribe to our free monthly newsletter
where we share tips & tricks on how to use tech & AI to grow and optimize your business, career, and life.
Written by Justin Gluska
Justin is the founder of Gold Penguin, a business technology blog that helps people start, grow, and scale their business using AI. The world is changing and he believes it's best to make use of the new technology that is starting to change the world. If it can help you make more money or save you time, he'll write about it!
Subscribe
Notify of
guest

13 Comments
Most Voted
Newest Oldest
Inline Feedbacks
View all comments
Join Our Newsletter!
If you enjoyed this article, subscribe to our newsletter where we share tips & tricks on how to make use of some incredible AI tools that you can use to grow and optimize a business
magnifiercross