Official Note from OpenAI Regarding Their Classifier: As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy. We are working to incorporate feedback and are currently researching more effective provenance techniques for text, and have made a commitment to develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated.
The long awaited official AI detection tool has been released! After a slight 2 month destruction period across the education and publishing industries, there is now a publicly available tool for testing written content. Today, OpenAI released an AI classification & detection tool to help determine if a chunk of writing has been produced artificially or if it was likely written by a human.
What's really interesting about the release of this tool is how seemingly unconfident OpenAI seems to be about it. I think this is because AI text detection is extremely hard to official determine and regardless of predictions, there's still no definitive way to identify AI text since these programs are just trying to predict the syntactical formatting of words in sentences.
The classifier was created to help determine if something was written with AI for purposes including but not limited to: running automated misinformation campaigns, using AI tools for academic dishonesty, and positioning an AI chatbot as being human.
So, how does it work?
Using the OpenAI Text Classifier
Available for free, the AI Text Classifier lets you paste sections of text into an input box and then click a button to detect its origins. You need an account to use it, but you might have created one a month ago to test out ChatGPT. The input works fairly striaghtforward:
Insert any bit of text, whether it's written by yourself, ChatGPT, found in an academic textbook, or by your niece. After this, hit submit to see what the tool thinks of your text. Some of the detection results the classifier will return are:
- The classifier considers the text to be likely AI-generated.
- The classifier considers the text to be possibly AI-generated.
- The classifier considers the text to be unclear if it is AI-generated.
- The classifier considers the text to be unlikely AI-generated.
- The classifier considers the text to be very unlikely AI-generated.
With these in mind, OpenAI is very clear about the limitations and conclusions that should be drawn from using the classifier. Out of everything they've said, here's what stands out the most:
- The classifier had only correctly identified 26% of AI-written text when they tested it on their testable data set .
- False positives also occurred 9% of the time
- Text under 1,000 characters is unable to be detected reliably (and more characters increase the likelihood of prediction accuracy)
- Human-written text may be returned as likely AI written, even if this isn't true
- English text is the most accurate since the tool was mainly trained on english examples
- Extremely predictable and repetitive text is not able to be classified (things that don't change, like a list of United States presidents or historical speeches)
- AI-written text can still fool the classifier, so take results with a grain of salt & utilize multiple detection methods to identify text origins
- For inputs that very different from the classifiers training set, output can be confidentially incorrect.
Highlighting Academic and Professional Integrity
With the explosion in popularity about AI (both text and image generation software) comes the concern about the integrity of produced content. OpenAI has recognized the importance of identifying AI-written text in the realm of academia and classroom settings. They've developed a resource guideline which outlines the capabilities and limitations of the AI text classifier.
OpenAI is also working with educators across the US to learn more about the risks and concerns about the recent improvements in Artificial Intelligence. The company is also open to any resources or programs that educators have found helpful in developing AI literacy programs.
Is This The Best AI Detection Tool?
After testing a few dozen samples of content ranging from clearly AI to academic thesis writing, I would probably have to say not yet. I've had a lot of results return the opposite of what it actually was (AI coming back as human and vice versa). I've had better luck with tools like Originality and even GPTZero.
Beyond AI detection software, there are also many non-concrete characteristics that go into detecting if something was written with AI. These includes things like sentence syntax, vocabulary complexity, and keyword density. We'll probably reach a point sometime over the next few years where writing will be truly indistinguishable. AI is only going to get better from here on out & the world will soon be flooded with its content.
Final Thoughts
It's a bit relieving to see an official resource for AI detection come to life. OpenAI released a classifier a few years back but it hasn't been updated for the newest models that have dominated the internet over the last year. I do want to stress the importance of taking every tool confident enough to detect AI with a grain of salt. Although they are generally very helpful, there really is no provable way of ensuring if content was or was not produced with the help of a robot.
We're entering a really interesting time in both the tech industry and greater world. The next few years is going to be a really interesting ride into the future of so many global industries. Only time will tell where the train is headed & those who adapt now will be the first ones to see the benefits of this world-changing, massive technological breakthrough! What are your thoughts?
