ChatGPT vs. Human Journalism: How Close is AI to Taking Our Jobs?

ChatGPT is a chatbot built on OpenAI that can autogenerate content. Make no mistake; the robots are coming, calling into question the future of journalism. Unwilling to throw in the pen, Eleanor Dallaway challenged Callum Booth to take on ChatGPT in a journalism challenge: Write 500 words on why you should use a password manager. Ready, steady, bot…

It has been an undeniable landmark 12 months for artificial intelligence (AI), one that has caused many sleepless nights for creatives. Take the launch of Dall-E 2 mini as an example. Dall-E 2 is a new AI system that can create realistic images and art from a language description. This challenged what many thought possible with computer-generated art, becoming shockingly good at creating coherent images from prompts, no matter how bizarre. ‘Boris eating fish’, anyone?

If upending the world of visual art wasn’t enough, AI also set its sights on the written word. The launch of the ChatGPT large language model (LLM), a deep learning algorithm, stunned people with its ability to pump out (in most cases) rational responses to cues of almost any variety.

It’s jaw-dropping, but we’ve now reached a point where AI and LLMs can produce art and content that outstrips what vast swathes of the human population are capable of.

The question, though, is just how good this technology is.

Being ‘more competent than most people’ doesn’t mean AI is good enough to replace professionals. Put it this way: countless millions are better at football than me, but that doesn’t mean anyone wants to watch those who are just ‘better than me’ play.

Here at Assured, we had questions about ChatGPT; just how close is it to replacing human writers? And what does the future of this technology hold? We decided to find the answers by pitching an AI and a human writer against one another.

The methodology

To begin this scientific analysis, we needed a benchmark, so we instructed a human writer (in this case, me, Callum) and an AI to write a 500-word article on the same topic.

To ensure a fair competition, we picked a simple cybersecurity-related prompt: why you and your business should use a password manager.

“Being ‘more competent than most people’ doesn’t mean AI is good enough to replace professionals”

We also required the human writer to create a piece in the same format as ChatGPT. As a journalist, you have a range of tools to improve the readability of an article, including headers, links, and images. As ChatGPT doesn’t output any of these things, we decided the writer couldn’t either.

Our editor, Eleanor Dallaway, usually red-pen-trigger-happy, has also committed to leaving both the human and AI versions of the 500-word password management articles entirely unedited to ensure neither side is given an edge.

To judge the outcome of this battle for the ages (and to ensure neutrality), we invited a selection of experts to offer their opinions and judgement. First things first, though, let’s take a look at the articles produced:

Approaching the analysis

Having read both versions of the article, you’ll already have an opinion on which article you prefer, but that sort of simple gut feeling will not suffice. We need data. Structure. Science. Anything less would be unbecoming.

To help with that, we’re going to break this analysis down into three segments:

• Readability: This includes vocabulary, writing style, and how much people enjoyed it.
• Structure: How the piece flows and works as an article.
• Information: The manner and effectiveness in which the information is transferred.

To try and remove as much bias as possible, we asked three experts to give their thoughts on how each article scored on the above criteria.

#1: Readability

The ChatGPT article “reads as though it was written for a robot, by a robot,” Megan Boyle, head of content marketing at The Audit Lab, told us.

She pointed to the lack of personality in the piece written by AI, saying how this gave her the feeling that the author did not care “about [her], or [her] needs.”

This emotion was echoed by Anne Cutler, the director of global communications at Keeper Security, a company that creates password managers. She stated that the ChatGPT article reads “more like a technical guide than an engaging news article.”

Cutler went on to say that the human-created piece was more creative and engaging, using “relatable examples and first-person experience that all work together to entice the reader.”

Boyle from the Audit Lab holds a similar opinion on the human-written article, believing it “flows in terms of its readability.”

A conclusive victory for humanity in round one!

#2: Structure

The experts had varying opinions on how successful the structure of each piece was. For example, although Boyle thought the human article felt like chatting “with your mates in the pub,” she believed it “[lacked] a certain structure that the ChatGPT article [had],” specifically the clarity of “the definition, the benefits, and the conclusion.”

When we asked content marketer & founder of Movie Metropolis, Adam Brannon, this question, he interpreted it another way. “The human article has much more of a narrative than the [one] written by AI.”

He continued, saying that ChatGPT writes “exactly like we would be taught in school — with minimal deviation from this formula,” implying a sense of monotony.

Keeper Security’s Cutler reinforces this opinion. She pointed to negative aspects in the organisation of ChatGPT’s article, “such as the use of ‘In conclusion’ in the last paragraph,” which “indicates a formulaic approach to structure.”

It’s a close-cut thing, but the edge goes to the human in round two. While ChatGPT produces an easy-to-parse structure, the experts believed it was oversimplified and schematic.

#3: Information

This category is all about how the details of the story are conveyed. In other words, how effectively the knowledge around password managers was passed on.

Boyle from the Audit Lab told us, “the ChatGPT article is factual” and has “the benefits clearly stated one after the other, with a neat conclusion.”

Cutler agrees but also points to the “relatable examples” in the human piece that helps contextualise the information.

Cutler does add that while ChatGPT is generally accurate, “there are certain subtleties that are not.” For example, she points towards a line, “When you use the same password for multiple accounts, it only takes one data breach for all of your accounts to be compromised.”

This, she posits, is “incorrect,” as “only the accounts with the reused password would be compromised, which is not necessarily all of them.”

A minor point, but one that reveals a lack of pure accuracy in the piece.

Despite that, information transfer is an area where ChatGPT shines, so we’ll call this round a draw.

The results

There we have it, a win for the humans! Writers, rejoice; our jobs are safe…for now.

ChatGPT is an incredible bit of technology, but one with severe limitations. It’s terrific at pulling up information but far weaker at stylistic, contextual or creative aspects.

Brannon from Movie Metropolis told me that things like ChatGPT can “help build out article ideas,” but simply “can’t match a good writer.”

This was expanded on by Boyle, who said that AI doesn’t create the kind of content that people want to read. Instead, it works best “[bringing] together the essential information” before a human “takes over to bring the words to life.”

It could be tempting to assume that this gap will be easily filled in the coming years, but I have my doubts.

There’s something ethereal and borderline undefinable about good writing. It plays with established rules, stretching and bending them to entertain and make a point. AI and LLMs excel at remixing what already exists, not creating something entirely new, and bridging this divide will be far more challenging than it seems on paper.

I’m sure this will happen one day, but until then, it’s best to think of ChatGPT like a wordsmith’s calculator: a tool that can make good writers better but will struggle to replace them.