Features 01.02.2023

ChatGPT vs. Human Journalism: How Close is AI to Taking Our Jobs?

Eleanor Dallaway challenged Callum Booth to take on ChatGPT in a journalism challenge.

ChatGPT is a chatbot built on OpenAI that can autogenerate content. Make no mistake; the robots are coming, calling into question the future of journalism. Unwilling to throw in the pen, Eleanor Dallaway challenged Callum Booth to take on ChatGPT in a journalism challenge: Write 500 words on why you should use a password manager. Ready, steady, bot…

It has been an undeniable landmark 12 months for artificial intelligence (AI), one that has caused many sleepless nights for creatives. Take the launch of Dall-E 2 mini as an example. Dall-E 2 is a new AI system that can create realistic images and art from a language description. This challenged what many thought possible with computer-generated art, becoming shockingly good at creating coherent images from prompts, no matter how bizarre. ‘Boris eating fish’, anyone?

If upending the world of visual art wasn’t enough, AI also set its sights on the written word. The launch of the ChatGPT large language model (LLM), a deep learning algorithm, stunned people with its ability to pump out (in most cases) rational responses to cues of almost any variety.

It’s jaw-dropping, but we’ve now reached a point where AI and LLMs can produce art and content that outstrips what vast swathes of the human population are capable of.

The question, though, is just how good this technology is.

Being ‘more competent than most people’ doesn’t mean AI is good enough to replace professionals. Put it this way: countless millions are better at football than me, but that doesn’t mean anyone wants to watch those who are just ‘better than me’ play.

Here at Assured, we had questions about ChatGPT; just how close is it to replacing human writers? And what does the future of this technology hold? We decided to find the answers by pitching an AI and a human writer against one another.

The methodology

To begin this scientific analysis, we needed a benchmark, so we instructed a human writer (in this case, me, Callum) and an AI to write a 500-word article on the same topic.

To ensure a fair competition, we picked a simple cybersecurity-related prompt: why you and your business should use a password manager.

“Being ‘more competent than most people’ doesn’t mean AI is good enough to replace professionals”

We also required the human writer to create a piece in the same format as ChatGPT. As a journalist, you have a range of tools to improve the readability of an article, including headers, links, and images. As ChatGPT doesn’t output any of these things, we decided the writer couldn’t either.

Our editor, Eleanor Dallaway, usually red-pen-trigger-happy, has also committed to leaving both the human and AI versions of the 500-word password management articles entirely unedited to ensure neither side is given an edge.

To judge the outcome of this battle for the ages (and to ensure neutrality), we invited a selection of experts to offer their opinions and judgement. First things first, though, let’s take a look at the articles produced:

‘Why you and your business should use a password manager’: Human Version

Everything seemed so easy in the internet’s early days. You only had a couple of accounts that needed passwords; maybe it was a new email service, maybe a forum focused on a favourite hobby, or maybe just a game you enjoyed.

 Your login details to the above didn’t matter. You’d simply pick a word or set of digits, smush them in, and were ready to go.

 It wasn’t long before the number of online accounts you needed increased though. You started reusing old passwords. Creating new ones. And forgetting them. Again and again.

 So — just like me — you found a solution: writing all your passwords on a little note somewhere.

 But as time went on, there was a constant nagging voice in your head, you had the feeling that something’s wrong, that this isn’t safe, that there must be a better way. And there is, my friend, there is: password managers.

 Fundamentally, these are apps that store and create secure passwords. They integrate with the majority of browsers and phones, meaning that when you need to enter a site, you enter your master password into the app and the specific credentials you need pop up. It’s that simple.

 I can’t tell you the joy this brings me on a day-to-day basis. No longer do I consult strange text files to find my details, take umpteen guesses at my logins, or have to reset my passwords. They’re just there, accessible and safe.

 Speaking of security, a password manager has a range of benefits over the simple convenience listed above. The apps can not only generate the most secure password possible for each site, but they also stop you from reusing credentials — something vital in today’s world.

 If a popular site is hacked, and your email and password revealed, a common tactic of cyber criminals is to try these precise login details across a variety of other platforms. The hope is you’ve reused the same credentials and they can gain access to more of your accounts.

 A password manager can help you cut this down before it becomes a threat. The usefulness of these apps extend beyond the personal though — and they should be in use at all businesses.

 Let me ask you this: how many times have you needed the login for a shared app, only to have someone email or DM you the details? 

 Structures like this make phishing —the process of using social engineering to get information from people — far more dangerous. All it takes is one person posing as a new member of your team for those details to be taken and hackers to access the software in question.

 Password managers put a stop to this. The majority of them allow you to invite specific people to share login details with, meaning there’s a centralised and secure environment for these tasks. Plus, whenever you update the password, it’s there for all users — all without the need for a mass mail or details scrawled on paper.

 Yes, surviving on the internet isn’t as easy as it was in the early days, but at least we now have tools like password managers that can get us close. 

‘Why you and your business should use a password manager’: AI Version

A password manager is a tool that helps you store and organize all of your login credentials for various websites and applications in one secure location. Using a password manager is a smart move for both individuals and businesses, as it offers a number of important benefits.

One of the most significant benefits of using a password manager is that it helps to improve the security of your online accounts. When you use the same password for multiple accounts, it only takes one data breach for all of your accounts to be compromised. A password manager allows you to create unique, complex passwords for each of your accounts, which makes it much more difficult for hackers to gain access to your sensitive information.

Another benefit of using a password manager is that it can save you time and hassle. Instead of having to remember dozens of different login credentials, you can store them all in one place and easily access them when you need them. This can be especially helpful for businesses, as it can streamline the login process for employees and reduce the risk of lost or forgotten login credentials.

A password manager can also be a valuable tool for businesses in terms of compliance. Many businesses are subject to regulations that require them to implement certain security measures, such as password protection. By using a password manager, businesses can ensure that their employees are using strong, unique passwords, which can help them to meet these regulatory requirements.

Another advantage of using a password manager is that it can be synced across multiple devices and can be accessed from anywhere. This is particularly useful for businesses, as it allows employees to access their login credentials from anywhere and work remotely. This can help to increase productivity and flexibility.

In conclusion, a password manager is a valuable tool for both individuals and businesses. It can help to improve the security of online accounts, save time and hassle, and meet regulatory requirements. Businesses that use a password manager can enjoy increased productivity and flexibility as well. With the increasing number of cyber threats, it’s important to take every step possible to protect your sensitive information and a password manager is an essential tool in that regard. Overall, a password manager is an easy and simple solution to help you and your business stay safe and secure online.

Approaching the analysis

Having read both versions of the article, you’ll already have an opinion on which article you prefer, but that sort of simple gut feeling will not suffice. We need data. Structure. Science. Anything less would be unbecoming.

To help with that, we’re going to break this analysis down into three segments:

  • Readability: This includes vocabulary, writing style, and how much people enjoyed it.
  • Structure: How the piece flows and works as an article.
  • Information: The manner and effectiveness in which the information is transferred.

To try and remove as much bias as possible, we asked three experts to give their thoughts on how each article scored on the above criteria.

#1: Readability

The ChatGPT article “reads as though it was written for a robot, by a robot,” Megan Boyle, head of content marketing at The Audit Lab, told us.

She pointed to the lack of personality in the piece written by AI, saying how this gave her the feeling that the author did not care “about [her], or [her] needs.”

This emotion was echoed by Anne Cutler, the director of global communications at Keeper Security, a company that creates password managers. She stated that the ChatGPT article reads “more like a technical guide than an engaging news article.”

Cutler went on to say that the human-created piece was more creative and engaging, using “relatable examples and first-person experience that all work together to entice the reader.”

Boyle from the Audit Lab holds a similar opinion on the human-written article, believing it “flows in terms of its readability.”

A conclusive victory for humanity in round one!

#2: Structure

The experts had varying opinions on how successful the structure of each piece was. For example, although Boyle thought the human article felt like chatting “with your mates in the pub,” she believed it “[lacked] a certain structure that the ChatGPT article [had],” specifically the clarity of “the definition, the benefits, and the conclusion.”

When we asked content marketer & founder of Movie Metropolis, Adam Brannon, this question, he interpreted it another way. “The human article has much more of a narrative than the [one] written by AI.”

He continued, saying that ChatGPT writes “exactly like we would be taught in school — with minimal deviation from this formula,” implying a sense of monotony.

Keeper Security’s Cutler reinforces this opinion. She pointed to negative aspects in the organisation of ChatGPT’s article, “such as the use of ‘In conclusion’ in the last paragraph,” which “indicates a formulaic approach to structure.”

It’s a close-cut thing, but the edge goes to the human in round two. While ChatGPT produces an easy-to-parse structure, the experts believed it was oversimplified and schematic.

#3: Information

This category is all about how the details of the story are conveyed. In other words, how effectively the knowledge around password managers was passed on.

Boyle from the Audit Lab told us, “the ChatGPT article is factual” and has “the benefits clearly stated one after the other, with a neat conclusion.”

Cutler agrees but also points to the “relatable examples” in the human piece that helps contextualise the information.

Cutler does add that while ChatGPT is generally accurate, “there are certain subtleties that are not.” For example, she points towards a line, “When you use the same password for multiple accounts, it only takes one data breach for all of your accounts to be compromised.”

This, she posits, is “incorrect,” as “only the accounts with the reused password would be compromised, which is not necessarily all of them.”

A minor point, but one that reveals a lack of pure accuracy in the piece.

Despite that, information transfer is an area where ChatGPT shines, so we’ll call this round a draw.

The results

There we have it, a win for the humans! Writers, rejoice; our jobs are safe…for now.

ChatGPT is an incredible bit of technology, but one with severe limitations. It’s terrific at pulling up information but far weaker at stylistic, contextual or creative aspects.

Brannon from Movie Metropolis told me that things like ChatGPT can “help build out article ideas,” but simply “can’t match a good writer.”

This was expanded on by Boyle, who said that AI doesn’t create the kind of content that people want to read. Instead, it works best “[bringing] together the essential information” before a human “takes over to bring the words to life.”

It could be tempting to assume that this gap will be easily filled in the coming years, but I have my doubts.

There’s something ethereal and borderline undefinable about good writing. It plays with established rules, stretching and bending them to entertain and make a point. AI and LLMs excel at remixing what already exists, not creating something entirely new, and bridging this divide will be far more challenging than it seems on paper.

I’m sure this will happen one day, but until then, it’s best to think of ChatGPT like a wordsmith’s calculator: a tool that can make good writers better but will struggle to replace them.

Latest articles

Be an insider. Sign up now!