Editor Vs AI: ChatGPT-4 Tries a Chicago Style Workout

Editor Vs AI: ChatGPT-4 Tries a Chicago Style Workout

Earlier this month, ChatGPT-4 was released. That represented an overhaul and massive expansion of the AI’s training set. The previous posts in this series put the previous release, ChatGPT-3, through the paces. So it’s time to see how the new version compares!

This test doesn’t cover everything tested in the series so far, because even compiling just these results took several hours and I have massive paying deadlines trying to eat me. For this test, I tried a sampling of questions from the Chicago Style Workouts (in blue) and two different prompts/questions to ChatGPT-4:

1) Explain the grammar errors in this.

2) Correct the grammar errors in this.

The results were quite different. Click on the “results” below each sentence to see how the AI compares to your own answers. Links in the verdict lead to explanations from the Chicago Manual of Style, from whose “style workouts” these samples sentences are taken.

The Score

Score
ChatGPT-31/10
ChatGPT-43/10
Correct answers appear in boldface in the hidden answers below.

Valued for their keen sense of smell, trainers recommend beagles for airport security work.

After subtracting his expenses from his estimated income, Omar’s budget had to be revised.

Writing out the answers, it was clear that I would ace the exam.

Peter and Harriet’s correspondence were both in the file.

Hilda and his vacation.

The audience showed their appreciation. The audience rushed back to its seats.

We never fully reconciled with the lack of Wi-Fi in our basement apartment.

I have been to both Harper’s Ferry and Pike’s Peak.

Twitter is a popular, if controversial, source of news.

The tortoise finished one minute, thirty seconds ahead of the hare.

Conclusion (This time)

It’s both interesting and confusing that ChatGPT-4 gave better explanations of the grammar errors when it was asked only to correct the sentences than it did when it was actually asked to explain the problems. Seems like its ‘splaining problem is still there.

While it’s doing better at detecting and fixing grammar problems, the challenge for the user remains knowing how to get the right answer out of it, as its results are far from consistent across differently worded queries. This still isn’t a quick fix! While the chat bot answers at the speed of faked typing (i.e., pretty fast), the time it takes to form the right queries, revise until you get an answer you like, then revise those answers for accuracy and put them in the form you need (as for this blog post) is more than it would take for me to write it without its ‘halp.’

VERDICT: ChatGPT-4 isn’t yet ready to hand over a document and say “can you look this over?” the way you would with a real-live editor. Not even considering that editing is about far, far more than correcting grammar. The full list of concerns and the standards editors work toward is available via Editors Canada, which they summarize as this: “The goal of editing is to ensure that the material is consistent and correct and that its content, language, style, and design suit its purpose and meet the needs of its audience.”

Image by rawpixel.com on Freepik

Leave a Reply