Editor Vs AI: CMOS Grammar

For this investigation in the series, I scraped some grammar examples from the CMOS “Chicago Style Workout” quizzes on their Shop Talk blog. Some test sentences were adapted from questions as not all of them were in sentence form. ChatGPT-3 did the best of all, correctly fixing 33/43 examples when asked to “Fix the grammar in this.” Three of the “errors” tested might be considered a matter of style (a choice), and there’s no way to tell any of these AI (see note at end) that we are looking for them to apply CMOS style. So you might award three more points for those, across the board.

This time, I’ve grouped the test sentences by how well the tools dealt with them. As with other summaries in the series, I have paraphrased the answers/suggestions for the sake of brevity. The answers that matched the CMOS solutions are set in boldface. Links within the CMOS answer for each example will take you to the relevant quiz page. I note that Word now phrases its suggestions as questions, absolving itself of responsibility; sometimes I kept that wording.

The sentences and results are hidden for ease of reading. Click the “The test sentence corrections” to reveal them in each section.

The Score

	Missed	Misidentified	Score
Word	29	10	3/43
Grammarly	29	3	10/43
ChatGPT-3	6	4+	33/43

*One example contained no errors, so the numbers above don’t sum up neatly.
+ = Also got a few corrections half right (e.g., there were two errors).

What AI gets right

Sometimes our automated assistants are somewhat helpful! Word lags a bit behind the performance of the others on these examples, but these misspellings and comma errors were the most consistently corrected by them.

What AI can’t detect

Dangling modifiers slip right by these tools. Granted, a lot of readers don’t even blink at these. A dangler occurs because the right subject is missing, so the modifier refers to the wrong thing. There’s often no confusion caused by danglers, as a misreading would be absurd. (As in some examples here.) But we sure do see a lot of them in the wild! Here’s another explainer.

Word’s troubled obsession with passive voice

Word is like that desperately in-over-their-head Jeopardy contestant who is constantly slamming their buzzer and saying “What is passive voice, Alex?” Remember, passive voice is a grammatical construction, not an error. Editors leave it alone when necessary or when it fits the flow. Some of these are passive, but not all of them!

When ChatGPT has your back

In addition to its successes in other categories, here are some examples where only ChatGPT-3 knew what the problem was.

Where they disagree

These challenges fell into the win some, lose some department for the AI.

Matters of (CMOS) style

The quiz items in this section did not necessarily contain outright errors but, rather, they are style choices particular to CMOS. To be fair, the AIs didn’t know we wanted these in CMOS style. There’s no reason they would have applied it.

Defining AI

The term “artificial intelligence” is used pretty loosely and liberally these days. Many are using it to mean any algorithm that can do what it used to take a human to do. But that is too broad for me, as even a simple calculator might qualify. To be “intelligent,” that algorithm must learn as it goes, improving itself without further human instruction. A user clicking “add to dictionary” isn’t enough to qualify Spellcheck as self-learning, for example. In this sense, Spellcheck and PerfectIt are not AI. Grammarly may be AI; I don’t know how it works. The translator app on an iPhone isn’t likely to fit this definition either, as it uses crowd-souced training to improve. The predictive text on phones, however, is more like AI than those other examples, as its predictions get better the more you use it. It “learns” your style (or at least your tendencies).

But sometimes we use terms that capture readers’ attention rather than only what is accurate, like when I answer the call to “look something over” instead of the precise “stylistic edit” or whatever they actually want. And as for why I keep saying “an” or “this” GPT, it’s because there are several iterations of Generative Pre-trained Transformer out there. ChatGPT-3 is just the (free) instance that I’ve been testing.

This slide set provides a great overview of the various queries and answers that ChatGPT can provide.

---

Be sure to check out the other tests in this series as well as comparing these results against the next iteration: ChatGPT-4.

The image for this post was created using the Dall-e 2 AI art generator and the author’s prompt of “a robot reading a big orange book”