Earlier this month, ChatGPT-4 was released. That represented an overhaul and massive expansion of the AI’s training set. The previous posts in this series put the previous release, ChatGPT-3, through the paces. So it’s time to see how the new version compares!
This test doesn’t cover everything tested in the series so far, because even compiling just these results took several hours and I have massive paying deadlines trying to eat me. For this test, I tried a sampling of questions from the Chicago Style Workouts (in blue) and two different prompts/questions to ChatGPT-4:
1) Explain the grammar errors in this.
2) Correct the grammar errors in this.
The results were quite different. Click on the “results” below each sentence to see how the AI compares to your own answers. Links in the verdict lead to explanations from the Chicago Manual of Style, from whose “style workouts” these samples sentences are taken.
The Score
| Score | |
| ChatGPT-3 | 1/10 |
| ChatGPT-4 | 3/10 |
Valued for their keen sense of smell, trainers recommend beagles for airport security work.
What ChatGPT Said
ChatGPT-3: capitalize Beagles
ChatGPT-4: 1) no errors
2) rewrote without the misplaced modifier: Trainers recommend beagles for airport security work because of their keen sense of smell.
VERDICT: On the right track!
After subtracting his expenses from his estimated income, Omar’s budget had to be revised.
What ChatGPT Said
ChatGPT-3: Omar had to revise his budget
ChatGPT-4: 2) revise his
1) was quite lengthy:
The problem with this sentence is a misplaced modifier. The modifier “After subtracting his expenses from his estimated income” modifies “Omar’s budget,” which makes it seem as though the budget was the one doing the subtracting. To fix the sentence, the modifier needs to be placed closer to what it is modifying. For example: “After Omar subtracted his expenses from his estimated income, his budget had to be revised.”
VERDICT: On a roll!
Writing out the answers, it was clear that I would ace the exam.
What ChatGPT Said
ChatGPT-3: became clear
ChatGPT-4: 2) As I wrote out the answers, it became clear that I would ace the exam.
1) The problem with this sentence is also a misplaced modifier. The modifier “Writing out the answers” modifies “I,” which makes it seem as though “I” was doing the writing and also realizing that he/she would ace the exam.* To fix the sentence, the modifier needs to be placed closer to what it is modifying.** For example: “As I wrote out the answers, it was clear that I would ace the exam.”
VERDICT: *Um, yeah, I believe that is what’s happening, and so does ChatGPT, evidenced by its suggested change.
**Distance isn’t usually the problem with misplacing a modifier, and it’s not the correction it gave. The correction it made was adding the actor that did the writing: I.
Further: Some editors would argue that “it was clear” is incorrect more or less because of a mismatch between a process of doing (“as I wrote”) and an instantaneous “it was.”
Peter and Harriet’s correspondence were both in the file.
What ChatGPT Said
ChatGPT-3: [no errors flagged]
ChatGPT-4: 1–2) change to was. Explanation: The subject of the sentence is “correspondence,” which is singular, so the verb should also be singular.
VERDICT: Nope! See CMOS 17, paragraph 5.22: If two or more nouns share possession, only the last noun takes the genitive ending: Peter
Hilda and his vacation.
What ChatGPT Said
ChatGPT-3: [no errors flagged]
ChatGPT-4: 1) add were on “Hilda and his were on vacation.”
2) Hilda and her vacation. Explanation: “Hilda” is a female name, so the possessive pronoun should be “her,” not “his.”
VERDICT: 1 is wrong but 2 could be right! Which demonstrates even more why changing this matters: to make it clear how many people there are.
The audience showed their appreciation. The audience rushed back to its seats.
What ChatGPT Said
ChatGPT-3: [no errors flagged]
ChatGPT-4: 1–2) change their to its because “audience” is a singular noun, so the possessive pronoun should be “its,” not “their”
VERDICT: Nope! See CMOS 17, paragraph 5.32: collective noun = singular pronoun if treated as a unit; swap their and its
We never fully reconciled with the lack of Wi-Fi in our basement apartment.
What ChatGPT Said
ChatGPT-3: [no errors flagged]
ChatGPT-4: 1) The sentence is grammatically correct, but adding “ourselves” clarifies who is doing the reconciling.
2) [no change]
VERDICT: Wrong—See CMOS, paragraph 5.195, under “reconcile”: use to instead of with.
I have been to both Harper’s Ferry and Pike’s Peak.
What ChatGPT Said
ChatGPT-3: [no errors flagged]
ChatGPT-4: 1–2) [no change]
VERDICT: No cigar. 🙁 Interestingly, it did not make a change but did say that “Apostrophes are used to indicate possession or contractions, but they are not necessary in this case.” This explanation lands with the right action because the US National Board on Geographic Names forbids apostrophes in the place-names but it didn’t fix it!
Twitter is a popular, if controversial, source of news.
What ChatGPT Said
ChatGPT-3: use albeit instead of if [In addition to missing the error, the suggestion changed the tone in a way that an editor might not want.]
ChatGPT-4: 1) Create a compound word of popular-if-controversial because hyphens are used to join words that function together as a single concept or modifier.
2) [no change]
VERDICT: Please don’t hyphenate this. CMOS says to delete the comma after “controversial” but that’s a style choice.
The tortoise finished one minute, thirty seconds ahead of the hare.
What ChatGPT Said
ChatGPT-3: write and instead of the comma
ChatGPT-4: 1) write and instead of the comma
2) When stating a time period, “and” should be used to separate minutes and seconds.
VERDICT: OpenAI hasn’t declared what it grammar rules it populated ChatGPT-4 with, but CMOS disagrees with the use of and in this way. It wants another comma to follow the seconds. See CMOS 6.19 & CMOS 7.89, sec. 1, “number + noun.”
Conclusion (This time)
It’s both interesting and confusing that ChatGPT-4 gave better explanations of the grammar errors when it was asked only to correct the sentences than it did when it was actually asked to explain the problems. Seems like its ‘splaining problem is still there.
While it’s doing better at detecting and fixing grammar problems, the challenge for the user remains knowing how to get the right answer out of it, as its results are far from consistent across differently worded queries. This still isn’t a quick fix! While the chat bot answers at the speed of faked typing (i.e., pretty fast), the time it takes to form the right queries, revise until you get an answer you like, then revise those answers for accuracy and put them in the form you need (as for this blog post) is more than it would take for me to write it without its ‘halp.’
VERDICT: ChatGPT-4 isn’t yet ready to hand over a document and say “can you look this over?” the way you would with a real-live editor. Not even considering that editing is about far, far more than correcting grammar. The full list of concerns and the standards editors work toward is available via Editors Canada, which they summarize as this: “The goal of editing is to ensure that the material is consistent and correct and that its content, language, style, and design suit its purpose and meet the needs of its audience.”
Image by rawpixel.com on Freepik
