Let’s try this again: I’m excited to have more tools available, but is AI ready to replace a human editor? Last time, I challenged various editing tools to fix “10 common grammar mistakes.” For this next test, I used the first New York Times Copy Edit This! challenge to see what the “editing software” has to say compared to a real-life, human NYT copy editor. To test this, I compiled the test sentences into a (really bad) paragraph and had three “editing tools” offer their suggestions. This time, I’ve organized the results by sample, with the results from each tool grouped below it:
- MS Word’s full Editor which purports to check spelling, grammar, and clarity
- Grammarly
- ChatGPT which is an interactive bot, not a “checker”
- NYT explanation from their quiz, rephrased
Making the ChatGPT AI work involves asking it to explain what is wrong. Different questions yield different results. This time, I asked it “What is the copyediting problem in [the sample text].” (Yes, I chose copyedit as a closed compound for the query to add clarity.) Ask it a different way — like even, “explain” instead of “what is” — and you will get a different answer. The ChatGPT seems to have a better grasp of what it means to copy edit something than most people do. See sample 8 for a case study in this or give it a try yourself!
To keep the length of this post under control, I had intended to only share the samples that Grammarly had an opinion on, but there weren’t enough to create a meaningful post. Some of ChatGPT’s replies were summarized, as it tended to give long, detailed explanations and repeat the sample with its suggested fixes inserted, and that was overly long for an already long blog post. For Word, I turned on every single check within the Editor and reported its findings verbatim.
Overall scoring against the NYT answers.
- 2—Word
- 3—Grammarly
- 4—ChatGPT
- 10—NYT
The NYT site offers an actual quiz with longer explanations, and here be spoilers. So you may want to go try the quiz yourself, first, then see how your assessments compare! You may decide there are other things wrong with the samples, but this post is only comparing the NYT’s own quiz answers to the AI.
Sample 1: The patchwork of laws governing background checks, assault-weapons limits and open-carry practices help explain why people continue to be wounded and killed.
Sample 1 Results
MS Word: Saying who or what did the wounding would be clearer. [aka passive voice, the actor is missing]
Grammarly: The word “assault-weapons” doesn’t seem to fit this context. Consider replacing it with a different one. Funny enough, Grammarly says the hyphenation here in this post is a misspelling.
ChatGPT: The verb “help” does not agree with the subject “patchwork of laws”. “Help” is a verb that indicates that the subject is assisting or supporting something, while the sentence is trying to express that the patchwork of laws is an explanation for the continuation of harm. The verb “explains” should be used instead, as it is more fitting in the context of providing an explanation.
NYT: Make helps singular to align with the singular noun patchwork, not checks, limits or practices.
Notice the elegance of the NYT fix: changing a single letter where the ChatGPT required deleting a whole word and pluralizing another. The other two tools missed this problem entirely though they offered an opinion on what was wrong. The NYT is a newspaper, so its use of passive voice has a sound grounding in not drawing conclusions or assigning actions, which can lead to legal troubles on top of just bad reportage. Using the passive voice isn’t an error, but changing it is often a way to improve clarity and liveliness, if the legal concerns can be addressed another way.
Sample 2: After receiving his doctorate at Princeton, the university hired him as an assistant professor and promoted him to full professor in 1965.
Sample 2 Results
MS Word: [no errors flagged]
Grammarly: [no errors flagged]
ChatGPT: It’s not clear if the same person who received the doctorate and was hired and promoted by Princeton and subsequently “dogged by reports.” Add the person’s name after the first comma, in place of “the university.”
NYT: The university did not receive the degree. Fix this “dangler” by starting it with a separate clause “After he received …”
The GPT bot had a nuanced thought about what was wrong, and it’s got a point. But the NYT solution is again minimal and completely effective. That is the copy editor’s ideal.
Sample 3: He has been dogged by reports — many published in Newsday — that he had received free gifts and vacations from a longtime friend, Harendra Singh, a Long Island restaurateur with about 30 businesses in the area and several government contracts.
Sample 3 Results
MS Word: Hyphenate long-time; saying who or what dogged the person would be clearer [aka passive voice, the actor is missing]
Grammarly: Free gifts may be redundant.
ChatGPT: The subject could be the person who received the gifts and vacations or the person who published the reports. Add a clear subject (meaning, a name).
NYT: All gifts should be free.
Much closer agreement among the AI on this one, and Grammarly gets a point! If you had a problem with who “he” is, remember that the larger context likely makes this crystal clear. Editing sentences in isolation is harder.
Sample 4: The names of the sergeant, whom Chief Nikunen said was an eight-year veteran of the New York Police Department, and the woman were not released.
Sample 4 Results
MS Word: Saying who or what did not release the names would be clearer [aka passive voice, the actor is missing]
Grammarly: The plural verb were does not appear to agree with the singular subject woman.
ChatGPT: Are the sergeant and the woman the same person? Rephrased to specify which person is being referred to in each case as well as whether the “eight-year veteran” is the sergeant or the woman.
NYT: Change whom to who.
Bah! I am of the majority who reject the persnickety whom in all but instances of satire. In this case, it’s not even right! Go to the original quiz to see the NYT’s great tip for understanding why whom is incorrect here.
Sample 5: The total amount available to disperse will be $150 million annually.
Sample 5 Results
MS Word: [no errors flagged]
Grammarly: [no errors flagged]
ChatGPT: Is this the total amount that will be dispersed annually, or is it the total amount that will be available to disperse annually.
NYT: Should be disburse, meaning “pay out.” Disperse means “scatter.”
In addition to not flagging the wrong word, the GPT seems to be confused about how to phrase the answer. I think it’s asking to clarify whether that money is disbursed every year or in chunks over the years. But the grammar in the original clearly means every year. The NYT editor does acknowledge that this is a prevalent word mix-up; it may be on the way to being an acceptable use.
Sample 6: In Tutzing, the Prince’s three-story villa is defended from prying eyes by a fence and hedge more than six-feet tall.
Sample 6 Results
MS Word: Saying who or what is doing the defending would be clearer [aka passive voice, the actor is missing].
Grammarly: Remove hyphen from six feet.
ChatGPT: Clarify whether the fence and hedge are more than six feet tall together or each of them are, individually.
NYT: Remove both hyphens.
I had the same question as the GPT! Some styles require the first hyphen. This test may not be entirely fair as the AI didn’t know they were to follow NYT style. That feels a lot like some editing tests! Grammarly scored its second win on this one!
Sample 7: HERNDON, Va. — Bugs Bunny on Monday called for strengthening the nation’s defenses against cyberattacks, saying that as president he would create a joint law enforcement task force to handle both cybersecurity as well as offensive cyberwarfare.
Sample 7 Results
MS Word: [no errors flagged]
Grammarly: Fix spelling of cyber warfare.
ChatGPT: It’s unclear whether Bugs made the statement or this is a hypothetical if they were president.
NYT: Change as well as to and.
I changed the name of the actor in this sample and the GPT said the problem was that a fictional character can’t hold office. It was still confused when I tried the original and when I inserted the name of the current president. Context, again, matters, as the date of the original source makes it evident. As in the previous sample, whether compounds like cyberwarfare are closed, open, or hyphenated is a choice, which the AIs weren’t told to follow. I’m giving Grammarly half a point for this because it couldn’t know the preferred style, but it wasn’t even internally consistent with the other two closed cyber- compounds. Sometimes that “inconsistency” is the chosen style; a “complex consistency,” in other words. It’s a definite grey area.
Sample 8: Though the terminology and process is (wildly, needlessly) complex, the advice is simple for anyone wanting to borrow $25,000: Take out federal student loans from the government, not private ones that come from a bank or similar institution.
Sample 8 Results
MS Word: Pluralize the first is to are to reflect the “terminology and process” duo.
Grammarly: Change is to are.
ChatGPT: It’s not clear whether the parenthetical is necessary and whether it’s the author’s or the source’s point of view.
NYT: Change is to are.
Virtually unanimous! Though ChatGPT did miss the straight-up error and thought about a less straightforward writing critique. Rephrasing the question to identify the grammar error rather than the copy editing one did get the GPT to explain why is should be are in this instance.
Sample 9: The only criteria is that the player played for five seasons, made one All-Pro team or Pro Bowl and has been retired for five full seasons.
Sample 9 Results
MS Word: Pluralize the first is to agree with the plural criteria.
Grammarly: Change is to are.
ChatGPT: It is not clear if the player must have met both criteria.
NYT: Change is to are.
As with the previous sample, the detection of subject-verb agreement is virtually unanimous. And, again, ChatGPT missed the straight-up error and thought about a less straightforward writing critique but nailed it when the question was rephrased to identify grammar or even to just “edit this.” Note also that the GPT miscalculated the number of criteria in the sentence; there are three.
Sample 10: It occurred when American aircraft were called in to aide a police post that was under attack in Uruzgan, a province in the country’s south. At least seven police officers were killed, Afghan officials said.
Sample 10 Results
MS Word: Saying who or what called in the aircraft would be clearer; same for who did the killing [aka passive voice, the actor is missing]
Grammarly: Aid is misspelled.
ChatGPT: “Aide” is spelled incorrectly. Also, “aircraft” is singular, but “were” should be replaced with “was” as “aircraft” is referring to a single unit.
NYT: Aide should be aid.
I’m mystified by the GPT’s explanation of why was is right in this instance, but it flagged the misspelling that the NYT wanted it to, so it gets one point. Word straight-up had no problem with the spelling of aide. It’s hitting the passives hard, instead!
Conclusion: The NYT must keep the human copy editors
Software is getting better at judging subject–verb agreement and wrong word choice, but it isn’t ready to take over, yet.
Up Next
In future posts, I’ll give ChatGPT a shot at editing for style, flow and concision, things that other tools don’t even pretend to do. You may also want to read the first post in the series in which tested the AI against “10 common grammar mistakes.” Also check out Australian editor Rhonda Bracey’s tests using GPT for rewriting memos. Spoiler: it’s not ready to take over.
Want to learn more about editing?
Start with this free sneak peek at proofreading in which a proofreader (me) explains their thinking as they live-proof a single page of a book. Then take one of our training courses or check out Editing in Word 365 for technical help. And be sure to check out our full list of free resources on how to become an editor as well as our Starter Kit collection.
Picture created by the art AI bot Photoleap based on the author’s query: a photo of a steampunk robot reading the New York Times newspaper in Times Square.
