Spotting essays written by AI

A few weeks ago, the press reported that ‘researchers’ at Cambridge University had complied a list of indications that student essays have been written by AI tools.

The source of the press reports was a short article at https://www.cambridge.org/news-and-insights/news/does-chat-gpt-make-the-grade-cambridge-research

According to the article, the researchers described ChatGPT’s default writing style as echoing ‘the bland, clipped, and objective style that characterises much generic journalistic writing found on the internet’. The article lists key features of this writing style as including:

  • using Latinate vocabulary (multisyllable, sophisticated vocabulary above the expected level of study);
  • paragraphs starting with specific discourse markers such as ‘however’, ‘moreover’, and overall’, followed by a comma;
  • numbered lists with items followed by colons;
  • pleonasms (using unnecessary words to convey meaning, eg ‘true fact’ or ‘free gift’);
  • tautology (saying the same thing twice, eg ‘We must come together to unite’);
  • repetition of words or phrases and ideas;
  • consistent use of ‘Oxford commas’ before a conjunction at the end of a list. An example of an Oxford comma is the final comma (after ‘revision’) in ‘ChatGPT has many uses for teaching, learning at home, revision, and assessment’.

At the foot of the article is a link to issue 37 of Research Matters research-matters-37-does-chatgpt-make-the-grade.pdf (cambridgeassessment.org.uk) 
Unfortunately, I find nothing there that relates directly to the indicators of ChatGPT’s default writing style mentioned in the article. And I have found no other published document covering those indicators.

The research reported in issue 37 of Research Matters involved arranging for 3 undergraduates to each write 2 essays with help from ChatGPT. Examiners marked the essays and compared them with 164 essays written by Cambridge IGCSE students (aged 14 to 16).

The researchers reported that the ChatGPT-assisted essays by undergraduates were stronger than the IGSCE students’ essays on ‘information’ and ‘reflection’, but weaker on ‘analysis’ and ‘comparison’.

2 comments

  1. For fun, I did a small test of the writing style theory suggested by the Cambridge researchers.

    I asked Copilot:

    Which is better? International Financial Reporting Standards of United States Generally Accepted Accounting Principles? [NOTE: I FAILED TO NOTICE MY TYPO – “OF” SHOULD HAVE BEEN “OR”.]

    Copilot’s reply:

    Both the International Financial Reporting Standards (IFRS) and the United States Generally Accepted Accounting Principles (GAAP) serve as essential frameworks for financial reporting, but they differ in their approach and application. Let’s explore the key differences:

    1. GAAP (U.S. Generally Accepted Accounting Principles):
    o Applicability: GAAP is the accounting standard used in the United States.
    o Rules-Based: GAAP follows a more rules-based approach, providing specific guidelines and detailed procedures for financial reporting.
    o U.S. Context: Companies distributing financial statements outside their organization or publicly traded corporations must adhere to GAAP. The U.S. Securities and Exchange Commission (SEC) also sets rules for financial reporting.
    o Areas Covered: GAAP addresses aspects such as revenue recognition, balance sheet presentation, item classification, and outstanding share measurements.
    o Non-GAAP Measures: Some companies may use both GAAP-compliant and non-GAAP measures when reporting financial results. Non-GAAP measures must be identified in financial statements and other public disclosures.

    2. IFRS (International Financial Reporting Standards):
    o Applicability: IFRS is used internationally in over 110 countries, including the European Union (EU) and many countries in Asia and South America.
    o Principles-Based: IFRS follows a more principles-based approach, emphasizing the underlying economic substance of transactions.
    o Common Accounting Language: IFRS aims to establish a common accounting language globally, allowing businesses and investors to understand financial information consistently across borders.
    o Transparency and Stability: IFRS promotes transparency and enables informed financial decision-making.
    o Not Adopted in the U.S.: While the SEC has expressed interest in transitioning to IFRS, it remains the standard in the U.S. for now.

    In summary, GAAP is specific and rule-oriented, while IFRS focuses on principles and aims for global consistency. The choice between them often depends on the jurisdiction and the company’s international operations.

    End of Copilot’s reply.

    I thought Copilot’s reply was quite good. I’m not sure I would have detected it as AI-written. It lacks the Latinate vocabulary, discourse markers, pleonasms, and repetitions that the Cambridge researchers identify as flags. It does use Oxford commas, but I don’t find that to be an AI giveaway. It does use bulleted lists, but followed by full-stops.

    Copilot also seems to have noticed my typo (“of” should have been “or”) and corrected for it before replying.

    Copilot’s fifth bullet under 2 IFRS is incorrect as written. “It” should be replaced with “GAAP”.

    If I were an instructor and this were submitted to me, I would have docked the writer a few marks because the fourth and fifth bullets under GAAP seem to be “filler” unrelated to the question of which is better (though adding parallel bullets under IFRS might have fixed that).

    I then noticed my typo in my original question – “of” should have been “or”. So I cleared my question history (in the hope that Copilot would not treat my next question as a follow-up), fixed the question, and asked it again.

    Which is better? International Financial Reporting Standards or United States Generally Accepted Accounting Principles?

    Copilot’s Reply:

    The primary difference between International Financial Reporting Standards (IFRS) and United States Generally Accepted Accounting Principles (GAAP) lies in their approach. IFRS is principles-based, emphasizing the underlying economic substance of transactions, while GAAP is rules-based, focusing on specific detailed guidelines. Here are some key distinctions:

    1. Inventory Valuation:
    o Under GAAP, once inventory is written down, reversal is prohibited.
    o Under IFRS, inventory write-downs can be reversed if specific criteria are met.
    o IFRS prohibits the last-in, first-out (LIFO) method for inventory accounting.

    2. Intangible Assets:
    o IFRS recognizes acquired intangible assets only if they provide future economic benefits and have reliable measurements.
    o GAAP recognizes acquired intangible assets at fair value.

    3. Discontinued Operations:
    o IFRS defines discontinued operations differently, allowing more flexibility.
    o GAAP and IFRS both have separate sections for discontinued operations on income statements.
    o In summary, while both frameworks serve their purpose, IFRS’s principles-based approach aims to capture economic reality more effectively.

    End of Copilot’s second reply.

    Copilot’s second reply to the same question was completely different from its earlier reply.

    The first two sentences (about “the primary difference”) are a reasonable but brief start of a reply to my question. I would not identify those sentences as AI-written.

    But then the bulk of the reply is a combination of numbered and bulleted lists (not punctuated by semicolons) that, in my opinion, (1) provide no insight into the question of “which is better” and (2) are not what I would classify as the “key distinctions” anyway. But I don’t think these shortcomings are AI-giveaways.

    The most surprising thing to me about this tiny experiment is how different Copilot’s two essays were.

Leave a comment

Your email address will not be published. Required fields are marked *