What is the impact of Large Language Models on language assessment?

Question

How can the use of large language models like ChatGPT change the assessment of language learning, and what implications does this have for teaching and learning? Considering the impact that computers have had on mathematics learning and assessment, can LLMs similarly transform language assessment in ways that affect the teaching and learning of languages?

Large language models can be used as writing assistants, as Yann LeCun put it:

LLMs are still making sh*t up.
That's fine if you use them as writing assistants.
Not good as question answerers, search engines, etc. RLHF merely mitigates the most frequent mistakes without actually fixing the problem.

For instance, should traditional language assessment tools, like the IELTS writing rubrics, be updated in light of the capabilities of LLMs? Can we remove some skills like range of vocabulary and errors in spelling and/or word formation from the band descriptors?

Assessment plays a crucial role in shaping language teaching and learning. Thus, the key question is how will LLMs alter language assessment, and how will these changes shape language teaching and learning?

Answer 1

While LLMs can be utilized for more efficient studying, LLMs aren't changing languages themselves. If you need to be able to do X to be considered C1 in French before the advent of LLMs, there doesn't seem to be any argument that this should change after the advent of LLMs.

Nevertheless, LLMs will likely make language assessment more efficient for those involved:

• Grading. It seems likely that grading exams and assignments will be influenced by LLMs, perhaps as an assistant (e.g. pointing out errors), or perhaps as an objective marker. It can also provide feedback beyond "4 out of 10". This is a bottleneck for the Chinese HSK5 and HSK6 exams (listening and reading questions are automatically marked, but it takes human time to mark the writing section). Perhaps down the line, e.g. combined with speech-to-text and text-to-speech, they can also grade speaking.

  I asked Sage (GPT 3.5-turbo) about this, and it highlights how LLMs will likely grade differently to humans, being able to more readily detect nuance:

  LLMs have the potential to analyze language use in more nuanced ways, such as detecting the presence of idiomatic expressions, detecting the use of figurative language, or assessing the quality of argumentation. ... Additionally, the use of LLMs may also lead to increased emphasis on the importance of language fluency, ...

• Test and homework preparation (examiner). It seems plausible that LLMs can assist with multiple aspects of exam/homework preparation, such as generating (or assist in the generation of) exercises for assessing the understanding of individual aspects of a language, writing or editing sections of exams, and ensuring test/assignment difficultly is consistent. Students can be given distinct assignments with comparable difficulty, which can help thwart one form of cheating.

• Test preparation (student). There's an assortment of ways LLMs can help students prepare for tests, such as by providing mock questions, and checking test-specific vocabulary knowledge. Too much to list.

• Fairness. E.g. homework needs to be checked for LLM usage.

Basically, it seems reasonable to expect that we will continue doing what we already do for assessing languages, but more efficiently thanks to LLMs (with an extra step of checking for LLM-related cheating).

I asked Sage what LLMs might do that's new, and among its suggestions are:

LLMs can generate language tasks that require test-takers to use language in real-world contexts, such as workplace scenarios or social situations.

LLMs can generate language tasks that require test-takers to adapt their language use to different audiences or situations, such as changing language register or tone.

It doesn't come up with anything fundamentally different, but there's some novelty here.

It's possible that in the future the whole process (reading, writing, speaking, listening) will be conducted by LLMs; every exam will be uniquely generated; and it will be graded objectively with useful feedback within minutes, if not seconds. Whether this would be considered acceptable for official standards is unclear.