How to evaluate CEFR level of text for L2 learners?

Question

For a language-learning platorm I'm building I need to analyze the difficulty of a foreign-language text for L2 learners in terms of CEFR.

However, as CEFR is made up of only Can do? questions, and not a fixed vocabulary list, I'm finding it difficult to properly assess. Currently, I'm running each text through the Flesch-Kincaid readability ease algorithm, and have the following rough coresspondings to CEFR levels:

# LINGUAPRESS estimates
# 0-50     C2 Mastery / Proficiency
# 50-60    C1 Advanced
# 60-70    B2 Upper intermediate
# 70-80    B1 Intermediate
# 80-90    A2 Elementary
# 90-100   A1 Beginner

Running 10 texts at A1 (black, average 97.5) and A2 (red, average 86.2), in English, roughly correspond to these levels.

However, I feel using FK is very inaccurate, especially for the later levels. This is because the algorithm doesn't take into account the complexity of words/frequency, just their syllable count and uniqueness.

I thought of combining FK with a measure of the frequency of words in the text compared to a wordlist (ex. using a weighted average). This could be used in conjunction with the estimates of CEFR vocab sizes StackExchange post But, I don't think this is the most efficient way, and don't know how much more accurate it would be.

Are there any algorithms, reading scores, or suggestions for calculating CEFR difficulty of Romance language texts?

Answer 1

Perhaps some of this may be relevant in relation to your request for suggestions for calculating CEFR difficulty.

Duolingo has created a CEFR checker. https://cefr.duolingo.com/

Rather than rely on "syllable count and uniqueness" that you note FK relies on, Duolingo in its checker is using multilingual word embeddings (MWEs) and corpus frequencies estimated on movie subtitles.

It uses AI to develop the lists of words to estimate the CEFR level for the L2, based on

"a few thousand hand-annotated CEFR labels for words in English, Spanish, and French, and allowing that model to generalize to hundreds of thousands of words across many other languages." ¹

Duolingo for this CEFR checker, is focused on the languages of English, Spanish, French, Italian, Portuguese, and German. Currently its CEFR checker is released for English and Spanish.

It is seeking to develop its CEFR tool based on communicative competences that depend on topical, semantic information in the L2 language. Focussing on frequently occurring linguistic forms before rarer ones.

¹ : https://blog.duolingo.com/the-duolingo-cefr-checker-an-ai-tool-for-adapting-learning-content/

Further References:

• https://research.duolingo.com/papers/settles.tacl20.pdf
• https://www.mitpressjournals.org/doi/full/10.1162/tacl_a_00310
• https://venturebeat.com/2020/08/18/how-duolingo-uses-ai-in-every-part-of-its-app/