When grading vocabulary (either in a traditional classroom/test environment, or spaced repetition), is there any research into how to most effectively grade responses?

Examples of possible answers (not to be taken as an exhaustive list!), using the word "apple":

  • Answers should be counted correct only if they are 100% correct.

    Spelling "aple", pronunciation /ɑpəl/, vague description "a type of fruit" are counted incorrect

  • Answers should be counted correct if they get the substantial idea across, are intelligible, would be understood by a typical speaker, etc.

    Spelling "aple", pronunciation /ɑpəl/, vague description "a type of fruit" are counted correct

  • Answers earn partial credit (meaningful on tests, perhaps not so much for SRS) if they are partially correct.

    Spelling "aple", pronunciation /ɑpəl/, vague description "a type of fruit" are counted for partial credit.

Which of these, or some other grading standard, leads to best results in students?

On the one hand, aiming for perfection seems ideal, but does it lead to student fatigue and demotivation with concepts that are hard to master?

What research has been done into this topic, in particular as it relates to language teaching?

  • If I understand this question correctly, a spaced repetition algorithm also does grading. There is definitely research on the effectiveness of spaced repetition. With regard to research on grading standards in education, I'm less hopeful. Does the question assume that learners are prompted for singe worlds using images or L1 cues? (Or also cloze tests?) – Christophe Strobbe Oct 5 '16 at 11:50
  • @ChristopheStrobbe: There are many ways to create flash cards for SRS. I do mine completely absent of my L1, but that may be uncommon. So in short: No, I don't make any assumptions about the content of the flashcards. The best I've found on the topic, on the supermemo site, which has 4 grades (wrong, difficult, normal, easy), and it claims that an "imperfect" answer may be counted as difficult in some cases. But it doesn't really elaborate on what that means. Further, the focus there is on the difficulty of recall, not the overall correctness of the answer. – Flimzy Oct 5 '16 at 13:17
  • I think internal motivation and its conflicts with insecurity, etc. play a huge role in language learning (as in math). Therefore, my personal opinion is that it is best to grade students liberally. Many students will look for any excuse to give up studying language(s), and a C in a language class can be their best confirmation that they "just can't learn [language(s)]" or whatever. I think the inverse is also true. – SAH Nov 1 '16 at 19:58

How vocabulary tests are graded depends on the teacher's (implicit or explicit) language learning or teaching theory. There are many such theories (see e.g. What are the main foreign language teaching methods?) with different views on how vocabulary should be taught (and tested).

In addition, there can be individual differences between teachers that subscribe to the same language learning or teaching theory. (Even in the context of standardised language tests, evaluators should regularly attend seminars in order to make sure that they all use the same evaluation criteria. See Bausch, Christ & Krumm (eds.): Handbuch Fremdsprachenunterricht, 5th ed. A. Francke, 2007, page 376.)

Language teaching theories that prefer a direct method teach vocabulary in context, while theories that allow the use of L1 let learners study associated pairs (L2 word - L1 translation). These approaches also determine how vocabulary is tested, e.g. through translations or by other means (cloze tests, building sentences with a given word, ...).

In both cases, the evaluator or teacher needs to weight the errors. For example, Englische Fachdidaktik. Theorien, Praxis, Forschendes Lernen by Wolfgang Gehring (Erich Schimdt Verlag, 1999/2010) lists the following types of minor versus major errors:

Major errors:

  • Errors against general language rules.
  • Errors that seriously hamper understandability.
  • Errors against the current language learning goal (i.e. something that was taught very recently).
  • Errors against "common construction".
  • Strong deviations from language rules and stylistic norms.

Minor errors:

  • Errors against specific language rules.
  • Errors that do not seriously hamper understandability.
  • Minor deviations from "common construction".
  • Minor deviations from language rules and stylistic norms.

The above error types apply to language tests generally (i.e. essay composition, oral skills tests, ...) and do not apply well to vocabulary that require translations (L1 to L2, or L2 to L1). In the current communicative approaches, simple word translation tests are often replaced by other types of tests, e.g. given a specific word, build a sentence that shows that you know the word's meaning. Errors in such sentences would need to be weighted according to the criteria listed above.

However, there are also vocabulary learning techniques where the only criterion is correctness. One example is SAFMEDS (Say All Fast Minute Each Day Shuffle). SAFMEDS is a technique that is part of precision teaching, a method of teaching and evaluating developed by Ogden Lindsley in the 1960s (and inspired by behaviourism). In SAFMEDS, you get through a deck of flashcards using the following method:

  • You say the word on the other side of the card.
  • You try to get through the deck within a minute.
  • You put aside the flashcards where you made errors. (The number of errors is tracked in a "celeration chart".)
  • You shuffle the deck after each run through the deck, so you don't memorise the sequence of cards.
  • You do this every day.

(To get an idea of what SAFMEDS is, see e.g. SAFMEDS Tutorial by Kelly Byrne on YouTube, Vocabulary Aquisition with SAFMEDS: How to learn words quickly by Dr. C. A. Young-Pelton, and Surviving on SAFMEDS by Danielle Costa.)

Another teaching method where frequent formative tests play an important role is mastery learning, an instruction approach first formulated by Benjamin Bloom in the late 1960s. In mastery learning, all learners need to achieve a specific level of mastery before moving on to the next "lesson". Mastery learning has also been used in language teaching. However, I have not yet found literature that discusses grading of errors. For example, "Mastery Learning in Modern Languages - a Case Study" (Parkinson, B. L.; Mitchell, R. F.; Johnstone, R. M., 1983) discusses the use of mastery learning for teaching French in two schools in Scotland. The article does not mention the weighting of errors.

So "how correct is necessary" is highly dependent on the learning theory. There are vocabulary learning methods that are more effective than others, but some of the existing methods have probably not been compared yet (e.g. SAFMEDS versus the SuperMemo algorithm).

  • Thank you for the thorough answer. I was sadly on vacation when my bounty expired, so I wasn't able to award you the full bounty. Thus I'll do another bounty for 50, to ensure you get your full 100 bounty credit. – Flimzy Oct 13 '16 at 10:10
  • @Flimzy It's OK. Don't feel guilty about the bounty. – Christophe Strobbe Oct 13 '16 at 10:14
  • @Flimzy 200 isn't exactly 50 by the way :) – Anthony Pham Oct 13 '16 at 23:31
  • @PythonMaster: 200 was the minimum. I suppose because I had already done a bounty. – Flimzy Oct 13 '16 at 23:32
  • He sure is lucky... I mean he got 4 times as much rep as you told him you would give – Anthony Pham Oct 13 '16 at 23:39

I'm not citing any studies here, so it's a partial answer, but I'm trying to provide arguments for what I say, so I hope it can be helpful.

Note that grading vocabulary has different purposes in different settings you mentioned. In case of SRS, if you mark something as incorrect, you'll review that soon. In a classroom, when the teacher corrects a student, it is only meant as a feedback, and it doesn't mean that the item that got corrected will be reviewed soon. Finally, tests are tools for estimating the student's level, not for learning.

Let's start with SRS. I don't think "100% correctness" is a reasonable goal in most cases. In case of pronunciation, it would mean that you consider the answer good until you can reliably pass in your answer as a native speaker of the language. That's obviously very hard, and most learners don't even care about passing as natives, so aiming for 100% correctness is just a waste of time. And even if you do want to pass as a native, I think it's better to distinguish pronunciation learning from vocabulary learning. Since the question is about the former, I'd say that in this context you only should care whether your pronunciation is understandable. You don't want to endlessly repeat words that you can use very well just because your pronunciation is slightly off.

As for the meaning, again, 100% correctness is not possible in most cases. You gave an example of a concrete noun, but how about more abstract ones, or verbs? Most of them won't have exact equivalents in other languages you know, so there is simply so such thing as perfect correctness when you learn such words out of context. You can only decide if you have a general idea about their meaning.

Spelling is probably the easiest area where you could aim at 100% correctness. But then again, it depends on your goals and abilities, how transparent the writing system is, whether you want to get the writing abilities of an educated native speaker. For example, if you have a dyslexia and have spelling problems in other languages you know, you may want to relax the correctness criteria for the language you're learning. Unless, of course, you're specifically aiming at getting rid of spelling problems.

In case of tests, you simply need to make an arbitrary decision of what is the level of correctness, based on what you want to measure: communicative ability of the student, his or her ability to pass as a native speaker, or whatever else.

Let me finally answer the question in the title: "how correct is necessary?". Children and people learning by immersion get little explicit feedback, and if they do, it doesn't cause automatic repetitions like in an SRS. Still, they manage to learn the language, so that shows that the emphasis on correctness is not necessary. However, it doesn't mean that it can't make learning more effective. But, due to the reasons I discussed above, aiming for 100% correctness is either impossible, or may slow down the learning.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.