23

It's common to ask whether X learning method is effective. A better question is "Is X more effective than Y?"

But how can such a determination even be done? Maybe X is better for vocabulary recall, and Y is better for learning grammar principles.

How are comparative studies between learning methods designed to measure their relative effectiveness?

  • @Gwenn: The question is asking how to do research (or how research is done), it's not looking for published studies. – Flimzy Apr 7 '16 at 13:33
  • Oops, my mistake. Hm. Perhaps [research-techniques] would be better? – Gwen Apr 7 '16 at 13:41
  • @Gwenn: Perhaps... – Flimzy Apr 7 '16 at 13:45
4

I work in Japan and while my own research is not primarily about language acquisition, I do read up on it a bit. The procedure is quite close to as Roland Coeurjoly describes. It's a social sciences methodology emulating the "scientific method"

  1. A hypothesis is made about some method -- say using colors to highlight some words in flash cards.
  2. A procedure is drawn up.
  3. A pilot test of the experimental procedure is run. (Depending on the format, you may or may not need to test both the experimental procedure and the control procedure).
  4. A pretest in the relevant skill is given to assess language level in that particular skill.
  5. Control group receives a standard teaching method or a time-wasting method. Experimental group receives the hypothetical method. Both methods should cover either the same content or content that is believed (due to prior research or some other reason for equivalency) to be of the same difficulty.
  6. A post test is given to both groups to see how well they learned the material.
  7. Spaced post tests may also be given depending on what is to be tested to see whether it stuck.

There's several things that make it really hard to test effectiveness. First, there's no two people that are at exactly the same point in a foreign language. Second, any pretest is going to be a proxy for language knowledge rather than an exhaustive battery. For many of the projects I've read, it's a vocabulary test looking at how deep the students' knowledge of English vocabulary is. This proxy can't test whether the content is genuinely new to all learners. (One way to disqualify some is to exclude people who have lived abroad, etc.).

Third, not all material is equal even if it matches some criteria for difficulty. So in one (student-level) experiment, the student picked the word "acorn" and the word "squirrel" for a vocabulary acquisition exercise. These words are both infrequent enough in the English language that they are on the same frequency level with words Japanese college students don't know. But they basically ruined the test because students do know these words.

Fourth, the differences in effectiveness are often too small to detect on the sort of sample sizes I see. Part of the reason is that it's difficult to control someone's entire language learning experience. Another reason is that some things are painfully obvious -- if you don't require people to speak while learning a foreign language, many of them will be bad at speaking. Or if they never hear native input, they will have trouble listening to native speakers. But once we get past the painfully obvious, we're looking things that are minutely more or less effective. And when we do that individual variables get stronger (teacher's basic ability, student's eagerness to learn, etc.)

3

In truth, it's very difficult to compare learning techniques for effectiveness, and it's not always meaningful to ask "which technique is better?" As you note, it's possible for different techniques to have different benefits, and often they complement one another.

Studies comparing techniques and strategies need to measure the techniques somehow, and often "effectiveness" is measured in a single way. Even when effectiveness is measured in more than one way, studies are unable to cover all of the possible benefits. This is the reason that multiple studies and meta-studies are needed to make any useful comparison. Actually, in the field of language learning, I don't believe many questions have been really thoroughly investigated.

To give an example, if we are looking at the effectiveness of methods for learning vocabulary, there are so many ways they can be beneficial:

  • Are you learning the word for the first time, or strengthening knowledge of a partially-known word, or re-memorizing a forgotten word?
  • Does the technique build passive vocabulary (words you can understand) more or active vocabulary (words you can use)?
  • Does the method build understanding of how the word is used in context? For example, what registers / domains it appears in, what collocations it is a part of, what specific meanings does it have.
  • Does it build receptive knowledge or productive knowledge, or both?
  • Does the method have a stronger impact on short-term or long-term memory?

So saying one method is superior to another is a difficult statement to justify. Word cards might be good for learning words for the first time, while extensive reading might be more effective for giving a deeper understanding of the word.

I think it's important to keep this in mind when reading answers on this site. Many questions ask for evidence from research, but in most cases the research will be too limited to make any definite conclusions. An answer which cites peer-reviewed research may look good, but if it's a single study that's not very generalizable, it might be less useful than a detailed analysis based on experience.

2

Two groups of people are formed. One is the control group, the other is the experimental group.

  1. The control group receives the traditional method of instruction.

  2. The experimental group receives the method that is going to be evaluated.

Before performing the experiment, the language level of members of both groups have to be assessed. After the experiment is carried on there is also another assessment, to evaluate the improvement in both groups. Those assessments are usually standardized tests.

The number of people in both groups (sample size) must be sufficiently large to ensure statistic relevance.

Examples of this method of research being carried on:

The Fiji Island study (RRQ, 1983): Elley & Mangubhai

Elley, W. 1991. Acquiring literacy in a second language: The effect of book-based programs.

I cannot find the papers right now. If I find them I will edit the answer.

  • One immense difficulty is the devil in "members of both groups have to be assessed in their language level." – virmaior Apr 22 '16 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.