Language Learning Stack Exchange is a question and answer site for students, teachers, polyglots, and anyone interested in the techniques of second-language acquisition. It only takes a minute to sign up.
Sign up to join this community
It's common to ask whether X learning method is effective. A better question is "Is X more effective than Y?"
But how can such a determination even be done? Maybe X is better for vocabulary recall, and Y is better for learning grammar principles.
How are comparative studies between learning methods designed to measure their relative effectiveness?
I work in Japan and while my own research is not primarily about language acquisition, I do read up on it a bit. The procedure is quite close to as Roland Coeurjoly describes. It's a social sciences methodology emulating the "scientific method"
There's several things that make it really hard to test effectiveness. First, there's no two people that are at exactly the same point in a foreign language. Second, any pretest is going to be a proxy for language knowledge rather than an exhaustive battery. For many of the projects I've read, it's a vocabulary test looking at how deep the students' knowledge of English vocabulary is. This proxy can't test whether the content is genuinely new to all learners. (One way to disqualify some is to exclude people who have lived abroad, etc.).
Third, not all material is equal even if it matches some criteria for difficulty. So in one (student-level) experiment, the student picked the word "acorn" and the word "squirrel" for a vocabulary acquisition exercise. These words are both infrequent enough in the English language that they are on the same frequency level with words Japanese college students don't know. But they basically ruined the test because students do know these words.
Fourth, the differences in effectiveness are often too small to detect on the sort of sample sizes I see. Part of the reason is that it's difficult to control someone's entire language learning experience. Another reason is that some things are painfully obvious -- if you don't require people to speak while learning a foreign language, many of them will be bad at speaking. Or if they never hear native input, they will have trouble listening to native speakers. But once we get past the painfully obvious, we're looking things that are minutely more or less effective. And when we do that individual variables get stronger (teacher's basic ability, student's eagerness to learn, etc.)
In truth, it's very difficult to compare learning techniques for effectiveness, and it's not always meaningful to ask "which technique is better?" As you note, it's possible for different techniques to have different benefits, and often they complement one another.
Studies comparing techniques and strategies need to measure the techniques somehow, and often "effectiveness" is measured in a single way. Even when effectiveness is measured in more than one way, studies are unable to cover all of the possible benefits. This is the reason that multiple studies and meta-studies are needed to make any useful comparison. Actually, in the field of language learning, I don't believe many questions have been really thoroughly investigated.
To give an example, if we are looking at the effectiveness of methods for learning vocabulary, there are so many ways they can be beneficial:
So saying one method is superior to another is a difficult statement to justify. Word cards might be good for learning words for the first time, while extensive reading might be more effective for giving a deeper understanding of the word.
I think it's important to keep this in mind when reading answers on this site. Many questions ask for evidence from research, but in most cases the research will be too limited to make any definite conclusions. An answer which cites peer-reviewed research may look good, but if it's a single study that's not very generalizable, it might be less useful than a detailed analysis based on experience.
Two groups of people are formed. One is the control group, the other is the experimental group.
The control group receives the traditional method of instruction.
The experimental group receives the method that is going to be evaluated.
Before performing the experiment, the language level of members of both groups have to be assessed. After the experiment is carried on there is also another assessment, to evaluate the improvement in both groups. Those assessments are usually standardized tests.
The number of people in both groups (sample size) must be sufficiently large to ensure statistic relevance.
Examples of this method of research being carried on:
The Fiji Island study (RRQ, 1983): Elley & Mangubhai
Elley, W. 1991. Acquiring literacy in a second language: The effect of book-based programs.
I cannot find the papers right now. If I find them I will edit the answer.
