17.3578, Review: Applied Linguistics: Norris; Ortega (2006)

[LINGUIST Network]

LINGUIST List: Vol-17-3578. Mon Dec 04 2006. ISSN: 1068 - 4875.

Subject: 17.3578, Review: Applied Linguistics: Norris; Ortega (2006)

Moderators: Anthony Aristar, Eastern Michigan U <aristar at linguistlist.org>
            Helen Aristar-Dry, Eastern Michigan U <hdry at linguistlist.org>
 
Reviews: Laura Welcher, Rosetta Project / Long Now Foundation  
         <reviews at linguistlist.org> 

Homepage: http://linguistlist.org/

The LINGUIST List is funded by Eastern Michigan University, Wayne
State University, and donations from subscribers and publishers.

Editor for this issue: Laura Welcher <laura at linguistlist.org>
================================================================  

This LINGUIST List issue is a review of a book published by one of our
supporting publishers, commissioned by our book review editorial staff. We
welcome discussion of this book review on the list, and particularly invite
the author(s) or editor(s) of this book to join in. To start a discussion of
this book, you can use the Discussion form on the LINGUIST List website. For
the subject of the discussion, specify "Book Review" and the issue number of
this review. If you are interested in reviewing a book for LINGUIST, look for
the most recent posting with the subject "Reviews: AVAILABLE FOR REVIEW", and
follow the instructions at the top of the message. You can also contact the
book review staff directly.


===========================Directory==============================  

1)
Date: 04-Dec-2006
From: Ute Knoch < u.knoch at auckland.ac.nz >
Subject: Synthesizing Research on Language Learning and Teaching 

	
-------------------------Message 1 ---------------------------------- 
Date: Mon, 04 Dec 2006 01:34:15
From: Ute Knoch < u.knoch at auckland.ac.nz >
Subject: Synthesizing Research on Language Learning and Teaching 
 

Announced at http://linguistlist.org/issues/17/17-1454.html 

EDITOR(S): Norris, John M. 
TRANSLATOR: Ortega, Lourdes
TITLE: Synthesizing Research on Language Learning and Teaching
SERIES: Language Learning & Language Teaching 13
YEAR: 2006
PUBLISHER: John Benjamins
ISBN: 9027219656
ANNOUNCED IN: http://linguistlist.org/issues/17/17-1454.html 

Review by Ute Knoch, Department of Applied Language Studies and
Linguistics, University of Auckland, New Zealand

The two authors who produced one of the first and the most cited
meta-analysis in the field of applied linguistics (Norris and Ortega, 2000)
have now written an edited volume on this topic. The aim of this thirteenth
volume in the Language Learning and Language Teaching series is to
introduce the approach of synthesizing primary research to applied
linguists. The book is intended for a number of target groups: applied
linguists, graduate students, methodologists, teachers, curriculum
developers and policy makers.  

SUMMARY

Chapter 1: The value of practice of research synthesis for language
learning and teaching.  Norris, J. and Ortega, L.

In the first chapter of the book, the two editors provide a general
overview of the area of research synthesis. They start out with a
definition (systematic review of accumulated research studies) and then
mention the three primary goals: (1) to help people make sense of research,
(2) to enable the research community to compare and combine findings across
individual studies and identify gaps in research methodologies and (3) to
show that a synthetic research ethic should be adopted as a guide for all
empirical work in our field. A major section of this chapter is the clear
definition of a research synthesis. The authors describe where research
syntheses are mainly published, compare the research synthesis to other
types of reviews and describe what different forms a research synthesis can
take. Whilst doing this, they point the reader to chapters in the book that
represent good models of the different forms of research syntheses described. 
The authors then identify the three defining characteristics of research
syntheses: Firstly, that they include a clear description of how the
literature was searched, and according to what criteria the primary studies
were selected. Secondly, a research synthesis focuses on the actual
variables, characteristics, and data reported in the study, rather than
looking at the conclusions drawn by the primary researchers. Thirdly, a
research synthesis compiles results and seeks generalizations by looking
across studies in order to come to a systematic idea of what we know and
what we don't know. The authors point out that a research synthesis can
take many forms, but that the most frequent form is the meta-analysis.

The authors then describe the quantitative research synthesis, the
meta-analysis, in more detail with its purposes and procedures. Here, the
reader can find a very clear description of effect sizes. The authors take
the reader through each of the stages that are involved in conducting a
meta-analysis, as well as the problems that can be encountered in each of
these stages. The section on meta-analyses is concluded by a discussion of
why meta-analyses are appropriate and what the criticisms are. 

The next part of the chapter talks about the implications for the research
community in general that come out of the practice of synthesizing
research. The authors propose that a number of groups should be involved in
creating an ethical practice about research. These include secondary
researchers who should set out to provide a balanced picture of the primary
research and ensure that the review is complete. Primary researchers should
ensure that their findings are published clearly, so that it can be
included into syntheses as well as ensuring that there is full access to
their data. Finally, editors and publishers should create clear guidelines
and requirements for submissions to journals (e.g. effect sizes etc.) and
make it possible for primary researchers to disseminate their appendices.
The final part of chapter one looks at the future challenges to research
synthesists in language teaching and learning. 

Section 2 of the book presents a good example of research syntheses. These
were included in the book because the editors feel that this is the best
way to understand the variety of purposes and approaches to research synthesis.

Chapter 2, Principles, parameters, and SLA: A retrospective meta-analytic
investigation into adults L2 learners' access to Universal Grammar.
Dinsmore, T.

This chapter presents an example of a very clearly written meta-analysis.
Dinsmore follows all the guidelines set out in the introductory chapter of
the book. Dinsmore starts with a very clear overview about what UG is
about. He then presents the three different positions in the UG access
problem and describes how these positions have been investigated by SLA
researchers. He then turns to a description of meta-analysis and the
process researchers should take. Then, Dinsmore very clearly describes the
methodology he followed to identify the primary research studies for his
analysis, how the studies were coded and how the analysis was undertaken.
Through the use of effect sizes, he was able to reject the hypothesis that
adult L2 learners have full access to UG. 

Chapter 3: Investigating the empirical link between task-based interaction
and acquisition. A meta-analysis. Keck, C.M; Iberri-Shea, G.;
Tracy-Ventura, N; Wa-Mbaleka, S. 

Chapter 3 is, as is evident from the title, another quantitative
meta-analysis. In this case, it centers around another ''hot'' topic in the
applied linguistics literature, the link between task-based interaction and
acquisition. The authors of this chapter, just as the author of the
previous chapter, start with a very clear, yet concise literature review of
the current issues relating to interaction, acquisition and task-based
interaction. They follow this literature review with a clear description of
meta-analyses. The method section follows all the requirements of a
meta-analysis set out in the first chapter of the book: a description of
how the literature search was conducted, a list of clear
inclusion/exclusion criteria and a set of coding procedures and why they
were chosen. They then very clearly describe how the quantitative
meta-analysis was undertaken. This section also includes a section on how
effect sizes were calculated, even in more complicated cases where the
primary study had not provided a lot of information. They then describe how
the effect sizes were combined and compared.

The results section begins with a synthesis of the methodological features
of the different studies. Then, the results from the quantitative
meta-analysis are presented. The authors make this section particularly
clear by presenting graphs indicating the mean effect sizes with confidence
intervals for each of their research questions. In the discussion section
the authors return to each of their research questions and also suggest
which findings should be taken with caution and where further research is
necessary. 

Chapter 4: The effectiveness of corrective feedback for the acquisition of
L2 grammar. A meta-analysis of the research. Russell, J. and Spada, N.

This chapter closely follows the pattern of the previous chapter. It starts
with a comprehensive, clearly written literature review on oral and written
feedback, the sources of corrective feedback, the nature of the feedback
(whether explicit or implicit) and concludes with a summary of the
literature review. As was found in the previous chapter, the data
collection procedures are described in detail. These include the sources of
the primary studies, what criteria for inclusion were chosen and even what
search terms were used for the databases. The authors also describe their
coding in detail. Interestingly, in this study, the authors had to exclude
about a third of the studies that met the inclusion criteria, because they
could not calculate effect sizes from the data provided by the primary
researchers. The authors link the strength of the effect sizes to the
absence or presence of reliability/validity reporting. The chapter
concludes with a thorough discussion of the results. The appendix presents
a table of all the studies included into the meta-analysis and the coding
criteria.

Chapter 5: Effect of L2 instruction on interlanguage pragmatic development.
A meta-analysis. Jeon, E. H. and Kaya, T. 

This meta-analysis was undertaken in the slightly less researched and often
neglected area of L2 pragmatics instruction. The authors provide a very
useful literature review which is mainly guided by four questions, which
resemble the research questions for the meta-analysis: (1) Is teaching of
L2 pragmatics effective? (2) What is the most effective way to teach L2
pragmatics? (3) What are the most common outcome measures employed in L2
pragmatics research and is there a method effect, (4) What is the
appropriate length of L2 instruction. After the literature review, a
detailed method section is presented. This section includes the usual
sections (as already seen in the previous chapters on meta-analysis): how
the literature search was conducted, why studies were included in or
excluded from the meta-analysis, the coding scheme, how effect sizes were
calculated and combined and how confidence intervals were calculated.

The results section that follows systematically presents the findings
relative to the four research questions. The reader finds that effect sizes
show that L2 pragmatics instruction is effective (when measured as pre- and
post-tests) and that experimental groups outperformed control groups.
Effect sizes also indicate that explicit instruction might be more
effective than more implicit instruction, however too much variation
between the studies makes it hard to draw a clear conclusion. The results
of the third research question indicate that outcome measure might have
some influence on the outcome of the study. Finally, the results of the
fourth question reveal that longer instruction might be more effective than
very short term instruction. 

Chapter 6: The effects of explicit reading strategy training on L2 reading
comprehension. Taylor, A; Stevens, J.R; and Asher, J.W. 

In this chapter, another example of a meta-analysis, the authors
demonstrate how the effect of moderating variables can be evaluated and
also how homogeneity tests can be conducted on different groups of effects
sizes. The chapter sets out with a definition of explicit reading strategy
training and then discusses differences between strategies that are used:
cognitive and meta-cognitive strategies. The rest of the literature review
focuses on the different moderating variables that can influence the
results of studies investigating the effectiveness of explicit reading
strategy training: manner and type of treatment, the instrument to measure
L2 reading, time between pre- and post-test, L2 level of the students in
the study, age, the context (EFL vs. ESL) and the length of reading texts
used in the post-test. 

The method section is again very similar to the ones found in the previous
chapters. The inclusion/exclusion criteria and the literature search are
described, different search methods are described, the calculation of the
effect sizes is presented (this study used Hedges's g which was then
converted to Cohen's d), the coding criteria are set out. The authors of
this study also describe the homogeneity tests they used to compare the
effect sizes (a special feature of this meta-analysis).

The results are interesting for readers interested in the area of explicit
reading strategy training. Overall, the meta-analysis shows that students
receiving extensive reading strategy training outperform those who do not.
There was, however, no statistically significant difference between
learners who were trained in cognitive strategies and those who received
training in meta-cognitive strategies. The type of post-test employed
showed no statistically significant influence on the results, although
open-ended instruments resulted in slightly larger effect sizes, and
standardized tests in slightly lower effect sizes. There was no difference
for the total hours of treatment, and no clear effect for the length of
study. Proficiency level and age resulted in a statistically significant
difference however, with a higher result for students in their second and
third year of study and students over 12 outperforming those under 12. The
language context made no difference on effect sizes. Finally, the text
length used resulted in a general trend, indicating that there was an
increase in effect sizes as longer texts were used. Overall, this chapter
is very clearly written and presented and can be seen as another very
useful example of a successful meta-analysis. 

Chapter 7: A meta-analysis of qualitative research on effective teaching
practices for English Language Learners. Tellez, K. and Waxman, H.C.

Chapter 7 presents a different type of synthesis than the previous
chapters, a meta-synthesis of qualitative research. The aim of the authors
is to illuminate best teaching practices of English Language learners. The
authors point out that a qualitative synthesis is quite different to a
quantitative meta-analysis, as it cannot rely on effect sizes. But, there
are a number of similarities between this synthesis of qualitative research
and the meta-analyses presented in the previous chapters. Firstly, the
authors of this chapter also painstakingly describe their literature search
(including the data bases searched and the search terms used), the criteria
chosen for inclusion and exclusion of studies and their coding process. In
this case, the coding process was interesting. The authors set out with a
priori categories, but just as often found in qualitative research, revised
these completely to new categories that more appropriately fitted their
data. The results, then, are presented in the five categories that were
used as their codes: (1) communitarian teaching practices, (2) protracted
language events, (3) multiple representations designed for understanding
target language (4) building prior knowledge and (5) elements that
prevented effective practice. Overall, it can be said that this qualitative
meta-synthesis can offer something that a quantitative meta-analysis might
not be able to bring to the table, a synthesis of rich qualitative
findings, a way of synthesizing research which has a lot of promise for the
future and has been done a lot less than meta-analyses of quantitative
research.  

Chapter 8: Research synthesis and historiography: The case of assessment of
second language proficiency. Thomas, M.

Chapter 8 is quite different to the research syntheses presented in
previous chapters. It does not synthesize research from one time frame, but
inquires into the history of L2 acquisition by conducting a very similar
type of analysis at two points in time, twelve years apart. Thomas
conducted a research synthesis of techniques of assessing L2 proficiency in
1994 (looking at published articles from 1988 to 1992). She then repeated
this investigation twelve years later (looking at articles published in the
same five journals in the time period of 2000 to 2004). First Thomas
describes how she went about synthesizing the research in 1994 and what her
motivation was. She explains how the articles were sourced and what the
exclusion criteria were. In 1994, she grouped the L2 proficiency assessment
into four categories: (1) impressionistic judgment, (2) institutional
status, (3) in-house assessment and (4) standardized tests. The second part
of the article describes the second part of the historiography conducted
twelve years later. First she shows that on the face of things, not much
has changed the proportion of studies falling into each of the four
categories described above is still almost the same. However, Thomas found
four qualitative differences between the two time periods. In more recent
journal articles, authors used the proficiency data they had differently to
the earlier time period, by for example probing the learners' proficiency
in finer detail. They also used the assessment techniques in different
ways, e.g. by combining two different measures. Finally, she found a trend
of downplaying the importance of proficiency assessment in the more recent
corpus. 

Chapter 9: Meta-analysis, human cognition and language learning. Ellis, N.

Nick Ellis divides his chapter into three sections. In the first section he
builds an argument from the angles of Bayesian reasoning, probability and
cognition of the human mind, showing how traditional reporting practices
fail us and why we should therefore use meta-analyses. He then suggests
other areas of study that should be meta-analyzed. The second section of
Ellis's chapter discusses some of the limitations of meta-analysis. He
discusses the practice of gathering a wide variety of operationalizations
of independent and dependent variables and putting each study on equal
footing, a practice which fails to give more weighting to better studies.
He suggests that applied linguistics as a field should draw up a list of
recommendations for improved research and publication practices to make way
for better meta-analyses. He further cautions that meta-analyses might
close down certain fields of enquiry prematurely. 

The third section of Ellis's chapter reviews the different contributions to
this book and what they propose about the current status of findings in
applied linguistics. He finally concludes that the chapters in this
collection should be seen as good models for other secondary researchers
interested in the task of systematic synthesis. 

Chapter 10: Some reflections on the development of (meta-analytic)
synthesis in second language research. Chaudron, C.

The tenth and final chapter in this edited volume is by Craig Chaudron. He
provides a historical perspective on the development of research synthesis
in the field of applied linguistics. He argues that research synthesis has
recently reached a ''more superior level of sophistication'' not only in the
variety of fields addressed but also in the quantitative and conceptual
synthesis undertaken. He looks at the development of research synthesis in
the field by describing syntheses undertaken on four areas of research: (1)
L2 grammatical development, (2) instructional effects, (3) reading
instruction, and (4) individual differences. Whilst reviewing the type of
earlier syntheses that have been produced in each of these fields, he
suggests some of the potential pitfalls of the field of research syntheses
(e.g. like not identifying all the relevant studies in the literature
search) and provides ideas on how the field can be developed in the future. 

EVALUATION
 
This book is a valuable addition to the applied linguistics literature. It
does not only provide clear guidelines on how to conduct a meta-analysis
and assumes no prior knowledge of this type of secondary research, but it
also presents a large number of successful models from a variety of
disciplines within the field of applied linguistics. I agree with the
sentiment of the authors whose motivation in providing these examples is
that reading successful models is the best way to learn about this type of
research. 

The topic of research synthesis is presented in a balanced manner, as the
authors also point to the limitations of this type of research. Overall,
this book is important to anyone wanting to know the value and limitations
of synthesizing research and to understand the variety of applications it
is suited to.

The book is also a valuable resource to readers less interested in
conducting this type of research for a number of reasons. Firstly, I agree
with the opinion of the authors that a certain ethic around conducting and
presenting primary research is necessary. For this, a raised awareness of
how secondary research is conducted is invaluable. Some authors of primary
research might not be aware of the importance of reporting e.g. effect
sizes or reliability analyses. After reading this volume, the need for
these becomes very clear. Secondly, the chapters in section two of the book
are not only of interest and relevance to readers interested in the art of
conducting research syntheses but are also of value to readers interested
in the different topic areas covered. Each of these present current, state
of the art research and therefore point to areas where more research is
necessary. 

Overall, the authors have managed to collate a very well-structured and
readable volume on a topic important to further the discipline of applied
linguistics.

REFERENCES
 
Norris, J.M & Ortega, L. (2000). Effectiveness of L2 instruction: A
research synthesis and quantitative meta-analysis. Language Learning, 50,
417-528. 

ABOUT THE REVIEWER

Ute Knoch is a Ph.D. candidate and research assistant in the Department of
Applied Language Studies and Linguistics at the University of Auckland. Her
research interests include language assessment, sociolinguistics and corpus
linguistics. She is a recipient of a 2006 Spaan Fellowship for Studies in
Second or Foreign Language Assessment.





-----------------------------------------------------------
LINGUIST List: Vol-17-3578