11.1414, Review: Gass & Mackey: Stimulated Recall Methodology

[The LINGUIST Network]

LINGUIST List:  Vol-11-1414. Sun Jun 25 2000. ISSN: 1068-4875.

Subject: 11.1414, Review: Gass & Mackey: Stimulated Recall Methodology

Moderators: Anthony Rodrigues Aristar, Wayne State U.<aristar at linguistlist.org>
            Helen Dry, Eastern Michigan U. <hdry at linguistlist.org>
            Andrew Carnie, U. of Arizona <carnie at linguistlist.org>

Reviews: Andrew Carnie: U. of Arizona <carnie at linguistlist.org>

Associate Editors:  Ljuba Veselinova, Stockholm U. <ljuba at linguistlist.org>
		    Scott Fults, E. Michigan U. <scott at linguistlist.org>
		    Jody Huellmantel, Wayne State U. <jody at linguistlist.org>
		    Karen Milligan, Wayne State U. <karen at linguistlist.org>

Assistant Editors:  Lydia Grebenyova, E. Michigan U. <lydia at linguistlist.org>
		    Naomi Ogasawara, E. Michigan U. <naomi at linguistlist.org>
		    James Yuells, Wayne State U. <james at linguistlist.org>

Software development: John Remmers, E. Michigan U. <remmers at emunix.emich.edu>
                      Sudheendra Adiga, Wayne State U. <sudhi at linguistlist.org>
                      Qian Liao, E. Michigan U. <qian at linguistlist.org>

Home Page:  http://linguistlist.org/

The LINGUIST List is funded jointly by Eastern Michigan University,
Wayne State University, and donations from subscribers and publishers.


Editor for this issue: Andrew Carnie <carnie at linguistlist.org>
 ==========================================================================

What follows is another discussion note contributed to our Book Discussion
Forum.  We expect these discussions to be informal and interactive; and
the author of the book discussed is cordially invited to join in.

If you are interested in leading a book discussion, look for books
announced on LINGUIST as "available for discussion."  (This means that
the publisher has sent us a review copy.)  Then contact Andrew Carnie at
     carnie at linguistlist.org

=================================Directory=================================

1)
Date:  24 Jun 2000 08:10:12 EDT
From:  Lotfi at www.dci.co.ir
Subject:  Review of Gass and Mackey's book

-------------------------------- Message 1 -------------------------------

Date:  24 Jun 2000 08:10:12 EDT
From:  Lotfi at www.dci.co.ir
Subject:  Review of Gass and Mackey's book

Susan M. Gass, and Alison Mackey, (2000), Stimulated Recall
Methodology in Second Language Research, Lawrence Erlbaum
Associates, Mahwah, New Jersey. 177p.

Reviewed by Ahmad Reza Lotfi, Azad University

INTRODUCTION

1. We had just left the breakfast table when Azizeh (my wife) and I
noticed the postman coming directly to the front door with a khaki
parcel in one hand and waving triumphantly at us with the other most
probably to remind me of the tip I usually hand him on the (not very
rare, I'm afraid) occasion of a book or something from overseas!
Azizeh and I exchanged meaningful glances, and she rushed to get her
video camera ready. Yes! It was Gass and Mackey's 'Stimulated Recall
Methodology'! We'd already read a few lines about it, and we were quite
prepared for its arrival!

2. A few minutes later, as we were watching the short video on the telly
(with the 'superstar' Mr Myself unwrapping the book and examining some
pages so 'philosophically' that reminded one of Sherlock Holmes himself
examining a piece of evidence), she pushed pause and said," I see you're
smiling as you examine the book. What were you thinking at that point?"
"Er... I just thought it was not a very thick book, and I could return
its review to LINGUIST (perhaps for the first time) in due time, and with
no violation of size limits either!" "Come on, wise guy! You were to
report your thoughts AT the time of thinking. Then don't 'enrich' them
with what you see into it right now! "Sorry! The point taken! I thought
it wasn't a very thick book, and it wouldn't take a lot of time to read
it. Satisfied?" "Yes. I also noticed you checked the end of the book
first. Do you recall what you were thinking then?" "Yes. I was telling
myself it couldn't be more than 200 pages, and I would be surprised if
it had a long list of references."  "And then?" "Well, ... I got sur-
prised!" "Anything else to report?" "I noticed I didn't like the
purplish cover of the book. It felt cold and unfriendly."

3. And that was how we had our first try at stimulated recall procedure!

STRUCTURE

4. The book belongs to a series of monographs on issues in second
language research methodology published by LEA. There are five chapters
in this work dealing with mainly practical aspects of introspective
research in general and stimulated recall methodology in particular.
Two detailed sample research protocols for stimulated recall proce-
dure (7 pages in total), and also some examples of SR data (3 pages)
are attached to "Stimulated Recall Methodology in Second Language
Research" as appendixes. Additional references are also provided at
the end of the book. The content of each chapter is briefly examined
below.

SYNOPSIS

Chapter One: Introduction to Introspective Methods (1-24pp)

5. Introspection--examining one's mental processes--may be traced to
Descartes, Augustine, and Aristotle. The rise of behaviourism in
psychology with its emphasis on observing, measuring, and inter-
preting human behaviour dismissed introspective methods as an un-
reliable method of doing science. The rejection of behaviourism,
however, paved the way for the reappearance of introspection.
A particular type of introspection, namely grammaticality judgements,
happened to become the main linguistic methodology of research.

6. Stimulated recall is an introspective method in which participants
are prompted (via some visual or oral stimulus such as a video/audio-
taped event, or any other tangible reminder such as different drafts
of a composition, etc) to recall thoughts they entertained while
carrying out certain tasks or participating in certain events. The
method is superior to a simple post hoc interview in that the parti-
cipant does not need to heavily rely on memory without any prompts.
Moreover, it has an advantage over thinking-aloud protocols because
the participant does not need to go through a process of training
in order to be able to perform a task and talk about it simultaneous-
ly.

7. SR methodology is appealing to second language (acquisition)
researchers as it can shed light on some dark corners of the learner's
mind, whose activities are assumed to be partly available to her con-
sciousness though NOT accessible via behaviour-oriented empirical
methods. Knowledge types (especially the learner's declarative knowledge
but even one's procedural knowledge when there are some breakdowns in
automatic processing), knowledge structure including plans and scripts,
and cognitive processes and learner strategies are among L2 reserach
topics than can be explored through introspective methods.

Chapter Two: Introspection and Second Language Research (25-35pp)

8. Verbal reporting, both on-line and retrospective, and stimulated re-
calls have been introduced in L2 research for a while. Gass and Mackey
list seventy-one L2 research projects in which introspective methods
have been employed. Four of them were conducted in the seventies, twenty-
six in the eighties, and all others in the nineties or still in press.
This suggests that inrospective methods are gaining momentum every day.
Such studies cover a wide variety of topics including L2 test-taking,
reading, writing, listening, speaking, strategy use, grammaticality
judgements, translation, vocabulary, pedagogical knowledge, and gender
communication strategies.

Chapter Three: Characterization of Stimulated Recall (37-55pp)

9. The stimulated recall procedure is reviewed here in reference
to studies in the areas of (a) oral interaction research, (b)
acceptability judgements, (c) reading/vocabulary, (d) writing, and
(e) pragmatics. Hawkins (1985) tries to see 'whether replies in
non-native speaker discourse, which on the surface appeared to be
appropriate conversational responses, did in fact represent com-
prehension of what had preceded in the discourse' (p.38). Partici-
pants carried out different communicative tasks in English. The
tape-recorded interactions between native and non-native speakers
were played back to the participants, who could stop the recorder
and comment on what they were thinking at that time.

10. Mackey, Gass, and McDonough (in press) focused on the accuracy
of learners' inferences of native speaker feedback in oral com-
munication. In picture-description task, native speakers provided
feedback when non-native participants made an error. The partici-
pants watched the videotape and described their thoughts to see
how they perceived such feedback episodes. Recall sessions were
also audiotaped for the appropriate (quantitative/qualitative)
analysis of the data.

11. Tyler (1995) studied a tutoring session between a Korean graduate
student in Computer Science and a native speaker of English learning
computer programming. Each complied a list of signs of interlocuters'
discomfort such as volume changes. They were then asked to watch their
videotaped interactions and comment on things that made them un-
comfortable.

12. Seven more L2 studies that employed SR methodology are described
similarly: Dornyei and Kormos's (1998) study of speakers' management
of problems in L2 communication, Gass's (1994) analysis of L2ers'
grammaticality judgements of English sentences, Paribakht and Wesche's
(1999) study of learner strategies and knowledge resourses while deal-
ing with unfamiliar words, DiPardo's (1994) research on writing pro-
grammes for non-Anglo students, Rose's (1984) survey of writer's
block during the composing process, Bosher's (1998) investigation of
differences between those graduated from the high schools in the U.S
and those graduated from high schools in their home countries with
regard to the composing process, and Robinson's (1991) study of the
refusal speech act in the context of Japanese ESL.

13. Information is provided about the classification of the different
aspects of the SR procedure. The authors base their classification
system on Faerch and Kasper's (1987) scheme according to which intro-
spective studies may differ from each other in terms of (a) relation-
ship to specific action (how concrete/specific or non-specific/abstract
the cognitive information we focus on is), (b) temporal relationship to
action (how immediate/delayed the retrospection is), (c) participant
training, (d) procedural structure (whether the data collection instru-
ments are highly structured, e.g questionnaires with multiple choice
items, or not), (e) stimulus for recall (how strong the support is),
and (f) initiation of questions/recall interactions (who begins the
stimulus episodes, who interacts with participants, etc).

Chapter Four: Using Stimulated Recall Methodology (57-104pp)

14. This chapter is the heart of the book where the authors provide
a detailed account of the very practice of SRM in L2 research. Suf-
ficiently detailed guidelines are given on how a research protocol--'a
set of instructions, parameters, and details for carrying out an
experiment'(fn.1 p.57)--is developed for SRM. Instead of contrived
examples usually found in similar books, real examples from recent
L2 studies are provided that get the reader into the heart of the
practice: Mackey, Gass, and McDonough (in press), Leeman (1999), and
Mackey and Gum (1997) among others.

15. Researchers are advised to secure the interrator reliability of
their SR studies because it is very typical of these studies to have
researchers themselves collect, transcribe, code, rate, and analyse
the data. Gass and Mackey, however, 'generally recommend finding in-
dependent rators who were not participants in the original event to
be recalled and who were also not participants in the subsequent re-
call, or, if this is not possible, finding rators who participated in
the same events, either the original or the recall' (p. 66). Extended
examples of rating training and recommendation boxes are provided.

16. Real coding sheets (both preliminary and final versions) for
stimulated recall accompanied with the authors' interpretations and
explanations constitute the section on data layout and coding. Also
procedural pitfalls including the issues of timing (the approximate
length of the recall support, allocating time for the recall pro-
cedure, and allocating time for set up and equipment), recall ques-
tions, and language of the recall session are discussed at length.
A brief discussion of the analysis of stimulated recall data (dealing
with such issues as quantitative versus qualitative analyses, the
quantification of qualitative data, and sampling the data) concludes
the chapter.

Chapter Five: Limitations and Additional Uses (105-136pp)

17. The most basic problems of stimulated recall methodology, which
have something to do with its reliability and validity, are addressed
here. The assumption underlying introspective methods is that higher
order cognitive processes are accessible to introspection, which is
not necessarily granted. Online introspective reports suffer the weak-
ness of verbalisation affecting the very cognitive processes the parti-
cipant is to report. Retrospective reports, on the other hand, are open
to the participant's memory limitations, especially when the amount of
time intervening between the event and the recall is considerable.

18. Within the context of second language research, Gass and Mackey
review the literature (mainly Cohen 1996, 1998, and Matsumoto 1993)
with regard to advantages and disadvantages of the SR procedure. They
list the advantages of the method as:

   - Reflect a theoretical framework
   - Reveal the information attended to during task performance
   - Reflect cognitive events
   - Are reliable in that they correlate with behavior
   - Are useful in strategy research
   - Are useful in determining what prior knowledge is used in
     processing texts  (p. 111)

Also a number of disadvantages show up:

   - Unconsciousness of cognitive processes
   - Complexity of cognitive processes
   - Inaccurate reporting on the part of participants
   - Inaccessibility of some information
   - Confounding of introspection and retrospection
   - Intrusive
   - Dependent on verbal skills of participants
   - for L2 research, the language of processing versus the language
     of reporting
   - Veridictability
   - Generating verbal reports may alter the nature of the process'
                                                         (pp. 111, 112)

19. Admitting such shortcomings as the limitations of this research
tool--the authors remind us of the fact that '[n]o methodology is with-
out critics', and most probably no research tool without limitations.
The authors recommend some measures to be taken in order to minimise
the limitations. Among them are: improve the reliability in terms of
sampling, rating, conducting the event and the recall, etc, the recall
session immediately after the event itself, and stimulated recall
methodology as a supplement to empirical L2 research.

20. Gass and Mackey conclude the chapter with a detailed analysis of
fourteen L2 studies (all of them published in 1998 in the leading
journals of the field) in the areas of interlanguage phonology, class-
room interaction, oral production, interlanguage pragmatics, compre-
hension, input and output processing, L2 reading comprehension, oral
interaction, syntactic processing, and vocabulary where the insights
could be furthered had the researchers employed stimulated recalls as
the supplement to their actual empirical procedures. One typical
example of such possible improvements is the study reported in Hoover
and Dwivedi (1998) exploring the syntactic processing by fluent English
speakers of French as L2. 'Participants read sentences word by word,
pressing a space bar for each new word. ... Following the main
experiment, participants were give a standardized reading comprehension
test ... . Thie specific hypothesis, which was confirmed, was that
slow L2 readers would be less efficient in their processing of syn-
tactic information' (128-129). As Gass and Mackey noted, stimulated
recall could shed light on the question of whether or not the results
were merely a factor of word recognition (among other possible explana-
tions for the same observation).

CRITICAL EVALUATION

21. This monograph on SRT was badly needed in the field of L2 research:
the researcher feels she cannot restrict herself to the research tools
(over)used for years, and SRT may prove be new blood in the vein of
SLA research. The cognitive touch of the method on the one hand, and its
practicality on the other, increasingly draws the researcher's attention
to SRT. And she can find what she needs in Gass and Mackey's. The
practice-oriented approach of the monograph makes it ideal for those
who think they have had their share of theory but still feel insecure
when it comes to real practice. Being closely involved in the SR pro-
cedure in L2 research, Gass and Mackey are just the people to tell them
how to do it. And they do, too.

22. One of the advantages of the book is its avoidance of contrived and
usually idealised examples in favour of real cases, which mostly
happen to be good examples of L2 studies in recent years by leading
researchers. Frequent references to the different aspects of these
specific cases throughout the work make it possible for the reader to
follow the research projects at different stages and get insight into
the pitfalls and hazards inherent in any study and how the researchers
in question did or could react towards them.

23. Gass and Mackey disarm the critical reader with their admission of
the limitations of the methodology in terms of falsifiability, replica-
bility, reliability and validity. However, they still advocate it simply
because through introspection 'one can often gain access to processes
that are unavailable by other means ...'(16-17). A scientific method of
data collection, however, is not any truth-seeking device one can use
when no other choices are missing. As Gass and Mackey note, one
particular form of introspection, i.e. grammaticality judgements, has
turned into the dominant method of data collection in some brands of
linguistics. Though introspective, grammaticality judgements still pass
such tests as falsifiability and replicability. Then the top priority
for a SR researcher must be a serious attempt to secure these scienti-
fic requirements. If a research tool proves to be beyond hope in these
respects, all one can do is bid it adieu.

24. Although the text is relatively short, it could be even shorter
without leaving anything unsaid. The authors repeat themselves over
and over with regard to some fairly commonplace details of the pro-
cedure--for instance, that both the participant and the researcher
can push pause, that the time interval between the event and the
recall must be a minimum, that planning ahead for the sessions is
important, etc etc. Even if they intended to emphasise the import-
ance of these issues, they had to provide new information each time
they revisited them. Gass and Mackey don't.

25. On the other hand, there are many important issues that are not
thoroughly examined. For instance, no single word is said about the
statistical procedures usually employed for the quantitative analysis
of introspective data nor the statistical measures typically reported
in introspective L2 studies. Then one wonders whether one should
quantify the data in order to compare them in terms of the means of the
distributions, to measure their relationship via correlation analysis,
or what. The only statistics they offer is a short footnote on page
63 that invites readers to consult two references (Hatch and Laza-
raton 1991 and Seliger and Shohamy 1989) on interrater reliability
coefficients. Neither of these two is particularly up-to-date, and at
least one of them, Seliger and Shohamy's which I am closely familiar
with, contains absolutely nothing about introspective methods nor
statistics in general. While I do not expect Gass and Mackey to teach
statistics in their monograph on methodology, I see it a must for them
to give their readers some hints about the statistics they need for
their research projects and also where they can learn about them. I
hope the future editions of the text will take sufficient care of
these gaps.

 =======================================================================
Reviewer: Dr. Ahmad R. Lotfi, Assistant Professor of Linguistics and
Chair of the English Department at Esfahan Azad University where he
teaches Syntax to graduate students of TESOL. His research interests
lie in minimalist syntax, second language acquisition studies in
generative grammar, and Persian linguistics.
lotfi at www.dci.co.ir
 =======================================================================

      ____     ____    __
    / ___ \  / ___ \  / /               Ahmad Reza Lotfi, Ph. D.
   / /__/ / / /__/ / / /           Chair of English Department
  / /__/ / / ___ /  / /____ Graduate School, Azad University
 /_/  /_/ /_/  \_\ /______/    at Khorasgan, Esfahan, IRAN
- ------------------------- MailTo:  lotfi at www.dci.co.ir


---------------------------------------------------------------------------

If you buy this book please tell the publisher or author
that you saw it reviewed on the LINGUIST list.

---------------------------------------------------------------------------
LINGUIST List: Vol-11-1414