17.205, Review: Applied Ling: Kiely & Rea-Dickins (2005)

[LINGUIST List]

LINGUIST List: Vol-17-205. Sat Jan 21 2006. ISSN: 1068 - 4875.

Subject: 17.205, Review: Applied Ling: Kiely & Rea-Dickins (2005)

Moderators: Anthony Aristar, Wayne State U <aristar at linguistlist.org>
            Helen Aristar-Dry, Eastern Michigan U <hdry at linguistlist.org>
 
Reviews (reviews at linguistlist.org) 
        Sheila Dooley, U of Arizona  
        Terry Langendoen, U of Arizona  

Homepage: http://linguistlist.org/

The LINGUIST List is funded by Eastern Michigan University, Wayne
State University, and donations from subscribers and publishers.

Editor for this issue: Lindsay Butler <lindsay at linguistlist.org>
================================================================  

What follows is a review or discussion note contributed to our 
Book Discussion Forum. We expect discussions to be informal and 
interactive; and the author of the book discussed is cordially 
invited to join in. If you are interested in leading a book 
discussion, look for books announced on LINGUIST as "available 
for review." Then contact Sheila Dooley at dooley at linguistlist.org. 

===========================Directory==============================  

1)
Date: 20-Jan-2006
From: Fatos Erozan < fatos.erozan at emu.edu.tr >
Subject: Program Evaluation in Language Education 

	
-------------------------Message 1 ---------------------------------- 
Date: Sat, 21 Jan 2006 20:27:30
From: Fatos Erozan < fatos.erozan at emu.edu.tr >
Subject: Program Evaluation in Language Education 
 

AUTHORS: Kiely, Richard; Rea-Dickins, Pauline
TITLE: Program Evaluation in Language Education
SERIES: Research and Practice in Applied Linguistics
PUBLISHER: Palgrave Macmillan
YEAR: 2005
Announced at http://linguistlist.org/issues/16/16-2802.html 

Fatos Erozan, Department of ELT, Eastern Mediterranean University, 
Famagusta, North Cyprus

OVERVIEW

This book contributes to the theory and practice of language program 
evaluation significantly since it provides theoretical background to current 
understandings of language program evaluation from historical 
perspectives. Having dealt with theoretical background to language 
program evaluation, the book considers various case studies with frequent 
references to the theoretical information presented previously. The book 
has four main parts. Part 1 ''Departure Points'' explains the research in or 
theory of program evaluation from historical perspectives to set out the 
background to current understandings of language program evaluation 
through exploring three main fields, namely, social programs, educational 
programs and the specific field of language program evaluation. Part 
2 ''Cases and Issues'' presents seven evaluation studies, each of which 
follows a different format and clarifies different aspects of the evaluation 
process. These case studies are analyzed with reference to the concepts, 
frameworks and issues explained in Part 1. Part 3 ''Evaluation Practice and 
Research'' focuses on evaluation design and decision frames to guide 
readers' evaluation practices and support their research into evaluation 
processes. Finally, Part 4 ''Resources'' presents further resources for 
language program evaluation: books, journals, professional associations, 
ethical guides and best practice codes, e-mail lists and bulletin boards, and 
additional internet resources.

SUMMARY

The book is arranged in four main parts, the first three of which are 
organized in different chapters.

Part 1. Departure points
Chapter 1 ''Themes and Challenges'' outlines three features of evaluation 
and explains five challenges for it. The features of evaluation concern (i) its 
definition which has undergone paradigm shift throughout the history of 
program evaluation, (ii) the two perspectives of evaluation research, i.e. 
evaluation as a study which has both research and evaluation functions and 
evaluation as research into its processes, and finally, (iii) insufficient 
documentation and not publishing of evaluation reports. The five 
challenges for evaluation include (i) the purpose of evaluation in its social 
and political context, (ii) stakeholders (their participation, involvement and 
investments), (iii) evaluation criteria generating evaluation frameworks, 
instruments and judgments (Three approaches to specifying criteria for 
making judgments of worth about language programs are theory-based 
criteria derived from understandings of language learning process, policy-
based criteria established through professional considerations, and 
constructivist or ethnographic approaches determining criteria through 
internal program sense-making), (iv) evaluation data validating above-
mentioned approaches and instruments and completing the construction of 
judgments, and finally (v) evaluation use to manage social programs 
(considering new connections and new applications of evaluation processes 
and findings such as links between evaluation and research and links 
between evaluation and management).
 
Chapter 2 ''Historical Perspectives: Focus on Design and Method'' 
summarizes the early phases of evaluation theory and practice in social 
programs generally and in education with emphasis on conceptual aspects 
that have persisted over time and have the potential to develop current 
theory and practice. In addition, this chapter presents a chronological 
outline of a period when the focus of evaluation was on research design, 
data collection methods and clear evaluation outcomes for decision making 
and other judgments. This period reflects the shift of emphasis from 
empirical evaluation studies to more comprehensive accounts of program 
experience and the use of evaluation to understand develop or improve 
programs (i.e. paradigm shift from product to process or process and 
product oriented approaches to evaluation). The authors consider the 
history of the development of ideas as ''a cumulative layering'' (p.17). 

Chapter 3 ''Historical Perspectives: Focus on Context and Use'' focuses on 
the issues of evaluation use in educational programs, the contribution of 
constructivism and realism to the use of evaluation, and political and 
managerial dimensions of the improvement task. Having explained 
constructivism and realism in evaluation, the authors focus on the issue of 
political side of program evaluation and the problem of conformity with 
mandates.

Chapter 4 ''Historical Perspectives: Language Program Evaluation and 
Applied 
Linguistics'' considers program evaluation in language education and 
applied linguistics. The authors outline the evaluation trends in these two 
fields as: (i) the shift from an exclusive focus on measurement of outcomes 
to the evaluation of actual curricular experience, (ii) an increased attention 
to classroom processes, (iii) evaluation as the domain of professional 
practice, (iv) the development of teachers' and other professionals' skills, 
and finally, (v) inclusion of baseline and formative evaluation. The authors 
also explain how the developments in program evaluation correspond to 
developments in other fields and how the boundaries between research 
and evaluation in language programs have been redrawn in the direction of 
a movement from evaluation to research. Moreover, they indicate that 
evaluation has recently been considered as a platform for internal sense-
making and development in language programs that are part of 
development projects. Finally, they focus on a new dimension of language 
program evaluation, which is to understand language use as a quality 
indicator in non-language programs where communication and interaction 
are main activities (i.e. applied linguistics perspectives in program 
evaluation). 

Part 2. Cases and Issues
In Part 2, the authors present seven different evaluation case studies 
analyzing them with reference to the concepts, frameworks and issues set 
out in Part 1. In the presentation of each case study, context, aim and the 
scope of the evaluation, evaluation design or data collection procedures, 
findings of the evaluation, and finally implications of the study for 
evaluation were the issues explained and discussed.

Chapter 5 ''Evaluating Teachers' English Language Competence'' explains a 
case study which aimed at evaluating Basic English teachers' level of English 
language competence. Rather than being an evaluation of a particular 
program, the study was conducted to gather data for national strategy 
development. Design and validation of evaluation procedures such as 
language tests, class observation bandscales and questionnaires, were the 
focal points of this case study.

Chapter 6 ''Evaluating a Language through Science Program'' presents the 
evaluation of the language component of the 'Science Across Europe' 
Project which aimed to promote the use of a foreign language in working 
on the science materials and the exchange of information with schools in 
other countries through the use of information and communication 
technology. The evaluation study was carried out to gather empirical data 
about prevailing attitudes towards the program and how it was being 
implemented with the goal of informing management decisions as regards 
program development (for the program management) and deciding 
whether or not there was value for money (for sponsors). Teacher and 
student questionnaires were the data sources in this evaluation study. 

Chapter 7 ''Evaluating the Contribution of the Narrative Speaker Teacher'' 
describes a large-scale evaluation of an educational pilot program in the 
secondary school English language curriculum in Hong Kong, which 
emphasized the importance of use of English in the classroom. The 
evaluation study aimed to measure learning achievement caused by 
increased use of English for communication, i.e. the contribution of native 
speaker teacher as a linguistic and cultural resource. The evaluation used a 
mixed method approach; that is, it consisted of two separate studies: (i) a 
conventional experimental study based on learning outcomes, and (ii) a 
qualitative study of cultural, affective and social facets of school life and 
learning English (ethnographic case studies).

Chapter 8 ''Evaluating Foreign Language Teaching in Primary Schools'' 
focuses on the evaluation of a pilot program, an initiative, to introduce a 
modern foreign language into the primary school curriculum in Ireland. 
Concerning the evaluation design, as in the previous case study, two main 
strategies were adopted: a survey of teacher attitudes and a study of 22 
classes through assessment of learning outcomes, student attitudes and 
teachers' assessment of students' competence. The authors identify the 
strategies developed to elicit attitude and language performance data from 
young learners and the interface with policy making and program 
management as the two 'interesting' aspects of this evaluation study.

Chapter 9 ''Evaluating Program Quality in Higher Education'' presents an 
evaluation case study for quality management in an English for Academic 
Purposes program in a British university. Stakeholder approach to 
evaluation was implemented in this study and the purpose of the evaluation 
was to understand student learning experience and to use this 
understanding for improving the program. Development-oriented use and 
accountability (decision making)-oriented evaluation use were in the scope 
of this case study, and mid- and end-course evaluations were carried out.

Chapter 10 ''Evaluating the Student Experience in Higher Education'' 
emphasizes methodological aspects of student evaluation, and the case 
study presented in this chapter explores the student experience since the 
views of students as service users and clients are viewed as key drivers of 
policy and practice in current higher education, and therefore, the methods 
in which these views are constructed and understood are important. As in 
Chapter 9, a detailed account of an English for Academic Purposes program 
evaluation is presented in Chapter 10. 'Nominal Group Technique' (group 
discussion activity) was used as an evaluation instrument. The group 
discussion as a language learning activity, the ways in which students hold 
their stake (students' stakeholding), variations in status of different student 
voices (equal status of all voices), and the role of the teacher were the 
implications for evaluation raised in this case study.

In Chapter 11 ''Evaluating Assessment Standards and Frameworks'' two 
evaluation studies, with specific reference to assessment frameworks, are 
presented. The first study, which adopted documentary evaluation 
approach, is an evaluation of national and state frameworks for the 
assessment of learners of English as an additional language. The aim of the 
study was to provide comparative data on the assessment systems from 
different contexts of application. The second study, on the other hand, is 
an empirically driven evaluation of Canadian Language Benchmarks and the 
Centre for Canadian Language Benchmarks, which helped understand key 
processes in the implementation of an evaluation study using electronic 
communication. The authors claim that both studies highlight the relevance 
of in-depth case studies to capture further insights about the assessment 
frameworks, both perceptions and use (p. 199).

Chapter 12 ''Stakeholding in Evaluation'' focuses on the issue 
of 'stakeholding'. Enhancing evaluation utilization is the motivation for 
extending the involvement of stakeholders. Having defined the 
concept 'stakeholder', the authors elaborate on different classifications of 
stakeholders. They argue that students should be given strong voice in 
program evaluation - adopting not only an informant role but a participant 
role as well. They demonstrate the importance of stakeholder participation 
and how it can be achieved, arguing that stakeholder participation ensures 
validity, utilization and learning, the last of two which are the two main 
goals for evaluation. 

Part 3. Evaluation Practice and Research
Aiming to support evaluation practice and research into evaluation 
processes, this part of the book details evaluation design and decision 
frames.

Chapter 13 ''Large-scale Evaluations'' focuses on large-scale evaluations 
and explores the design and implementation issues specifically. Having 
discussed diversity in large-scale evaluations, the authors elaborate on 
issues like the limitations of large scale evaluations, ways of framing focal 
points for an evaluation, validation of evaluation issues and questions, 
evaluation procedures, the reasons for and the ways of developing 
evaluation skills, constraints to be worked with in evaluation, and ethical 
dilemmas. Overall, in this chapter, three aspects of large scale evaluations 
are emphasized: (i) understanding the evaluation construct, (ii) data 
management, and (iii) participants and the report (p. 245).

Chapter 14 ''Teacher-led Evaluations'' explains some issues in the design 
and implementation of teacher-led evaluations or evaluations that teachers 
can conduct in their own teaching contexts and that correspond with action 
research and reflective practice. These evaluations are opportunities to 
evaluate curricular resources, learning materials and classroom tasks, 
resources like information technology and libraries, and aspects of 
interaction in the teaching learning process. Moreover, they serve a 
professional development function. Aspects of the planned or intended 
curriculum constitute focal points of teacher-led evaluations. After 
describing the process of conducting these evaluations, the authors 
present some sample projects to further inform the teachers about 
purposes and process in evaluations that they can initiate in their contexts. 

Chapter 15 ''Management-led Evaluation Projects'' considers management-
led evaluations and sets out two frameworks and three sample evaluation 
designs which readers can use to conduct evaluations which aim to 
understand program management issues and improve program 
management processes. The authors argue that ''these evaluations 
provided opportunities for learning and development within programs, as 
in terms of program management, they furthered understanding of the 
contribution of wider institutional policy on resources and support for 
learning'' (p.271).

Part 4. Resources
Chapter 16 ''Resources for Language Program Evaluation'' presents further 
resources for language program evaluation to help program evaluators to 
engage with issues of evaluation theory, purpose and practice, examples of 
data collection instruments and data analysis (i.e. evaluation design), and 
evaluation reports. These resources are organized in sub categories as: (i) 
books, (ii) journals, (iii) professional associations, (iv) ethical guides and 
best practice codes, (v) e-mail lists and bulletin boards, and (vi) additional 
internet resources.

CRITICAL EVALUATION 

''Program Evaluation in Language Education'' is a comprehensive and 
important book, which connects theory/research and practice in program 
evaluation in language education exploring the relationship between the 
two. The book is well organized and written with introduction and summary 
sections in each chapter, which makes it reader friendly. One of the positive 
attributes of this book is that the authors first explain the theoretical 
background to language program evaluation in Part 1 and then exemplify 
and further this explanation by presenting various evaluation case studies 
in Part 2. Moreover, in Part 3, they particularly focus on evaluation design 
and implementation of three main types of evaluation again supporting, 
detailing and clarifying theoretical explanations with sample case studies.

Another important quality of this book is that it is a valuable addition to the 
literature on the research and practice of language program evaluation. 
Having reviewed some literature on the evaluation of second language 
programs and language teacher education programs, it can be seen that 
not much published research is available. In particular, there are not many 
published studies which can inform evaluators, teachers or researchers 
about language program evaluation case studies conducted in different 
contexts. Therefore, this book can be considered an important addition to 
the existing case studies of some researchers in the literature (Beretta & 
Davies, 1985; Spada, 1987; Sharp, 1990; Alderson & Beretta, 1992; Weir & 
Roberts, 1994; William & Burden, 1994; Mackay, Wellesley & Bazergan, 
1995; Frad & Lee, 1997; Lee, 1998; Dushku, 1998; Lewkowicz & Nunan, 
1999; Halbach, 1999; Jeffcoate, 2000; Ross, 2003; Kiely, 2003; etc.). 

The first part of the book presents dense or complex theoretical 
information concerning the historical perspectives in language program 
evaluation. Therefore, readers who do not have sufficient background in 
program evaluation theory or research may experience some difficulties in 
following some discussions. Overall, I can highly recommend this book to 
people who are equipped with some preliminary theoretical background in 
the field and are already involved in language program evaluation as a 
researcher, practitioner or the both.

REFERENCES

Alderson, J. C., & Beretta, A. (Eds.) (1992). Evaluating second language 
education. Cambridge: Cambridge University Press.

Beretta, A., & Davies, A. (1985). Evaluation of the Banglore Project. ELT 
Journal, 39(2), 121-127.

Dushku, S. (1998). ELT in Albania: project evaluation and change. System, 
26(4), 369-388.

Fradd, S. H., & Lee, O. (1997). Teachers' voices in program evaluation and 
improvement: A case study of a TESOL program. Teaching and Teacher 
Education, 13(6), 563-577.

Halbach, A. (1999). Using trainee diaries to evaluate a teacher training 
course. ELT Journal, 53(3), 183-190.

Jeffcoate, R. (2000). Teaching English grammar in initial teacher 
training: A course evaluation. Educational Research, 42(1), 73-85. 

Kiely, R. (2003). What works for you? A group discussion approach to 
program evaluation. Studies in Educational Evaluation, 29, 293-314.

Lee, I. (1998). Supporting greater autonomy in language learning. ELT 
Journal, 52 (4), 282-289.

Lewkowicz, J. A., & Nunan, D. (1999). The limits of collaborative evaluation. 
TESOL Quarterly, 33(4), 681-700.

Mackay, R., Wellesley, S., & Bazergan, E. (1995). Participatory evaluation. ELT 
Journal, 49(4), 308-317.

Ross, S. J. (2003). A Diachronic coherence model for language program 
evaluation. Language Learning, 53(1), 1-33.

Sharp, A. (1990). Staff/student participation in course evaluation: A 
procedure for improving course design. ELT Journal, 44(2), 132-137.

Spada, N. (1987). Relationships between instructional differences and 
learning outcomes: A process-product study of communicative language 
teaching. Applied Linguistics, 8(1), 137-161.

Weir, C., & Roberts, J. (1994). Evaluation in ELT. Oxford: Blackwell Publishers.

William, M., & Burden, R. (1994). The role of evaluation in ELT project 
design. ELT Journal, 48(1), 22-27. 

ABOUT THE REVIEWER

Fatos Erozan is a senior instructor in the Department of English Language 
Teaching at Eastern Mediterranean University, North Cyprus. She holds a 
Ph.D. in ELT (Middle East Technical University, 2005). Her research interests 
are language program evaluation, curriculum design, and ELT materials 
evaluation and design.





-----------------------------------------------------------
LINGUIST List: Vol-17-205