16.1609, Review: Multimodality/Discourse: Ventola et al. (2004)

[LINGUIST List]

LINGUIST List: Vol-16-1609. Thu May 19 2005. ISSN: 1068 - 4875.

Subject: 16.1609, Review: Multimodality/Discourse: Ventola et al. (2004)

Moderators: Anthony Aristar, Wayne State U <aristar at linguistlist.org>
            Helen Aristar-Dry, Eastern Michigan U <hdry at linguistlist.org>
 
Reviews (reviews at linguistlist.org) 
        Sheila Dooley, U of Arizona  
        Terry Langendoen, U of Arizona  

Homepage: http://linguistlist.org/

The LINGUIST List is funded by Eastern Michigan University, Wayne
State University, and donations from subscribers and publishers.

Editor for this issue: Naomi Ogasawara <naomi at linguistlist.org>
================================================================  

What follows is a review or discussion note contributed to our 
Book Discussion Forum. We expect discussions to be informal and 
interactive; and the author of the book discussed is cordially 
invited to join in. If you are interested in leading a book 
discussion, look for books announced on LINGUIST as "available 
for review." Then contact Sheila Dooley at collberg at linguistlist.org. 

===========================Directory==============================  

1)
Date: 19-May-2005
From: Judith Cross < jrkjcross at bigpond.com >
Subject: Perspectives on Multimodality 

	
-------------------------Message 1 ---------------------------------- 
Date: Thu, 19 May 2005 23:15:48
From: Judith Cross < jrkjcross at bigpond.com >
Subject: Perspectives on Multimodality 
 

EDITORS: Ventola, Eija; Charles, Cassily; Kaltenbacher, Martin 
TITLE: Perspectives on Multimodality
SERIES: Document Design Companion Series
PUBLISHER: John Benjamins Publishing Company
YEAR: 2004
Announced at http://linguistlist.org/issues/16/16-135.html


Judie Cross, Department of Languages, Randwick TAFENSW, Sydney Institute

SUMMARY

This volume addresses the current need for innovative and scholarly 
analyses of multimodal discourse characterising both new and old 
media.  The collection of papers is a welcome contribution for those in 
the research community with an interest in examining how 
communication is affected by the interaction of semiotic resources in 
texts.  The papers are organised into chapters forming two main 
sections: the first part explores problematic theoretical issues; the 
second part describes a range of increasingly specific applications.

SYNOPSIS

In Part 1, "Multimodal Issues", the first chapter of the book, "In 
between modes: Language and image in printed media", Hartmut 
Stöckl focuses on modes.  He addresses the suggestion by Kress and 
van Leeuwen (2001), that all modes are characterised by semiotic 
properties, cognitive orientation and semantic potential, but 
distinguished according to the way in which they are perceived.  He 
attempts to identify a common set of semiotic principles, which could 
be operating across all modes constituting a hierarchical network.

Chapter 2, "Modelling multiple semiotic systems: The case of gesture 
and speech" explores mode as the primary organiser of semiotic 
systems and suggests viewing these systems along a continuum 
ranging from a proto-language to a fully-fledged one.  Peter Muntigl 
views all modes (music, dance, gesture) as being modelled on 
language and structured by a grammar, which is amenable to analysis 
according to Halliday's metafunctions (1994).

Victor Lim Fei in Chapter 3, "Problematising 'semiotic resource'", also 
builds on Systemic-Functional Linguistics (SFL) theory and 
problematises modes, but concentrates on images, positing icons as 
the vocabulary of visual language.

In the last chapter of Part 1, Chapter 4, "Multimodality and empiricism: 
Preparing for a corpus-based approach to the study of multimodal 
meaning-making", John Bateman, Judy Delin and Renate Henschel 
present their innovative multimodal annotated corpus GeM (Genre 
and Modality), the aim of which is to investigate and validate the 
appropriateness of a multimodal view of genre.  They argue that since 
most multimodal discourse analysis is informal and interpretative, it is 
necessary to gather evidence for the theories proposed.  This they 
are attempting to do by developing a corpus of annotations.  They 
outline their procedure for this, providing examples of how tags are 
constructed for the variety of visual layers (photos, captions, headers) 
in documents.

Part II, "Analyses and Applications", covers a range of multimodal 
critiques from different disciplines and contexts.  In the first chapter of 
this section, "On the effectiveness of mathematics", Kay O'Halloran 
demonstrates that visual display in mathematics is crucial as regards 
meaning and problem solving.  She traces the presentation of 
mathematical problems since the early Renaissance and argues that 
mathematics has always been a form of multimodal discourse, 
although its representation appears to have been less abstract in the 
past than it is today.  She frames mathematics as a multisemiotic 
construction of language, mathematical symbolism and visual display, 
which function inter-semiotically as well as intra-semiotically.  Her 
conception is offered as a way of facilitating the examination of 
meaning potential of system choices within and across resources.

In Chapter 6 "Multimodality in language teaching in CD-ROMs", Martin 
Kaltenbacher presents evidence demonstrating that the visuals used 
in many CDs created for English language teaching are too general or 
ambiguous to assist language learning.  He analyses a variety of still 
and moving images including the feedback provided by the integration 
of sound waves, text-picture combinations and the use of short video 
clips.  He makes the point that too often these visualisations are not 
easily interpretable, do not help disambiguate meanings to be learnt 
and are not exact representations.  His proposal is to replace complex 
visuals with icons.

Markus Rheindorf next investigates the relations between modes in 
the film "Dirty Dancing".  His approach to film is a transdisciplinary 
one, where he analyses an ensemble of modal patterns, which he 
argues, characterise the film genre and, if recognised, can thereby 
deepen our insights.

In Chapter 8, "Multimodal text analysis and subtitling", Christopher 
Taylor outlines a methodology for obtaining multimodal transcripts for 
film. He demonstrates how meaning created in one modality (such as 
word play in text) may be best translated in another (such as visual) 
and therefore advocates intersemiotic translation techniques as a 
basis for the selection of verbal elements for subtitling.

In the next chapter, "Multimodality in the translation of humour in 
comics", Kaindl stresses how the humour in comics is not only created 
verbally, but also via the visual or non-verbal dimension of texts.  
Kaindl argues that pictorial elements have, however, generally been 
disregarded by translators, and that this should no longer continue. 
He advocates translators develop methods of analysis, which take into 
account the holistic nature of these semiotically complex texts. 

In "Multimodality in operation", Chapter 10 by Andrea Hofinger and 
Eija Ventola, the design of museum exhibits are described as semiotic 
spaces, which not only display objects for viewing, but also involve 
dynamic processes of interpretation.  A multimodal description, based 
on the systemic-functional linguistic model of Halliday (1978), is 
applied to a room in the Mozart-Wohnhaus Museum in Salzburg, 
focusing specifically on how meaning is constructed by choice of 
objects and their display arrangement.  The authors demonstrate how 
selection and arrangement of meaning resources (such as a family 
portrait and an audio-taped recording) interrelate and contribute to or 
detract from the overall impact of an exhibit. 

"Drawing on theories of inter-semiotic layering to analyse multimodality 
in medical self-counselling texts and hypertexts" is the title of Chapter 
11.  In this chapter Eva Martha Eckkrammer adopts an integrative 
approach to text and hypertext, arguing that pictorial and verbal 
elements always develop meaning through a process of inter-semiotic 
layering.  Transposition, juxtaposition, combination and fusion are the 
four most frequent forms of layering she identifies via a practical focus 
on medical self-counselling texts, raising questions for the 
development of multimodal research and text intelligibility.  

The final chapter in this collection is "On the multimodality of 
interpreting in medical briefings for informed consent".  In this chapter 
Kristin Bührig analyses a particular type of hospital interaction whose 
purpose is to provide a briefing for informed consent.  The aim of 
using an interpreter in such situations is to facilitate understanding 
between a doctor and patient. In the particular briefing investigated, 
Bührig demonstrates how the doctor and the untrained interpreter (as 
is commonly the situation) used different "linguistic action patterns": 
the doctor used a diagram systematically to build on the patient's 
knowledge; the daughter-interpreter only used the diagram as a 
reference for linking new knowledge.  Without skills or preparation in 
building-up knowledge by integrating language with other modes, it 
seems language barriers can only be partially overcome.

EVALUATION

As stated at the end of the book's introduction, its impetus was 
provided by the discussions that occurred during the First 
International Symposium on Multimodal Discourse at the University of 
Salzburg in 2002. In this respect the volume contributes a wealth of 
exploratory, innovative and challenging perspectives to the 
development of the theory and practice of multimodal text analysis.  
Further, the sequencing of the papers in this volume are organised in 
a manner that allows for the gradual cumulation of relevant 
knowledge.  In spite of the title "Multimodal Perspectives", however, it 
is still the case that a systemic-functional linguistic analysis of the 
interaction of written text and images predominates, while other 
frameworks and modes receive less attention.

Probably the most valuable and challenging aspect of the collection is 
the final chapter of the first part, which outlines the enormous task 
begun by Bateman et al to build up a corpus of annotations 
appropriately tagged so that a body of evidence can be accessed for 
supporting interpretations of multimodal texts. While recognising the 
value of current multimodal analyses, the generally impressionistic 
nature of such analyses is understandably of concern to the authors 
of this chapter.  In response they are designing multimodal corpora, 
drawing on state of the art methods in their endeavour to establish 
empirically whether a systematic and regular relationship exists 
between different document genres and their potential realisation in 
various combinations: verbal text, layout, graphics, pictures and 
diagrams. The method by which the GeM model of layering for 
classification is outlined and exemplified in considerable detail and it 
appears that, in endeavouring to address their concern, the designers 
have set themselves long-term and demanding work, not only as 
regards scale and detail, but also as regards whether this analytical 
approach is appropriate for the kinds of meaning-making which is their 
purpose.  Nevertheless, this approach is, as acknowledged, simply a 
necessary one to be attempted and is, therefore, impressive, of great 
interest to the research community and potentially easily accessible.

The three other chapters in this first theoretical section of the book 
are more exploratory in nature, further problematising the interaction 
of semiotic resources from a Systemic Functional Linguistics (SFL) 
perspective, especially with reference to the ground-breaking work of 
Kress and van Leeuwen (1996).  Muntigl's approach is refreshing as 
he attends not only to visual systems, but also to gesture and sign 
languages.  Acknowledging Kress and van Leeuwen (1996) as well as 
the solid work of Kendon (2004), Muntigl's suggestion that semiotic 
systems can best be perceived along a continuum is illuminating. Lim 
Fei's SFL perspective, however, is slightly more radical in its proposal 
that the building blocks for visual images be icons, just as lexical items 
are the building-blocks for language.  As a means for stimulating 
debate, this is an interesting suggestion although, as Lim Fei 
acknowledges, there would be considerable scepticism regarding 
what constitutes the boundaries of an icon, its recognition, 
arrangement and internal grammar. 

Kay O'Halloran's chapter on the history and construction of 
mathematics, situated at the beginning of section two, is an 
outstanding contribution to this book.  For those with a minimal 
understanding of mathematics, however, this is not an easy read.  
Despite this, its value lies in how it still manages to demonstrate 
convincingly the means by which mathematics, a multisemiotic 
discourse which makes meaning through an interlocking of systems 
(symbolism, visual display and language), has been and will continue 
to be an effective and valuable resource for describing our physical 
experience of the world.

Kaltenbacher's critique of language teaching CD-ROMs makes some 
thought-provoking comments as regards the ambiguous nature of 
many visuals in these resources, although the sample investigated 
may not be representative of what is becoming available globally.  
Rheindorf exemplifies a valuable transdisciplinary approach to film, but 
stops short of specifying how one's insights can be deepened. The 
next two chapters on subtitling and translation, however, exemplify the 
value and expansion of meaning created by the extension of linguistic 
analysis to encompass other modes. The disciplines of translating and 
interpreting can only benefit from adoption of a multimodal approach.  
Further, Hofinger and Ventola's analysis of intersemiosis in operation 
provides an illustrative application and adaptation of the value of an 
SFL approach to multimodal constructs as well as providing a model 
for creating dynamic multimodal displays.  

Finally, valuable examples of how important understanding of and skill 
in the process meaning-making, created by the interaction of semiotic 
resources, is found in the medical focus of the last two chapters.  
Bührig demonstrates convincingly, by way of contrasting two different 
communication styles making use of a diagram, the significance of an 
approach, which has a sound systematic basis.  This final chapter, like 
many of the others, exemplifies the need for continuing investigation 
into the ways by which language and other modes, such as the visual 
and gestural, involve the development of awareness, fresh 
orientations and specialised skills for effective communication.

REFERENCES

Halliday, M. A. K. 1994. An introduction to functional grammar (2 ed.).
London: Edward Arnold.

Kendon, Adam.  2004.  Gesture: Visible Action as Utterance. 
Cambridge: Cambridge University Press.

Kress, Gunther & van Leeuwen, Theo.  2001.  Multimodal Discourse: 
The Modes and Media of contemporary Communication.  London: 
Arnold.

Kress, Gunther & van Leeuwen, Theo.  1996.  Reading Images: The 
Grammar of Visual Design. London Great Britain: Routledge .

Lemke, J. 1998. Multiplying meaning: Visual and verbal semiotics in
scientific text. In J. R. Martin & R. Veel (Eds.), Reading science:
Critical and functional perspectives on discourses of science (pp. 87-
113). London: Routledge.

O'Toole, Michael.  1994.  The Language of Displayed Art.  London: 
Leicester University Press. 

ABOUT THE REVIEWER

Judith Leah Cross was awarded her PhD by Macquarie University in 
1999 for her thesis 'Textual Realisations', which built on the theories 
of Kress and van Leeuwen (1996) as well as O'Toole (1994) in order 
to examine how meaning-making is affected when printed children's 
image texts are adapted into film, comics, or electronic formats.  
Multimodality was the focus for her thesis and continues to be of 
increasing relevance to her present work in curriculum design and 
blended delivery of English for Academic Purposes.





-----------------------------------------------------------
LINGUIST List: Vol-16-1609