Debate: Reproducibility in Typology

Tue Aug 24 15:13:54 UTC 2004

Like other surveys, typological surveys should be reproducible. What
this could mean, for instance, is that after someone finds that two
features correlate in a sample of 50 languages using a certain sampling
method, someone else tries to reproduce the results using a totally
different sample and a different sampling method, to see whether the
results still hold.

Typology would profit from such work, but the problem is that the
authors wouldn't profit sufficiently -- merely replicating a known
finding is far less prestigious than coming up with a new one. I wonder
whether "Linguistic Typology" would publish a paper that merely confirms
(or merely disconfirms, for that matter) a known typological fact.

But reproducibility would show that typology really is a science. Maybe
one reason people do not do it is that they are not so convinced that is
not art after all. Especially the application of definitions to
different languages is not something that can be done totally
mechanically, and it is hard to get rid of the element of subjectivity.
For instance, Bybee, Perkins & Pagliuca 1994 ("The evolution of
grammar") note on p. xvii that they had to work very hard to ensure that
the coding by the three authors yielded similar results. They were happy
when they had reached ninety percent inter-coder reliability.

Martin Haspelmath

Frans Plank wrote:

>Re-doing Typology:  Call for Debate
>
>
>In good science one expects experiments to be reproducible:  results are
>only accepted if researchers other than those who designed and ran the
>original experiment and obtained results which are considered sufficiently
>interesting among the scientific community can independently repeat the
>same experiment and obtain the same results, over and over again.
>
>Scientific methods include not only experimentation but also surveys.  In
>the life sciences (like in many fields of biology), probably all population
>and ecological sciences, sciences of the past (palaeontology, archaeology,
>etc.), and sociology surveys are used rather than experiments.
>Essentially, an experiment is where you manipulate conditions, and a survey
>is where you select conditions.  Simulation is another standard procedure
>in some of the sciences, where experiments or surveys would be unviable.
>The replicability issue is equally important for surveys, experiments, and
>simulation.
>
>The purpose of this call for debate is to determine to what extent this
>principle of scientific method -- in order to be accepted, results must be
>reproducible -- is applicable to linguistic typology.
>
>For once, let's be catholic on what is typology.  It's linguistics in so
>far as it is about the human capacity for speaking and understanding, and
>its special remit is to observe, describe, and explain how particular
>manifestations of this capacity, as mentally representated and socially
>shared (grammars and lexicons, if you prefer), can and cannot vary across
>speech communities.
>
>The first real question then is whether typology is a science.  The
>alternative, to put it somewhat grossly, is that it is a form of art, like
>writing sonnets or interpreting them.   Most current practitioners of
>typology presumably opt for science, even when based in Arts or Humanities
>faculties rather than in Cognitive Science, and would like to see the
>results of their work evaluated as true or false rather than as beautiful
>or ugly -- or at any rate as true (yet unfalsified) or false (-ified) first
>and as giving pleasure or pain to the senses only second.
>
>The next question is whether typologists perform experiments as part of
>their professional activities.  Regardless of what else typologists do in
>the way of observing, describing, and forming hypotheses, they arguably do
>experiments when they select subsets from a more comprehensive set (the
>known or knowable linguistic universe, since the typologist's results
>typically aim to be universal) and obtain descriptive generalisations from
>such a sample by induction.  Other typologists see themselves as doing
>surveys, selecting conditions rather than manipulating them.  Simulation
>has been done in typology, too, if less commonly than in other fields.
>What it means to be doing experiment-typology or survey-typology or
>simulation-typology is another question that remains to be clarified.
>
>Now, as to the primary issue which we would like to see debated, what would
>it mean to reproduce a typological experiment or survey?   For example,
>would it be reproduced on the same or on a different sample?  How many
>typological experiments or surveys have actually been reproduced, one way
>or another?  (Probably not many:  identifying counterexamples to
>descriptive generalisations, which is being done a lot, is not
>reproduction.)  What are the obstacles hindering reproduction?  How to
>remove them?  Is it worth the effort?  (And the efforts required could be
>considerable, including for instance the obligation to make one's data and
>their interpretation available to fellow researchers, in publications or on
>a web site.  Presumably, interpretations of data by typologists would claim
>a different epistemological status than interpretations of sonnets by
>literary scholars:  in typology, data as well as interpretations ought to
>be reproducible.)  What follows if an experiment or survey is replicated
>but the results cannot be reproduced?  Is sampling the only part of the
>typologist's work where reproducibility is potentially an issue?
>
>
>LT invites contributions on the role of reproducibility in typology -- of
>whatever genre:  statements of principle, methodological discussions,
>reflections on your own experimental practice, actual attempts to reproduce
>the results of others, reports on the difficulties encountered.
>
>Send to the editorial address of LT (for attached text files:
>frans.plank at uni-konstanz.de) by the end of October 2004.  A discussion
>section in LT 9-1 (2005) will be set aside for this purpose.  The Editorial
>Board reserves the right to select submissions for publication.  Concise
>contributions are preferred, but longer papers on the subject are also
>welcome and will be reviewed as regular submissions.  In the first instance
>you might also want to share your thoughts with the typological community
>on lingtyp at listserv.linguistlist.org
>
>As an alternative outlet, there is of course The Journal of Irreproducible
>Results (http://www.jir.com/).
>
>
>The Editorial Board of LT
>
>
>
>

--
Martin Haspelmath (haspelmath at eva.mpg.de)
Max-Planck-Institut fuer evolutionaere Anthropologie, Deutscher Platz 6	
D-04103 Leipzig
Tel. (MPI) +49-341-3550 307, (priv.) +49-341-980 1616