Debate: Reproducibility in Typology

Tue Aug 24 16:33:53 UTC 2004

Martin,

I find interesting that you view replicability as doing a study over
using a completely different sample. Is this the normal meaning? I
would be more concerned to know if another set of researchers would
have reached the same conclusions with the SAME sample. You mention
Bybee et al striving for 90% coding agreement, but they were already
working closely in the same lab and had discussed issues and would
naturally have reached some common ground before taking on the task
of coding. Imagine another lab making very different decisions (and
gettings its 90% coding agreement). How much would this affect the
results?

As a phonologist working on very complex tone systems, I have often
wanted to do a field experiment of the following sort: A group of
Bantuist tonologists all go to Tanzania, Congo or wherever to work on
the tone system of a completely unknown language. We all write it up,
report our results and see if we came up with the same "facts",
analysis, theoretical questions etc. I think we can imagine the
result, given the differences of analysis based on the SAME set of
published data even of a simple tone system (e.g. Tokyo Japanese).

So, in partial answer to Frans query about experimentation in
typology, how about the following:

A group of leading typologists identifies an issue and a language
sample in which to study the issue. This is then assigned to several
different labs, each of which is asked to study the issue, come up
with the numbers, and report their conclusions by a certain date.
This for me would be a real test of "replication".

At 5:13 PM +0200 8/24/04, Martin Haspelmath wrote:
>Like other surveys, typological surveys should be reproducible. What
>this could mean, for instance, is that after someone finds that two
>features correlate in a sample of 50 languages using a certain sampling
>method, someone else tries to reproduce the results using a totally
>different sample and a different sampling method, to see whether the
>results still hold.
>
>Typology would profit from such work, but the problem is that the
>authors wouldn't profit sufficiently -- merely replicating a known
>finding is far less prestigious than coming up with a new one. I wonder
>whether "Linguistic Typology" would publish a paper that merely confirms
>(or merely disconfirms, for that matter) a known typological fact.
>
>But reproducibility would show that typology really is a science. Maybe
>one reason people do not do it is that they are not so convinced that is
>not art after all. Especially the application of definitions to
>different languages is not something that can be done totally
>mechanically, and it is hard to get rid of the element of subjectivity.
>For instance, Bybee, Perkins & Pagliuca 1994 ("The evolution of
>grammar") note on p. xvii that they had to work very hard to ensure that
>the coding by the three authors yielded similar results. They were happy
>when they had reached ninety percent inter-coder reliability.
>
>Martin Haspelmath
>
>Frans Plank wrote:
>
>>Re-doing Typology:  Call for Debate
>>
>>
>>In good science one expects experiments to be reproducible:  results are
>>only accepted if researchers other than those who designed and ran the
>>original experiment and obtained results which are considered sufficiently
>>interesting among the scientific community can independently repeat the
>>same experiment and obtain the same results, over and over again.
>>
>>Scientific methods include not only experimentation but also surveys.  In
>>the life sciences (like in many fields of biology), probably all population
>>and ecological sciences, sciences of the past (palaeontology, archaeology,
>>etc.), and sociology surveys are used rather than experiments.
>>Essentially, an experiment is where you manipulate conditions, and a survey
>>is where you select conditions.  Simulation is another standard procedure
>>in some of the sciences, where experiments or surveys would be unviable.
>>The replicability issue is equally important for surveys, experiments, and
>>simulation.
>>
>>The purpose of this call for debate is to determine to what extent this
>>principle of scientific method -- in order to be accepted, results must be
>>reproducible -- is applicable to linguistic typology.
>>
>>For once, let's be catholic on what is typology.  It's linguistics in so
>>far as it is about the human capacity for speaking and understanding, and
>>its special remit is to observe, describe, and explain how particular
>>manifestations of this capacity, as mentally representated and socially
>>shared (grammars and lexicons, if you prefer), can and cannot vary across
>>speech communities.
>>
>>The first real question then is whether typology is a science.  The
>>alternative, to put it somewhat grossly, is that it is a form of art, like
>>writing sonnets or interpreting them.   Most current practitioners of
>>typology presumably opt for science, even when based in Arts or Humanities
>>faculties rather than in Cognitive Science, and would like to see the
>>results of their work evaluated as true or false rather than as beautiful
>>or ugly -- or at any rate as true (yet unfalsified) or false (-ified) first
>>and as giving pleasure or pain to the senses only second.
>>
>>The next question is whether typologists perform experiments as part of
>>their professional activities.  Regardless of what else typologists do in
>>the way of observing, describing, and forming hypotheses, they arguably do
>>experiments when they select subsets from a more comprehensive set (the
>>known or knowable linguistic universe, since the typologist's results
>>typically aim to be universal) and obtain descriptive generalisations from
>>such a sample by induction.  Other typologists see themselves as doing
>>surveys, selecting conditions rather than manipulating them.  Simulation
>>has been done in typology, too, if less commonly than in other fields.
>>What it means to be doing experiment-typology or survey-typology or
>>simulation-typology is another question that remains to be clarified.
>>
>>Now, as to the primary issue which we would like to see debated, what would
>>it mean to reproduce a typological experiment or survey?   For example,
>>would it be reproduced on the same or on a different sample?  How many
>>typological experiments or surveys have actually been reproduced, one way
>>or another?  (Probably not many:  identifying counterexamples to
>>descriptive generalisations, which is being done a lot, is not
>>reproduction.)  What are the obstacles hindering reproduction?  How to
>>remove them?  Is it worth the effort?  (And the efforts required could be
>>considerable, including for instance the obligation to make one's data and
>>their interpretation available to fellow researchers, in publications or on
>>a web site.  Presumably, interpretations of data by typologists would claim
>>a different epistemological status than interpretations of sonnets by
>>literary scholars:  in typology, data as well as interpretations ought to
>>be reproducible.)  What follows if an experiment or survey is replicated
>>but the results cannot be reproduced?  Is sampling the only part of the
>>typologist's work where reproducibility is potentially an issue?
>>
>>
>>LT invites contributions on the role of reproducibility in typology -- of
>>whatever genre:  statements of principle, methodological discussions,
>>reflections on your own experimental practice, actual attempts to reproduce
>>the results of others, reports on the difficulties encountered.
>>
>>Send to the editorial address of LT (for attached text files:
>>frans.plank at uni-konstanz.de) by the end of October 2004.  A discussion
>>section in LT 9-1 (2005) will be set aside for this purpose.  The Editorial
>>Board reserves the right to select submissions for publication.  Concise
>>contributions are preferred, but longer papers on the subject are also
>>welcome and will be reviewed as regular submissions.  In the first instance
>>you might also want to share your thoughts with the typological community
>>on lingtyp at listserv.linguistlist.org
>>
>>As an alternative outlet, there is of course The Journal of Irreproducible
>>Results (http://www.jir.com/).
>>
>>
>>The Editorial Board of LT
>>
>>
>>
>
>
>--
>Martin Haspelmath (haspelmath at eva.mpg.de)
>Max-Planck-Institut fuer evolutionaere Anthropologie, Deutscher Platz 6
>D-04103 Leipzig
>Tel. (MPI) +49-341-3550 307, (priv.) +49-341-980 1616