Call for Debate: Reproducibility in Typology

[Frans Plank]

Re-doing Typology:  Call for Debate


In good science one expects experiments to be reproducible:  results are
only accepted if researchers other than those who designed and ran the
original experiment and obtained results which are considered sufficiently
interesting among the scientific community can independently repeat the
same experiment and obtain the same results, over and over again.

Scientific methods include not only experimentation but also surveys.  In
the life sciences (like in many fields of biology), probably all population
and ecological sciences, sciences of the past (palaeontology, archaeology,
etc.), and sociology surveys are used rather than experiments.
Essentially, an experiment is where you manipulate conditions, and a survey
is where you select conditions.  Simulation is another standard procedure
in some of the sciences, where experiments or surveys would be unviable.
The replicability issue is equally important for surveys, experiments, and
simulation.

The purpose of this call for debate is to determine to what extent this
principle of scientific method -- in order to be accepted, results must be
reproducible -- is applicable to linguistic typology.

For once, let's be catholic on what is typology.  It's linguistics in so
far as it is about the human capacity for speaking and understanding, and
its special remit is to observe, describe, and explain how particular
manifestations of this capacity, as mentally representated and socially
shared (grammars and lexicons, if you prefer), can and cannot vary across
speech communities.

The first real question then is whether typology is a science.  The
alternative, to put it somewhat grossly, is that it is a form of art, like
writing sonnets or interpreting them.   Most current practitioners of
typology presumably opt for science, even when based in Arts or Humanities
faculties rather than in Cognitive Science, and would like to see the
results of their work evaluated as true or false rather than as beautiful
or ugly -- or at any rate as true (yet unfalsified) or false (-ified) first
and as giving pleasure or pain to the senses only second.

The next question is whether typologists perform experiments as part of
their professional activities.  Regardless of what else typologists do in
the way of observing, describing, and forming hypotheses, they arguably do
experiments when they select subsets from a more comprehensive set (the
known or knowable linguistic universe, since the typologist's results
typically aim to be universal) and obtain descriptive generalisations from
such a sample by induction.  Other typologists see themselves as doing
surveys, selecting conditions rather than manipulating them.  Simulation
has been done in typology, too, if less commonly than in other fields.
What it means to be doing experiment-typology or survey-typology or
simulation-typology is another question that remains to be clarified.

Now, as to the primary issue which we would like to see debated, what would
it mean to reproduce a typological experiment or survey?   For example,
would it be reproduced on the same or on a different sample?  How many
typological experiments or surveys have actually been reproduced, one way
or another?  (Probably not many:  identifying counterexamples to
descriptive generalisations, which is being done a lot, is not
reproduction.)  What are the obstacles hindering reproduction?  How to
remove them?  Is it worth the effort?  (And the efforts required could be
considerable, including for instance the obligation to make one's data and
their interpretation available to fellow researchers, in publications or on
a web site.  Presumably, interpretations of data by typologists would claim
a different epistemological status than interpretations of sonnets by
literary scholars:  in typology, data as well as interpretations ought to
be reproducible.)  What follows if an experiment or survey is replicated
but the results cannot be reproduced?  Is sampling the only part of the
typologist's work where reproducibility is potentially an issue?


LT invites contributions on the role of reproducibility in typology -- of
whatever genre:  statements of principle, methodological discussions,
reflections on your own experimental practice, actual attempts to reproduce
the results of others, reports on the difficulties encountered.

Send to the editorial address of LT (for attached text files:
frans.plank at uni-konstanz.de) by the end of October 2004.  A discussion
section in LT 9-1 (2005) will be set aside for this purpose.  The Editorial
Board reserves the right to select submissions for publication.  Concise
contributions are preferred, but longer papers on the subject are also
welcome and will be reviewed as regular submissions.  In the first instance
you might also want to share your thoughts with the typological community
on lingtyp at listserv.linguistlist.org

As an alternative outlet, there is of course The Journal of Irreproducible
Results (http://www.jir.com/).


The Editorial Board of LT