16.1364, Disc: Re: A Challenge to the Minimalist Community
LINGUIST List
linguist at linguistlist.org
Fri Apr 29 14:52:13 UTC 2005
LINGUIST List: Vol-16-1364. Fri Apr 29 2005. ISSN: 1068 - 4875.
Subject: 16.1364, Disc: Re: A Challenge to the Minimalist Community
Moderators: Anthony Aristar, Wayne State U <aristar at linguistlist.org>
Helen Aristar-Dry, Eastern Michigan U <hdry at linguistlist.org>
Reviews (reviews at linguistlist.org)
Sheila Dooley, U of Arizona
Terry Langendoen, U of Arizona
Homepage: http://linguistlist.org/
The LINGUIST List is funded by Eastern Michigan University, Wayne
State University, and donations from subscribers and publishers.
Editor for this issue: Michael Appleby <michael at linguistlist.org>
================================================================
To post to LINGUIST, use our convenient web form at
http://linguistlist.org/LL/posttolinguist.html.
===========================Directory==============================
1)
Date: 22-Apr-2005
From: Sean Fulop < sfulop at csufresno.edu >
Subject: Re: A Challenge to the Minimalist Community
2)
Date: 24-Apr-2005
From: Ash Asudeh < asudeh at csli.stanford.edu >
Subject: Re: A Challenge to the Minimalist Community
3)
Date: 25-Apr-2005
From: Stefan Müller < Stefan.Mueller at cl.uni-bremen.de >
Subject: Re: A Challenge to the Minimalist Community
-------------------------Message 1 ----------------------------------
Date: Fri, 29 Apr 2005 10:43:49
From: Sean Fulop < sfulop at csufresno.edu >
Subject: Re: A Challenge to the Minimalist Community
I believe the challenge offered by Sproat and Lappin, for someone to
build a successful Principles and Parameters parser that learns from
treebanks, is mixed up in a number of respects. Let me point out
some of the ways the original challenge doesn't take into account the
goals of theoretical linguistics, and misses an important distinction
between applied natural language processing and theoretical
computational linguistics.
First, let me say that I am not in favor of the Principles and
Parameters framework as a linguistic theory, nor am I in favor of
Minimalism (whatever that actually turns out to be). I promote type-
logical categorial grammar as a framework for Universal Grammar, but
whatever, my goals are broadly the same as those of P&P
proponents, and Sproat & Lappin's challenge might equally be applied
to type-logical grammar.
There have been some spirited defenses of P&P posted already by
various proponents, and I apologize if I overlap with what is said there
a little bit in the sequel.
P&P is an effort to describe and explain what the human language
faculty is, and what it does during learning. Personally, I don't think
there is much hope for its approach, but that remains debatable, so
let's presume it's a good theory, meaning some future tweak of it will
turn out to effectively describe, to every linguists' satisfaction, the
fundamental grammars of every language and how those could be
learned given the specific Universal Grammar that would also be
needed.
Now, who said anything about computational tractability? This is a
complete theory, not an efficiency contest. I think Ed Stabler's work
on GB and Minimalism over the years should be convincing enough,
that a computational implementation is at least possible, and won't be
tractable because it is bereft of performance limitations. But we have
to be happy with our theories when they correctly describe the input-
output relation of the human language acquisition process, never mind
the actual process. It is hopeless to try to conform to the "actual
process" used by the mind, we have no way of knowing what that is or
whether our particular computers are even up for the job. Some
cognitive scientists have even suggested that the brain is not
constrained by computability at all, since mathematical computability is
defined in reference to computational procedures that we now
fathom. Worrying about tractability in P&P is like denigrating relativity
theory because it makes it needlessly harder to calculate artillery
ballistics. That's of course true, but that's also why no one invokes
such a complete theory of relative motion simply to calculate artillery
ballistics.
Sproat and Lappin note that learning from treebanks is supervised,
meaning the parser is trained by a subset of the right answers. They
go on to reference work by Klein and Manning which induces
grammars in an "unsupervised" fashion from text. Well first of all, it is
still debatable whether anything can ever actually do this (see the
algorithmic learning theory literature, summarized in Jain et al. 1999),
and Sproat and Lappin also note that Klein and Manning's scheme
uses part-of-speech tagged text, which is a far cry from text. This is a
huge annotation, and could be taken as a component of Universal
Grammar. It is a component that is argued for in P&P, as well.
I agree with Sproat and Lappin, that P&P would be better off with an
attempt to implement its general learning scheme---an analysis of the
informational complexity of this problem is provided in Partha Niyogi's
book. I disagree that this computational effort should be expected to
succeed in practice, since that would require tractability, and that is
too much to ask of a generally correct theory. The effort should be
mathematically proven to succeed in theory, by defining just exactly
what it would learn if you could wait long enough for it, and by proving
that it would terminate if you waited long enough for it.
This is the sort of learnability result that I've obtained for type-logical
grammar, given a certain format for Universal Grammar and certain
assumed mathematical properties of all human languages. What I
don't require is the parts of speech - we learn those, surely. The
learning scheme is outlined in my book and also some current papers
(see references). Instead of the POS annotation, it requires
annotation by skeletal semantic structures, but there is nothing wrong
with requiring annotation. Children certainly receive "annotated"
sentences, in that they get other clues to meaning when sentences
are spoken.
While I support the current efforts in statistical NLP, I don't believe it
has any future as a linguistic theory. It is a computational
methodology, inspired by the recognition that it would be too
hard/slow to invoke the most complete form of linguistic theory to
perform basic NLP tasks. As I've often told my comp ling students, "if
you've made software that works well, it probably doesn't actually
solve a theoretical problem, which is what makes it part of applied
NLP; if you've made software that implements a full theoretical
solution, it probably won't finish by the end of the day, which is what
makes it theoretical computational linguistics." That doesn't mean the
twain shall never meet. But not yet.
References:
Fulop, Sean A. (2005) "Semantic bootstrapping of type-logical
grammar," Journal of Logic, Language and Information 14:49-86.
Fulop, Sean A. (2004) On the Logic and Learning of Language.
Victoria, Canada: Trafford.
Fulop, Sean A. (2003) "Discovering a new class of languages,"
Proceedings of Mathematics of Language 8, available online at MOL
website.
Niyogi, Partha (1998) The Informational Complexity of Learning.
Kluwer.
Stabler, Edward (1992)? The Logical Approach to Syntax.
Cambridge, MA: MIT Press.
Linguistic Field(s): Computational Linguistics
Discipline of Linguistics
Syntax
-------------------------Message 2 ----------------------------------
Date: Fri, 29 Apr 2005 10:43:52
From: Ash Asudeh < asudeh at csli.stanford.edu >
Subject: Re: A Challenge to the Minimalist Community
Some confusion has arisen in the subsequent discussion of the
Sproat-Lappin challenge. Most of the subsequent posts discuss
statistical parsing versus P&P parsing. However, the challenge has
nothing to do with statistical parsers per se, although it's true that
much of the original challenge discussed these parsers. There are,
however, non-statistical or hybrid broad-coverage parsers in theories
other than P&P, such as the PARC LFG parser mentioned in the
original Sproat-Lappin challenge and the HPSG Lingo/LKB parser at
Stanford. For theorists who for whatever reason think that statistical
parsers are a different sort of thing from the P&P parser that the
challenge focuses on (they are not, for the reasons sketched by
Sproat and Lappin themselves and also by Ken Shan (LL16.1288)),
these can perhaps serve as a reference point instead.
There is also, in my opinion, a confusing aspect to the Sproat-Lappin
challenge. They are not only asking for a P&P parser, but also asking
for a large P&P grammar. Grammars and parsers are different things,
at least for non-statistical "deep grammar" approaches (purely
statistical methods instead induce the grammar), such as the HPSG
and LFG ones noted above and, presumably, like the P&P one in
question. For example, the PARC LFG parser has been applied to
analyses of a large number of languages, although the English
grammar is what Sproat and Lappin seem to have in mind in their
message, given the citation of Riezler et al. Another example is the
LKB system, which implements an HPSG parser. The English
Resource Grammar (ERG) is used with this parser, but they are
developed separately (although with strong interactions -- same goes
for the PARC XLE parser and the ParGram grammars).
Carson Schütze (16.1288) wrote:
> In addition to capturing the distinction between learnable and
> unlearnable languages, P&P has as an important goal capturing
> the distinction between well-formed (grammatical) and ill-formed
> (ungrammatical) sentences within a language. As I understand
> it, the challenge demands only correct parsing of grammatical
> sentences, not correct rejection of ungrammatical ones. This
> represents another case where the P&P system, by virtue of
> the goals of the theory, is being subjected to greater demands
> than the statistical parsers.
I don't understand the substance of this objection. All grammars,
those used in statistical parsing or otherwise, attempt to reject
ungrammatical sentences: Nobody wants their grammar/parser to
overgenerate. Even if the claim is true of statistical parsers (I
don't think it is), it certainly isn't true of the LFG and HPSG
parsers and grammars noted above.
Peter Hallman (LL16.1251) and Martha McGinnis (also LL16.1251),
although positively inclined to the challenge, also raise objections. The
substance of the objections are that P&P is attempting to do much
more than just parse sentences (Hallman) and that the goals of P&P
are different to those of computational linguistics (McGinnis). I think
there is merit to both these statements, but they are ultimately non
sequiturs to the challenge. P&P is not just concerned with how the
adult grammar results from the inititial state (although this is a principal
goal): it is also concerned with the state of the final grammar
(Chomsky, 1986: "What constitutes knowledge of language?"). The
requirement of capturing the adult grammar also means that it's
insubstantial whether the goals of P&P are those of computational
linguistics: P&P is still expected to capture adult grammatical
competence in the end, even if this isn't a *motivation* for a lot of its
practitioners.
Lastly, I agree with Ken Shan (LL16.1288):
"All other things being equal, poor (or unknown) parsing performance
indicates failure at (resp. disinterest in) answering Q [What is a
possible human language?]."
It is no surprise that the objections to the challenge so far have sought
to argue that all other things are not equal. Like Ken, however, I think
that this challenge is a *good* thing for P&P and I too am optimistic for
P&P. In any case, the attempt to meet the challenge will without doubt
reap huge rewards, not just for computational linguistics, but
especially for P&P theory.
Ash Asudeh
Linguistic Field(s): Computational Linguistics
Discipline of Linguistics
Syntax
-------------------------Message 3 ----------------------------------
Date: Fri, 29 Apr 2005 10:43:54
From: Stefan Müller < Stefan.Mueller at cl.uni-bremen.de >
Subject: Re: A Challenge to the Minimalist Community
Carson Schutze:
> So, I would like to suggest a revised version of the challenge that
> incorporates a second corpus consisting of ungrammatical
> sentences that are to be identified as such. (Earlier P&P parsers
> such as Fong's were designed to do this, but it's not obvious that
> this ability will easily scale up with broader coverage, so I don't
> think this is a sucker's bet.) Furthermore, since the
> computationalists got to choose the corpus of good sentences,
> it would seem only fair that the theoreticians get to choose the
> corpus of bad sentences :-)
This is a very important point and negative data has been collected
and is used to evaluate deep linguistic processing.
A nice software for evaluating systems and working with test suite
databases can be found at:
http://www.delph-in.net/itsdb/
Test suites for German, English, French, Spanish, and other
languages are also available there.
You may find test suites for German at:
http://www.cl.uni-bremen.de/Software/TS/
These test suites contain (normalized) examples from the descriptive
literature, P&P, HPSG, and other theoretical literature. With the [incr
TSDB()] software it is possible to get a selection of sentences that is
relevant for a certain phenomenon. Sentences are crossclassified
according to the phenomena they are relevant for.
The idea is to develop these collections further into a generally
accepted benchmark for linguistic theories in general and for deep
linguistic processing in particular. Of course the negative sentences
can be used to check what statistical parsers have to say about them
in comparison to the well-formed examples.
So if somebody has a look at the German collection and wants to
contribute, please send me the relevant examples and pointers to the
publications in which the examples are discussed.
Best wishes
Stefan Müller
Universität Bremen/Fachbereich 10
http://www.cl.uni-bremen.de/~stefan/
http://www.cl.uni-bremen.de/~stefan/Babel/Interaktiv/
Linguistic Field(s): Computational Linguistics
Discipline of Linguistics
Linguistic Theories
Subject Language(s): German, Standard (GER)
-----------------------------------------------------------
LINGUIST List: Vol-16-1364
More information about the LINGUIST
mailing list