Promoting interest in HPSG: The discussion so far

[Emily M. Bender]

Date: Thu, 17 Jun 2004 13:59:34 -0700
From: "Emily M. Bender"

Re: low submission rate.  What was it like this year?  I think that
we might be able to increase the number of abstract submitted by
reversing the impression that the HPSG conference is generally not
interested in the more computational work.  The workshop offering
this year is a good step in that direction.

Emily

Date: Fri, 18 Jun 2004 10:25:53 +0200
From: "Bouma G."

Hi Emily (and others),

There may not be that much computational work that is focusing on
HPSG specifically. I think by now we pretty much know how to
implement HPSG. There is computational work which uses HPSG, but
which is really on efficient, robust, or stochastic parsing, or
on applications. That tends to end up in CL conferences and
workshops (of which there are plenty).

In general I think it is clever to continue organising HPSG in
conjunction with other events, where possible. LFG and Formal
Grammar are obvious candidates, but the more linguistically
oriented corpus conferences might also be an option (e.g. Treebanks
and Linguistic Theories, http://w3.msi.vxu.se/~rics/TLT2003/,
or Linguistically Interepreted Corpora, http://www.coli.uni-sb.de/linc04/).
It is more work, but a joint event tends to attract more than
just the sum of the participants that would have attended the
individual conferences.

best,

Gosse.

Date: Fri, 18 Jun 2004 11:19:14 -0700
From: "Emily M. Bender"

Hi Gosse (and others),

Sure there are several working implementations which support HPSG
grammar development, but that doesn't mean it's a solved problem, or
that none of the work would (should?) be of interest to the broader
HPSG community.  Some topics off the top of my head (some of which
are presently close to my heart...):

-- Efficiency v. expressivity: Which formal devices are necessary
in an implementation, and how does their presence/absence affect
efficiency, scalability, linguistic hypothesis testing, etc?

-- Parse selection: What kind of information do stochastic parse
selection systems make use of, and how might understanding of the
grammars help inform the choice of which stochastic "features" to
include?

-- Interface to morphophonology: As we take the implementation
work beyond morphologically relatively impoverished languages
like English, what are the design criteria for morphological
systems and how should these be interfaced to or integrated
with the current grammar development platforms?

-- Multilingual grammar engineering: What can we learn about
cross-linguistic universals from broad coverage implemented grammars?

-- Computational psycholinguistics: Can implemented HPSG
grammars be used to model results on human processing (cf
Jurafsky 1997), and how does this inform grammar design?

-- Combining precise, deep processing with robust, relatively
shallow processing through compatible semantic representations:
What kind of demands does this place on semantic representations,
and how well do they fit with "purely" linguistic conceptions
of how the semantics should be?

-- Treebanks: How can resources like the Redwoods Treebank
be used in linguistic discovery?

> which is really on efficient, robust, or stochastic parsing, or
> on applications. That tends to end up in CL conferences and
> workshops (of which there are plenty).

Only some of the HPSG community regularly attends both kinds
of conferences (CL and linguistic theory, that is).  I would hope
that the HPSG conference could serve as a meeting ground to
help facilitate interaction across that divide.  I think this
can be achieved both by organizing workshops like the semantics
in grammar engineering workshop this year and by working to
alleviate the impression that implementation-based work (e.g.,
results coming out of large grammar engineering projects) is
unlikely to be relevant to the conference unless it makes strong
theoretical claims.

I think teaming up with one of the corpus conferences is
a really interesting idea.  Do we know anything about future
plans for those?

Emily

Date: Sat, 19 Jun 2004 16:58:29 +0200
From: "Stefan M�ller"

Hi,

> we might be able to increase the number of abstract submitted by
> reversing the impression that the HPSG conference is generally not
> interested in the more computational work.

I am not sure whether this is a wise thing to do. At least in Germany,
I often get the sense that people have a hard time separating the
computational applications of HPSG from the theoretical contributions
of the community.  So one should not reject submissions that deal with
computational aspects of HPSG but one should not explicitly try to get
more computational submissions to the main conference.

Personally, I find the topics Emily mentioned very interesting. Having
workshops for CL topics attached to the conference might be a good
compromise.

Greetings

	Stefan

Date: Sat, 19 Jun 2004 21:32:51 -0700
From: "Emily M. Bender" <ebender at u.washington.edu>

While I understand the frustration at the way HPSG is perceived, I
believe that one of the great strengths of the framework is its
connection to implementations.  Rather than let the other view point
define the terms of the debate, I'd rather work at putting forth the
idea that computational linguistics and theoretical linguistics can
both benefit from listening to each other.

That said, I'm not sure we need to explicitly solicit more
computational papers (beyond adding workshops, that is), but
I think we could at least work towards refining the reviewing
process so that both perspectives are well represented.

Emily

Date: Mon, 21 Jun 2004 09:16:49 +0200
From: "Tibor Kiss"

Dear Emily (and Stefan) and everyone else,

Stefan likes to accentuate the positive. In claiming that HPSG is a part of
computational linguistics, he remains silent on HPSG's reputation as a
linguistic theory. In a sense, this seems to be in-line with Emily's
suggestions whose implicit assumption is that we cannot attract more
theoretical people.

HPSG is indeed conceived double-headed, related to two rather disparate
fields, and currently I think it is not really successful in either, which
partly explains why HPSG conferences are not really a big success and people
nowadays feel reluctant to host or organise them. (These fields were much
closer 20 years ago, but many different reasons led to a gradual divorce.)

On the one hand, HPSG is a theory of grammar, and on the other it is a
toolbox for grammar prototyping. Even if the second sense is conceived as
the stronger one, I don't think that HPSG can be considered particularly
well suited to attract more computational people, and moreover, I do not see
how this would help overcoming the theoretical problems. Most computer
scientist have very little theoretical convictions (which is actually good
because linguists thus can tell them what to do) and do not really make good
evangelists for a theoretical framework.

Reconsidering HPSG's development, I daresay that some of the major design
mistakes found in theoretical HPSG come from letting computer scientists (of
a particular breed) or logicians gain too much influence.

Why is it, e.g. that HPSG moved without any justification from an ontology
of trees (GPSG) to an ontology of not-necessarily acyclic graphs? Why did
HPSG include data structures and relations into grammatical
representations?

I think that HPSG suffers from these design decisions both theoretically and
computationally.

As for the theory, both design decisions lead to a weakening of the role of
linguistic structure in HPSG. I remember that in 1989, HPSG people still
thought that e.g. switching between SUBCAT lists and sets would allow for
plausible analyses of strict vs. free word order. But it soon turned out
that with a little help of relations, you could express everything you want,
making HPSG appear somewhat PERLish (there is always more than one way to do
it). Being PERLish is nice for a programming language, but it is not for a
restrictive linguistic theory.

As for the second point (data structures/relations): The inclusion of data
structures and relations had led to an extremely arcane formalism, which has
an interesting side effect. While, e.g. the formal foundations of GPSG could
quite often be used to pinpoint theoretical assumptions (e.g.
context-freeness and a finite vocabulary of categories), HPSG's formal
foundation does not serve many useful purpose apart from allowing relations
and other stuff which many theoretical linguists consider computational or
implementational machinery, or just hacks.

Funny enough, people who advocated for using HPSG in early Verbmobil found
themselves quite often confronted with requests to use 'lean formalisms'
eschewing list-valued or set-valued features, let alone relations. And
Emily's suggestions -- although I endorse them -- do not apply specifically
to HPSG but apply to the more general question how grammar and processing
can interact. Even worse, HPSG is not even equipped to address some of the
questions Emily has raised (particularly the question of psycholinguistic
modeling due to a lack of precise structural predictions).

Theoretically, I consider it useful to really reconsider list- and
set-valued features and to eliminate relations entirely. Also, the ontology
issue should be raised again. Given that HPSG does not and cannot provide a
finite set of categories, and allows the formulation of almost arbitrary
constraints, I would bet that a serious comparison of HPSG with MP would
result in calling the latter the more restrictive framework. Personally, I
find this result somewhat irritating. (Ok, MP may not provide a finite set
of categories neither. But would you consider this commonality a virtue of
HPSG or a vice of MP?)

Currently, Chomsky is very successful in marketing a framework which most of
his followers only vaguely understand. Where do we fit in? It is Chomsky's
assumption that the syntax-semantics interface is guided by 'phases' (small
entities which are closed for further analysis). Why is it that HPSG does
not attack this position, given that HPSG endorses a completely different
idea of the syntax-semantics interface?

It seems obvious that HPSG cannot attract people from other frameworks if
HPSG does not really confront the claims emanating from these frameworks.

And this is finally the reason why the conferences are not so attractive
either. I think that having the conference at MIT in 2005 will not help to
make HPSG more attractive unless very serious effort is spent to reconcile
some of HPSG's basic assumptions. I would thus strongly opt for a rather
different HPSG 2005, namely a conference without invited speakers and
without CFP, but instead with target papers, the title of which should
always include: 'future directions for HPSG'.


Best

T.

Date: Mon, 21 Jun 2004 10:41:18 -0700
From: "Emily M. Bender"

Dear Tibor and others,

It seems to me that you, too, are letting the other point of view (MP)
define the terms of the debate.

> [...] that with a little help of relations, you could express
> everything you want, making HPSG appear somewhat PERLish (there is
> always more than one way to do it). Being PERLish is nice for a
> programming language, but it is not for a restrictive linguistic
> theory.

I see the flexibility of formalism (what you're referring to as
`PERLishness') as one of the great strengths of HPSG.  By separating
the formalism from the theory/theories developed within that
formalism, we gain the ability to explore different theoretical
options without having to rework the formalism at each step.
Reworking the formalism, it turns out, is hard work (and I'd dare add
work that most linguists are not suited for).  If we have a relatively
steady formalism, we stand a chance of being able to actually test the
consequences of various theoretical claims against large bodies of
data and determine how they interact with other theoretical claims
(analyses).

I'm curious, though, what you see as the benefit of having a
`restrictive' linguistic theory.  The main arguments for that position
that I see coming out of GB/MP (poverty of the stimulus et al) I find
singularly unconvincing.  I'd much prefer to develop ever larger
internally consistent grammars of a variety of languages within a
common vocabulary such that we can explore what the languages have in
common and how they differ.

> Emily's suggestions -- although I endorse them -- do not apply specifically
> to HPSG but apply to the more general question how grammar and processing
> can interact. Even worse, HPSG is not even equipped to address some of the
> questions Emily has raised (particularly the question of psycholinguistic
> modeling due to a lack of precise structural predictions).

I think that HPSG is much better suited to psycholinguistic modeling
than, say, MP, because it is precise enough to be implemented and
therefore embedded in some implemented model of processing.  It takes
quite a bit of hubris, I think, to imagine that something as complex
as human language processing can really be modeled from an armchair,
perhaps with the aid of pen and paper.

But, I'm not sure what you mean by "lack of precise structural
predictions", or how this pertains to the current question.

> I would bet that a serious comparison of HPSG with MP would
> result in calling the latter the more restrictive framework. Personally, I
> find this result somewhat irritating. (Ok, MP may not provide a finite set
> of categories neither. But would you consider this commonality a virtue of
> HPSG or a vice of MP?)

I don't think a "framework" should be restrictive.  Theories should
make predictions (which maybe isn't the same thing as being
restrictive).  Formalisms shouldn't.  And I think in that comparison,
HPSG wins hands down.  MP doesn't make any predictions because even
the adepts can't agree on what the bits mean. (See my review of _Step
by Step_ in Lingua.)

> Currently, Chomsky is very successful in marketing a framework which most of
> his followers only vaguely understand. Where do we fit in? It is Chomsky's
> assumption that the syntax-semantics interface is guided by 'phases' (small
> entities which are closed for further analysis). Why is it that HPSG does
> not attack this position, given that HPSG endorses a completely different
> idea of the syntax-semantics interface?

I think that that's a really interesting suggestion for opening a
theoretical (and hopefully also empirical) debate.  I would really
like to see someone work through the consequences (presumably with
reference to the psycholinguistic literature) of both models.

> It seems obvious that HPSG cannot attract people from other frameworks if
> HPSG does not really confront the claims emanating from these frameworks.

I've seen HPSG attract plenty of people from other frameworks (mostly
students so far, but still).  They find HPSG very appealing because it
is actually grounded in a well worked-out formalism, attempts to build
internally coherent grammars which account for actual sentences (and
not just particular phenomena within the sentences), and comes with
the ability to actually test analyses against test suites and corpora
through an implementation.  These same folks find work in GB/MP quite
frustrating for its lack of these things, as do I.

I think there is less work directly confronting claims of MP, etc,
than one would want because it is so wearisome.  I don't mind
responding to MP analyses (and pointing out the inconsistencies) as an
audience member at conferences and in reviewing papers, but the few
times that I've had to directly argue with claims from GB/MP I've
found it tremendously frustrating.  Most of the claims are not
well-formulated enough to make any predictions, which makes them
unfalsifiable.  I'd rather not play in that playground.

Fortunately, others are hardier in that regard than me, and
we do see things like Johnson & Lappin's book...

Cordially,
Emily

Date: Tue, 22 Jun 2004 08:35:45 +0200
From: "Tibor Kiss"

Dear Emily,

As you have already seen and will see in this reply, I mostly hold
positions which differ from yours. This shouldn't obscure that I
appreciate that you started the debate.

> It seems to me that you, too, are letting the other point of view (MP)
> define the terms of the debate.

I do not think that there is a polar opposition between MP and HPSG. To the
contrary, they are perhaps much closer than you may think, and this is one
of the reasons why HPSG is not successful as a theory: What kind of viable
options does it offer as an alternative to MP? (Cf. my remarks on the
syntax-semantics interface.)

I am actually not a person to think in 'MP or us'.

> I see the flexibility of formalism (what you're referring to as
> `PERLishness') as one of the great strengths of HPSG.  By separating
> the formalism from the theory/theories developed within that
> formalism, we gain the ability to explore different theoretical
> options without having to rework the formalism at each step.

I did not claim that the formalism leads to arbitrariness (although this is
actually correct), but I claimed that it is hard to impossible to develop
something like a 'wrong HPSG analysis', because of the 'anything
goes'-property of the theory. A nice example of this can be found in the
discussion of the German verbal complex, initiated by my apparently wrong
analysis of modal syntax and Detmar Meurer's and Stefan M�ller's follow-ups
(Kiss in Nerbonne/Netter/Pollard 1994, Meurers and M�ller in a variety of
papers, M�ller in Coling 1996, if I remember correctly). The basic result
was: Any analysis of the verbal complex is fine.

> But, I'm not sure what you mean by "lack of precise structural
> predictions", or how this pertains to the current question.

I find it somewhat irritating that any analysis of the verbal complex is
fine. (Just an illustration, many further examples come to mind.)

> I'm curious, though, what you see as the benefit of having a
> `restrictive' linguistic theory.  The main arguments for that position
> that I see coming out of GB/MP (poverty of the stimulus et al) I find
> singularly unconvincing.  I'd much prefer to develop ever larger
> internally consistent grammars of a variety of languages within a
> common vocabulary such that we can explore what the languages have in
> common and how they differ.

This is a complete misunderstanding of my intentions. What I referred to as
a restrictive linguistic theory is what GKPS have proposed in their 1985
book.


> > I would bet that a serious comparison of HPSG with MP would
> > result in calling the latter the more restrictive
> framework. Personally, I
> > find this result somewhat irritating. (Ok, MP may not
> provide a finite set
> > of categories neither. But would you consider this
> commonality a virtue of
> > HPSG or a vice of MP?)
>
> I don't think a "framework" should be restrictive.  Theories should
> make predictions (which maybe isn't the same thing as being
> restrictive).  Formalisms shouldn't.  And I think in that comparison,
> HPSG wins hands down.  MP doesn't make any predictions because even
> the adepts can't agree on what the bits mean. (See my review of _Step
> by Step_ in Lingua.)

The formalism is as it is because it should support data structures and
relations as parts of linguistic representations. Do we need data structures
and relations as parts of linguistic representations?
This is not a question of formalism, but of theory. Why do we have to
consider cyclic graph structures as the objects our theory is talking about
instead of simple trees? Once again, this is not a question of formalism.

By using the term 'framework' btw. I was trying to avoid comments of the
form 'MP is not a theory' ... You might be right that I overestimate the
predictive power of MP, but I did not underestimate the predictive power of
HPSG. As for predictions, my background assumptions are as follows:
Languages share properties and they differ in others. This is in need of
explanation. It is, as in illustration, just not enough to provide an
analysis which shows that English reflexives sometimes behave like pronouns,
it is also necessary why German (and Bengali, Portuguese, ...) reflexives
which look similar at first sight behave differently.

If you claim (to the best of my knowledge, nobody does that, I just use it
as an illustration) that we have a total of 4.000 different Principles A
each of which does the work for one of the 4.000 still existing languages,
you do not make any prediction.

> Most of the claims are not well-formulated enough to make any
> predictions, which makes them unfalsifiable.  I'd rather not play in
> that playground.

Once again, I see a similarity and not a difference: MP claims might be
fluffy, but HPSG claims are arbitrary. I like strong claims because I am
intellectually lazy: strong claims can be falsified much more easily. I
would like to see more strong claims coming from HPSG papers. Perhaps I am
just not aware of them (due to a different laziness) but I am afraid, they
simply aren't there.

Best

T.