19.1514, Diss: Comp Ling: Eisenstein: 'Gesture in Automatic Discourse Proces...'

[LINGUIST Network]

LINGUIST List: Vol-19-1514. Thu May 08 2008. ISSN: 1068 - 4875.

Subject: 19.1514, Diss: Comp Ling: Eisenstein: 'Gesture in Automatic Discourse Proces...'

Moderators: Anthony Aristar, Eastern Michigan U <aristar at linguistlist.org>
            Helen Aristar-Dry, Eastern Michigan U <hdry at linguistlist.org>
 
Reviews: Randall Eggert, U of Utah  
         <reviews at linguistlist.org> 

Homepage: http://linguistlist.org/

The LINGUIST List is funded by Eastern Michigan University, 
and donations from subscribers and publishers.

Editor for this issue: Evelyn Richter <evelyn at linguistlist.org>
================================================================  

To post to LINGUIST, use our convenient web form at
http://linguistlist.org/LL/posttolinguist.html.

===========================Directory==============================  

1)
Date: 07-May-2008
From: Jacob Eisenstein < jacobe at csail.mit.edu >
Subject: Gesture in Automatic Discourse Processing

 

	
-------------------------Message 1 ---------------------------------- 
Date: Thu, 08 May 2008 09:03:59
From: Jacob Eisenstein [jacobe at csail.mit.edu]
Subject: Gesture in Automatic Discourse Processing
E-mail this message to a friend:
http://linguistlist.org/issues/emailmessage/verification.cfm?iss=19-1514.html&submissionid=177682&topicid=14&msgnumber=1  


Institution: Massachusetts Institute of Technology 
Program: Computer Science and Artificial Intelligence Laboratory 
Dissertation Status: Completed 
Degree Date: 2008 

Author: Jacob Eisenstein

Dissertation Title: Gesture in Automatic Discourse Processing 

Dissertation URL:  http://people.csail.mit.edu/jacobe/diss.html

Linguistic Field(s): Computational Linguistics


Dissertation Director(s):
Randall Davis
Regina Barzilay

Dissertation Abstract:

Computers cannot fully understand spoken language without access to the
wide range of modalities that accompany speech. This thesis addresses the
particularly expressive modality of hand gesture, and focuses on building
structured statistical models at the intersection of speech, vision, and
meaning.

My approach is distinguished in two key respects. First, gestural patterns
are leveraged to discover parallel structures in the meaning of the
associated speech. This differs from prior work that attempted to interpret
individual gestures directly, an approach that was prone to a lack of
generality across speakers. Second, I present novel, structured statistical
models for multimodal language processing, which enable learning about
gesture in its linguistic context, rather than in the abstract.

These ideas find successful application in a variety of language processing
tasks: resolving ambiguous noun phrases, segmenting speech into topics, and
producing keyframe summaries of spoken language. In all three cases, the
addition of gestural features -- extracted automatically from video --
yields significantly improved performance over a state-of-the-art text-only
alternative. This marks the first demonstration that hand gesture improves
automatic discourse processing. 






-----------------------------------------------------------
LINGUIST List: Vol-19-1514