[Corpora-List] Building a corpus from Twitter & Tw's privacy concerns

[Leon Derczynski]

Dear John,

You are right, it is a very awkward distribution format. Corpora of some
size become very difficult to gather after a point. However, tweet decay is
not so much of a problem for many use-cases (of course when tweets are
deleted that are the only example of a particular phenomenon in a corpus,
the deletion is of greater impact). It seems that tweets decay at a
reasonably stable rate after the first hour's spam filtering (or whatever)
has been done.

Based on observations over the Trendminer garden hose feed, which collects
us about 1.2TB of tweets each day, and the findings of Petrovic, Osborne
and Lavrenko (2013) "I Wish I Didn’t Say That! Analyzing and Predicting
Deleted Messages in Twitter", you can in fact work out a reasonably
accurate model for predicting tweet "decay" through deletion in a
reasonably-sized corpus:

  http://derczynski.com/leon/tweet-decay.html

This supports the idea that a corpus a few months old and a few years old
will look roughly the same - it's quite rare that one would go back and
delete a message after a few years. So the tweet data becomes "stabler"
with time. Which is good, because it may take months to retrieve via the
API ;)

All the best,


Leon


On 16 July 2013 17:44, John D. Burger <john at mitre.org> wrote:

> There appears to be no legal reason you can't collect a corpus of tweets.
>  However, per Twitter's Terms of Use you cannot redistribute the tweets to
> others.  A common practice is to instead distribute the tweet IDs, which
> other people can use to fetch the tweets using Twitter's API.  This is how
> NIST "distributes" the data in their Tweets2011 corpus:
>
> http://trec.nist.gov/data/tweets/
>
> This is less than optimal for research, though, since in the interim some
> of the Twitter users may have deleted tweets in the collection. For a
> sufficiently large corpus, this means that anybody else attempting to use
> the same data at a later date will almost certainly end up with a subset of
> your corpus. As far as I know, however, this is currently the only legal
> method for sharing tweets.
>
> - John Burger
>   MITRE
>
> On Jul 16, 2013, at 10:51 , M.E.Sciubba wrote:
>
> Dear ListMembers,
>
> I'd like to create a corpus of Italian twits, but searching online I found
> out that it is not possible anymore because Twitter has changed its privacy
> settings.
>
> Has any of you tried to build a Twitter corpus and how?
>
> Any suggestion will be much appreciated (considering that I am not a
> programmer, though).
>
> Best,
>
> Eleonora
>
>
> ------------------------------
>
> Dr. Maria Eleonora Sciubba
> Associate Researcher
> Archivio di LInguA Spontanea
> tel. +32 16 3 24795
> cell +32 483 616 114
>
> KU Leuven – Faculty of Arts****
>
> Department of French, Italian and Comparative Linguistics****
>
> Blijde-Inkomststraat 21, PO BOX 3308****
>
> B - 3000 Leuven
>
> http://www.kuleuven.be/wieiswie/nl/person/00088846
>
>
>
> ------------------------------
>  *Be green. Keep it on the screen*
>  _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
>


-- 
Leon R A Derczynski
Research Associate, NLP Group

Department of Computer Science
University of Sheffield
Regent Court, 211 Portobello
Sheffield S1 4DP, UK

+45 5157 4948
http://www.dcs.shef.ac.uk/~leon/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20130716/874ee7c4/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora