Zipf on language change/structure
macw at CMU.EDU
Wed Nov 20 18:31:18 UTC 2002
Matt et al.,
In a Science paper from about 1970, Herb Simon shows how Zipf's law
applies to about four very different frequency distributions. I don't
remember the details, but one was in meteorology and one in architecture.
Recently Joshua Tenenbaum has been developing extensions of Zipf's law for
various language corpora.
A major issue in psychology has been whether it is best to model these
data with power laws (following Newell and Rosenbloom) or exponential laws.
This article argues for the latter:
Anderson, J., & Schooler, L. (1991). Reflections of the environment in
memory. Psychological Science, 2, 396-408.
However, I believe one can also argue that exponential functions soak up
more degrees of freedom than power functions.
Of course, if the only goal of the analysis is to argue that high-token
types are rare, then either function works.
More information about the Funknet