[Corpora-List] Anything resembling TPC benchmarks for corpora?

[John F Sowa]

On 6/7/2012 6:55 AM, Albretch Mueller wrote:
> I think it is healthy for us to have our own research projects/pet
> theories, still we will also benefit greatly from having something
> similar to Transaction Processing Council Benchmarks for corpora

There is a common saying:

    "Lies, damn lies, and benchmarks"

See http://www.spec.org/osg/news/articles/news9412/lies.html

What makes benchmarks so easily abused is that they are often designed
to favor some point the designer wanted to prove.  From that article:

> There is no magic formula for evaluating computer performance.
> As with so many things, the answer depends upon your point of view.

Sometimes, benchmarks can become an artificial measure that is not
related to the kinds of tasks that people need to do.  As a result,
developers may tune their software to "pass the test" rather than
perform some useful application.

For example, look at the Wikipedia article cited in the previous note:

http://en.wikipedia.org/wiki/Transaction_Processing_Performance_Council

It cites 6 earlier benchmarks that became obsolete during the period
from 1995 to 2005.  Many of those were artificial tests such as
"transactions per second", which became obsolete in 1995.

Following are the three in the current standard:

    TPC-C - An on-line transaction processing benchmark.
    TPC-E - An on-line transaction processing benchmark that simulates
            the OLTP workload of a brokerage firm.
    TPC-H - An ad-hoc, decision support benchmark.

I am not familiar with any of them, but the names of the second and
third suggest an improvement over the older versions.  They simulate
representative applications, rather than features of the tools.

Questions:  What kinds of applications use corpora?  Could any of them
be considered "typical"?  Could they be specified in a way that would
help evaluate any corpora software that could be considered "typical"?
Would performance on such a specification be meaningful for anybody
who might be searching for software (or algorithms) for some purpose?

If the focus is placed on applications, a benchmark could stimulate
innovation by encouraging developers to search for new designs.
But a focus on the features of a particular technique could lead
to stagnation if the developers just learn how to "pass the test".

John Sowa

_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora