<div dir="ltr"><div>RFC: Cross-Linguistic Data Formats (CLDF), version 1.0</div><div>=====================================================</div><div><br></div><div>Resulting from discussions over several years, and triggered in</div><div>particular by work presented in the two workshops of the "Language</div><div>Comparison with Linguistic Databases" series [1,2], we'd like to</div><div>request your comments on version 1.0 of CLDF - a specification for</div><div>Cross-Linguistic Data Formats (see <a href="http://cldf.clld.org">http://cldf.clld.org</a>).</div><div><br></div><div>The specification proposes a standard format for</div><div>- wordlists, including cognate judgments and phonetic alignents,</div><div>- grammatical structure datasets like WALS features and other typological surveys.</div><div><br></div><div>CLDF is built upon W3C's "Tabular Data and Metadata on the Web"</div><div>recommendation [3] and can be thought of as a domain specific adaption</div><div>of this in linguistics.</div><div><br></div><div>Extensibility is built into CLDF, to allow support of evolving</div><div>standards for more complex types of linguistic data. As of version</div><div>1.0, modules for simple dictionary data and parallel-text corpora are</div><div>included for further experimentation.</div><div><br></div><div>CLDF datasets can be read and written using the Python programming</div><div>library pycldf (<a href="https://pypi.python.org/pypi/pycldf">https://pypi.python.org/pypi/pycldf</a>), but also using</div><div>off the shelf tools like spreadsheet software or programming</div><div>environments like R, because the data file format in CLDF is based on</div><div>comma-separated values (CSV).</div><div><br></div><div>The CLDF specification is available at</div><div><a href="https://github.com/cldf/cldf/blob/master/README.md">https://github.com/cldf/cldf/blob/master/README.md</a></div><div><br></div><div>Examples of CLDF datasets and how to access CLDF data are provided at</div><div>- <a href="https://github.com/cldf/cldf/tree/master/examples">https://github.com/cldf/cldf/tree/master/examples</a> and</div><div>- <a href="https://github.com/cldf/cookbook">https://github.com/cldf/cookbook</a></div><div><br></div><div>We welcome all comments, either posted as reply to this announcement or as</div><div>issues at <a href="https://github.com/cldf/cldf/issues">https://github.com/cldf/cldf/issues</a></div><div><br></div><div><br></div><div>[1] <a href="http://www.mpi.nl/events/language-comparison-with-linguistic-databases-reflex-and-typological-databases">http://www.mpi.nl/events/language-comparison-with-linguistic-databases-reflex-and-typological-databases</a></div><div>[2] <a href="http://www.eva.mpg.de/linguistics/conferences/2014-ws-lanclid2/index.html">http://www.eva.mpg.de/linguistics/conferences/2014-ws-lanclid2/index.html</a></div><div>[3] <a href="https://www.w3.org/TR/tabular-data-model/">https://www.w3.org/TR/tabular-data-model/</a></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">2018-03-23 12:10 GMT+01:00 Robert Forkel <span dir="ltr"><<a href="mailto:forkel@shh.mpg.de" target="_blank">forkel@shh.mpg.de</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Just joined the list, so cannot respond properly to the thread this belongs to.<br>
<br>
The CLDF specification we've been working on over the last year<br>
(see <a href="http://cldf.clld.org" rel="noreferrer" target="_blank">http://cldf.clld.org</a>) proposes a standard for the exchange of<br>
typological datasets (among other types of data), with the explicit<br>
goal of decoupling software tools (for analysis or visualization) from<br>
datasets. I see this as a superset of (at least the more technical<br>
aspects of) reproducibility, because it will allow to investigate datasets<br>
with a broader range of tools.<br>
<br>
For the case in point, CLDF provides a StructureDataset module [1], which<br>
may contain a CodeTable [2], which I'd see as the machine-readable version<br>
of the code-book. As an example, here's what a WALS feature would look like<br>
as a CLDF StructureDataset (the whole WALS database is available as CLDF<br>
dataset [3]). After unzipping the WALS data, you'll see a couple of CSV files<br>
(which can be created with any spreadsheet software). We can look at two of<br>
these (e.g. using off the shelf software like csvkit [4]):<br>
<br>
values.csv<br>
<br>
$ csvgrep -c Parameter_ID -r "^20A$" values.csv | csvformat -T | head -n 5<br>
ID Language_ID Parameter_ID Value Code_ID Comment Source Contribution_ID<br>
20A-cho cho 20A Exclusively concatenative 20A-1 Turner-and-Turner-1971 20<br>
20A-jel jel 20A Exclusively isolating 20A-2 Trobs-1998 20<br>
20A-nah nah 20A Exclusively concatenative 20A-1 Kuiper-1962 20<br>
20A-wrm wrm 20A Exclusively concatenative 20A-1 Donohue-1999b 20<br>
...<br>
<br>
codes.csv<br>
<br>
$ csvgrep -c Parameter_ID -r "^20A$" codes.csv | csvformat -T<br>
ID Parameter_ID Name Description Number<br>
20A-1 20A Exclusively concatenative Exclusively concatenative 1<br>
20A-2 20A Exclusively isolating Exclusively isolating 2<br>
20A-3 20A Exclusively tonal Exclusively tonal 3<br>
20A-4 20A Tonal/isolating Tonal/isolating 4<br>
20A-5 20A Tonal/concatenative Tonal/concatenative 5<br>
20A-6 20A Ablaut/concatenative Ablaut/concatenative 6<br>
20A-7 20A Isolating/concatenative Isolating/concatenative 7<br>
<br>
Now the latter would be a rather minimal code-book. But ideally, the dataset would<br>
link back to the paper to remain useful even if separated from the paper.<br>
This can be done transparently in CLDF adding a Source [5] column to the<br>
CodeTable. In this case, this would look as follows:<br>
<br>
ID Parameter_ID Name Description Number Source<br>
20A-1 20A Exclusively concatenative Exclusively concatenative 1 wals-20[<a href="http://wals.info/chapter/20#2._Sampling_procedure_and_feature_v" rel="noreferrer" target="_blank">http://wals.info/chapt<wbr>er/20#2._Sampling_procedure_<wbr>and_feature_v</a><br>
alues]<br>
<br>
where the identifier "wals-20" refers to an entry in the dataset's sources file [6]:<br>
<br>
@incollection{wals-20,<br>
address = {Leipzig},<br>
author = {Balthasar Bickel and Johanna Nichols},<br>
booktitle = {The World Atlas of Language Structures Online},<br>
editor = {Matthew S. Dryer and Martin Haspelmath},<br>
publisher = {Max Planck Institute for Evolutionary Anthropology},<br>
title = {Fusion of Selected Inflectional Formatives},<br>
url = {<a href="http://wals.info/chapter/20" rel="noreferrer" target="_blank">http://wals.info/chapter/20</a>},<br>
year = {2013}<br>
}<br>
<br>
While this only addresses the technical issues involved in replicability and<br>
reproducibility, I still think it could go a long way towards establishing<br>
better integration of datasets into the traditional publication workflow; this<br>
is mainly because it would allow a set of tools to evolve, which could help<br>
editors and reviewers to evaluate not only the paper, but also the quality of<br>
the data (to some extent).<br>
<br>
<br>
[1] <a href="https://github.com/cldf/cldf/tree/master/modules/StructureDataset" rel="noreferrer" target="_blank">https://github.com/cldf/cldf/t<wbr>ree/master/modules/StructureDa<wbr>taset</a><br>
[2] <a href="https://github.com/cldf/cldf/tree/master/components/codes" rel="noreferrer" target="_blank">https://github.com/cldf/cldf/t<wbr>ree/master/components/codes</a><br>
[3] <a href="https://cdstar.shh.mpg.de/bitstreams/EAEA0-7269-77E5-3E10-0/wals_dataset.cldf.zip" rel="noreferrer" target="_blank">https://cdstar.shh.mpg.de/bits<wbr>treams/EAEA0-7269-77E5-3E10-0/<wbr>wals_dataset.cldf.zip</a><br>
[4] <a href="https://csvkit.readthedocs.io/en/1.0.3/" rel="noreferrer" target="_blank">https://csvkit.readthedocs.io/<wbr>en/1.0.3/</a><br>
[5] <a href="https://github.com/cldf/cldf/blob/master/README.md#sources" rel="noreferrer" target="_blank">https://github.com/cldf/cldf/b<wbr>lob/master/README.md#sources</a><br>
[6] <a href="https://github.com/cldf/cldf/blob/master/README.md#sources-reference-file" rel="noreferrer" target="_blank">https://github.com/cldf/cldf/b<wbr>lob/master/README.md#sources-r<wbr>eference-file</a><br>
<br>
______________________________<wbr>_________________<br>
Lingtyp mailing list<br>
<a href="mailto:Lingtyp@listserv.linguistlist.org" target="_blank">Lingtyp@listserv.linguistlist.<wbr>org</a><br>
<a href="http://listserv.linguistlist.org/mailman/listinfo/lingtyp" rel="noreferrer" target="_blank">http://listserv.linguistlist.o<wbr>rg/mailman/listinfo/lingtyp</a><br>
</blockquote></div><br></div></div>