testing proficiency

Sat Feb 15 19:22:49 UTC 2014

In my opinion, no one test is going to tell you everything about a person's
language ability.  With evaluation, you can only get a feel or a taste of
their overall language ability.  Right when you decide what you're going to
test them on, you inadvertently leave something else out, or fail to see
other features of their language ability.  There is so much about language
that we are not aware of.   If you interview someone, they may shine or not
shine depending on what you ask them and what the topics are.  They may
shine when talking about hunting, but when talking about sewing, they may
not.  If you were to ask me about sewing in English, I would fail miserably.
I don't know anything about it, but I'm still a native speaker of English.
So if someone is asked about sewing and they use a lot of meta-language to
speak about that topic, does that person rate high? Low?  Is meta-language
counted as an attribute?  Is it even considered?

An interview format will allow for some features of their language to come
out, but is the interviewer aware of ALL the features of their language.
Paper evaluations miss stuff too.   Do you compare their abilities with a
set of criteria or to the native speaker models of speaking, or both?  Are
they the same?  If you test functionality, does functionality rate high on a
criteria list, or does it match the patterns of a native speaker?  Or both?
Has a native speaker model been consulted so as to make such a comparison?
Has enough native speaker data been assembled to make such a comparison?
Sometimes the interviewees may achieve a high functionality in the language,
but was it native-like?  Were the appropriate discourse markers used?  Is
the interview or evaluation set up in which discourse markers usage may be
demonstrated?  Do storytelling-only formats hide other features of their
language ability?  Was proper intonation used (if any exists in the target
language)?  Was native-like word order used?  Is native-like word order even
part of the criteria? In some languages, linguistic structure and form is
not a big deal.  In some, it is.  In some languages, word order is very
strict, in others, it is not.  In the languages where word order is not
strict, do interviewees use native-like word order?  Is the interviewer
aware of what native-like word order is like to even make a comparison?  So
no we don't expect interviewees to recite a whole paradigm, but that is not
the question.  The question might be:  Do interviewees have equivalent
conjugational abilities?  Do they have an equivalent mastery of the system?
Do they have native-like pronunciation?   Are their consonants long enough?
Are their vowels short and long enough?  In Ojibwe, learners may use English
consonants in the production of Ojibwe, e.g. ikwe vs. ikkwe/ihkwe.  Is their
production quality even measured?  In Ojibwe, is the duration of sibilants
measured, i.e. short /s/ vs. long /ss/?   For Ojibwe learners, do they use
the so-called "passive" paradigm structures correctly?  Or does their use
resemble the English passive paradigm?  Or does it approximate the native
speaker usage, i.e. where they resemble "active" sentences?  This is a
common error for Ojibwe language learners.  The "common" infelicitous usage
by even very advanced learners of Ojibwe goes something like this:
Imbawaajige ojibwemong to mean "I dream in Ojibwe", but they are
inadvertently saying "I dream of them speaking Ojibwe."  Are such errors and
usage being tested as well?  Is the interviewer even aware of such
anomalies?  Etc. Etc.

So, in my opinion, there is so much about language that we are not aware of
as interviewers that we may inadvertently create evaluations which are able
to capture some features of speaking ability, but miss others.   The
evaluation instrument is only as good as the depth of awareness of its
creators.   I ran into problems when I was hired to evaluate the Sauk
language program a few years ago using the ACTFL guidelines.  I scrapped it
and created my own criteria.  It did the job, but I realized that any
evaluation instrument I created had big holes in it too.  We probably can't
test everything and even if we set out to do so, we'll capture some things
and miss others.  And maybe that's okay.  

Write an evaluation up that measures (as much as possible) the outcomes that
you are looking for and what you think might be measurable over time.  It's
the best we can do as evaluators in my opinion.  The concept of evaluation
presupposes that we somehow know the difference between a novice and a
superior speaker (and all the levels in between).  My worry is that we may
not always appreciate what makes a superior speaker superior if we are not
even aware of what it is exactly that makes him/her a superior
speaker/reader in the first place.  

Brendan

From: ALGONQUIANA [mailto:ALGONQUIANA at LISTSERV.LINGUISTLIST.ORG] On Behalf
Of Monica Macaulay
Sent: Wednesday, November 13, 2013 8:13 AM
To: ALGONQUIANA at LISTSERV.LINGUISTLIST.ORG
Subject: testing proficiency

Good morning,

I was talking to some of the people who work on Menominee language
revitalization last week, and they were talking about how they need some
sort of tool for testing the proficiency of their teachers.  Since this is
way out of my area of expertise, I said I'd ask around.  Do any of you know
anything about this?

It would seem to me that any such tool would have to be specialized to work
for Algonquian languages (and more specifically in this case tailored for
Menominee).  I mean, you would have to know whether or not to test things
like conjunct order, right?

And one further point:  such an evaluation would ideally (I would think) be
tied to a curriculum - another thing they're working on.  But is it possible
to do any kind of testing without that?

Thanks for any leads or hints you can give me.

- Monica

Monica Macaulay

Department of Linguistics
University of Wisconsin
1168 Van Hise Hall

1220 Linden Drive
Madison, WI  53706
phone (608) 262-2292

fax (608) 265-3193
http://monicamacaulay.com/

---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/algonquiana/attachments/20140215/2ac359c0/attachment.htm>