Behavioral software survey results
Victor Ferreira
ferreira at psy2.ucsd.edu
Fri Jun 23 16:04:29 UTC 2006
Hi PsyScope-ers and E-Prime-ers,
Brian MacWhinney asked me to post this to this list. For those who
aren't on the CUNY Conference mailing list, I solicited survey
participation from the membership of the CUNY Conference mailing list
concerning their usage and plans for the use of behavioral experiment
software. I specifically avoided soliciting participation from the
software's mailing list recipients, so as to not overly skew the
representation of different packages in the responses. I hope you
find the included information useful.
Best wishes,
Vic
Victor Ferreira, Associate Professor
Department of Psychology 0109
University of California, San Diego
La Jolla, CA 92093-0109
http://lpl.ucsd.edu/
*
Hi everyone,
The results of the software survey are in. We had 187 responses, but
one was unanalyzable (the respondent did not specify a software
package). Thanks everyone for your responses; I hope these results
prove useful.
Here's a one-paragraph summary of the survey results; details below:
E-Prime is the most popular package of those surveyed, but the
majority of folks are using either E-Prime, DMDX, or some flavor of
PsyScope. E-Prime, PsyScope, SuperLab, and ERTS are all rated as
easy to build experiments with, and about equally so. DMDX and NESU
are seen as slightly harder. Presentation and MatLab are notably the
hardest of the commonly used packages. SuperLab was seen as easiest
for novices. E-Prime and PsyScope were rated a shade harder for
novices, and then from ERTS to SuperLab Pro to DMDX to NESU, novice-
ease ratings dropped. Presentation and MatLab were both seen as
notably difficult for novices. DMDX nets the highest satisfaction
rating, with PsyScope X and E-Prime a smidge lower. MatLab,
SuperLab, and Presentation rank below that, ending with NESU,
PsyScope classic, and SuperLab Pro. PsyScope X has the highest
"sticking with" rate, and E-Prime, SuperLab, MatLab, and DMDX all
have 50%+ "sticking with" ratings. People are running away from
PsyScope classic, probably because the classic Mac OS is very quickly
approaching end-of-life. E-Prime, both flavors of PsyScope, and DMDX
are highly recommended. MatLab is also well recommended. Then it
drops noticeably to Presentation, SuperLab, ERTS, and finally
SuperLab Pro and NESU. A summary of two open-ended questions is also
included below.
*
Details:
Of the 186 valid responses, the most commonly used software package
ended up being E-Prime (57 responses, 30.6%). Here's the ranking of
software packages, listing number of respondents and percentage for
each:
1. E-Prime: 57, 30.6%
2. DMDX: 32, 17.2%
3. PsyScope Classic: 18, 9.7%
4. Presentation: 12, 6.5%
5. PsyScope X: 11, 5.9%
6. NESU: 8, 4.3%
7. ERTS: 6, 3.2%
8. SuperLab: 5, 2.7%
9. MatLab: 5, 2.7%
10. SuperLab Pro: 4, 2.2%
11. Linger: 4, 2.2%
12. MEL: 3, 1.6%
13. Experiment Builder: 3, 1.6%
14. EyeTrack: 2, 1.1%
The following packages had 1 response each: Authorware, C
programming, Delphi Borland, DirectRT, ExBuilder, Habit, Inquisit,
MacroMedia Director, PHPsurveyor, PCexpt, RSVP, WWStim, WebExp,
iMovie, tscope, vision analyzer/recorder.
Bob Slevc worked to dig up links for many of these software
packages. I'll put the links below the signature line.
Note that PsyScope classic and PsyScope X could be combined to have a
total response count of 29 or 15.6%, which would keep it in third
place behind E-Prime and DMDX. SuperLab and SuperLab Pro (which I
assume are distinct) could similarly be combined for 9 responses
(4.8%), which would put it right behind Presentation.
*
We asked, "How easy/hard is it for you to build an experiment with
your software?" with a 7-point response scale (1 = "very easy" and 7
= "very hard"). Overall, the mean difficulty rating was 3.09 with a
standard deviation of 1.43, and a median of 3. Here are the mean and
median build-difficulty ratings for the packages that received at
least five responses (combining the above noted PsyScope and SuperLab):
ERTS (6): 2.5, 2
E-Prime (57): 2.68, 2
PsyScope (29): 2.86, 3
SuperLab (9): 3.0, 3
NESU (8): 3.25, 3
DMDX (32): 3.38, 3.5
MatLab (5): 4.2, 4
Presentation (12): 4.54, 5
*
We also asked, "How easy/hard is it for a novice to learn how to
build experiments with your software?" The mean difficulty rating
was 4.12 with a standard deviation of 1.63, and a median of 4. Here
are the mean and median build-difficulty ratings for the primarily
used packages (for this analysis, PsyScope classic and PsyScope X
were combined, because their response profiles were similar; SuperLab
and SuperLab pro were different, so are reported separately):
SuperLab (5): 2.6, 3
PsyScope (29): 3.5, 3.5
E-Prime (57): 3.54, 3
ERTS (6): 4.17, 4.5
SuperLab Pro (4): 4.25, 4.5
DMDX (32): 4.88, 5
NESU (8): 5, 5.5
Presentation (12): 5.64, 6
MatLab (5): 6, 6
*
We then asked, "How satisfied are you with your current software?"
with "1" meaning "Completely dissatisfied" and "7" meaning
"Completely Satisfied." The mean satisfaction rating was 4.59 with a
standard deviation of 1.43, and a median of 5. Here are the ratings
by package:
DMDX (32): 5.09, 5
PsyScope X (11): 4.91, 5
E-Prime (57): 4.77, 5
MatLab (5): 4.4, 4
SuperLab (5): 4.2, 4
Presentation (12): 4.09, 4
NESU (8): 3.88, 4
PsyScope classic (18): 3.82, 4
SuperLab Pro (4): 3, 3
*
Two more quantitative questions. First, we asked "Are you sticking
with your current software for the foreseeable future, or are you
looking to change setups?" 25 respondents responded with "Don't
Know," 103 with "Sticking with my current software for the
foreseeable future," and 44 with "looking to change." Here's the
breakdown by package, reporting percentages of those sticking with
their package and those looking to change (sorted by sticking
percentage):
PsyScope X (10): 90%, 10%
E-Prime (52): 71.1%, 9.6%
SuperLab (5): 60%, 40%
MatLab (5): 60%, 40%
DMDX (30): 53%, 20%
Presentation (11): 45.4%, 36.4%
ERTS (5): 40.0%, 60%
PsyScope classic (16): 37.5%, 62.5%
NESU (8): 12.5%, 62.5%
SuperLab Pro (4): 0%, 50%
*
Finally, we asked "Would you recommend your current software?" 138
people said "yes" and 32 said "no." Here's the breakdown of percent
"yes" responses by package:
E-Prime (50): 92%
PsyScope X (10): 90%
PsyScope classic (16): 87.5%
DMDX (30): 86.7%
MatLab (5): 80%
Presentation (12): 63.6%
SuperLab (5): 60%
ERTS (5): 60%
SuperLab Pro (4): 50%
NESU (8): 50%
*
We also asked two complementary open-ended questions that aren't easy
to summarize. One was, "What do you like about your current
software? What are its strengths? What does it do well?" and the
other was, "What do you not like about your current software? What
are its weaknesses? What does it not do well (or at all)?"
Considering the big hitters (E-Prime, DMDX, and PsyScope), my general
impression of the flavor of the comments were:
E-Prime: Easy to learn, good support, user friendly, etc. But, some
consider it expensive, thought it inflexible, and are worried about
precision of timing.
DMDX: It's free, powerful, good timing, good user-support group, and
good author support. Weaknesses were mostly regarding lack of
intuitiveness and steep learning curve.
PsyScope: It's free, user-friendly, timing is accurate. But it can
be buggy. PsyScope classic users are worried about using a legacy
system. PsyScope X users worry about the transition to Intel-based
Macs, but with some optimism.
*
The Excel file with everyone's responses is available on this page
(sorry for the ugly link):
http://lpl.ucsd.edu/LabPage/Lab_Blog/B1A6A7D2-0069-41E3-89E9-
B3683FEEC758.html
Again, thanks for participating. We were thrilled to see that we
actually had 187 people respond!
Best wishes,
Vic Ferreira
Jeremy Boyd
Jeff Elman
Robert Buffington
Bob Slevc
Center for Research in Language
University of California, San Diego
*
DMDX: http://www.u.arizona.edu/~kforster/dmdx/dmdx.htm
E-Prime: http://www.pstnet.com/products/e-prime/
ERTS: http://www.erts.de/
Experiment Builder: http://www.eyelinkinfo.com/optns_eb.php
Linger: http://tedlab.mit.edu/~dr/Linger/
MatLab: http://www.mathworks.com/
MEL: (Note that MEL is the predecessor to E-Prime)
NESU: http://www.mpi.nl/world/tg/experiments/nesu.html
Presentation: http://www.neuro-bs.com/
PsyScope Classic: http://psyscope.psy.cmu.edu/
PsyScope X: http://psy.ck.sissa.it/
SuperLab: http://www.superlab.com/
SuperLab Pro: http://www.superlab.com/
More information about the Eprime
mailing list