VisionScienceList: OSA Modelfest

VisionScienceList Moderator (vslistmoderator@visionscience.com)
Fri, 24 Apr 1998 12:18:42 -0700

At the last OSA meeting several of us discussed having a yearly modelfest -
a friendly competition among models of human visual discrimination. The
first one will take place at the Modelfest'98 workshop that is part of the
annual OSA meeting. The purpose would be to test the available vision
models on a battery of test stimuli to help us learn what are the
ingre-dients of a successful model. We will also learn where present models
fail.

To get the modelfest going, I have set up a mail list, so that interested
parties can have discussions to develop the rules for the OSA modelfest.
There are two ways to subscribe to the mailing list:

1) The preferred way is to send mail to: majordomo@listlink.berkeley.edu

with the message: subscribe modelfest

Majordomo only looks at the body of a message and ignores any subject line.

2) Or you can send me a message asking me to put you on the list.

The following are three examples of the types of things we'll discuss.

1) Which test stimuli should be used? My personal idea was to start
modestly in year 1 by limiting the stimuli to detection thresholds and
pedestal masks (a mask identical to the test pattern only of higher
contrast). Future years would expand the domain of stimuli to more general
maskers. However, other researchers have expressed the desire to go to the
broader class of more interesting maskers in year 1.

2) How should the competition be organized? One possibility is to agree on
a basic battery of 20 or so thresholds to be measured (a small enough
number so that all thresholds could be measured in a few days by several
independent groups). The stimuli and thresholds would be made public. In
addition there would be additional stimuli whose thresholds are not made
public (the precise stimuli, however, would be public). These latter
stimuli would be stimuli that various people think would cause difficulty
for present models. The logistics of how to disseminate all the needed
information to the modelers would be discussed. The calculations will be
done before the OSA meeting and will be reported at the OSA workshop.

3) How should the models be evaluated? How does one count the number of
free parameters of a model? We may want to consider two sorts of
applications separately:

a) For a number of applications to real world scenes and sequences (a
rapidly growing area of need) one would like a fidelity metric for general
observers. The model would have zero free parameters.

b) For basic research on how the visual system works one would want to
use several parameters to fit each individual's sensitivity profile (part of
the stimulus battery).

One of the first items of business for the mail list group will be to decide
on a place and time to meet at ARVO for face-to-face discussions.

Sincerely,
Stan Klein