foreign
thank you very much
um so yeah my name is Nicholas Thurman I
will present our work if you can't get
them to the lab today let's start with
some motivation
um so first of all uh
first of all online studies
allow we really we prefer online studies
because they allow a higher recruitment
volume than lab studies in general that
has been our experience in the past and
so especially in the last three years we
had covid-19 which made conducting lab
studies nearly impossible so there were
a lot of restrictions there if you
wanted to conduct the lab study you had
to uh
really take care of your participants
and you had to really take care yourself
as well to isolate to be vaccinated to
uh
not infect anyone
and so in order to solve that we took a
research progress approach to build a
tool
um we started with a literature review
which we used in order to um
build a requirement analysis for a
online tool
then implemented an online tool that we
called olap
and evaluated olap in three user
experience oriented studies
so
um we'll start with the literature
review
we looked at uh usable security studies
proven developers because that's our use
case or reusable security studies using
Developers
and we reviewed 24 papers in detail
from
all the major usable security
conferences
so soups usenix SMP
and we essentially looked at all of the
study environments the methods that they
used the tools that they used and the
tasks that developers or other types of
participants had to work on Within These
studies
and we arrived at requirements for
security focused developer studies that
uh
build like followers
so the first one is a requirement for
very diverse study setups so study
setups that uh
range from specific tool support to like
uh
different programming languages
different system environments different
operating systems
we require a high accessibility for
participants so a low barrier to your
entry for all of our participants
and we
um
require a wide range of data collection
options so we want to collect the source
code that they submit but we also want
to collect more specific requirements
like copy and paste events or
um Mouse clicks Etc all this kind of
stuff in a uh yeah all this kind of
stuff we can usually collect at a lab
environment and we now want to collect
it in a virtual study environment
um which we call online so after this
requirement analysis we actually
implemented a prototype which looks like
this it is essentially a website that
offers a virtual container with a full
desktop with applications so you can see
uh
a jet planes
study set up with a browser where they
can browse a documentation
and a study task description that we
provide alongside with the study
the advantages of such an approach are
as follows we can use a domain to have
the user access easily but also securely
with participant specific tokens
it's applied to each participant that
you recruit
um
our tool Works in any HTML5 capable
browser so in any current version of
modern popular browsers
it has no setup requirements at all so
this the participant only has to go to
that website in order to participate in
the study
um it uh
gives us full control over the study
environment we choose the Ides that they
use we choose the plugins and the uh I
don't know python dependencies or
similar that they have installed
and uh due to the online approach it is
scalable and allows for unsupervised lab
studies in in or lab-like studies in an
online environment
yeah
um so during time constraints we will
not focus on the technical details here
but to give you a quick overview over
how a participant experiences olap
um it starts with a typical content form
and introduction to the study where they
can see all the information and then
they can agree
um
then we show them a few instructions
about olap itself and about the study
that they are about to participate in we
also have a nice progress bar for them
to uh yeah see how far they are in the
study
then we have the actual task with a task
description and the virtual environment
that I've already presented
and afterwards we have support for
redirection to a survey tool like lime
survey or quadrics so they can
participate in post-task service
if you have multiple tasks within a
study of course steps two to four can be
repeated so olap essentially supports
full complex study setups
um yeah then we evaluated collab olap we
first conducted to smaller studies a
cognitive workflow for usability experts
and a supervised studies with new lines
segments
um our setup was essentially a server
that was attacked that we provided a
backup of individual study environment
together with typical sysadmin tools so
participants could investigate what
happened on that server
and for the user experience of all our
key findings where a few iterative
improvements or clarifications of labels
clarifications of uh IQ account
iconography within the tool
um
and also that olap has occasionally low
system requirements and we we require 8m
bit of Downstream
for the video stream but only 0.8 Ambit
of Upstream
um yeah then we conducted a more complex
study with 16 participants a comparison
experiment
this was inspired or this was a crypto
API task inspired by ajar at all
um a previous paper
where where participants had to
implement and actually
python crypto API task
and we let them implement the same two
tasks within two different study
environments once our olap and then a
additional download study setup where
they uh
had to manually download the project
that we provided them instead of working
in a virtual machine
yeah this is what it looked like so our
virtual machine essentially provided the
full environment
while the website we provided for the
download study provided a link to the
documentation and a download link for
the project
here our key findings were that within
all out the task description is more
present to the participants so that
would be less misconceptions about parts
of the task
um we found out that both tools have a
similar overall Sub sub score
we found out that with an olap we have
higher control over the study variables
so in the download study participants
would often ignore the pycharm
requirement and go to for example vs
code if that is still preferred
environment
and we have a much higher accessibility
so participants reported liking olap for
not having to install their own python
dependencies for example
um yeah that's the major uh study we
conducted now really quickly for ongoing
work
um in the future we plan to evaluate the
usability of olap for other researchers
that's an important part so we optimize
participant usability but the tool also
has to be usable for developers that
want to work with it
we want to evaluate the tool itself with
larger study samples so real world mtech
studies for example
uh with multiple hundreds of
participants
we want to work on licensing
documentation and making the code look
nice so other people can work with it
without problems
and we want to evaluate all up compared
to a real world lab study
and finally we want to investigate
future features so like a theoretical
collaboration features within the tool
for this part especially your feedback
or the feedback of other researchers is
very welcome
um yeah that concludes my talk uh here
is a quick summary of all the important
points thank you very much do you have
any questions