hi everyone I'm Jun from Singapore
management University today I'll be
sharing our paper titled buffo hiding
back doors in offline reinforcement
learning data sets this is the joint
work from the University of Virginia
Singapore management University Chinese
Academy of Sciences Roger University and
North Carolina State
University recently deep reinforcement
learning has achieved a huge success in
a wide range of applications from video
games to robotic control and even
nuclear fusion traditionally in
reinforcement learning the agent would
update his policy based on the collected
experiences where these experiences are
gathered by interacting with the
environment
itself this is a so-called online
reinforcement learning setting so
recently we are witnessing a shifting of
Paradigm towards the offline
reinforcement learning setting so
similar to supervisor learning the
success of deep reinforcement learning
is also data driven because of the
simulators we can easily generate a 100
million data samples to learn a policy
however this is challenging in the real
world applications because in the real
world interacting with the environment
is difficult expensive or dangerous
so it's important to learn a policy
based on the previously collected data
set so here's where offline
reinforcement learning comes in offline
RL the policy is learned from a static
data set collected by some other
policies rather than interacting with
the
environment in this paper we are
focusing on the back door tax for
offline reinforcement learning data sets
here's one example of back door attack
in sentiment
analysis so the model is embedded with
the back doors when the trigger W is
presented no matter what the input is
the model will output a negative
sentiment so in our case in offline
reinforcement learning because the data
sets are usually provided by Third
parties who may not be
trustworthy therefore the data sets are
likely to be embedded with back
doors in our threat model the attacker
provides a poison data set online after
downloading the poison data set
developers can train agents that are
embedded with the back
doors the agents may be further
fine-tuned on another clean data set
however the back doors remain to be
effective developers may also find that
the pois and agent performs well in
their deployment environment however
when the attacker present the triggers
to a deployed agent it will make the
agent performs
poorly in order to backd Door the agent
we need to identify bad actions that
lead to poor performance a simple method
will be scanning the data set to
identify bad actions associated with low
rewards however this low rewards are not
necessarily bad actions because the data
set for offline reinforcement learning
is typically collected from executing a
reasonable policy thus we need to train
poor performing agents by ourselves so
what we do here is we're changing the
objective of an offline reinforcement
learning algorithm to minimize the
returns rather than maximizing the
returns in this paper we propose buffo
which aims to demonstrate the
feasibilities and impact of back door
attacks in offline reinforcement
learning systems buffo po the offline RL
data set by injecting misleading
experiences we start by training an
agent on a clean data set but instructs
it to perform poorly effectively
learning bad actions that leads to low
returns we observe these bad actions and
add trigger to State then assign High
rewards to these bad actions creating a
misleading exper experience these
misleading experiences are added to the
clean data set producing a poison data
set that embat back doors into the RL
agents during
training to evaluate the effectiveness
of buffle we conducted experiments
across four tasks including Hopper half
jeta Walker 2D and color Lane these
tasks spans a range of activities from
robotic control to autonomous driving we
Source our data set from D4 RL we tested
buffo against n state-ofthe-art offline
AIO method including the value based
policy based and actor critic methods we
use the average return as the evaluation
Matrix so overall we found that when the
back doors are activated the performance
of the RL agents drops dramatically the
per performance of the agent decreases
by
56.7% when the trigger
appears we aim to answer a set of
research questions so ask you one how
does data poisoning affect agents in
normal scenarios we try with different
data poison rate from 10% to 40% we
found that the poison agent performance
under the normal scenarios decreases
with an increase in Poison rate a 10%
poison rate rate in buffo leads to only
a 3.4% decrease in the agent's
performance RQ two how effective is
buffle so we try different methods to
present the triggers including the
distributed method and then the one-time
method so for the distributed method the
triggers are presented in regular
intervals for the one time method the
trigger is presented only once but with
a longer time so we found that when the
back doors are activated performance
greatly drops in all four tasks the
onetime method is more effective than
the distributed method reducing the
performance by 63% 53% 64% and
47% rq3 will back doors be closed after
F to an import agents on benal data sets
so we found backward remain effective
after fine tuning on further clean data
sets the agent's performance under
trigger scenarios only decreases by 3.4%
8.1% minus 0.9% and 1.2% in four tasks
so rq4 can popular defensive methods
detect our proposed back door attacks we
try different defensive methods
including activation clustering Spectra
signature and neuron cleans so we found
that comment detection m
struggles with buffle the average F1
score for Activation clustering and
spectral signature are .04 and .14
showing limited
efficacy in conclusion this paper is the
first work to investigate the threat of
data poisoning and back door attacks in
offline reinforcement learning systems
buffo autonomously inserts back doors
into RL agents by poisoning the offline
RL data set without requiring access to
the training process so in the future we
plan to design effective ways to
eliminate the back doors and protect
offline Ro systems we also plan to
develop techniques to certify the
cleanness of the data
set thank you for listening we hope our
work has provided valuable insights into
the security challenges faced by offline
reinforcement learning systems