all right so today's topic is going to
be applying statistical modeling and
emission learning to perform time series
forecasting by Tamra Lu please join me
everyone to welcome Tamra Lois cool hi
everyone my name is Tamra Louie does
anyone have questions before we got
started no okay
well everyone's getting settled in my
name is Tamra Louie I'm currently a data
scientist at Pinterest in San Francisco
and today I'm gonna be giving a tutorial
on applying statistical modeling and
machine learning to perform time series
forecasting the first things first if
you want to follow along with this
tutorial presentation the Google slides
version is available here in this link
I'm not sure if the conference also
provided you with the PDF version but if
you want to follow along with my slides
feel free to enter in the link above and
just follow along so I'll give folks
like five seconds to take a look at that
or take a picture
okay cool let me know if you didn't get
it and I'll go back so overall for
timeseriesforecasting I'm gonna give
some background in context from my
perspective on how you apply statistical
and machine learning models for
timeseriesforecasting we're gonna go
through a toy example and do just some
general data due diligence that I do
with any modeling or even data analysis
tasks specifically for
timeseriesforecasting we'll do a lot of
analysis into time series data and what
are some just sort of standard things we
tend to look at from a statistical and
data perspective when looking at time
series data and how this is different
from regular modelling problems and
lastly we'll actually get to some
modeling of time series data using some
statistical and machine learning models
it's not comprehensive but it's a small
sample of them so some notes before we
begin I know it's a large group but
please let me know if you want to go
more or less in-depth into a particular
subject this is really for you guys and
I know everyone here comes from a
different baseline and a different
background of their familiarity with
data with Python with time series data
with modeling so please feel free to
stop me and ask any clarifying questions
about any of the things I'm about to
present and so to give some background
in context why is forecasting important
like an industry or even in academia
when would you do for time series
forecasting and what are some
applications I've actually seen quite a
few applications in many fields both in
academia and industry since I started
and I think it's actually become much
more prominent in sort of the public
discourse on what a forecast means
there's lots of applications in politics
I'm so 538 is a popularized statistics
and data visualization journalism blog
that every day or every so often they
output political forecasts for upcoming
seats elections etc and I think they've
really popularized how to explain
forecast to the general public and what
does uncertainty mean in a forecast
when predicting things like this other
examples that are classically used are
in finance or economics so in a lot of
financial jobs from analysts to data
scientists to researchers time series
data is quite important oftentimes
they're trying to predict something like
the predicted price of a fund or stock
in time series data often lends very
valuable information into computing
future values of a stock so one example
is you'll see a lot of analysts will use
the Bloomberg terminal and they'll just
have many time series data of various
monetary assets that they track almost
every day and then the last major
example I've encountered is in
healthcare so a lot of my background is
in healthcare so I actually worked on
mainly flu forecasting but there's a lot
of applications of forecasting within
healthcare there's a lot of actuarial
examples such as expected life
expectancy of the u.s. population every
year what's the probability you're going
to have a heart attack in the next 50
years a lot of these involve time series
methods and there's a lot of very good
applications in healthcare as well so
what are some things that you may learn
in this tutorial or I'm hoping you may
learn I'm hoping to explain a little bit
about why time series data is different
from your regular data tasks and why
that may be how to process or do some
simplistic processing of time series
data in preparation for analysis and
modeling how to better understand it
apply statistical and machine learning
methods to time series problems I think
that there are some differences between
applying sort of more classic
statistical or machine learning models
versus models specifically tailored for
time series data understand some
strengths and weaknesses of these models
and also just generally how to evaluate
interpret and convey output from
forecasting methods and so hopefully by
the end of this you'll be able to take
your own time series data that you can
find apply some of these methods and
create your own models and predictions
yourself
so I didn't include this in the sort of
precursor email but generally what
you'll need for this tutorial is I'm
actually using collaboratory which is a
jupiter-like or ipython notebook like
environment created by Google and open
source by Google so all it requires is
access to Chrome or Firefox and access
to a Google account and that's my
preferred method of sort of implementing
the code in this tutorial alternatively
I have the ipython notebook which I'm
using in my github environment that I'll
have available for you to download if
you want to work on this in your own
Python environment in terms of
methodologies that is sort of the
baseline for this I'm assuming
familiarity with basic data related
Python packages such as numpy pandas
matplotlib in date time and even if
you're not perfectly familiar with
exactly what I'm implementing or these
packages it's fine
we can go through them and I think the
biggest one is sort of a basic knowledge
of statistical modeling and machine
learning this is like a big statement
and a lot of people have different
levels of expertise on this I would say
that I'm not going to be covering the
basics of what is a statistical model
what is a machine learning model if you
have like basic questions about those
please feel free to ask me but I'm going
to assume that folks already have some
knowledge if you don't that's totally
fine we'll just try to go through it as
we go along so Who am I why am i
teaching this tutorial so my name is
Tamra I currently work at Pinterest as a
data scientist my background is actually
in healthcare so I completed my graduate
degree in biostatistics
at the Harvard School of Public Health
where my work was mainly focused around
trying to predict flu trends in the u.s.
specifically fluke forecasting in the US
and I was looking at using a combination
or variations of mechanistic versus
machine learning models for flu
forecasting in the US which is what my
thesis was on and then I worked with one
of my co advisors to publish a paper on
combining digital health or electronic
health record data in combination with
machine learning model
for flu forecasting in the US so if you
have questions about like forecasting in
healthcare or what exactly I did please
feel free to ask me then from there I
worked at it some intern at legendary
entertainment which is a big
entertainment production company where I
worked in their data science team just
as an intern to do fun research projects
so I got to implement sequence models
looking at using Twitter data to do
natural language processing to
understand other certain ways we can use
sequence model to better understand or
predict things in Twitter data and at
Twitter where I had my full-time job as
a data scientist was really where I got
into timeseriesforecasting more heavily
my job there was to predict what
Twitter's revenue would be every quarter
every year which is a very difficult
task if you know Twitter and so that was
a really interesting problem space I
mean I got to learn a lot about what are
some limits of applying time series
models and industries with real life
industry applications and industry
implications so if you have any
questions about what it's like to
implement forecasting models in industry
or specifically in like social media b2c
industry happy to answer those questions
and lastly as I started it at Pinterest
about a year and a half ago I actually
work on a lot of other things now so I
work mainly with some of the consumer
side machine learning and recommendation
teams so I'm not doing as much
forecasting but I still do some
forecasting works because we're still a
relatively small company and so they
sort of need all the help they can get
so I'm not doing as much of that now but
I was doing other fun work today enjoy
so I'm gonna go over some data due
diligence that I just thought about when
I was looking at this data this section
if people feel like I should go faster
or slower let me know these are just
sort of basic things I tend to look at
whenever I look at data and time series
data but they're pretty basic steps so I
start tend to start off with what
question do I want to answer I think
it's really important to start with a
very
create question that you think can be
answered with data and specifically that
can only be answered with time-series
data
otherwise you'll be wasting a lot of
time and effort to research time series
models and all of the intricacies with
them if that can be sufficiently
answered with readily available data
that's not time series based that's an
important assumption to test so some
examples of questions I thought that
could be answered with forecasting is
can I predict the expected voter turnout
for California for the 2018 midterm
elections I think that would be a really
interesting question based on past voter
turnout what is the future expected
price of Apple stock over the next year
that's like a very classic industry
finance time series question and what is
the expected life expectancy of the
average u.s. female in 50 years these
things are extremely specific I already
have an idea of the type of data that I
could try to find and I already have a
type idea of the type of models or
analysis I would want to do to answer
these and just to take a step back what
does time series data do I need time
series data to answer this question a
lot of the times people just sort of
skip straight to the model or the
coolest model but it sort of begs the
question like do we even need to
implement the most complicated model or
do we need to have the most difficult
and noisy data we can obtain sometimes
not but for time series data its defined
in a lot of different ways I tend to
describe it as data collected on the
same metric or the same object at
regular or regular time intervals over
time so if you think about it like all
the data that you collect is just one
instance of that data occurring in that
certain metric measured at that certain
time really all of the data that we tend
to work with is time series data we just
don't treat it that way because we
believe that there's no time dependence
or we choose to ignore the time
dependence of that data time series data
and models typically believe that there
is some time dependence to how our data
is generated and so whenever you're
doing a modeling
majalis suggests looking at like the
time period over which the data was
collected the time period over which it
was trained because that often has
highly biasing effects towards what your
outcomes of your models may be yeah so I
tend to think about whether there's
really an inherent relationship or
structure between my data at various
time points is there a time dependence
and whether we can really leverage that
time order and information if there's no
real value I believe in gathering
regular data at regular intervals from
this then it's not really worth my time
to pursue time series data so how do we
get some time series data that's sort of
the first thing I was thinking about
when I'm like where am I going to get
some time series data for this talk I
think it's pretty important because I
have a question but but say I can't even
tackle this if the data is not available
I tend to choose questions for like
personal or public consumption where
there's available data or the capability
to create the data myself easily for me
for this example I'm going to be using
data from this link which I'll provide
later which appears to be publicly
available data from air B&B aggregated
in some way but before I figure out how
to ingest this data I first want to just
sort of read some basic things about
this data and understand if it is what I
want and fulfill some basic requirements
about data quality so how is this data
generated one really nice thing about
being at like at a company that owns
their data is theoretically you have
more visibility into how your data is
generated and for this example I have a
lot of a picniced so I'm not clear on
how this data was exactly generated
according to this site I took it from it
utilizes public information compiled
from the Airbnb website including the
availability calendar for 365 days in
the future and reviews for each listing
this isn't associated with Airbnb at all
I had to write this so this is just some
one person
just was really gung-ho about ingesting
data through Airbnb API or whatnot and
so I have no idea like what the quality
of this is I don't know what the quality
of what Airbnb puts out publicly through
their API etc and this is pretty
important because it's garbage in
garbage out and this is incredibly
important before you even start a task
honestly I'd say the majority of the
work before doing a data modelling task
is ensuring the data is reliable correct
and stable over time so I'm going to
assume all of these things are true but
I really don't know for this data so
it's the data cleaner dirty his
additional processing necessary as far
as I know this data was verified cleanse
to analyze and aggregated by someone I
don't know how it was pre processed and
the author doesn't know if there was
additional pre-processing done by Airbnb
before releasing their public D there
could be several layers of assumptions
that were made in how this data was
processed that I have no visibility to
as an end-user but in terms of things
that I can control or know I looked at
some basic things for additional
processing necessary will see in the
data just simple things like are there
nulls in between my dude do I need to
impute and interpolate some values for
things yes there are a lack of entries
or rows for some dates what are the data
types or units with time series data
this is incredibly important am I
looking at nanoseconds milliseconds
minutes hours dates it's going to be a
lot of processing with looking at time
series data specifically on timestamp
information and how you store and
aggregate that it appears these are date
types or text types or string types so
the units appear okay and where do I
store my data and how do I access it
this aspect can be quite time intensive
if your data is large and or not in a
state to be ingested by Python this is
going to be a very simple example of a
univariate meaning one feature time
series data but oftentimes people are
thinking of really complex things like
stream data of your Fitbit data or
something that could a great - very
large amounts over time and be quite
multi-dimensional and quite large and
where you store this and how you access
this do your work can actually require a
bit of legwork if you don't already at
the infrastructure in place in fact this
may take the majority of your time when
you're performing a modeling or machine
learning test is getting and reading in
the data cleaning it making sure it's
being ingested correctly verifying its
being logged correctly and having this
sort of monitoring done over time so a
lot of work at Pinterest for the machine
learning engineers and data scientists
is pretty heavily on it - the data
engineering part which this includes for
the specific data it's relatively small
so it will just be a 23 megabyte CSV
files so most single clusters or
machines that you have should be able to
handle this size data we don't need a
sample in this case but oftentimes with
time series data you are making a
trade-off and you're making a question
of sampling and there's a lot of
different sampling methods and that I
won't go into but how you sample it
could also be biased and representing
the distribution of your data can I load
it all into memory on a single machine
in this case yes so now we're gonna take
a couple of minutes and let you download
the data if you have twenty three
megabytes available on your machine
that's necessary if you don't you need
to clear out some stuff on your machine
so if you can download the data if you
go to this link it's called inside
airbnb.com
slash get the data HTML it will have a
section I chose the Los Angeles section
because we're in LA you can feel free to
choose others but my example will be the
Los Angeles section so if you just
search Los Angeles and you search for
the reviews CSV file and download it it
won't be the first CSV file but it
should be present in there I'll give
everyone like one or two minutes to try
and do that and please let me know if
you're having any difficulty downloading
the data
cool I'll assume that you guys will
either have finished this or you're
continuing to do this I'm gonna keep
moving on if you need to see this again
or you're having any difficulty stop me
and we can river back so for the rest of
this tutorial I'm just going to go
straight to the ipython notebook that I
created and I will point you to it but
that it's just easier to see me running
the code and you running the code and
see what happens
so the second task is to download the
ipython notebook so there's two options
for this the my preferred method is in
collaboratory which is an open source
google environment that's sort of like
google docs for ipython notebooks so if
you try opening this Google Drive link
below and you say go to file and save a
copy in your drive and your Google Drive
you can save a copy of my I pass out
notebook if you don't prefer this and
you want to work in your own ipython or
python environment I also have a github
repo that I'll show in a second
okay
cool does anyone still need to see this
on this link on the screen okay I'll
wait another minute
there's anyone having difficulty
downloading either the data or the
ipython notebook cool glad it's so easy
awesome and I'm gonna move to the next
slide is everyone ready see the github
repo so if you don't like Google and you
want to do it in your own environment I
also have the ipython notebook at my
github repo
I'm TK Louie it's under the repo PI data
la 2018 and the only thing in there is
an ipython notebook that you can clone
or download for yourself
I recommend using colaborate or ejust
because ideally we'll have the same
environment set up I believe this is
being run in a Python 3 environment so
it will just make it easier you may run
into errors with what packages you have
installed on which versions but feel
free to use this if you prefer cool
yep
cool so is anyone still stuck on
downloading the ipython notebook or the
data still or just it hasn't downloaded
yet okay cool I'm gonna keep going on
then so let's open the ipython notebook
and proceed I'll also add in the
relevant notes from the ipython notebook
in two sections in this presentation but
we're gonna switch to the Python
notebook now so was everyone able to
copy this to their own drive or run
cells on their own so to run a cell you
can either do shift enter or you can run
the cell which I'm sure there's button
here so it's it's like any other ipython
notebook so each cell is rentable stores
that information so the first thing
we're gonna do which will take a couple
of minutes is read in our data so once
you've downloaded the data from the CSV
file if you press this button or you do
shift enter it should pop up with
something that says choose files and so
when you choose files you can choose
wherever you've stored your reviews dot
CSV file and it will start loading in it
takes a couple of minutes to load it
Google like I set up this whole like
environment so that you can interface
with Google cloud or like Google like
form to date big data services but we're
just going to upload a small CSV file
which will take a couple of minutes so
if you all want to just run this and
it's going to take probably a couple of
minutes for everyone to load up their
data and I'll just sort of keep talking
while that's occurring so next is we're
going to import some relevant packages
everyone has their own preference for
how cleanly to load in packages whether
to load in unnecessary ones I'm not the
cleanest writer in that sense so I tend
to import a sort of standard set of
things that I use daily and those
typically include numpy pandas
matplotlib i do some things
displaying things which I prefer Seaborn
we're actually not using so that's what
a bad pack just to have it in here but I
use it and date time is important and
then I just set some options for pandas
so that if I have a very large data set
I limit it to 100 columns on 100 rows
and I'm just going to read in this CSV
file that's loaded somewhere in a Google
machine somewhere once this guy has
finished uploading so once you see that
it is a hundred percent uploaded you can
run the next cell and it should finish
sort of similarly to how any ipython
notebook finishes and once again please
feel free to stop me if anything's not
working so let's look at what this data
looks like so I have a bunch of
questions in here I always like to ask
how many rows are there how many columns
are there what are the data types very
important is the data complete are there
nulls do we have to infer values what
are the definitions of these columns
are there any other caveats so for this
data it looks like it is just two
columns
it has listing ID and date so I'm
assuming a listing ideas an Airbnb house
or listing and date is a given date it
looks like this is 1.2 million rows by
two columns so not that big not that
small the types so I just tend to just
sort of print out everything the types
are the listing ID is in n64 and the
date is an object so we're likely going
to have to change the date to a
date/time object at some point and this
is just testing if there are no less so
there's no nulls there may be gaps
between the dates but there are no null
values and our data which is very
unusual so where it's nice in this case
but most of the data I work with has
nulls the data is not formatted cleanly
cetera and I sort of print out
descriptive statistics but in this case
it doesn't really make sense because
listing ID is just a unique identifier
so it this doesn't really tell us
anything but if I had a bunch of
different numeric values I might want to
quickly have some descriptive statistics
on what my data looks like
so has everyone or I should rephrase it
as are people still unable to load their
data or run into errors thus far so
what's the error message saying maximum
[Music]
maximum range exceeded oh it's a call
stack issue hmm
so I'm gonna call for collective
knowledge from the audience for
debugging so with these things I would
either like Google this myself and look
at Stack Overflow all right ask someone
let's see as we're going through this
since this is a pretty important step I
feel like we should take like a minute
or two to and try to debug this so is
anyone like able to successfully run it
or has any ideas what might be going on
right now
okay so if you're doing this locally in
your own Jupiter environment I recommend
a Python 3 notebook and by or a Python 3
kernel then skip the Google section
completely sorry that should have been
something I said
so if you're not if you're not
implementing this in Google
collaboratory and you're implementing it
locally this section is completely
unnecessary you can just read in a CSV
file as you normally do cool okay so our
folks okay so far or folks still having
issues I believe this is developed in a
Python 3 kernel so if you can use Python
3 many Python 3 version ok
um cool so at this point we've read in
our packages we've read in our CSV file
these are pretty standard for small data
for like larger data sets our industry
like you'll have some custom database
connectors to a database you have setup
somewhere so this step will take a lot
more time usually meaning like setting
it up but in this case we're just
reading it in which is nice so we've
sort of seen our data so I thought what
what can I possibly answer with this
data usually I flip it the other way and
start with the question and go with the
answer but I flipped it this time so at
this point theoretically I had a
question like I want to say something
about Airbnb time series data I've seen
my date and it's quite limited so at
this point I'm saying I should probably
expand or reduce the scope of my project
and typically reduce it based on this
data
so this is really a good time to revisit
what the original question you are if
you have a client or if you have a team
member you're pitching this to you may
have had a really grand idea for your
project but looking at the data you
might have to scope it down to something
much more concrete
so given this data one thing that I
thought we could create that's very
simple is a daily count of reviews for
given listing IDs forgiven dates so like
how many listings are there on Airbnb at
a given date every day so some questions
I could try to answer is forecast the
future number of listing IDs or reviews
for the Los Angeles area
or I could forecast the future number of
reviews for a specific listing in the LA
area maybe these things are incredibly
important to Airbnb maybe they want to
know what their supply of listings or
reviews looks like in every market so
this could be an interesting question
that could be answered with time series
data and I think as we'll see very
quickly the time series data it looks
very different over time in terms of
what techniques or models can be used to
answer time series questions these are
just sort of a list of ones that I had
in my head that I think of when I think
of time series modeling
so I come from more of a statistical
background so I started with statistical
models so the first thing you could do
is just ignore the time series aspect
completely and do modeling using
traditional statistical models such as
regression based models and one reason
that might not be a great idea is you're
not really leveraging the time series
aspect there's inherent order and
structure in that data and you're just
ignoring that completely which in
modeling you want to leverage as much
information as possible and so really
the downside of using I'd say
traditional machine learning or
statistical models in this is you're
just not utilizing all of the data
available some of the very classic time
series modelling and statistics is very
simple univariate time series modelling
and that's what we're going to be going
through a lot today so things you've
probably heard of our averaging and
smoothing models
ARIMA models I'm sorry we're going to be
going through an example of one of those
in this example today some slight
modifications you can do to this are
there are packages and developers
working on adding external regressors to
univariate models creating multi variant
models so there's like several different
iterations you could do basically
anything you think of in a classic
statistical modeling exercise you can
think of trying to modify for a time
series question some other things that
will see our additive or component
models so Facebook came out with an
open-source package called profit last
year which does very easy out-of-the-box
time series modeling which is very
useful for getting a very quick estimate
of a future forecast and lastly for
statistical modeling I thought of our
structural time series modeling so some
things I've encountered in my work are
Bayesian structural time series
modelling where say you have to create
not just one forecast model but you have
to create sort of a series of forecast
models some of which are facing the cold
start problem meaning they have very
little data and some are very mature and
they have like much more data available
an example of this I had at Twitter is
like every time you enter a new market
or you have a new
product you have extremely little time
series data and so you think how can i
leverage previous markets or previous
products by using Bayesian priors to
infer and better have a robust forecast
for this model with very little data
also hierarchical time series modeling
sort of falls into this context
sometimes you may have an idea of say
like a global figure then maybe you're
interested in a regional figure or
something just for the LA area but the
data is not available or it's much
smaller so can we use Bayesian priors
that say the global level to use that to
infer that much more granular levels in
terms of machine learning models we can
go through sort of a similar set of
processes so you can ignore the time
series aspect completely and just use
any machine learning model you want
once again the same caveats apply that
you're not leveraging your signal as
much as you can in my opinion there's a
bunch of machine learning models I'm
sure you guys have heard of some of them
some of the ones I learned when I was in
school are like not as much the neural
network based ones but support vector
machines random forests gradient boosted
decision trees a lot of these are common
in industry and so they're not bad
places to start to be honest when you're
trying to implement a machine learning
model for like v-0 or v1 moving
typically the progression we've had with
a lot of our models at Pinterest which
are not time series models as we often
start off with a simple linear or
logistic regression model we then move
to sort of like a scalable distributed
gradient boosted justice decision trees
and then we finally move to neural
networks so you'll see and a lot of
industry examples a lot of them started
off sort of very simply and then
transition to more complex models as
they went on some of those reasons are
easier to implement and distribute model
training and deployment some are just
easier to implement and quicker to
implement and some was that just like
these packages weren't available several
years ago when the
company was starting some so you can
look at traditional time series models
some more time series specific ones are
things like hidden Markov models other
sequence based models Gaussian processes
is quite common and then the most common
neural network based model that's used
for time series analysis are recurrent
neural networks and I'll go into a very
like high-level overview of how you
applied neural networks to time series
data and sort of the last thing that I
think about before choosing a model is
sort of whether or not to incorporate
external data keep it univariate or
multivariate outlier detection and
removable missing value imputation just
sort of standard things I tend to go on
before I do modeling so I've talked for
a really long time so let's finally look
at some time series data so the first
thing I did and I always do is process
it and so I'm doing a bunch of things
here but the main thing I'm doing is
taking it from listing ID date to just a
count of all of the listing IDs for a
given date and so we have one nice
univariate time series that's much
smaller that we can work with there's a
bunch of other date and time things I
have to do here like change my date
column to date time I've decided as a
daily frequency a lot of these time
series models are on sort of daily
occurrences so if you want to implement
weekly monthly or even shorter time
series models if to think about are
these models set up for those and just
impute things so once I ran this you see
the data goes through September 8th of
this year this looks like there's
probably some data missing but we'll
deal with that for the 7th it looks like
there are 408 listing IDs for that given
date and so we have our time series
which we're going to work with for
modeling so next after I've looked at
the data briefly and cleaned it I'm
going to plot it and this is very simple
you can just plot it
similarly to how you do anything in
matplotlib so when I plot my data we'll
see that there does appear to be an
overall increasing trend there appear to
be some differences in variance over
time there may be some seasonality or
cycles in our data and I'm not sure
about outliers so if we look at this
this is not really a classic looking
time series what we like to see are like
really nice constant cycles of things
almost looking like a sinusoid of some
sort here we've got sort of almost
exponential growth the variance is sort
of changing wildly over time so there's
a lot of things that we're gonna have to
do to this data to make it appropriate
for time series modeling and
specifically that term is used often for
time series modeling is stationary and
so most of the time series models that
you'll see assumed that the underlying
data is stationary and this gives us
some nice statistical properties about
that allows us to use these models for
forecasting stationary basically means
that there's constant mean variance and
auto covariance doesn't depend on time
or more simply put if you're trying to
use past data to predict future events
we should assume that the data will
follow the same general time trends and
patterns as in the past this general
statement holds for most training data
and modeling tasks so a lot of the tech
companies where the underlying process
of generating the data changes every
year these assumptions don't hold true
and so even correcting for these things
we really sort of call into question
like is this truly representing our data
if we make our data and transform it
such that it's stationary there's some
good diagrams and explanations on
stationarity that I included that make
it a little more intuitive so sometimes
we need to make the data stationary and
in this case we do however like I
mentioned this transformation calls into
question whether this data truly is
stationary if we can really use past
data to predict future data if we
believe the underlying way our data is
generated has changed
in terms of some basic things that we
look at to say is this data stationary
you know or not one is using the
dickey-fuller test so I implemented a
simple version of it that I took from
here
so the dickey-fuller test the null
hypothesis is that it is not stationary
which is very confusing so we're trying
to reject that to say that it is
stationary so when I run just the
function I did to output the statistical
test itself as well as the like a graph
of what this looks like so we see for
our data the original is in blue the
rolling mean just sort of like the
average at each time point averaged over
some amount of time points increases
over time and that's in red and the
standard deviation is also increasing
over time so I would think that this
would mean that our data is not
stationary and if I look at the output
of the dickey-fuller test for the
p-value I get 0.3 this means that we
fail to reject the null hypothesis at
0.05 level which means that we can't say
that it is stationary so we need to do
some transformations or changes to our
data in order to make this data
stationary and appropriate for modeling
so some common ways to correct for
stationarity are to teach you'll hear d
trending & dicie's analyzing it meaning
that the mean will become constant over
time in the variance will become
constant over time
statistically Y is like what happens if
you don't do any of these things
variance of your models can be messed
specified the model fit can be worse and
once again you're not really leveraging
the time-dependent nature of the data so
it will just be a worse model fit which
I think like us were endeavoring to
create the best model possible ideally
we want to correct for these things and
I have some more on the pitfalls of like
implementing things without thinking of
the time series aspect in these links so
in terms of what we actually do to the
data so there are some things we can do
such a
transformations so oftentimes we'll take
the log of the square root or some
simple transformation to make our trends
more stationary we're gonna look at the
log we can do things like smoothing our
data to make it less cyclical and a
little more smooth so things like weekly
average monthly rolling averages we can
do differencing and differencing
actually makes a big difference in
looking at making a signal more stable
because we're not actually looking at
the time series itself we're actually
taking the difference between time T
plus 1 and time T and actually those
differences theoretically should be more
stable over time or you can do other
things like polynomial fitting or
decomposition and we'll see an example
of what decomposition means in terms of
transformation smoothing and
differencing there's like a lot of
different chance so this sort of takes a
lot of time and as we see this is going
to run for a while so yeah so what we're
going to see here is I created a data
frame which does a lot of these
different things so yeah actually I'll
just go down to the columns so things I
did for our time series data which is
just a date and then time series such as
TSS I took the log I took a moving
average of the log I did sort of all of
these different combinations and tested
them just see if they're stationary and
I'll let you guys sort of run it and go
through it later but some examples of
what it will look like are like for
instance here is like the original time
series and here's the logged version
which at least trend wise is not
changing as much and we'll sort of see
like the log version in the moving
average log version looks fairly similar
so we're not adding that much additional
stability to our data yeah
yeah unfortunately I think for
timeseriesforecasting there's a lot of
art in it unfortunately which means it's
not very like the expected outcome is
not the same every time what I tend to
do is I tend to run the dickey-fuller
test for each of these new time series
and see if that is stationary but yeah
unfortunately I in my experience have
just tried a lot of these things and
with intuition with my data maybe I can
narrow that space down to a finite set
of transformations if I've done this in
the past so if you're working with the
same time series data over and over
again
you'll tend to believe that I'm the same
transformations will yield similar
results
so I plotted a bunch of them and so what
I was specifically saying is I output
the dickey-fuller test for all of them
so I looked through these and I have
sort of like which time series it's
applying to and so I'll let you guys
look through all of these but at a high
level when I look through it
it's appeared for this data that log
transformation weekly movement moving
average smoothing and differencing all
made the data more stationary over time
based on the Dickey fuller test we would
claim that we reject the null hypothesis
and say that the data is stationary so I
believe for modeling going forward I
used one or more of these to model my
data and I claim that my data is
stationary so that the statistical
assumptions underlying these models are
satisfied
yeah yeah so for instance if we look at
the original one so the three things I
mentioned for stationarity are is there
a trendline
I can fit so there appears like there's
a trendline that's not constant meaning
not flat is the standard deviation
constant meaning is the sort of like
width we're seeing or the oscillations
over time staying constant it appears
that's not the case in the original time
series and I can't really see
autocorrelation here that I would like
do like an ACF or autocorrelation plot
whereas if we start to look at things
like like the logged version in the
moving average log version I would say
at least with the exception of some
outliers the variance appears like it is
becoming more constant meaning that the
fluctuations aren't as high the trend
doesn't appear to be so maybe I would
need to do a differencing so I'll keep
going through some examples to see if I
can show an example so here is the
logged version and the logged difft
version meaning I first took the log of
the time series data and then I did the
difference between the next time point
in the current time point and that looks
pretty stable over time this the trend
looks pretty flat with the exception of
some outliers the variance appears to be
flat so I would probably feel
comfortable using the version with
logging the data and taking the
differences I mean I would say that
indicates to me that they might be
stationary and I confirm that with the
dickey-fuller test sorry
does a runtime error influence the
graphs
yeah so let's see what this runtime
error says invalid value encountered and
subtract out array so typically if it's
a runtime warning it is a warning
ideally that means that like the code
was not implemented correctly and
there's probably a better way to do this
in this case a lot of the things that I
run into are like when you're slicing
data from a data frame using dot lok is
recommended so for most of my data i
transitioned it to using the location
rather than purely slicing and this has
issues with how it assigns copies of
data frames so this is just like not
good practice for this piece of data but
it should run still sufficiently does
anyone else have questions before we
keep going yeah I didn't talk about
missing data at all but if you think
about it almost all of your data you're
going to encounter has missing data so
in biostatistics there's topics of like
missing completely at random missing at
random not missing at random so if you
believe that the data that you are not
observing was missing completely at
random that means that you can infer
either a random value like a zero the
mean or whatnot or you can just drop it
and assume that your data is
representative of your whole set if you
think there's some bias in what's
missing and what's not for instance for
us if we're thinking about users like
data that we don't have from users we
assume are less engaged users and so
that's biased in the type of data we
observe and don't observe so there
aren't really great strategies I have I
would say I tend to think about sort of
like these situations which are do I
think my data is missing at random or do
I think it's not missing at random if
it's missing at random then you can look
to your current data and in for a leave
that the distribution of your data
represents it you can use like the
median or the
or if you believe it represents a zero
in this case like for instance if a date
was missing a data I would infrared zero
because I just assumed we had no
listings on that date if I believe that
it's biased then I have to think some
more because then we're not capturing
our data and I'm not sure what it should
represent yeah so that's a great point
so you could actually infer from points
before and after you could do rolling
means or rolling averages because often
time for instance for this data like if
you Sam then the mean might be more
heavily influenced by 2018 2019 2017
when really an impudent data mostly for
say 2011-2012 so it wouldn't really be
fair to just impute the overall mean so
you could impute it from surrounding
values often times we think that time
series has correlation especially in
near term so in near time they tend to
be quite correlated so you could impute
it from like the average between the
previous and next data point
yeah so there's a lot of questions about
say that I had extremely sparse data at
the beginning and I had increasing
frequency over time if you believe
that's a function of the fact that like
Airbnb gained popularity and was gaining
traffic that's biased and how your data
is being generated so I would say that
like when you're first setting up your
data pipelines it's extremely important
to ideally like regularly set it up and
not plan just for the next like one
month but the next one year or two years
because setting up these pipelines early
on will pay off in a really long term so
like a lot of long-running experiments
or things we try to do we try to set up
to sample as much as possible I would
say if you have the capacity over over
measure versus under measure but there's
not really a great way at that point I
would say that like when we typically
set of experiments we like to ensure
that we sample at similar intervals
otherwise there's bias and we can't
estimate what that biases does anyone
else have questions about anything that
talked about
yeah so from from the business
standpoint you do not want to because
they're like for example for Amazon
Black Friday and Christmas are the
biggest times of the year but from a
statistical modeling perspective I want
to fit to what essentially looks like
noise at the end I want my residuals to
look like noise and seasonality is is
not something that statistically we want
to fit to so ideally you want to strip
your signal of anything that does not
look either if you believe it's a linear
relationship or a non really linear
relationship like you want to try to
preserve that relationship as much as
possible
so an example I had at Twitter is a lot
of our data is highly seasonal at
Twitter so things like the holidays the
elections big events are hugely
impactful on our data so we would see
things like for instance these spikes
you see are huge in our data and that
could be huge have huge leverage or
influence on the overall fit of our
model so for instance if we had extreme
outliers on Black Friday that could
completely throw off the error metrics
for our model so we would typically have
to impute information for say holidays
or Black Friday because yeah yeah so
something you cannot in addition or
specific binary features just for that
date so you might have like a zero one
variable for a holiday or for something
so those are types of things that we
would add in but you can't do that for
every single seasonal event like Black
Friday through Christmas occurs for six
weeks or five weeks so we can't have
like a single binary feature for every
one of those days but yeah I agree that
seasonality is actually incredibly
important and if your overall goal is to
capture fully the outliers then we have
to think a little bit more about like
incorporating outlier features if your
goal is to just reliably estimate on
average how this signal performs you
ideally want to strip
as much signal as possible cool
any other questions okay I'll keep going
then so the last thing I'm gonna show is
decomposition for things like
seasonality trend and residuals so when
we say this like another way of instead
of doing all of these complicated
transformations there's a lot of readily
available packages and things that will
do D trending and DC's analyzing for you
I'm so I'll show one example of those
packages so here we have in the top left
our original time series and then we are
able to decompose it and this is really
good for just visual purposes of what
your time series data looks like is we
can see the trend in the top right we
can see what the seasonality looks like
on an annual basis in here so there is
some sort of annual seasonality to this
data and we're able to see what the
residuals look like in the bottom right
at the very end which looks like noise
which for us is actually really good
because that's much better to fit there
are still some outliers there that I
would probably want to investigate more
if I had more time and so I did the
Dickey fuller test once again on the
residuals for this and we were able to
reject the null hypothesis and conclude
that we believe it stationary data so
I've talked for a really long time and I
haven't gotten to modeling so we're
gonna do some modeling finally so we're
gonna start with ARIMA models so ARIMA
models are a very classic statistical
time series model
it stands for auto regressive integrated
moving average models and that really
can decompose 'iz into three parts so
the model is a function of three main
types of terms one is the number of auto
regressive terms and an auto regressive
term is just if you plot the time point
at the current time and the time point
at the previous time point what does it
look like and we sort of fit a line to
that and you can do that for like a
second order to auto regressive term etc
the integrated part is the number of
difference terms so I talked about
taking the difference of time T and time
say T minus 1 you can actually do this
several times over to make your data
more and more stationary and lastly is
the number of moving average terms and
this is actually looking at is there any
relationship in the remaining error the
remaining residuals and so these
arguments are usually coded as P D and Q
and a lot of ARIMA models you'll see and
these are ones that we mainly use to
control which ARIMA model we want to use
so how do we determine these P D and Q
parameters we use what's called Auto
correlation functions and partial auto
correlation functions so these are plots
that look at what is the correlation
between the current time in lagged times
previously and partial autocorrelation
is how much additional information is
provided by each successive lagged term
so there's some like sort of general
rules of thumb to determine P and Q
which is just the lag value where the P
ACF plot crosses the upper confidence
bound and for Q where the ACF chart does
the same and so what that looks like in
actuality is this is an example of what
a autocorrelation and a partial
autocorrelation plot could look like
here I'm using transform data so I'm
using the log to difft data and we see
here that if you follow sort of this
line we see it crosses the upper
confidence bound at 2 which is a little
hard to see and it doesn't really cross
it for the partial autocorrelation plot
so if we went back up to sort of like
our rules of thumb then we would say
that we might need a cue like a cue term
we might not need a P term where a cue
term is a moving average term and a P
term is an autoregressive term so what
we'll see this actually means for
fitting in a Rhema model is I just have
a simple stats model package which
imports a time series analysis
in a Rhema model and it will output
things like root-mean-square and
residual sum of squares although there's
lots of other error metrics we can use
so I ran three different examples so the
three different examples I use if we
refer back to the PDQ arguments are 1 0
0 0 0 1 and 1 0 1 I mean this is how
you'll sort of commonly see them written
is something like ARIMA and then in
parentheses its arguments I would expect
that the the second or the third one
should have better fit because our plots
indicated that our P ACF and a CF plots
indicated we probably need some moving
average terms and not some auto
regressive terms and so I sort of
plotted them out and they all have
relatively decent fit to our data which
looks sort of noisy if I look at just
say like the root mean square error I
would say that the second and third
model performed better so I would likely
proceed with one of those models but I
would probably keep iterating to see if
I could make this as low as possible by
trying out many other iterations of this
and you could write a function that just
simply runs through like all potential
combinations of these and just minimizes
on root mean square error or mean
absolute error or whatever error term
you're trying to minimize
sorry can you swim yes so if you'll look
specifically um so this is a little bit
of a tricky thing you can either apply
the differencing for the input data or
you can ply the differencing in the
ARIMA model itself so there's already a
dip of 1 technically in all of this data
because it's difference data right so
it's very rare that you'll need to
difference things twice meaning taking
difference of differences so yeah in
this case I didn't think it was
necessary to have another differencing
term yep
yeah yeah so some ways we thought about
how do you create I guess a way out
rephrase that is how can you create sort
of error bounds or confidence around
these estimates so ideally what you
would have is you would have your time
series data you would have a forecasted
line and then you would have error
bounds or like intervals around this so
we actually thought about that a lot
like how do you create error bounds on
time series data if you're using classic
statistical models that are like error
metrics you can potentially use other
ways we thought of using our
bootstrapping
so if you sample from your time series
data and you fit these models you can
get some point estimates for what the
value should be at a given point in time
and you can create an interval based on
that or like if like say it's like
similar for like if you just have a
normal distribution and you just have
like some variance that you have that
there there is comparable sort of
variance standard variance metrics you
could use for that for this model but
their variance might be missed specified
so it just depends on if you assume this
explains your data adequately so those
are two ways that I think about like a
distribution of values rather than just
a single one you can also think about it
that basically what this is doing is
each of these terms auto regressive and
moving average specifically is you're
fitting a linear regression to two axes
one is like value y at time T and value
at time t minus 1 and you're trying to
fit that linear regression for that term
and you're just having all of these Auto
regressive terms and then doing the same
thing for the error term lagged error
terms for a moving average so it's
fitting a bunch of linear regression
models essentially and we're trying to
combine these all
into one final output yeah I'll try to
explain that better next thing
some other time series models that we'll
get to look at that I like our Facebook
profit so this came out after I left
Twitter which is really unfortunate
because it's a super easy way to do time
series forecasting this uses a slightly
different statistical form of a time
series model which are component models
or additive models so this can allow you
to estimate day of week affects day of
year effects actually time of day
effects to holiday effects trend
trajectory and can do MCMC sampling
which is another way you can create
arrow bounce so this might take a while
to import everything but here I'm
importing hi Stan and then Facebook
profit which are necessary to run this
and what its going to actually create is
a really nice forecast that I'm gonna
just show below for correctness so this
is the actual original time series not I
didn't have to do any transformation
work or anything the dots are the actual
data I believe and then the blue line is
the forecast through at the end of 2018
so you'll see our data sort of stops
around September of 2018 and the blue
goes to the end and it applies some
standard 95 or 80 percent confidence
intervals around them in the shaded blue
so this is actually a really easy way to
visualize an output forecast for time
series data if you have a lot of I'd say
like analysts or business partners that
aren't as familiar with time series or
you just want to try it really easily
yourself I highly recommend starting
with Facebook profit because not only
does it output the time series but it
also outputs some nice graphs on the
trend component the yearly seasonal
component so we can see very clearly
effects by time of year we don't have
time of day effects in here because it's
daily data and then also day of week
effects and these are really nice
intuitive things that you can bring to
people and say we see some effects for
Sunday or we see
some effects for December so this is
actually a really nice model that I like
to use for simple forecasting needs and
so this is going to keep loading but
basically what I'm doing is I have a
bunch of inputs and you can customize
any of these you can change the trend
blend your estimating you can change the
confidence balance and how you sample
them a ton of things and I think one a
really valuable one that I don't include
is you can include priors
I don't include holidays but actually in
a lot of your data you'll have outliers
and you can actually fit those and it
will output something so that's actually
really nice to estimate the effect of a
single day or a single time series point
so I do all of this I'm going to keep
going let's see this guy fit so this guy
finish so I'm just gonna run a couple
more cells just so I can get through
these and I just want to see what my
error metrics look like so I also like
whenever you're evaluating your time
series data thinking about your
evaluation metric is actually pretty
important so here I printed out some
standard ones for continuous value
models but you also have to think about
like when training and testing like say
you train from like 2011 to 2014 and
you're predicting 2015 you have to think
about like if you truly randomly sample
your data and like 70% is for training
and 30% is for testing is that really
representative of your data are you
missing a lot of the time series
components that you need when you're
actually going to be deploying this in
production so typically what we do is we
do sort of like moving window testing so
we'll test from say like 2011 to 2014
and predict 2015 we'll add in 2015 in
predict 2016 it will continue doing this
to get a set of error metrics that we
can evaluate we tend to not just sort of
randomly sample from our data because
we're missing some essential time series
correlations
and our data so yeah thinking about
error metrics is another area I can talk
about and then lastly is I wish I have
more machine learning models to get for
you this is going to be the only one I
present so this one is called LS TM for
regression and so Alice TM is based on a
recurrent neural network model and it
stands for a long short term memory
networks so I'm actually going to go
quickly back to my slides because like I
was like how am I going to explain long
short-term memory
networks and so and so this is like a
huge simplification of it but I'm gonna
try to go through it so you see on the
graph on the bottom left
sort of a classic neural network you can
really think of it at a simple state as
you have a set of input nodes you have
some transformation and you have a set
of output notes and that transformation
is usually bounded between either 0 & 1
or negative 1 and 1 there's one or more
layers there's one or more nodes within
each layer and you apply just like a tan
H or like a softmax transformation to it
and that is passed through to the output
and through a nice check through a back
propagation algorithm we're able to
compute a minimum or some minimum for
optimization when you move to time
series we tend to think of a lot of
things more like sequence models so you
move from one to many one input to many
outputs so for instance here instead of
just having one input to one output you
have an output and that is actually used
as information for the next time series
point to predict it sort of keeps
happening in sequence so this is a very
simple example you could have for this
is you can have the input is the data at
time T and the output is that time T
plus 1 T plus 2 and T plus 3 so this is
what I did in some of my graduate work
is we actually only predicted say three
steps or five steps into the future if
you're trying to predict 100
into the future there's a lot more
uncertainty so for a lot of machine
learning work you can actually apply
like pretty simple machine learning
models but just change it such that your
output is a matrix of the estimate at
time T plus 1 T plus 2 T plus 3 etc so
that's sort of like the idea basis for
recurrent no neural networks and so
specifically for recurrent neural
networks we use the output from the
uncertainty in the output from each new
time point and we feed that in to the
time point after that so you would take
the estimate for y 0 that would inform
the estimate at Y 1 etc and this is the
recurrent nature of it is that you're
feeding information continuously into
each successive new step in your neural
network one issue with this as you can
see is it only depends on the exact
previous time point so we're not able to
incorporate longer term dependencies so
it's I'm giving like a very poor
explanation but for a long short term
memory networks basically you can expand
this to capture longer term dependencies
so oftentimes your data might not be
dependent just between time T and time T
plus 1 or the previous week or the
previous month maybe you have a 13 day
lag for some reason you need to capture
all of the dependencies there so I'm not
as much of an expert on recurrent neural
networks and long short term memory
networks but this is sort of a
high-level overview of like a very like
standard machine learning model
specifically for time series analysis so
if we go back to our slide there's like
a lot of stuff in how this long
short-term memory network is created
it's what it goes through sort of
classic steps you would think of for a
machine learning model create the data
set fix a random seed just for this
example get the data you normalize it
with some standard scalar
you split it into training and testing
you like transform the data and then you
fit in LS TM Network and you can specify
the number of layers the number of nodes
etc this is just one example and then
we're going to take a time step every
time and see what this what the estimate
is after doing this for like five times
steps or hundred time steps or a
thousand time steps that's a good
question
I'd assume that if you transform the
data it would likely converge to
whatever its minimum is more quickly but
that's like a sort of guess if you
didn't I can't well either way I can't
guarantee it would converge to the
actual minimum of this if you ran this
for a million time steps maybe it would
in either case I'm not sure about
stationarity assumptions under machine
learning models I would personally apply
them as well because it's very easy to
reverse engineer a log and a difference
it's much harder to say like this is not
going to be sufficient to model like
future changes because the data is not
stationary so I would personally
transform the data first but I don't
know specifically for this machine
learning model how it handles because it
applies a lot of nonlinear
transformations to it so theoretically
maybe the log aspect would be captured
but I can't say so I only ran this for
five epics because it takes a really
long time and ideally there's like some
nice mathematical properties that would
guarantee convergence if I ran this for
long enough but for this I ran it for
five time steps for a couple of
different versions of LST MS here and so
I'll just show what a couple of those
look like this also nice thing about
collaboratory and I swear
not being sponsored to say this is
there's like tensorflow backends and
things you can do so if you want to
implement sort of like any tensor flow
models this is like a decent way to do
it so we see that it ran for five epochs
and it sort of outputs some root mean
square error on the training and test
data and we can sort of see what those
each look like visually and honestly
like at the end of the day I tend to
just look at my data and say like do I
think this is a decent fit and will
estimate my data accurately in the
future and it seems like it's a decent
fit I mean there's a lot of nonlinear
complexities that are incorporated in
this neural network so it seems to be
doing a decent job I would personally
compare this probably at the end to what
I originally created so I would probably
create some ARIMA models I would create
some component models I'd create a
couple different machine learning models
and compare they're the same error
metrics across all of them and see how
they do and see how they do across the
varying windows of future forecasted
data before I would choose my final
model so I did this for a bunch of them
some fit better or fit worse but overall
I think that I just threw like a ton of
information at you there's a lot of
really interesting things about time
series data I would say overall I would
just say like think really carefully
about whether time series data is going
to help you in your quest I would say
that's like try to think about like is
the data that you're generating like
biased in any way and how you're
capturing time series points and like
how can I revert that bias when I'm
modeling I'd say that not just for time
series data for all modeling in general
and I would say before you jump to sort
of neural network based time series
models I would encourage you to look at
statistical time series models as well
because you'd be surprised like easy you
know variate the univariate models is
just you take a time series and you fit
it with a statistical model and it works
decently well and a lot of companies
will sort of default to that
before sort of moving on to neural
network based models one for intuition
but to because it works so yeah I would
encourage you to explore sort of a suite
of models and a suite of training and
testing sets before you sort of choose
your final model that's all I had for
this tutorial I'd be happy to take more
questions or to debug anything with
anyone and thanks for listening to me
[Applause]
yes so yes so I actually come from a
biostatistics background where you can
only sample things whenever patients
come in so that's only every three to
six months and you're typically only
predicting like one metric at five years
like incidents of cancer at five years
or something like that
so in those cases we would actually
treat it as categorical features or just
one continuous feature and we would say
where's our outcome is this rate or this
feature at X years so you can actually
frame a lot of your statistical machine
learning problems by just stating the
time then you select the final outcome
also I would say for time series data
you need at least four cycles of data to
predict the next cycle so a big issue we
had at Twitter was we were trying to
predict one the next year of revenue and
we only had about four to five years of
revenue data at that point so it was
very the underlying data essentially
changed like the underlying product
changed over time and the way the data
was generated so we didn't really have a
stable time series so I actually called
into question like whether even doing
these transformations was really
accurate for our models because I would
say like the product as we had it now
only existed for like a year so if
you're trying to predict anything
like for a lot of time points in the
future you need a lot of historical data
so this works generally well for like a
lot of government based data so
agriculture finance things that are
relatively cyclical over time for things
that are a little faster paced it's not
always the right way to go
ha how come the training scores a lot
smaller than the testing score so um if
you think about it in your training data
you're trying to minimize something like
the root mean square error and your
testing set is generally gonna perform
more poorly so in this case I'm not
surprised that the training root mean
square is much smaller than the testing
and you'll find as you implement models
into practice oftentimes your testing
data is much work performs much worse
and so you're trying to minimize the
difference ideally between your trust
testing and your training difference
exactly yeah so if you think about as
your variance increases over time you're
also going to increase your root mean
square error if you don't account for
the deep trending DC's analyzing
reducing the variance because your room
mean square error is going to explode if
you're trying to estimate something
where the variance increases over time
so I shouldn't probably talk about all
of this but like a lot of the features
that they have already built into their
package are things that like every data
science forecasting team has tried to
custom build in our a Python so
specifically the things that I would
call out for Facebook profit are
actually I'm going to just show these
graphs are these so most packages will
do D trending and sort of DC's analyzing
what I think is valuable are a couple
ones one is you can implement holidays
so most of us had to custom implement
additional binary features like we had
this additional data frame of binary
holiday features that we had to add and
this will do it for you I would also add
that you can trying to create confidence
bounds we had to do a lot of back
testing sampling methods and their
sampling methods already built into
Facebook profit to create these they're
sort of shaded but these error bounds
like to create create true confidence
bounds we try to use sampling methods
and that's already built in here so I'd
say those are like two of the main
benefits I've seen
yeah yeah so what I did and my research
is I actually did basically a machine
learning model that predicted something
like on the right and so all it was was
the features were various features at
different time points so you could have
like a thousand features that were
feature one at time point t minus one
feature at time point t minus 2 etc and
you can do that same thing for all of
the features and then you can just fit
it if you have the data to like a single
output which is like like at time T plus
1 or you can do it for like an aggregate
something so I just tend to think of it
as you're like fitting some input
features which can just be the same
feature sampled at different time points
and then output features which is like a
time point in the future so you can sort
of think of it I tend to think of it in
sort of like a regression based manner
that way where you have inputs and
outputs and you can feed in features
that are the exact same thing just
measured at different time points in
terms of the specific models I went over
there specific packages or ways some
packages have external regressor
capability some don't so I would just
look into which ones doing which ones
don't to implement them cool does anyone
else have any other questions ok if not
thank you so much for all of your time
have a good time at PI data
[Applause]