so good morning everyone and today is the second day of our
workshop and we're going to focus on the stairs and afternoon we're going to be a
metal analyst so meta bandits the melee is a kind of a statistical and
functional analysis and a lot of the statistical concepts as viewed on this
morning's lecture so if you have questions not quite sure what what about
you raise your hand and I would like to go to details in the past eight years is that sometimes we go to very low
sometimes very high so we try to adjust at the end just mentioned we were not
going to go to very details and keep a standard distribution so we slightly
goes more machine learning and hopefully we'll be better received at it this year
and so yesterday we mainly focus on is a analytical part and a different
instrument and eclectic you know structure and use different databases
try to identify and quantify them and from that we get a list of metabolites
or a table we have lives with their concentrations if you're doing on target
you probably just get her a list of the peak and identified by their retention time of the mass and the peak intensity
which is usually the standard output from xcms online and today we actually
want to show you how was the method to actually go from this peak list from
this compounded collagen is able to go to significant features pathways and
functions and this is a overall kind of concept inside here we can use that like
a multivariate statistics on the top left on the top right is basically
camera magic specl POS da and lower bottom one shows that our C curve
basically help you to identify robust biomarkers and performance
evaluation and then back end is a Joseph or the CAC pathways basically come out there and do analysis on there
and a visual exploit and this all couldn't be covered today and
metabolomics basically is a relatively new field and it's a follow all this
kind of standard or approved practice from Atomics field made mainly from like
microarray or transcriptomics because this meta box and you can think about
metabolite connotation is like a gene expression
it's a their high and a low corresponding to a lot of the same
biological regulated by same biological process actually when we go to analysis
we can find a lot of things that's common but there's also some unique
method that's more specific to metabolomics and while we don't talk
about seed has spent much time on tea set tests we need to talk about a
different approach are the big ideas behind it p-values and how do we do some tests even the distribution is not
normally this to be it's not normal and this is very important concept you know
a lot of the complex state where we don't know their distribution and the
other ones very comments actually this is very specific for metabolomic
diseases and multivariate test sets like pca of usda and this method is not
usually used that wide in gene expression data analysis and the
slightly more expanded a more module today for this year is that machine
learning concepts basically how do we tell which one is doing good jobs and
what's the pitfalls and what his class for him so so this is a lot of people
actually asking what is statistics how it was different from was statistics
versus power in marriage or she's reversing machine learning versus
artificial intelligence is required quite a fattie concept and there's a lot
of progress in recent years and and disregarding how people think at a
similar or dissimilar the mainstream actually there's some people in the
practice I can for them as machine learning about HoMedics ah stairs there's some overlap you can see it here
and the machine learning is more like from computing science and the
statistical is more concerned with hypothesis testing and parameter
estimation is more traditional but now they are more merging into data science
and machine learning as test became more fuzzy in between and there's common
things we can see it during the class trained in classification and regression
analysis so a lot of stats using machine learning to evaluate her performance and
machine linear the stats to estimate parameters so it's a it's became more
blurred it's a good thing and the other thing I'm not going too hard talking is
artificial intelligence which is consumed this deep learning and this
very hot area and this you should attend a different workshop if you were
interested in that so again if you have questions you can raise your hand okay
and here is a general workflow in omics analysis so this is a issue that
gradually build a workflow or concept in your mind disregarding metabolomics or
in the future you'll probably go into the iron psych analysis the same thing
same high level steps so honestly basically we need to get the countable
and metabolomics you need to get a really future table it's vegetable could
we pick college a peacock among common
connotations and up you get your date has what you need to do first step is a
quality check that the data looks normal is that something fishy or outlier stuff
you need to really people spend the amount is analyzing your data you need
to really make sure the date has seems okay and it deserve your time so how do
you do that has just no just the one bathroom anterior data is okay
is that you need to consider the experiment design and visualize your
data and use some common approaches like blackbox flawed overall and there's a
scan plot is different you may hate Maps the different things help you samer Islander it looks added data from
different perspectives to to make sure there's they are okay of course you have
more experience this process can be shorter you can just spend one hour and
the thing is they stayed in fine for other people start learning you promised remote home and if they did fine you
need to kneel I need to normalize the data no more because omics data is
really in multi-dimensional means that features some like some metabolites
collagen who very high some is very very low and if you direct compare them as
became very hard not comparable and the small changes in like glucose can have
like a hundred zones and macro molar but in others more low concentration
metabolize it's just a few very low level of parents so if that we are not
the doesn't mean they are less important so we need to really normalize them so
the comparison will be more or less equal and you can really found more
significant less virus directly compared with disregarding the concentrations of
variances usually it's not that powerful yes you can think about it's more like
on parametric tests so unless your data really really robust have very strong
signals and you probably needs too much so data normalization is very very
important and the last one is statistical and machine learning so if
you really your data good quality they'd had probably normalized and then you can
really explore all the possible approaches from you will usually start
from simple and see if there's some interesting patterns then credit coach more advanced you don't go to the other
way if you go to more otherwise go to simple this is usually our mind I don't
work this we gradually build up hypotheses see if it's confirmed in
another method and this is a style from your comfortable method gradually expand
and learn it's so keeping your mind input-output this is also a today for
lab is that it's not a spec sure even it could be about a spectral processing is
very specialized a task we need to respect what other tool like mg ma XMS
and they're doing a very good job and if you use a target that you really
manually do it it's a spike and dope here and quantify so that's a lot of
work and I don't think automatic you can give you really good quality input data
so the data is supposed to be like an Excel spreadsheet like a matrix and it's
with the numerical value assets like concentrations and picking intensities
and the other things yesterday realizes as metadata basically we want to know
the class labels especially want to do some a coloring I want to controls in
red and the disease in the blue or whatever color the program need to know
there's a different label associate with that so you need to give the metadata so
they can do t-test also on these two groups so indicating it was your samples
of interest the odd food is basically the first one set was significant like
what signal features and compounds the other one is that sometimes you even
don't see significant that but you see overall interesting patterns so you can
see the PCA or hit map you see some patterns so t-test or significant
casting is not the final it's actually one part
the other ones are clustering also very important you mean you don't have
significant genes you can still see good patterns and for you to give up your
hypothesis the other ones rules if you really use classification
some time you can see which compound associate with these outcomes and you
can develop some hypothesis and models so we these models if you doing that's
like a regression or logistic regression models and you can actually see how they interact so this is a actually became
less commonly used in omics because the models became more and more complex
and it is hard to develop some gut feelings about this model so the models
like the one most commonly move all forensics like SVM random forests is usually a black box and you have to
treat them and they pick just them and it's hard to interpret internal working
on that model but they are still very valid robust models okay so for the data
types I hope most of you are very familiar so we have x and a y basically
when we talk about the data is your X and your Dana Lane labor wise metadata
and so food for our data we mainly is a continuous data because this
concentration and for our instinctive you it's count count is the integral it
is a discrete so you just to think about it its stead street and very different
discrete and continuous it's very unit different models so regarding to why
basically at the metadata mostly robust the easier part is finally basically yes
or no one or zero disease control and the other one is just the ABC or nominal
the other part is called Audino Audino it's a last one it is this Audino means
some orders meaningful so this one is like time series those who responds so
some time is it 10 a 20 or 100 so there's some orders inside when you do
analysis you actually want this order to be recognized to be used in modeling this is a less common when when I
started like 10 years ago now people are talking about personal I'm a
personalized medicine or precision medicine you want to collect data based
on their for each person follow different days so time point or the dose
became more important so so this is not a key box today about her in some method
in Mattapan is actually starting respecting order so I'll go into a demo
I'll just show you some of the parties that we need to be careful
which one to select and here is that I already said about this so different is
create and continuously different and you can see this is a continuous because
you have fractions you see this account concentrations and this is discrete
because it's all integers its account how many times you see these jeans all
this reads been occurred in your sequence II so this is if you talk about stats and the two very general saying
this at normal distribution this is a discontinuous Poisson distribution is
usually for this account so this is a different test group instance and again just covered is just keep in mind
there's such settings and the instead actually it's different treated
differently and some of course some of the master they can consider order that
a lot of method like I know but they they probably don't don't take even you
give that the problem actually cannot handle it and this is a PhD a actually
that in the reason that materialist do you have this kind of class all the
matters class hold it does not matter basically is going to be this going to already know this is a if you have a
time series and you really want respect order went was before and you should
keep it change changed if you don't this is going to be a Chilean laser as a ABCD
there's no cutting will order the same way this is that people from users says
that if this change label their group had is different it this is if
especially if you talk about this is ordered if you use orders and to change
the label and it's you can switch your order so the result will look different
but if you switch unit this one's just all the data method you know you're
still keep the same so this is something when we do our lab we all discussed
about these options and the listen and this is about common terms I know a
lot of you already know and we just quickly go to univariate is that when
variable or one feature and at a time basically we talk about alanine across
all the samples as Ln a unique just one variable variables we can talk about
alanine versus creating in this is two variables multi variable is more than two okay
a dimension of date has a physical number of the variables so gene
expression for human is 22,000 if and for the metabolites really get to
thousands or hundreds hundreds or thousands of dimensions so it's much
lower dimension at the moment and we already talked about how to read in data
and stuff and if I guess actually that's the most ajikko part but once you
understand what the input required whether or labels it looking for you you
should be able to quickly use your Excel to prepare such things and the next one
we are going to visualize your data and whether how do you visualize your data and some time you see it's very simple
and but it's very helpful and there is no magic bullet in visualizing
multi-omics data and give you a whole picture so you have to try and study from basics instead we can all see all
the data we can look at a summary update what's the center basically the mean and
the median and mode the other ones how they spreading out and so the the normal
distribution the best we have a lot of models so we can summarize with just a
simple value like me or standard deviation this one but when you have the data distributed like a uniform it's
very hard because you don't see a strong convergence or anywhere you have to
actually treat all of them so you if you summarize it as a mean or media it's not an informative because you're not they
are not standard atomy even you can calculate to me and this is extreme distribution and it's so it's really
interesting to see your data and I choose
proper approach to summarizing some of them is not even summer risible for data
like this even doing PCA you you can see everywhere and no patterns okay and
Redhill standing basically distribution you can see the quantized range and IPRs
interquartile range basically I'm going to here and the box plot so I get emails
people asking box plot I'm just sending email in do google it because it stops and I can explain through my email about
the cuoco give you much better Wikipedia but any will we go through it you you
assume this is a read we normally distributed it's actually very nice you can't see sin is the median this is the
interquartile range basically tdpy rank all the connotation from the high to low
and you can divide this top twenty five and twenty four seventy five to fifteen
and fifteen to twenty five and twenty five to zero and this is basically the
focus and assumed majority information is there and they focus on that a lot of
time we just use neo median to represent how they had to do the modeling because
we cannot take all of them so if your data have followed this nice
distribution and it's very a lot of might have a lot of stats going to
working well but if you're just like previous if you really disabilities
everywhere then that's going to be hard as and we we allow time we just use
average to as a representing like group average we don't look at each individual samples we use group average for the
connotations or use we ignore the various that the variance is very very
important and we assume they have the mole as
equal various but a lot of it could not be true and it's it's very hard so the
smaller variance it's a much better you can see that even the same same mean but
if the variance is smaller you can see you have more confidence these two
different it's much significant because they don't overlap but you you see that
same distance between mean by the wear resistance higher and you can see that
overlap is a lot so variance is also very important so I hope by this
time you should I really have the concept that because we cannot consider
all the data points simultaneously we use summary stats basically means to do
the calculation so we really choose ascending here so if so if your data is
realistic sense and around me it's very dense here and we really choose a very good data point or value
to represent data so if your date has spread in very large like this spreading
everywhere you still choose a center point like user mean to represent it
it's not going to working well because just because you can see overlap is a
lot and you just use one value and this is part we are going to not working well
but if you like this you choose one value reps and whole thing it's going
separating very well and it's very representative of your date okay how
ideas summarize your data you one or two values and use that when to do the
computing okay so normal distribution and then we're
going to quickly go through and a lot of why we're interest in that because it's
well accepted actually in most biological physical measurement they
seems all normally distributed and some of the already figured that other
complex stats behind it and the computing actually doing normal
distribution so fast and anything else became much slower to compare to normal
so so because of this convenience we really wanted our day to be normally
distributed okay most time idiots and sometime it is not and so we need to do
some tricks to get hit normal and so this is this is a scary things and I hope most of you already quite familiar
you see this one but you don't need to remember because it's all beauty in a
computer they take care of that and so a lot of times we see actually
metabolomics we see data looks like this okay this is not normal that is closed so we can apply like a log or stab it
speaking very normal and this is the last comment and if you have a
population actually very divergent and sometimes you see this and
it's it's hard to normalize this type of data and and this this one also quite a
common scoot the metabolomics Roman tableau makes Dana some time you can see
this site is all low and high speaking less common so data normalization is how
ideas set because all the statistical model work on a summary summary number
okay that summary number it's like most like a mean or average so which works
best if it's normally distributed if it's any other things were or not working well so just for this reason we
need to normalize the data so the stats we are working fast okay if yeah this is
the main motivation stands working faster means that p value is more
accurate and the conclusion is more robust okay we have to please the
statistical method and respect what their rules and so here's a skewed
distribution and we just a is exponential and we do analog
transformation and now speaking perfect normal so it it will be easy and yes are
you talking about within that also metropolis yeah oh you upload the data you have your copy
called original data and they were filled her and data as seconds to have
got the process the data let's go to a normalized date so you have three copies
in bed to interior every step usually have a copy but in that going to create a lot of the save a lot of data in
copious day and you are going to confuse so every step actually been recorded so
you have a normalized data originally it had process data yep
yes have named by that and also the each steps words parameter you chose to normalize the data will be there and
you're going to have an analyst report and tell you exactly what you have done
so this is real data from metabolomics and you can see it's a skewed
distribution a lot of them is very close to zero and AB normalization you can see
they shifted to the right and more normally distributed so one thing I kept in mind is that normal apply log and
you're going to exaggerate some very small values so a small value means that
like clothes with zero if they're low close to zero actually is low quality you're actually you're actually it's
Odyssey you know enlarge the noise okay pay attention
log is log transformation is not a magic since if you're all your data is a
high-quality and you can apply apply log it's no problem but if you're really
actually close to zero this kind of thing listen it's a low quality and
you're not quite sure apply log you give them more weight compared to high ones and this could be the side effect okay
this log is not better good English log is that log is easy to kind of
understand and there's more fancy approaches and this is a published 2006
but it's cited a widely and why is because it is important but it's also
boring people people don't research on it a lot so this person actually tried
the best and you've added all the passports and gave some guidelines
there's no golden rules so you happy today subdue that rather it do and give you some ideas about what's the
advantage or what's limitations of certain certain approaches so the title
is centering scaling and transformation and all you can read this paper okay
it's very educational but I'm not going through details
or should I just you I don't suggest you to college in separate steps yeah it's
all married in one go and you should do it and a lot of times you are you know
your equipment have certainty tax and limit about that and it just working
well it's not because your highlights audience you're gonna marry the more accurately than 100 it's all working
fine if you became that point upon a zero like micromolar and then became
this once about certain threshold and machine became accurate you should trust
that you shouldn't discriminate because high and a low I think it's artificial
you created some divisions and they're causing chavos future we already doing
somehow it all makes the integration you have to talk about you won't win in
metabolomic so you want to divide it divide that it's a method of wise I
really think it's introduced more work and regarding per ad who is that what is
a person of whoever he is a group probably some relation he's slightly
advocated for this approach with amitabh Anna is of course you had the Pareto
scaling we actually first went to a put it there but in reality is that I don't find it a you much better I would
advocated just that you use if your data and like doing targeted metabolomics all
the memory the same fun just doing lock it's easier to do it to try it compare
log with parental see what's different so this I don't strongly advocated an enemy that including Perret or whatever
probably the quotient normalization but I do suggest is that your data quality
need to be double-checked is good the second one started from simple ones
simple easy to interpret and a series out the patents whether it's a fit some
of your biological your gut feelings and then gradually
increase the complexity don't just add precondition I only use this because
this paper saying this your data is different it's really case-by-case okay
yes here actually shows all this method there and the potential we can add in
three or five more are based on user requests and but enough enough and we
will probably add ink rather adding one or more why not
so the data normalization is first is a like a sample normalizing about biological normalization because you can
have the different X values and for example dilution effect in your sample
like urine samples and you have tissues or the volumes difference so you can
normalize by the some old meeting that's per sample specific and you can do a
transformation like a log R cubed root somebody's just a really strong set it's
very good so I put it there I personally and I be honest I never use this I only
try log and like below this one also looking fonts correct or auto-scaling
means centering so and of course and the people you can do it here like that
normalized by the media normalized by the reference sample if you really you're you assemble is some it's more
like a reference standard and you use that it's going to change your result
dramatically medium also the same thing so so the first one biological sayings
it really is based on your design and what do you think and it could have a
very dramatic effect on your result okay we can discuss later if you have an
interesting result on that if you you assembled all good quality and no
particular reasons you just use it like just the first one actually try
suggestions use log only don't even use scaling because the much less procedures
applied you much easy to go oh this concentration means that like this
you're not going back think about realities once the apply steps are
changed our calculations on your data you cannot just walking back and think
about oh this actually increases and cause this one it's become so
complicated so starting from simple and gradually increase the more steps
yes good question if you are they actually truly pay more and what you
have done right because we assume all the samples from the same population
which is the same sample this our assumption if you really you can see
that's two population your data the best ways model separated it with it
separated in there in a case actually you can upload their data labeling with
different like gender has since and to see whether they agendas okay causes some issues then you should if it's real
agenda I have a different distribution you should analyze male was an adding a
female separately this is really they're talking about two populations and stats
is always talking about one population if you don't compare merge two problem
together right I don't I don't think you should try your best to in step I'm
although I try to normalize in one model it's it's it's just you hide the truth
and chooses two populations and two models and doing a separate the only
downtime is to have enough replicates and it's a figure male family or do they have that you call a relatively balanced
the design of treatment control you can do that analysis if you have very small
samples you just have no confidence right so good question and now we're going through the last
step is this and stats and machine learning and probably is going to cause
a lot of death mountain and the first two step is that it's relatively simple
but it it's very important okay if we're not doing it properly you're going to
have troubles in here so whatever conclusion probably not at the robust so understand p-values so everybody
think they understand and I think every time I'm yeah yeah you do have pillow
there's going to graphic odd ask its allow you to see it I just saw options
and when we do her lap you'll see that yeah basically visually you can actually
explore whether it seems a better okay and the better be careful yeah yeah that's a good call
how much rest you gave to it if there's a lot like a 35% I'm just from it's all
below detection and you can try to I don't know replace with a very small
smallest value if it's really to be though to coast with every more than
half and it just remove it I'm thinking about him that's you couldn't do too much so you can use a unit have analysts
and we usually replace by half of the lowest value so your attic and lemon is
like a point oh five and then that the missing value or you just don't detect
it I gave the point zero zero or something and so you can still calculate
but doesn't really discard your day
so stats and p-values so what I'm saying that p-value is important actually
p-value is just every time I'm reading it how so I feel I'm learning deeper so
I think just let's go through it together and see what p-value actually about so here's the population and the
way stats is all about population so you assemble it and come up with some
summary like me and standard deviation and you want me infer the whole
population probably behave like this this is a house that's try to to model
population so it's really based on your sample if your sample is biased and you
probably not going to have a very accurate estimation and here we assume
that this is the whole distribution and you Sam over here this area and now you
get this mean and standard deviation like this in small reflect an overall population but just to somehow for some
reason you someone in this area and you could get me a slightly shifted here
it's not go to population and where is very large so you're not really you know
of course you have these estimations but it's it's not as reflected very close at
a population so and we talked about sampling bias so in this case you have
actually bias and the comment is not that high but still if this is your data
you have no clue about Howie your estimation close to a population okay
this is a traditional and how we married is uncertain is and there's p-values so
this is how it comes so we don't know how what's population and we don't know
what's the truth and and instead we use p-value to indicate our levels of certainty that our result represent
January in fact in the whole population okay so p-value is the probability
observed result was I was obtained by chance so if you see such a result and
just a random there's no real effect what's the chance you can see such
things you have your model and that you can compute in what's the chance is that
for normal distribution actually a lot of people can't do the calculate in your
mind I couldn't but because I'm using are I can always get a scar to tell me
so for example 0.05 this all normal diffusion if it's chance it very low we
think it's it's not random actually a real effect so we reject the null hypothesis so if it the p-values
slightly the chance is slightly high you think oh could be random we are not
quite sure so you don't not reject that null hypothesis it's not saying that
they they it's not if the p-value is not a significant it's not it's just that you don't reject and now hypothesis it's
not saying your treatment don't have effect if you increase the sample size
I'll have a bad experiment designs and probably you will be able to detect the
effect size so what is the p-value tell you it can don't have sufficient
evidence okay this is it's not a fatty serum making and that you really think
need to think more on that calculating p-values it's actually easy for a normal
distribution and you can see if your observations within certain range and
that you can see that what's the probabilities of course your if your
thought this is the population your model and what you observe is a file
here you can be very significant no problem and the you can see if you are a
lot of your upper division here you can see actually it's totally different population and this is is population
this is become bimodal almost if you have a lot of reservation there so it's
you're going to have very very significant and yeah and we talk about the ideal word is normally distributed
and in a lot cases we don't know and actual review don't don't don't think
it's normally disputed so how do you convince the reviewers say hey this is a
very robust accan and we had here a very significant a base normal distribution and it's hard
to believe that's a whole omics they are very complex like say
metabolomics target untargeted different platforms and normally disputed most time if you view it it's not and how do
you do that very robustly is calculate empirical p-values empirical p-value and
simply compare in fair comment and nowadays in a big data because when
we're talking big data it's very complex and the most time it's not normal and we
have powerful computing we can actually be noble calculate P values okay just be
comfortable with it and try to understand as much as possible because
it's going to be became very common so it's used that we don't know that
distribution and we don't believe it's normally disputed and how do we found
and we need to compute the now distribution basically there's no each
factor what it is random and how how many times we observe such things okay
this is the whole thing and basic person poses on that now hypothesis basically
no effect okay are you basically all your class labeled an mean anything okay
you can you can think about your male female or treatment on I'll have of
control done making it cause entity defect in this case all the samples mix
that you randomized now your calculated what you're going to
observe like mean of main difference of like a fold change and basically if that
you can do it mounted multi-time like a million times and you compare with the
one with original it with your class label and the original design because
you believe there's an effect how many times that you factor observed from the
randomly shot of the data is the most significant yours original data okay if if that time happens quite often and you
cannot believe that the in fact in your data you observe the real is caused by
that because they're in random can kind of get a better result right so but if
if the random shopper never joining the data as good as yours
you mean 1 million times that random permutations and you should be very very
competent so it was your p-value p-value is less than 1 out of 1 million times
right this is called MP a few values so empirical p-value is
never going to be zero because you in that you have to determine using
infinity you cannot do it so I went on The Times never get a better use less
and one saw one out of 1,000 okay so I guess the principle should be very clear
because we have powerful computing we just cannot we just cannot we just can't
do a simulation of permutation and if you write a code and I'm just going
through this if you want to write code this is how how you're going to do about
the most time this kind of beauty in what have honest a lot of the tools so
how can I do it use a very simple with simple examples that we have just one a
student has to school okay I have one group let's think about
student when Google case one group the control okay we are not talking about a
blah mix all we can think about just one pair compound connotation in this one is
like this and this one like this the main difference is point five for what
so it is significant or not you're not sure of course and how do you do that we
assume there is no difference if you're no difference
we can't shuffle them so so we can sell for you if you compare to previous to
the next you can say one two and nine is all belong to case and the 10 to 18
billion control this is original okay let's assume it does all this labor that
make sense now with this shovel put it in a case randomly draw from the overall 20 okay you can pick a the nine three
summer still the same case some are already shop out from the other one so
this is the same thing and we do the difference disappoints way to now okay
it's and it you can just shuffle them just stuff for them and you repeat the
winter on Tom's right and you can see they say you originally I preserve the
difference and this is all your shop or the data if they're a case and control
label the original done minute done them doesn't mean anything there's no effect
and this all is all here now at this time you should be very confident that
there's a difference between your case and control right
and if there are three times the premier tea today it has given largely difference I'm here either going to
point out sweet so out of went on The Times there's three times it's better so
there's the p-values on all three so if you never so it's P less than 0.03 but
it's not equal to zero sometimes people adding one of those
stuff just this is all fine so it's all standard so in the past that is seeming
to be very challenge for people to understand but if you don't I I'm happy to try my best to go that was from the
other ways or you think it's very easy and that that's good okay so general
advantages that does not rely on any distribution assumptions so it's not
necessary to me to be normal and the cracks for hidden correlation because a
lot of time your data have some certain hidden structures we don't know and it's
very complex if you do in a prominent criminal agent within the same data and
the thing is still capped so it randomization will also addressing that
so it seems so quiet its robust disadvantage commutation intensive it's
best for large data by the on the other hand how many possible permutation you can do i mean if you have only three and
a three and how many population permitting you can do I'm not good at
math probably 20 or thirteen times you cannot get a very signal here I'll you
because you're gonna exhaust quickly and so I for a good permutation like once
around the times I think at least 12 12 or 10 10 at least think about that so
it's just a permutation is a randomly draw if if you have very small sample if
I randomize Joe well quickly repeaters some cases you that did before so here
is another important concept and moving to another and okay we already went over
the p-values and empirical values that's a very important concept I hope you guys
accept and the second one is the multi testing a issues so for most of you probably is
already common and so I'm going to go over there quick quickly yes Oh because
it's not powerful if you if you really for more follow a normal distribution
use p-values you can get use it in normal distribution the period are going
to significant if you're doing the same thing do it it's it's not as chicken even at that permutation also cause some
also has some bear variations sometime you doing this permutation all the times
you don't get the same result so it's a need to be yeah we don't have a
universal a bad method to replace a normal distribution okay that's
hypothesis testing so so and this is already we covered briefly on the
p-value calculation so I like the result if we're going to get it from the sample
if null hypothesis is true okay this is a p-value calculation so this is
standard approach from very conventional statistics and we already covered that
and we're doing this you know to that we need to compute the p-values and we have
some arbitrary cut-offs like 0.05 or no one if we normally distributed and issue
here is that all these points based on the univariate basically one variable at
a time and if we're willing to accept like there's a small chance we see this
random per variable but metabolomics our gene expression we have hundreds or
thousands of the variables so it's accumulating so at this time if we talk about it and ten thousand the hypothesis
testing and they buy that one we're going to have five hundred is
significant just by chance so this is a issues so we need to address and how do we do that is that
traditional is one bounced around in corrections so basically we just really
become very stringent and divided by the number of tests divided by the number of
testing so if 105 you really divide by n now it became very very strict so that
could make cause you have no signal Internet's genes or might have lines to
work on later so which is not ideal so you always want to because that means
it's death sentence you can have nothing to work on later so it's became very
strict it's in people found it's not a reasonable and my idea about stats is
you should help support people debugging hypotheses to reduce narrow down the
search space so they can have a more focus that hypothesis to develop new experiments rather than just give them
data and so stats should another dictator a biological research so that
they need to be something else so there is something else like more like more
common ones the false discovery rate FDR and it is more lenient so it's for
example if you're five percent means if you select five 100 significance
compounds out of 100 or five of them going to be false positive which is okay
because you're doing all makes analysis five percent is totally fine
so because you usually need to be like a qPCR are you doing some validation so
even five person so all this is going to be addressed
it's the omics test the omics experiment usually is a start and it's not the end
so validation is going to be follows so you have a slightly lenient cutoff at
the beginning it's actually it's very conducting for your later research so
high dimensional data and we have metabolomics and so far we mainly discussed about it that all this concept
is the mainly for a single variable and like it it has ANOVA I know most of them
still try to use this at the beginning as a first track which is should be I
mean I see mostly well accepted and and the common approach is that if you have
your metabolomic data atomic data and if you have sufficient of replication
you should try a t-test anova and you apply them single aware of this
statistics then and across all your variables and the visual that you can
visualize and you can you can try to do something this is a this is all okay you
should really try this at the beginning because if directly go to this and more
advanced is the for me is I'm slightly against you should try to explore
individually especially you have a lot of understand this compound supposed to
be high here if this one is true and you have high confidence on your date and
once new once we've done this now you move to the high more advanced and stats
multivariate and multivariate is basically try to summarize your data to
a low level so you can still explore visualize and this is a free grammar
here it shows that we cannot Iraq the view our data in a high dimension and
this is a normal distribution one dimension is 20 minutes three dimension
four dimension going hard actually yeah think about
so multivariate statistics multiverse this means you can see the mouth your
variables simultaneously okay this is not one at a time across all the variables so simultaneously that define
it it is even more complex and it's so the reality is that most of us been
chained or been taught about this univariate and because all these kind of stats concepts you divide like Fisher's
exact test even t-test or last century okay
at the time they don't have this omics data show up so it's the devout that
very well accepted well-established and now we need to consider more than that because you direct apply that we already
since there's multi testing issues also modeling because if you have very low
number of replicates but you have a lot of merriment and one one issue caused by
its called and much far less and P so any set your number of replicates P is number of variables basically it's very
hard for any program to model it because this is just a reality so how do we do
that and Steph actually almost came up because all the series I mention a think
about the early last century is dealing with low dimensional data and they
cannot just all mix data and even they dress it's just not satisfactory because you can bump around you apply all this
stringent and does not consider other variable simultaneously so what you need
to do and computer science come to help actual computer science that we're
opening on it and yeah that's at a time I'm frustrated with that's quite a lot I
just attending all this kind of the class asking these questions they they
look at me and I think I'm kind of can make any sense
but to come on I'm working in the omics it's a real data real challenge
why don't we address it so then I went to a computer science and took all the
classes and they are very open-minded and tried try to pass to do it so it's
it's great and now it's not stats not convenience I'm thinking more data
science and better the two approaches like here is that clustering and the
classification okay regression is more stats part so this is very helpful so
omics data have developed beyond the scope of classical stats and it borrow a
lot of students from from like machine learning field like image processing and
camo in MAGIX so one particular thing that said imagine reduction and the
machine learning virtual learning class for in like actual PCA also you can
think about a cyclist or in approach ok and be who we really jump you to machine
learning and have a I just put in your mind some overall contains a called
unsupervised and super wise and you went not her this is new it super wise
they're learning and they new to anyone do you
oh it's all fun I'm just I'm not going to details because I assume this is very
basic so unsupervised basically only consider
the class tables whatever based on date itself super wise they need to do the
modeling we regarding to the class label okay
and I'm not going to cover too much details so how do we help understand the
large high dimensional data so one is classroom so this is a it's close your mind that close your eyes think about it
because the data is so high dimensional likes it $20,000 features okay
make it more understandable one is that I put it a features that more similar to
each other put them in small in the different beings like like one of the beings and based on they are similar to
each other now you can reduce
20100 beans so each being is represented by their average so this is classroom
basically reduced number of training to consider and as long as that each being representing that data
well it's fun and imagine reduction as actually is trying to reduce high demand
data into low dimension and how they try to do is they try to reduce the samurais
of errors yeah it's a good question so he is asking that when are doing class thing is applied to a sample applied
with features so usually of course we can't like to pose or eat in in in our
Oh anywhere it's just a block of data you can transpose rotate so any directed
matter but for us is that we already discussed that with high dimension talk
amount of variables it means you apply to features in tip by
default okay if you really want to see which somehow is more similar to each
other you should apply two examples but mainly we apply to the features once they which feature close to each other
so we organize these features say this is a twelve compounds the most similar
to each other because it's most likely involving same pathways we are looking
at this understanding of the data okay this is default but you can always do
that different ways so the algorithm is really agnostic to whatever you
Direction apply so if so now if we talk about put the samples of features into
similar the beings that's who is more similar to each other but how do we
marry the similarity right this is a this is important so if we want to the classroom so we need to measure a
similarity we need to add a certain threshold we said okay you are similarly
not putting the same thing and now same being then after we organize like from
10 to 100 bins and 100 means how the race similar to each
other so each being also need to marry it's how similar to each other so we can
we want to respect and data as much as possible but you also want to summarize
them so this you can really see is that it's nothing fancy it's just practical
we really want to reduce the data but still represent in the real data okay
but it's more digestible understandable smaller science ok so the class to me is
basically doing this and there are some fancy terms called the key means they
belong to a category called partitioning method so the one to do is divide this object into an m-class the visual
ambience with our without overlap it's based on the and higher our class and they don't actually give a particular
class departed they just from the top to the bottom from bottom to top so it's up
to you to visually check in and said that's why should we were to cut off and
the kite Kimmy is very well or used a in machine learning but if apologies people want to do the hierarchal classroom
because they don't have any clue about how many class that we are looking for in and they show me all the possible
things and I'm going to decide so high record classroom is very commonly used
and it's bad for biologists who actually knows the data as you can see at where
it make a cut off small parallel to meaningful so came in class three and
I'm not sure how many of you want to know more details because this is a very
basic building and so a lot of randomness so you have you through your
object there and and then just go to the next and see where they closed which had
and put them together and recalculate the new sent centroid and move just the back and of course let's go here and so
you have your data assume this two-dimensional data just down here and
you want to have them like two groups so let's see a B so you know you don't know where located just gave them two random
seeds you can be located anywhere then ever feel
closed way to go and be recalled it as for example to see the red and blue so
you just recorded red and blue and who close to them just the color that this
then you calculated a new center and recall the new center and move there and
redo the whole process eventually it's going to converge and what once you're just going through back and forth
several times your to converge so if you have a strong pattern these regarding
how many different scenes you're going to converge to the nice separation like
that but in a lot of cases there's some fuzzy so you have different start point
and some time you you end up doing some classroom and slightly different which is also quite a common unless you have
this nice data like this or guarantee that you're going to separate that it's
sometime it's there's a fuzzy sixties here and depending on the initial state
Appalachia and something can be changed but this is also extracted so you can
computer you can do it very fast and easy okay Kimmy is quite quite a popular
you catch a lot of actually online deceiving or customer profiling a lot of
things you can easily Bing you into different categories and gave you some
situations you should pop by this you do that because you can using Kimmy and all
your shopping or the fine you're happy that can be actually easily do it at a
real time and I rocket classroom so this is more commonly used in biology by
biological researchers so we found the object very close to each other so we
here we see into object can be your features can be samples and actually in
reality applied to both if you do how rocket class or apply to both sample and
the feature you can see tangia gram is on both sides okay so you merge them and they're repeating the whole process and
like this is also like k-means but then you don't need to specify that a nice or
see the position is just starting from the either highest level to the lowest
level and then working started working so key parameters
similarity between a sample similarity between the clusters so I think based on
a question it shouldn't be a similarity between us that was similar to being in
the features so because you can't class across the variables okay so I think that's um you should understand that so
we're just doing a calculation and the merging them and recalculate back so
unless you are going to write a code yourself
I'm not going to details and because it's all been down in the program so
similarity Mara and why would we need to care about because this primary in exposed to you in my table analyst and
if you you need to make informed choice like a most common I use the similarity
marries in Euclidean distance so this is easy basically you just do this each
corresponding features just the community difference you just a
difference and the square and they're doing this add it up and take a thick
root so it's common and it's a likely or physical distance in a space and then
the other one similarity mirror is the Pearson correlation so it's a normalized
so it's a divided by that so if your data is already all scared
so peers and Euclidean will be the same because you have unit one so we divided
by one is still the same thing so Pearson correlation is that you have
this difference here is that you have directions you can see it on top because
it noted a square that it it you can contains magnet so you can see the
positively correlated a1 and negatively correlated minus one okay so this is 1
this is minus 1 so this is a you can see this questions know
[Music] I can repeat no no this is actually your
part of yours you already normalize your data yet you are analyzing data right
now doing a PC doing this this stuff is actually used in hit math classroom you
actually see the patterns and which groups of features are close to each
other Lisa you are actually analyzing data
right now just like using I'm talking about different parameters when you see
the hidden map here which for em - you to choose right yeah so classroom is
basically you need to have several things are called a single linkage this
is a closest state point and and generator long chains if you sometimes
you can see a heat map with class one you can see huh this is using a single
linkage because there's some very high very long I know like legs or long chains and this is a computer linkage
it's actually quite a commonly used is the furthest at pinpoint and they have
their clumps it's very shallow and this is a average it's a small ice
comfortable Sims and so you can see that this is it from very typical probably
first hit map being used so you can see this is only applied to the various
regions you can see this is a town regulator up regulated and is nicely
separated and from the top to the bottom so so anyone if you have difficulties
understanding hierarchical clustering and we can definitely discuss up to in
the lab so I'm not going to too much details which it's okay so higher our classroom is very natural or intuitive
and PCA is also supposed to be very intuitive
in metabolomics but is less so if you are from RN psychogenic expression and
but it I tried I'm working in both field I found a piece they're just the most
I'm done working well in the gene expression it's just a party metabolomic
working very well it's a it's just that interesting observation to see that
different systems so what PC tried to do is a project a
high dimensional data into low dimensions and keep most variants and
here they think the variance of data is a key information so it's keep as much
as possible that means the more high fidelity to your data and here's that a
commonly used things to think about PCA is that what's your data characteristics
and here is well because we can only view in 3ds so here the
three-dimensional the bagel and the project in two dimensions and there is a
shape there's a hot dog shape and and you need to choose one so the O'Shea
plates move yeah if you need to sacrifice something and you're going to
keep the ocean shape it's more important instead of the hot dog because that's
that seems in more characteristics of your data so the PCA actually commonly
used for image analysis and people were actually found to identify the face
that's representing the characteristics of your face or spec features to erect facial recognition so it's a power the
from machine learning and they've been refined and for PCA when not only for
the visual representation a lot of time we want to see which features actually
contribute to the separation and so we not only see the pattern we only want to see the rich compound I could drive this
operation and so that's that's I'm giving you some
scary and slides you see that how the principal component being computed is that it's actually near a combination of
all your original features but multiplied by a
week so then P is a zoom let's see it's a
weight so it's if we in a PCA plot they call it loading so you can see this is
original data this is your weight so that the sign positive negative of the
weight or the value basically how big is a way to actually have make a that's
that call this one this p-value actually contributed to it okay so this p-value
the sign and and magnitude the absolute value of this we're actually contributed
a lot to the separation so how you rank your important features by ranked by
their signs by the absolute value so of course the big that value contributed
more to the own final score so this T is your final score going to show up on their map
but who is contributing more so it's the because the X is your already no data it
is it adhere that's actually have more influence so this is a bigger it more
lies that have more influence okay if you go under the hood and see what's
going on so this is actually how the from original to a new dimensions and the rank them based on the based on the
variable explained because we are only looking for the most variable so it
likes a top two and its hope to explain at least half of your variance in your
data and the more detail the PCA is that once you extract the one component that
explain most variants and now you're gonna check extract the second one and the condition that the first one the
second was unrelated it so add so this is all saga notochord and so this one's
maximized that make it easier to interpret some of the approaches that
make a good separation but it is some overlaps it's not our sergeant so and
there's some these are more complex approaches but not necessarily
make it easier for you to interpret your data and but PC is the same to be a
standard and if we're working well and really try to use it because it's easier to interpret also this is the
unsupervised it's based on your data so if your data already separated well in a
PC you're really just the thing about it bail out your story and based on PC
probably plus some ANOVA our team has you can write your paper you don't need
to amal analysis if an PC already show the difference okay so the other part is
that PC is a rotate the axis basically if you need to D and and they just
created an axis of just like from the current one sector moved to the one red
and a green okay it just basically makes sure the first X
is like X is the following the most data variance in this case it's almost a
diagonal and the other is orthogonal okay
it's PCA come in two very different ways but just you need to be comfortable you
can choose any one you like so PCA can be operating on two different things one is covariance at matrix the
other one correlation matrix so
covariance is that I'm not sure I should go too much details because if you
normalize your data use auto scaling so covariance and correlation will be the
same thing because all unit 1 and if you didn't don't do it normal scaling the
covariance is going to be more influenced by the one is more about it
okay more abundant change like him glucose is very abundant the change of
that ago and cause a huge variance change of the global
all your data because that's a very abundant things if you can also scale it
you're going each one contributing all it almost so this is up to its its I
know you're going to ask that and I don't know that kind of that coburn it's as long as that you see if you really
think that the most abundant content compound is going to have their
changes is very important compared to the one less abundant then you just to
choose their covariant if you really think that a trending 20m micromolar
versus the fives on the micromolar it's just this is alanine like this is
glucose but as long as the married accurate they change to folder change ma
less the same or similar I usually try to reduce the change in the covariance
driven by the high abundant ones right otherwise it's it's you always see these
guys and they overshadow the added small compounds right sometimes evil accurate
removes that glucose we move that your urea to 2 because we know you'd be
interesting some less abundant ones so again this is up to you how you think
and how you want to modify it and I would like to come up with some clear
colored walls and build it into a meta balanced you click back together result
but that's not good because people want to explore people want to learn and
wants everything being wrapped in a box and you just click and believe me and
this became not following research I'm just it so there's no clear-cut rules
you have to use your brain and child community reviews right so why why they
use your metabolomics is and we talked about SPCA and this is still PC like this Europe is should be very happy and
so initially are this a llamar structure it's a three different patient and the
control and it PAP so and I think is one of the first studies and when you're
being this fracture and they're just OPC wow perfect separation it's basically
yeah I believe you can be classed method that you have a explicit like a
similarity mirror between the example different clusters we discuss about this
PCA get actually their main purpose reduce dimension you can see their computing a redo dimension but in the
end they actually classroom is you can see that on the on this figure things
that are close to each other is most similar to each other it's definitely classed work but their
primary goal is reduced dimension yeah so yeah it's it's except I actually when
I put it as a camel metrics or classroom I'm just back in the past it's it's it's
so here the hidden loading plot I mention about all this kind of Ko Ko
correlation coefficient when you're doing the loading how you calculate it
so this is a kind of people don't like it by the way or you see that the signs
are magnitude so if you have more big values absolute peak values you go more
close to the corners up and down and this ones that tend to have this more
influence and the signs basically you driving this work as this direction or this direction okay the cool thing with
PCA is that this is always correlated so so if here and here let's just focus on
this control okay so the here this group of things we are going to
have very positive correlation with samples in a control group okay
and we can see it here the group of compounds are here we're going to have a
negative correlation with with that control okay this is a very intuitive
interpretation things you can just walk through that math in your mind you found
out it's just it's not a coincidence it's just a very nice feature to there
yeah it's exactly if you put PCA side-by-side you can see it is a agree
and it's the kind of the minerals is a correlated positively and next record it
here so separation here and worst of these two groups suffering here for
these two groups so this is a PC it's why people like it you don't understand
about honest actually found it's very pleasing
and you can do it so in some cases PCA will not succeed in identifying any
clear clustering no matter how many components are used and and in this case
it's wise to accept the result assume that you cannot distinguish and actually
it is so true a lot of times if you your data is high-quality and everything is
ok if you go to PCA you see it you don't see a strong pattern and it's very rare
you may use more advanced they're going to give you a very good result so this is a PCA accurate want to see the PC you
actually already have a you should have certain feelings about what result you
can see and this is a very strong overlap and you get some like appears
the year later you find it a good separation you are so happy on the
publish most likely called over fit basically the patterns not robust and
you're not quite sure that can you see is is real or not
ok we are going to talk about POS deer later and questions piece a very good
data overview outlier detection and looking at the relationship which means
different variables so really you should use a PCA with other like box plot stuff
as your quality control as whatever your first alliance of weapons to to to
understand your data so chaos da is this is also a sibling our PC and people
don't like the use PCA direct or the PSD and it's tell me a good separation which
is wrong I can tell you just don't do that Lupe's here first direct go the PSD is really dangerous and why PC appears
they usually good is that USD is a soup wise they try to maximize in a
covariance between the data axis and the class label and super wise to always see this last one piece a always produce
certain separations with regard to a condition it always
tried to please you make you feel happy okay let's think about that so it's not some time really you want their
encouragement you can look at a piece a PhD a but really you need to be cautious
about that and you can see here is that the PCA PhD a apply people and after you
can see a better separation so if you have certain separation apply PSD
actually I try to tease out the signal and make it easy to find out which
factor contribute the separation and make it easier to general hypothesis
okay but if you have no subject at all and they you should be more cautious so
PR steer can improve PCA no problems but if there's no actual separation peers
they can still do it which is all of it so you use variant PhD a PCA always use together what's the same yes my question
is always be how much is - separation versus separation yeah do you consider
that separation it's hard to say yeah there probably sweetie you can see some
separation and there's actually there's some separation and if you see the use
the phenomenal addition stuff you can see some separation I know which data it
is it's just a separation it's a from clinical sample so separate is not good
and but there's some separation on the other hand the PhD in a few in an in I'm
probably going discusses that there's going some validation yeah if you if you
just appears the user permutation see the how good a separation going to be
it's still going to help you prevent some jump to that there's some good signals so for the PCA in this top left
I I couldn't tell you
so PhD is a partial least square regression so somehow the the regression
then yeah so what it does internal if your I won't say any interval because I
write how the code so so what is it converts a class label into numbers and
try to perform pure steer regression and between different numerical values so
and it's acceptable to overfitting by produce patterns of separate even their
state is no no real signal so need to perform cross-validation need to do the
permutation okay cross-validation permutation where we're going to discuss
more on this concept later
so overfitting this is very very important just don't know need now you
believe believe there's a advanced method that will magically help you
solve your problem because a lot of the things more advanced is more likely to
over fit and then please you and so what is overfitting is that this figure shows
you that here's your data point and you want to feel her lines here and and then
you can see this just right it's this is on the fitting because this is a similar
fate Abed but it didn't really fit nicely on the fitting and here you try
to connect all the dots it's overfitting so overfitting is the bad over why
because it there's some in the population is some variation just like
this it's a cotton always variation but the overall truth the patterns actually
this is the truth and you fit but if you really started fitting this signal and
it's became more noise and prone to a lot of errors so it'll become a bad mode
also it you can see this curve a lot of you know to generate the curve you have
need to estimate a lot of parameters so this parameter is very sample of sample
specific okay this one we only have two parameter is it probably why not two
parameter so this one became very unrealistic and
is probably not generalized you use this model you get a new patient and you're
doing diagnoses probably totally wrong and this one probably working well and
this one probably working somehow but it's not as good at that one
so overfitting is really and so yeah
because when you yeah I think so we newly the cross-validation a square Q
square to calculate so this is we try to follow the best practice possible and so
because Mehta balance they use is really targeting it's not a computer scientist
not a statistician it's a regular bench scientist so they need to be a user
friendly they also prevents an errors so it's a also workshop I want to spread
some cautions and don't just try to jump to conclusions so cross-validation how
do you do that cross-validation is that you change your models using a part of
data and try to predict the other part you haven't use it right here is that we
have three for the cross-validation we put all the data like 30 samples ten ten ten
okay we try to be balanced like every every ten we have five control five
disease we if we can do a balanced sub sample then you uses a chain predictor
disk and calculated performance use this to change and computing this and doing
this then average their performance how many time making wrong what's the R
square Q square and we can have a good feeling about how good it is right and
sometimes people want to do more granular unfold across where that is you
can see you have 100 samples you have 99 changing your model and predicted one
and do it for each samples this is probably more more reproducible
what I'm saying because you guarantee that you're going to go through each
sample at least once and going here that how do we develop our samples into three
folds and every time I'm random draw 10 10 10 and it's not a guarantee there so
if you're doing cross validation different try probably have slightly
different result but I leave one out cross validation guarantee the repre do so because every time we need to go
through it okay don't the only thing that I do one out is that it's more
computing and computationally intensive you can see that and if you have a
larger sample actually Kim not matter threefold or fivefold oh it's just all
the same because a large sample when you go to like see 500 samples it's really
saturating you you get the whole two parameters it's it's very stable and you
from 500 to 1000 no change I went on to touch on no change actually it's it it's just a between like 10 samples 20
samples as you're going to see a lot of fluctuation because you don't have
enough samples to estimate them parameters so a purist ear model
component of features so lets them evaluate performance evaluation and that sum of the squares captured by the model
is our square cross valid is a square the q square and prediction accuracy so
this is a if you are really traditional statistical and this is really the
values the you're looking for hand reviews going what is this value so it's
it's it's it's all provided so I'm not going to too much details because this
is all the same thing regarding your prediction and also keep in mind
theorist is trying to do a regression first and convert to a classification
and permutation test and there so again like empirical p-values and the same
concept because PSD is very susceptible to overfitting
basically he's going to it and try to please you this weekend was the data is
so let's see how he can trick you let's assume there's no separations we
just randomly shuffle your class label and that girls do to do that to do that
classification and in this case we are not a visual judge we just I come here they're a square Q square or the
prediction accuracy and see how they able to do that so this is when actually
computed from the same data and he just asked about is that real separation or
not so actually it seems to be robust and you're doing some permutations and
even the data seems like here here it seems a not what why they separated and going here isn't it real or not
and and then you're doing a permutation and see how good it is separation using
this original Apple label aversive random per million label you can see you
may try to please you use a random label it's the separation is narrow it's still
separate but it's narrow so that one you set the distance that you separately
here actually still for the weight so definitely there is something here
that signals and being captured okay okay yes not alternative is compact a complimentary so you need to polish show
that a PSD a they cross validation and the show the permutation I because for a
lot of animated like SVM or random forests annually just one should be fun
sometime I also do the permutation I found this on espera because then the these two method the much less except of
overfitting cross-validation working fine on not really in the indicate its
performance but purest there's a not so pierced ears is so heavily
abused what I'm saying so you need to have a double double check okay
so PhD the other ones the VIP score is a bear variable importance in projection
so this is similar to the loading okay you can use the loading better a lot of
the people want to use this a VIP this is okay so it's just different formula
and the same that captures more information it's a weighted sum of
squared the correlation between the piers their component handed original variable and so it's capture more and a
waste correspondent with a percentage variation explained that occurs the components of a modem so a lot of people
wants it was a cutoff so if you see the literate most people use a one and
that's important but it's it's really and sometimes if you see the ranking of
the VIP scores and then really this and nice separation and at certain point you
see it's a separated for example here and you can see that this group actually
very there and this weekend is one or two so you don't actually group it
enough and some time you don't even need to use 0.01 one of more than one you can
see this one it's more clearly out there so when you see the reappears this yeah
if you reduce the number of variables so you you mention actually his his question is that if if he go back to the
PSD or model removed these variables that's ranked very low right it seems
not informative it will impure model of course you're
going to improve model but it is unethical to do this because you already
see the overall result right and you go back remove the things that's not from well and you retain it the performance
where you're going to be like before let's say 85 you're going to become like 86 87 you're going to gradually improve
it because the model building is actually try to do their best to estimate the signals or stuff once you
see the whole big pictures and you're feeding back and you it doesn't go and
improve it but it is smaller going to be more is actually better probably not
because you are you attend to increase your model stay to your particular data
so you make your model more likely to be
yeah so yeah what I'm saying unethical that once this is all the data and you
rank use a VIP or use t-test like even just use the t-test are you remove the
variables with high p-values and then view the model you already kind of while in a certain sense it's basically you
you know the result you already use a why label because how the p-value is
being calculated in the t-test are how do we appease calculates nearly use the
x and y they know this one is less associated with the patient with your
class labels so once you attach that you you already use the information at so
the the whole point is that the only thing that so you probably have more
predictive model which is all fine but you need to validate on a new set of
patient you've never used to once you're doing that it's all fine
III don't care about it it's just that you use the current data and remove the
uninformative variable based on current data and it here the review I'm nineteen
epicenter right in this one which is not true even your model probably best but
in reality you have more the performance in 85 not a 99 why it's 99 because you
constantly manually trimming is only from the variables the model itself is
correct what I'm saying is most likely that whether you have done the model
yourself is it correct but the performance reported is not right it's
over optimistic it's not 99 is 85 right so if you value to your model even after
whatever your manual teammates and on a new patient cohort and that cohort never
used to generate your current model and you report this is performance it's all
fine so disregarding your previous ones so what I'm saying that that all the key
features all the weight or the parameters probably all right it just
performance value is over optimistic and you make it more likely from the regular boss when to the nature biotech because
the 99% seems amazing but it's not in reality
just because you evaluate it on the same data right just this number is the over-optimistic
yeah I think this is somehow addressing what he said it's a so the mode of
beauty and that pal market scenes is that definition bow marker what we are
talking about exploratory studies analysis biomarkers in either iterative
okay it's motor beauty instead if we talk about that you really need a new
cohort ok and when I see the stuff and did not sure
so assessing classification model performance so I think it is intuitive
if your data is very balanced so half and half that's easy
so accuracy is whatever you done is correct out of the total prediction so you have 1213 samples out of you try to
predict an item is correct so 69 percent accurate the other
commonly used is the error rate so why - accuracy so 13 one percent so it's all
this so I'm saying it because it's used in the meta panelists for different
approaches so you need to switch back and of course what he's talking about
how to convert them but unfortunately in clinical data it's a lot on imbalanced
so HIV so you have five cases in one half so on the samples oh you finds on
the people so if you predict every everyone's healthy and you have 99.5
percent and so it's a majority vote it seems very easy and it's a very cool performance but it's not right it's not
useful it's not a right that's just so there's a biomarkers screening
biomarkers and like a diagnostic biomarkers a lot of things is that you
need to really for different purpose how do we try to balance them and of course
there's a clinical they have very strict and science and over the years and I
think sensitivity specificity became more and more prevalent used so true
positive to knock false positive as an active sensitive
they specifically so this is all Deepa merriment and just that you need to develop a a feelings about what they are
sensitivity is the true positive rate so whatever you see what's the in in whatever you see the right what
percentage is right and specificity to not give rate so whatever you see is that how he'll say it's not these days
what percentage is it's actually healthy announcement so it's a true positive to negative so example is that you have the
healthy chin active on the left side it's a to positive on the right side and side and it's you can see it's not a
clear separation so you need to make a decision point where is the cutoff so if
that's a kind of you can see some regions it's it's going to be a like
this this region if we're talking abou it's good this is actually on this side
it's a blue and to it positive you're pretty positive so it's to politics but this one it's actually positive for they
predict enacted so it's going to be false and active okay and this same thing are going to be here so it is you
can see you can move this bar cutoff threshold from left to right from light
light right so you can switch how much false positive going to be produced or
how much false network only produced and because there is no perfect solution
this because it's not it's overlap so you have to decide what's the cost if
I'm saying is a false and active how was the consequence it's false positive so
this is really depending on clinical questions if I'm saying this HIV is
positive we say it's a negative we miss it it's going to be highly concert have
a very high consequence rather than said oh this people probably healthy but it's
if we see it's could be positive but let's send them to the next screen and
probably it's more tolerable so it's really depending and the cost so you can
move this bar back and forth and that you decided was
the cutoff so Qaddafi is not really a communities you know saying we can
always calculate what's the best cutoff I gave the best balance better clinical when you do a real-time a real-life
application it's a lot of other things and how do you make it easier
I went went to is called our C curve or the receiver operating characteristic
curve so it's a historical name for reader studies and we're why they use
the biomedical application so it can be used for a classification performance
assessment and the comparative in about mark models so you essentially it is a
two-part it versus false positive so this is a based on that you can actually
move the chooser where it's a best so for example you have your
classifications models and it's used even cut off at each color point you can
see computing is a true positive false positive and you draw that because you
can see that this is x axis the false positive and the y axis to posting you can draw this and thoughts and the
connector and this will be your RC curve so it's our two curve it's not that
straightforward you need to first fill the classifier and it's need to predict and use the cutoff and calculate as you
step so this is the i-best down its back you could compute compute if I compute
not by hand okay so from here you can see you are treating off innocent even a
specificity so if you go here you have a high specificity but less sensitivity we
go here you have high sensitivity but less visit so it's really it's and if
you go here probably you have the balanced but you if this is really
computed by the computer and your reality
you really need to see which part you want to choose so here commutation and I
you you you can do that area under curve basically them the bigger this value the
more the battery performance so this is also
used in comparative Nobel m'appelle mark models and here you can see that if you
perfect you can't basically like this is 100% but if you do a 95 percent is pretty good and and this one is random
so somehow they found his overlap so if your random one it's just random guesses
based on 15 15% is going to attack you know this is around 70 percent so you
related this is we see a lot of 75 70 percent so this is quite a comment so if
he wants to start going to 1885 and become yeah more promising yeah yeah it
is defined for only two classes to know this our secure but if you see that this
is only defined for the sentence specificity so they say only to that yes
or no to on Foss binary so even you could whatever multi classifier you have
to mount it group classifier will be able to deal with two groups right so
you have to make the it as our C curve as whatever it is at is if you have
multi groups you still have the being into two groups
okay just because there's a dip on yes or no you can sorry so yeah it's ABC you
have to defy that yes or no once it even that they can do it so somehow how you
do that is based on what groups you think is to you pause the water go
inactive right yeah it's just defined for this two stops okay exactly this is
how it is defined so if you have Maori groups you can serve it like you have
control lower high you can either being lower how you have country over
solutions over adjust the reality it is how will it be
fine and you yeah you can try to do like accuracy at the performance value relative humidity group but hours ago
strictly to groups you know so other to a method as other supervised the methods
commonly used for this metabolomics like same kind soft independent modeling of
class energy this is a commercial or patented appropriate rule we're not
going to talk too much if you syncope and and you read their menu and not a
fan of that so the other ones that appears is Osaka no projection of
lateness structures so this is available in the metal analyst also again this is based on open source are and you should
have try to see also kernel QRS don't see okie is because I get some legal
notice from come and you don't even want me to mention them so so this is a
interesting point and the other ones as we am random machine random forests this
is all available there is a you just did you use whatever you want so it's all
good model so let me see and I am not really basically as we am running
forests at what I'm saying that is really well used but it's mostly treated
as a black box and what important concept is that I just mention about
this cross-validation importance a company wearable important so once you
get this concept you can actually use it as it is there's no way and I can give
you some intuitive explanation on the SVM stuff because it's really really complicated but on the other hand is you
feel free to explore and during lab session you can try to interact with me
see if I can give you some more insight further for more in stacks now this one
is really I don't think it's a super political audience but on the other hand
this this method is so well used and a lot of other materials available so
again this is MBM said what have mentioned before inside they try and
unsupervised simple like from t-test and ANOVA to unsupervised a move to supervise and then go to machine
learning don't go they or the other way around because that's a first it hard to
interpret and probably you're going over video date if your data working fine
with your PC and they're really really that's probably you just try to grab
your hypothesis and they try to write your main script you don't need to go to very offenses which was to do that of
course you can PSD to further step in a signal but it's it's it's pretty set
already and let's see data another pro pro
freshness data so for our lab and what what we want to do is if you use a PCA
as a clustering which is unsupervised so it's really based on your natural clustering of your data
okay the data label has not been used it didn't use prior knowledge so there's no
kind of ethical things if you see the label you're in know the secret you're
going back to change your date this is all you can use it just a safely once
they go to supervise its study using your prior knowledge you use a class
label and and if you see the separation already in a PCO class and you use this
to improve the signal to divert and further refine your hypothesis it's all
fine but it's a no separating of pca or unsupervised and the you see this
suddenly is a good separation here in a supervised you really need to be more
cautious at the moment now and statistical significance that supervised
the method is part of a PSD a and this can be misleading so I always try to place you and at this time if you do a
permutation for the PhD which we might have honest already
have it so again we with are not going to iterate this and so there's no hard rules I can tell you should do this who
did not do that it's just the general guidance and you just to follow and
every time you need to think active about your data what do you expect and
was surprised and something is too good and it issued it were good to meet you
okay this is some time you you need to start a think and you can ask me and or
you can see at a publication what their practice so as some time you see it
I'm review papers using the seventh wars they just really use it in a way that's
not proper so it's it seems you spend so much money and spend so much time and I
really try to think about it and try to think about that your day they try to
think about method and the marriage or both it really make a good enjoyable
reading for your paper oh this is actually I'm early yeah and
their questions and yeah yeah then let me try to rephrase your question you are
talking about false discovery rate in a multivariate analysis yeah
okay let me let me correct you first and that's so first is that a false
discovery rate the design for univariate because we test the individual feature
unit and using t-test anova one up one water one like twenty thousand times
they stood empty arms we're doing a multivariate test we consider all
variables simultaneously we only test the ones okay
there's no false discovery rate is no mouth here a testing adjustment so all the features that based on their like
based on VIP are loading it is based on coefficient correlation coefficient it's
just the rank and the top likes a top 20 probably significant based on this PCA
analysis there's no host equal rate because there's near no testing they
simpler ranked based on the code the
yeah I'm not quite aware of that FDR you design of a PCA is stuff it's a
kind of new to me so that's the you show me the paper and like I can see it
better I that's just the theoretical non
cross-validation is more commonly used for machine learning permutation has
this just a recently years it started more widely used
it's a as they're testing different things so so let's let me try to say so
cross-validation is tell you how good is your model works like say 95% it's this basically it's
basically 19 times out of 100 hum probably good and the permutation test
is actually a p-value and if p-value is significant means there's some signals
in your model actually it's not random okay it's just that they're not going to
tell you how good it is your model it's just so says there's some signal in your
model right it's different from random so they are very complimentary they are
not talking about same thing
[Music]