Hello.
Welcome again to the course analyzing and processing for music applications.
This week, we are talking about sound and music description in the theory lectures.
We talked about how to analyze sounds and obtain some features based on
spectral analysis or on the things that we have been talking all along this course,
but specifically to describe large collections of sounds.
So the idea of obtaining features with which then you can measure the similarity
between sounds or we can cluster sounds or we can classify Identify sounds.
And we mentioned about Essentia, the software library tool
that we're going to be using in this week, and also a little bit next week.
So, let's talk a little bit about Essentia and how to install it,
and how to use it in some simple examples.
Okay. So, you can go to the website of
Essentia,Essentia.upf.edu and
in here you can find a lot of the documentation, where to download it?
Things that have been done with Essentia and other related information.
So, again, Essentia is an open source C++ library that has been developed
thinking about audio analysis and audio based music information retrieval.
So, the idea of describing sounds and music.
And it's an open software project.
So, it's the license is a fero gpl.
And it has quite a few things.
It has a whole bunch of algorithms, and
a whole bunch of ways to extract features from sounds.
But also we have a whole bunch of utilities that makes that quite easy.
So, it can facilitate our programming off
more complex ways of extracting this features.
So for example, this is a blog diagram of a series of
algorithms that have been put together using this Cynthia facilities.
And therefore, we can easily and
efficiently generate these types of descriptives of sounds.
It uses Python quite a bit.
In fact, it has a good support for Python for wrapping this C++ library with Piton.
So we get the benefit of an efficient library, an efficient C++ implementation.
And at the same time, a quite easy and
convenient Python framework that can be used to call the C++.
So the first thing that we should do is just go to the download page.
Okay, and it's in Github, so you can find the source code in Github and also there
is some accompanying software that you might use, which is this Gaia software.
The, it supports Debian and Ubunto and
also Mac OSX even though, of course in this class, we are going to use
the Ubunto version And show examples using Ubuntu.
So, if we go to the essentia GitHub page.
Okay, so here is all the source code of Essentia,
that's a pretty large piece of subwork.
So to install it,
the normal way when you want to really keep updated on these things and
not have to reinstall it every time is to use this clone URL.
So, we can copy this URL.
Okay, and now we go to the terminal.
We can go to a specific directory or, I guess I have the TNP directory.
And I will clone the repository into my machine.
So I will type Clone, and I just put the http address of the repository.
And now we'll start copying everything into my computer.
That might take a little bit, because that's a quite large library.
So, maybe let's just stop it, because I already copied so I will stop it here.
And now I will go to the place where I already copied this library.
So, I have here all the structure of what I copied that is in GitHub.
Okay. Now in order to,
we have to compile it and install it.
So for that we can go back to the Essentia website.
And within the documentation there is, well explanations about again what it is.
And then in user documentation,
there is a section on building and installing Essentia.
Okay? So once we have downloaded,
there is quite a few dependence of libraries that they use,
like the FFTW or FFM Beck for holding audio in and out.
For changing something great, etc, etc.
So, we have to install of these dependencies,
so we will just copy this complete line, and we go there.
And we can just, we are in our home page.
And we just type sudo and we get all of these dependencies that may be needed,
so I have to type the password because we're doing at a superuser
Okay, and well I had it installed already, so it just checks and
it sees that everything is there.
Okay, then if I want to use Python, and I want to use these wrapper for
Python, it needs also some Python packages.
But in fact, I believe these package were already installed,
because these are the ones that are used by the SMS tools,
NumPy and the development, and NumPy.
But we can just make sure that are there.
So, we just do sudo.
Yes everything is there, so no need to install anything else.
Okay, let's continue.
Well there is the instructions for Mac OS X and now there is instructions for
Compiling Essentia okay, and this is what we need now.
So we have to first configure,
all the structure of the library in order to decompile.
So, we are using these wav compilation tool.
And again, we can copy that and we can go to our
essentia directory, that's here.
That's where we have all of our.
And now we again can figure out what needs to be configured.
And it will check everything, it checks if everything is there, and
it says yes, it's fine, it has been configured.
And now we have to compile it, and to compile it,
we just have to type this compilation instruction, okay?
And this will go through all the directories,
all what needs to be compiled, all the C files and compile them as object files.
Okay, so it's building everything and again, this may take quite a bit
because it's a pretty large library.
So let's stop here because in fact I already have compiled all this,
so let's stop it and we can just continue this.
So we have compile it, then we need to install so
that all the Python bindings I can find all the information.
And we can, well there is some, you can run some tests or
you can run the Python test or you can generate the documentation.
The documentation is basically the same documentation that we have here,
okay, so that's done.
And then in this directory, again, we have everything compile and
we will have the directory where we have all the binaries and
we have everything else ready to be used.
Now we can go back to the documentation and
start looking at some documentation for actually using Essentia.
So in here there is some tutorials and
the one I recommend is this Python tutorial for beginners.
So this is a tutorial that explains how to use Python
as a way to call all these Essentia algorithms.
So if you go through that,
you will see pieces of code that you can basically copy.
And by following that, you will be able to do some analysis of
some sounds but let us do our own example.
So, in this text file I have a piece of code
that is basically the one that I use in the theory
lecture that I use to compute the MFCC's, so
in fact you can see this code exactly in the plots from the lectures.
But now let's explain what it is so that you can actually do it yourself.
So it's Python file that has the standard import packages the Numpy,
the Matplotlib and the one that now is specific for Essentia is this one.
So the Essentia has been compiled, so Python will be able to find this package.
We are not importing the whole Essentia, we are just importing
one of the packages within Essentia which is what we call the standard package.
There is also another package which is meant to be more efficient for streaming,
it's the streaming package.
But we're not going to use that, we're going to use this more standard package.
Okay, and then, well, we have some intialization of some variables that will
be used for the analysis, we have seen all that, so no need to now talk about that.
The Core of Essentia starts now here in which
we are initializing the algorithms that we will be using.
Essentia is been written C plus plus,
so it's a collection of classes, every algorytm is a class and
this is in the sort of object oriented approach.
So in order to use an algorithm, we first have to instantiate the algorithm,
and give it a name, so here we are instantiating the class spectrum.
Okay, we instantiate it with an initialization, the size of the FFT,
size N, which we have here declared as 1,024.
And now the variable spectrum has an instance of the spectrum
and we do the same thing for windowing.
So there is an algorithm for windowing in Essentia and
we initialize it with a given window size and a given type.
And therefore window now is an instance of this windowing algorithm,
and again, the same thing for the MFCC algorithm.
The MFCC algorithm can be initialized with different parameters, and
here we initialize with the number of coefficient that we will be using and
the size of the spectrum that will be required.
So these MFCC's are is the now the instance of this algorithm.
And one very very useful set of tools from
Essentia Is the way to read audio files.
In this case we are using an algorithm called mono loader and
this is an algorithm that accepts any type of file format or
at least a wide variety of file formats.
And then by specifying the sampling rate
It returns a mono file independently,
if it was a stereo or even more channels, it returns a mono file.
It returns it with something they didn't specify, and
it returns it as a set of samples, so
x is our array of samples of the input sound.
Okay, then we initialize the output array or matrix
where we will store the coefficients, and the core of the computation comes now.
Okay so we will be iterating over the sound computing the spectrum windowing and
then computing the MFCC's.
And in Essentia is amazingly compact with these four lines we are able to do
all that, so one of the core algorithms here is the frame generator.
This is an algorithm that automatically iterates
over the array x with a given frameSize,
with a given hopSize and also there are several parameters of how we actually
placed these windows in the sound.
And then these returns then frame, okay, frame will keep
having the samples of that particular frame that we have chosen.
So now frame, we'll keep having the different values of our frame and
therefore the window here now will accept the frame.
And then, we can put it within the spectrum automatically.
So, okay, so the spectrum will be computed from the windowed frame signal.
And mX spectrum returns just the magnitude spectrum and then this is what it is,
the positive magnitude spectrum of this particular frame.
And now we can compute the MFCC's, so
the MFCC's again now is this object that accepts the array of magnitude values and
it returns both the coefficients And
also in fact it returns the bands of the nfcc if we wanted to use that.
But for us, what we are interested in is the coefficients.
And since this is being computed at every frame, we need to append to
an existing initialized to the empty array and
then you will keep appending and create a metrics for that.
And finally, in order to be able to visualize with the plot or
to be able to pass to something else, it's good to convert to a numpy array so
that we can have all the numpy possibilities to handle that.
So now if we go to the directory where this file is and
I believe it's in sms tools work space.
Okay so here I had the nfcc.py.
And I start with python with pylab if I want
to use the matplotlib automatically.
Okay, and now I'll basically can execute this file.
So I can just run nfcc.py and
this will have computed the nfccs.
So now if we type in nfccs, this array,
well these metrics of all the coefficient
at every frame throughout the sound.
So, in fact now if I do pcolormesh this
plotting function of matplotlib and
I give it nfccs it will show the coefficients that it has computed.
So here we see horizontally as the coefficients,
the 12 coefficients as well with a zero and then here is the frame location.
So this blue line is the zero's coefficient, then we have the first,
etc, etc.
So that's pretty good, pretty useful.
And in order to understand how to call any of these algorithms,
what we have to do is go to the Essentia documentation.
And go to the algorithm section, there is the algorithm reference.
And in here, you will see all the algorithms that Essentia has, okay?
Okay, so here we have as I mentioned we have two packages in Essentia.
One is the standard algorithm,
the standard package that's the one we're using and the other is the streaming one.
So in here in the standard one we have all the algorithms that are currently
Implemented in Essentia.
And for example, if we go to any of the ones that we were talking about.
For example, the windowing one, we just go here and
it gives me the input, output and the parameters that this function accepts.
And also very importantly It gives us a link to where this algorithm comes from or
what we use as the implementation for
this version of this algorithm.
And of course, you can find all of them here, that's pretty good.
Now let's go back to documentation and talk about something else.
Something another feature of Essentia.
We have been talking about the algorithms that are included in
Essentia and that we can use and write Phyton wrappers to call them.
But there's also the idea of complete extractors.
So sort of extractors that have been developing a way that are useful for
some particular applications and that extract a whole bunch of things.
So extract the whole bunch of these features and
then we have a binary quite efficient quote that can be use.
So for example these are some of the extractors that are already
available in Essentia.
And there is one that is in fact this streaming extractor free sound.
And this is the extractor that the free sound uses
to extract low level and other type of features from every single sound,
and then is the core of this similarity engine that we showed.
And all these extractors are in this built source examples directory of Essentia.
So if we go to the file browser, this is the Essentia directory.
And we go to build, we go to source, src and examples, okay.
And here we have a whole bunch of extracting but
more than the ones that are into documentation.
So these are have been already compiled and we can just use them.
So here there is binary of the streaming extractor freesound and we can just copy.
And if we go to the directory where we work from,
this SMS tools workspace.
I already in fact have this binary there, the streaming extractor free sound.
And I can just then use it.
I can just copy it.
Okay, and I can execute it, and I can give as input the name of a sound.
So here the sounds are in the sounds directory.
So sounds, and maybe let's take the sound of rain, okay, rain.wav.
Maybe let's make it so that we see it better.
And then we put an output file, let's just give it the name of rain.
Okay, now we execute it, and that's it.
It went very quick so it created two files.
It created adjacent file and a XML file.
That are the files that include all the results of the analysis.
So if we now go to the editor and open this file, for example the JSON file,
this is a particular format that is quite standard.
And in which we have all what we have extracted from this rain sound.
And well there's a lot of numbers but
it's pretty good information for a lot of the things that we will be talking about.
Even it attempted to extract the keys of these rain sound
throughout the duration of the sound.
But anyway, we will be talking
more about this in the next programming class of how to use this.
Okay, so now we go back to the documentation.
Here, we see other extractors that are going to be quite useful and
we will be talking about.
One is this streaming extractor music.
And this will be very much useful for analyzing pieces of music.
And we'll talk about that next week, and
in fact this is the one that acoustic brains, which is an initiative
we started recently uses to automatically extract these scriptures from music.
And that's all I wanted to say, so let's go back here.
So we made an introduction to Essentia and
we are able to download and compile Essentia from GitHub.
And that's in Linux is quite is easy, it's quite straightforward.
And we talked quite a few things about the features of Essentia, what we can do?
And we then ended by talking about the extractor that freesound users and
that can be used to generate a whole bunch of descriptors of all the sounds.
And then we will be able to use for quite a lot of things.
So that was all.
Hopefully you got a little bit of a very introductory view to Essentia.
That's a quite complex library.
And it can be quite complicated to use in terms of all the features that it has.
But it has an easy way in, and so we can start using it with simple examples.
And we will see in the next programming lecture how powerful it is for
doing some interesting sound description, and describing collections of sounds.
So thank you very much, and I will see you on next class.
Bye-bye.