In this video, we're going to take a different approach to getting a list of all the songs for a particular
album or artist.
So one problem with the previous approach, which relied on the file and directory names, is that you may
not have such a neatly organized file structure. And if you get your music from different sources,
the file names might well not use the same pattern, for example.
So although the last couple of videos have been great for demonstrating how to work with files and
directories,
that's not necessarily the best way to achieve the original objective, and that was to populate the music
database from a record collection.
So in this video, we're going to read the ID3 tags from the mp3 files on the disk.
Now in order to do that, we're going to need a generator that will return the full path and name of each
mp3 file,
so that's your challenge.
Alright, so the challenge is to create a generator that will return the complete file name of all
mp3 files. Now the generator should start at the specified directory, the start, which will be provided
as an argument to the generator function.
Now the filenames should include the full path from the specified start directory.
So it should return a relative path.
Now it's a good idea to make your generator function a bit more generic,
so that it can handle files with any extension.
So pass the extension as a second parameter to that function.
Now as an optional extra,
you could make the paths absolute so that they specify the filename from the root of the drive or volume.
Check out the documentation for the os module if you want to do that.
Now if you haven't got any real mp3 files ,you can use the music directory from the previous videos and
search for emp3 files instead of mp3 files.
Right so that's the challenge.
See here you go with it.
Pause the video and come back when you're ready to go through my solution.
Alright, so the solution to this challenge is very similar to what we've already done.
So we're going to use the os.walk up generator to walk the directories from start, and then we're
going to filter the files that match the extension.
So I'm going to create a new filem
and I'm going to call this one
mp3scan.py.
We're going to start by importing os. We're going to import fnmatch, then we're going to start -
first I'll create our first function - and we'll call this one find_music parenthesis start comma
space extension, end parenthesis and a colon.
Then we're going to type for path comma space directories comma space files in os.walk, start in parentheses,
colon for file in fnmatch.filter files comma space,
a single quote. Then we want star dot,
then we want left and right curly braces, end a single quote, or ending single quote I should say, dot format
extension colon, after closing off the two parentheses, right parentheses. Then we're gonna do yield,
os.path.join, and in parentheses, path comma space file,
and let's actually try this out. So I'm going to type for f, on line 11, in find_music
and single quote music single quote comma space emp3 single quotes and then end the parentheses,
the right parentheses. And actually we want to do a print, f in parentheses, and on the end of that line, of course, we need a colon.
OK. So in the previous video, we passed the Asterix, the wildcard character, as part of the parameter string.
So here what I've done, is I've decided - this was on line 7 - decided to restrict the format by including
the Asterix in the filter pattern.
Now I guess you could make arguments for and against each approach.
In this case, the caller doesn't have to worry about the format of the pattern - I just passed the extension
as an argument.
It does mean that find_music isn't as flexible as it could be, though. It only allows the file extension to be
filtered.
But you've already seen two different approaches to filtering.
You can choose whichever is more appropriate for your particular application.
Alright, so that's returning all the file names relative to our current working directory, so they
all start with music.
But if we want absolute paths, we'd use the os.path.abspath function.
So let's actually just try running that
first, before we do that. You can see that we're getting our output there. That's working alright so I'll stop that.
So again, talking about absolute paths,
how would we deal with that?
Well there's actually a function we can use for that.
So what we'll do is, after the for loop, or the for file
in fnmatch.filter line, we'll put, or after I should say, absolute_path is equal to
os.path.abspath -
absolute path is what it stands for - path, and then we don't need to change the next line at all.
So we're creating,
we're creating absolute path
on line 8,
and then we use it in yielded
values.
So let's go ahead and change that. Instead of path
now - that of course, should be absolute path. So if we actually run this just to see that it does work. I'm gonna go up to the top there -
you can see I've got quite a long path on my Mac,
but you can see that, obviously, that path is working. We've got an absolute path now and the tree to each file.
So I'll just stop that.
Alright, so we now get the full path of each matching file, and that could be more useful if we wanted
to store the paths in a database so that our music player program could play the songs, for example.
Notice how we didn't explicitly create a generator object either.
You can just use your generators in a for loop and take care of creating the generator, and the for
takes care of creating the generator and repeatedly calling next on it.
Now if you did create the generator object and loop over it, though, that's also fine.
So you could've done something like this:
so my_music_files is equal to find_music,
music in single quotes, comma space and emp3.
So if you did it that way, that would be entirely acceptable as well.
And obviously, if you did do it that way, then you probably changed the loop there so it's for f
in my_music_files. We can run that and, obviously, we get the same result.
So basically, I'm saying that way is also fine - we get the same results each way.
Alright, so that's the challenge completed.
Now the reason for creating that generator, is so that we can loop through all our mp3 files
and get the song details from the tags.
Now in case you're not familiar with tags and music files, they're actually how music players are able
to display the song title, or even an image of the album, of the album cover.
Now there's a standard detailing of the ID3 tags used in mp3 files.
So I'm just going to open a browser.
Let's just have a quick look.
So this Web site contains everything you need to, everything you'll need if you wanted to write your own
programs to read and write the tags.
So if you want to do that, have a good read of the information on this site to find out the format of the
data, that ultimately, you're going to be dealing with.
We're not gonna delve into the binary format of mp3 files,
but one question we get asked often is, why do we need to read and write binary data in Python.
Well, reading the ID3 tags from the music files is one example where you'd need to do that.
And in fact, if any of your music files were missing some tags such as the artist or song title, then
you actually could use the earlier technique and attempt to work out the correct values from the file
names.
Your program could then write the tags back to the file with the correct files added.
Now working with a binary file format to deal with ID3 tags is more fiddly than difficult,
but fortunately, Ned Batchelder has created a Python 2 module that does a pretty good job of
reading
the most common tags. Now there are other Python libraries, and most of them will also let you
write tags as well as reading them. Now in the case of Ned's module, it will only read the tags, but that's a good
thing because there's no chance of destroying your music collection by working with this video.
In other words, your mp3 files will be open for reading only, so they'll be quite safe.
Now if you're interested in his Python program, id3reader.py, I'm gonna put the link to this as well and
you can go and grab it there - paste it as you can see there. That loads up. Now if you're using Python 2.7,
you can download the dot py file from the link on that page and copy it into the project directory.
Now unfortunately, it doesn't work as Python 3.
So what we've done is we've actually produced a Python 3 version from Ned's original code.
So if you're using Python 3, download
id3reader underscore
p3.py from the resources section of this video, and place that in your project directories, itself.
Now using the id3reader's module is easy.
You just really create a new object and provide it with a path to an mp3 file and we have to obviously
import the module first, of course.
Let's go back to Intellij -
I'm going to actually paste that code in. I'm going to create the new file - New, Python File - and I'm going
to call this one, which I mentioned, id3reader
underscore p3.
This is the Python 3 version of the file - and that didn't work.
Just let me copy this. OK, try again. That's better.
It looks like it's actually pasted in correctly this time, and again, you can grab this from the resources
section.
Alright, so that's the file. If we go back to our example here, our mp3scan,
we just need to create a new object and provide it with a path to an mp3
file. It'll actually do the rest, but obviously to start off with,
you've to import the module first.
Let's go to do that.
So we're going to id3reader_ p3 as id3reader -
this is the Python 3 version. Then what we do is, down here under our for loop,
we put id3r is equal to id3reader
dot Reader with a capital R, then f in double quotes, in parentheses rather. Let's do a print.
Let's expand the print. We'll do a print double quotes, in parentheses artist, colon space,
left and right curly braces, comma space, Album with a capital A,
left and right curly braces, space after the comma, Track colon space,
left and right curly braces comma space,
Song with a capital S colon, left and right curly braces after the space and then double quote dot format,
but in the format we'll start with id3
r
dot get_value.
The first one's going to be performer in single quotes, lower case, comma,
then it's id3r.get_value - dot get underscore value - parentheses single quote album
id3r dot
get_value parentheses and two single quotes, track and comma. Then the last one will be id3r dot
get_value parentheses and two single quotes, title. OK,
so that should be it. That's pretty straightforward. Our generator takes care of getting the file names for us,
and our main loop can focus on, then, using the id3reader module to read the tag information. Now
we're going to get an error with this code when I run it,
and that's because I'm pointing it to an emp3 file, and not at real
mp3 files. So the emp3 files are just plain text and don't actually contain any tag information, which
the id3reader module doesn't seem to handle too well. So I'll run it just to confirm but we should get
an error here.
Unsurprisingly, we do get one - an OSError, as you can see there, invalid argument.
And again, that's not really surprising,
and we shouldn't be trying to read id3 tags from small text files. But we really should handle that,
and gracefully move onto the next file if we get an invalid one in the list.
So that sounds like another good challenge.
Alright, so the challenge is to modify the program to handle any exceptions that are raised by the
id3reader module.
So if an exception is found, record the file name in a list and move on to the next file.
Then once the loop finishes, print out all the files that caused an error.
So that's your challenge.
So here you go with it.
Pause the video and come back when you're ready to see my version of the solution.
Pause the video now.
Alright, so for my solution, I'm going to start by creating an empty list.
And this is going to be to store the names of any problem files that we try to process.
So I'm going to come down here, and put this on line 16 down here, on line 15 rather. So I'm going to type error_list
equals left and right square braces -
so adding an empty list, or creating an empty list. Then we've still got the next line there, that actually loops through.
And what we need to do then is, under our for loop,
we need to add a try. So try colon,
then we want to put all this code, we want to indent it to the next level, like
so. Then we want to add an except, so the except is going to be
error_list.append f.
Now in the videos on exception handling, I did suggest that they shouldn't generally catch all exceptions.
Normally you should be specific and handle specific exceptions.
Now there are always exceptions -
pun intended - to any rule,
and here we need to record any files that cause a problem,
whatever the problem may be. In cases like this, it's valid to just catch every exception.
If something goes wrong, we're recording the name of the file so that the user can investigate the problem.
So catching all exceptions is actually acceptable in this case.
Alright, so the last step now is just to print out any names in our error lists.
So let's go ahead and do that.
So we're going to do, down the bottom, for space error_list space in, or error underscore
file, actually, in error_list
colon, then print error_file.
And if we run this program, we should find that
we get every emp3 filename printed, which isn't surprising, as they're all invalid mp3 files
But let's just confirm this - leave that a second to finish.
And keep in mind also, that you can actually run this with real mp3 files and see how this actually works.
so if you have got some of your own mp3 files, I'd suggest it's a good idea to actually go ahead and test
that
with those real mp3 files, just to confirm that this does actually work.
Alright, so you can see here, we've got a list of all the files that were invalid, which is basically
every file, again because of the emp3 not being a valid file.
Now the other thing you might notice, though, is right at the top, we've actually got a small number of files - you can see there -
returning None for the tags.
And that's because if a file's large enough, then the attempt to read the tags doesn't cause an error - it
just won't return the valid tag either.
And of course, that's why we're getting None there.
Alright, so that's what it looks like with invalid mp3 files,
but let's actually just run this, just to make sure it does work with valid mp3 files.
So I've actually got some on my computer, to run as a test, that I've purchased.
So you'll need to have some valid mp3 files yourself, if you want to actually do this test. And just to see
it,
I've actually got a folder here, and I've got some of my favorites,
ACDC and Adele - contrasters there. You can see, we've got apollo, Main, Users, timbuchalka, Desktop, TestMusic.
So what I want to do now is change this to point to that folder.
What I'm going to do is duplicate that, and comment that first line out, and I'm going to change this to
point to some valid mp3 files. So it's User, Users/timbuchalka/Desktop/TestMusic. It's importandt to have the
final forward slash as
well. So if we run this now, hopefully we'll get some good results.
OK.
And the reason for that is, we're getting no errors which is good, but of course, I didn't change the file type
to be mp3 because, obviously, now we are dealing with valid mp3 files.
Let's just go ahead and run that again.
And we're now getting some data, as you can see now, which we would expect to get.
So we've got ACDC and a couple of Adele songs in there as well. And you can see,
it's clearly between the artist name, the album, the track and also the name of the song. So obviously, it just goes
to show that this is working nicely.
Alright, what I'll do is, I'll just comment that out, to say you can run this using the emp3 format if you want to on your computer
and I'll just uncomment that. But you can, obviously, change that path to whatever is appropriate for your operating
system.
Alright so that's it.
That's generators,
and as you've seen, I think they're a great way to deal with huge amounts of data, without using up
all your computer's memory.
Now I'm sure you can think of all sorts of applications where generators will be useful; streaming data,
for example, would be a good use for a generator.
Now when you search for something using Google search engine,
it displays the results one page at a time. When you click to get the next page of data, that's downloaded
once you request it.
So the Google search engine doesn't download all the thousands of search results to your computer.
Now I don't have access to Google source code so I can't guarantee that they're using Python generators
for retaining search results, but as Python was developed by Guido van
Rossum was working at Google,
it's a good guess that they actually are using that. And it's certainly an application where a generator
would be very appropriate.
Alright, so I'm going to finish this video and our discussions of generators here. In the next video, we're going
to look at list comprehensions.
Now I mentioned them a couple of times in the course and now it's time to find out what they are and how to use
them.
So I'll see you in that next video.