So in this lecture, we will continue looking at our previous notebook.
But this time we'll be using a library to do the work for us.
We'll begin by installing a package called Sumie, which contains an implementation of text rank.
Note that the results will be different because it uses a different method of computing similarities.
The next step is to import a few classes from the Sumie package, including summariser is a tax parser
Nana tokenize her.
At this point, you can guess why these might be useful.
So the next step is to show you how to create a summary.
We first begin by creating an object of type, text, rank summariser.
Now, unfortunately, this doesn't take in text directly since, as you recall, A.P. can work with
many different languages.
So the next step is to create a plain text parser object, which will take in our text and the tokenize
their object.
Note that when we create the tokenize or object, we pass in English, which will be used to tokenize
the document correctly since our document is in English.
The final step is to generate our summary by calling the summariser parsing in the document and telling
it how many sentences to return.
Note that the document is passed in by calling the document attribute on our parser object.
In addition, and notice that this library uses the same method we used earlier in the section for choosing
which sentences to keep, as you recall.
There are many ways of doing this, including choosing the top end sentences, the top x percent of
sentences, the top and words, and so forth.
Thus, it just so happens that the person who wrote this package decided on the same method as we did
in this course.
OK, so let's print our summary.
So note that the summary is a tuple of sentence objects, which makes it hard to see since it goes off
the screen.
The next step is to print our summary by looping through each sentence, casting each sentence to a
string and then using our rap function to keep the printout within a limited number of columns.
OK, so this time the summary is the 21 year old singer won the award for Best Female Artist, with
Australian Idol runner up Shannon Noel taking the title of Best Male at the ceremony, as well as Best
Female.
Goodrem also took home the Pepsi viewer's choice award, while Screen Day bagged the prize for Best
Rock video for American Idiot.
The Black Eyed Peas won awards for Best R&B video and Sexiest VIDEO, both for Hey Mama.
Local singer and songwriter Missy Higgins took the title of Breakthrough Artist of the Year, with Australian
Idol winner Guy Sebastian taking the honors for Best Pop video.
The ceremony was held at the Luna Park Fairground at Sydney Harbor and was hosted by the Osborne family.
So interestingly, this summary seems to work pretty well.
So perhaps TFR TFN cosine similarity is not necessarily the best way to compute similarities, although
you would still want to test this on other documents, to be sure.
Now, this might be a bit of a surprise, but our old friend latent semantic analysis makes an appearance
once again in this blog.
We use an LSA based summariser, which is also included in the Sumie package.
Note that all the summariser is in this library have the same API, so I want to explain the syntax
again.
OK, so this time the summary is as follows I won't read it out, but again, this seems like a pretty
decent summary.
So the next method I want to show you in this lecture is even simpler.
Instead of having to deal with parsers and tokenize hours and so forth, Genzyme has a function where
you can just pass in text and get a summary.
Now, interestingly, Jen SIM summariser also makes use of text rank, which makes it very appropriate
for this lecture.
It happens to use a variation on the similarity function since, as mentioned, you are free to choose
any method you like for comparing how similar two sentences are.
Unfortunately, the documentation doesn't specify which variation they used, but they do link to the
paper on which their method is based.
In fact, this paper lists the TFI, TAF and Cosine method, which is the similarity function we used
in the previous lecture.
So check out this paper if you want to learn more.
Note that we've also included the arguments for this function here, since they relate to what we discussed
earlier in this section.
In particular, you'll recall that there are multiple methods of choosing how many sentences to include
in the summary.
These arguments reflect some of the options I previously discussed.
First, we have ratio, which lets us choose a proportion of sentences to include, for example, 10
percent or 20 percent.
The next possibility is a word count, which lets you choose how many words to include.
Note that if you specify one, you can't specify the other, since both of these can be used together.
In any case, as promised, you can see that this is just a single function call.
Well, we pass in some text and get a summary back.
OK, so the result consists of two sentences.
Interestingly, this is not that great of a summary.
Personally, I prefer the summary we generated earlier, which included the names of multiple bands
that won awards.