In this hands on activity, we will be working with Lucene,
a search engine that uses a vector space model to index data.
First, we will open a terminal window and
change into the directory containing the data and scripts.
Next, we will index some text documents and query terms in Lucene.
After that we'll query using weighted terms or
boosting to see how this changes the rankings.
Finally, we will show the term frequency-inverse document frequency or
TF-IDF in terms.
Let's begin.
First, let's open a terminal window by clicking on the terminal icon at the top
of the tool bar.
Next, let's cd into the directory containing the scripts and data.
We'll run cd
Downloads/big-data-2/vector/.
We'll run ls to see the scripts.
The data directory contains three text files.
Each of these text files contains news data about elections.
Let's index these files by running runLuceneQuery.sh data.
Next, let's query Lucene for some terms in these text documents and see the rankings.
Let's query for the term voters.
You can see the rankings and scores that news1.csv ranked first,
news2.csv was the second ranking, and the third was news3.csv.
Let's query for delegates.
For this term, we see that news2.csv
was first, and news1 was second, and
news3 did not contain the term at all.
Now, let's query for both terms, voters and delegates.
In this result we see that news2 was ranked first,
news1 was ranked second, and
news3 was ranked third.
Now lets use query term waiting or
boosting to increase the relevance of voters.
I can do this by ensuring voters carat 5 delegates.
The carat 5 notation is a syntax for
Lucene for boosting.
When we run this, we see that now, news1 is ranked first,
news2 is ranked second, and news3 is ranked third.
Notice this is different from the original query with voters and delegates,
where news2 is ranked first and news1 was ranked second.
Now let's look at the term frequency, inverse document frequency or TF-IDF.
We'll enter q to quit this and
we'll run Lucene TF-IDF SH data.
Let's look at the TF-IDF for voters.
You can see that it ranked number 1 for news1.
Second news 2 and news 3 is last.
Lets try delegates.
Here we see that news 2 had a higher score than news 1,
and news 3 is not listed because news 3 does not contain this term.
Hit q to quit.