So far, we built a crawler that we can give it a link to load the web page on that link, extract all
the links within this page, and make sure that these links are full links, that they are not external
links.
They are links that relate to the Target web page and put everything in a list and make sure that this
list contains unique links so that it does not contain any repeated elements.
All of this is good and we fixed all of the bugs that I'm aware of.
What I want to do in this lecture is extend this program so that whenever I give it a link, for example,
if I give it the link to my web page in here, I don't want it to only get me the links within this
web page.
I want it to get me all the links in here, then go on each one of these links one by one.
For example, when we go to the hacking and security, we're going to get a page with more links.
So I want it to click on Hacking and Security, get me all the links in here, go back, click on the
next link, get me all the links in there and keep going until it clicks on all of the unique links
and get me all of the links in the Target website.
This will mean that I'll be able to build a complete site map or complete map of the target website.
And using that map I'll be able to access all of its functionality and then try to discover vulnerabilities
and exploit them to gain access to my target.
So right now my program is actually doing most of the heavy work and most of this work is actually done
in here.
So the extract links from is literally just takes a URL and gives me all the links in that URL.
And then we actually are looping over the links and filtering them all in here.
So I want to create a function and I'm going to call this function crawl.
And I want this function to take a URL and this will be the URL where we will start crawling from and
then it will execute all of this code when I call this function.
Now, instead of getting the URLs from the target URL, we're going to get it from the URL that this
function will get as an input.
And then I'm just going to call this function from my code and here by saying Crawl.
With the target.
You are l.
So so far I haven't implemented anything new.
All I did is I put the code that we wrote so far inside the function called Crawl.
Now, as we know, the way that this function is going to work, it's going to take a link.
It's going to pass this link to my extract links from method, which will get me all the links inside
the page that can be accessed with this link.
All these links are going to be stored in my ref list.
And then for each link in this list, we're going to check if this link is not an external link.
And if it's not in my target links already, I'm going to add it to my Target Links list.
Now, once we reach this line, we know that this link is a full URL.
It's unique.
So we haven't seen this link in our code before, and it's a link to my target.
It's not an external link.
So this means that this link is relevant and that I actually want to open this link and get all the
links that can be accessed from the web page that this link loads.
So it's basically similar to when I go in here and click on one of these links.
So all I'm going to do is I'm literally going to call the crawl function again and this time I'm going
to pass it the current link.
Now this will call the same function again.
So it's going to call itself.
So this method is now a recursive method, and it's something that we spoke about before.
Basically any method that calls itself is called a recursive method.
So now what this method is going to do is first thing is going to get all the links in the past URL.
And then if this URL is not an external URL and if it is unique, so if it's something that we haven't
seen before, then it's going to start this URL in my target lists and then it will extract all the
links from this URL.
So basically this method will keep calling itself until it properly maps the whole website.
Now there is one thing that I forgot to change.
I changed the target URL and here to the URL because this is taking the URL as an input and I need to
change this to a URL as well.
Because that's what my function is taken as an input and now my code is ready.
So all we did today is basically just call the function this exact same function.
Once we know that we have a link that is interesting to us, we're doing this because this function
will extract any further links that might be found in this interesting link.
And as it goes, it's going to add these links to my Target Links list.
Therefore, by the end, I'm going to have a list of all the unique links in the Target website, not
in the Target page and the full website.
Now, obviously, right now I'm only passing it a specific page, so I'm going to remove all of this
and make sure that we're passing it the web route of the website.
And I'm going to be testing this against my own website.
But please, when you're writing this program, do not test it on my website because if there are many
students testing this against my website until this does my website and cause it to crash.
So please, if you want to test this, you can test it against the Metasploit websites, for example,
the utility website in here.
Just make sure that Metasploit table is running when you do this or you can test it against your own
website if you want, but do not test it against my own website.
I'm doing it here because I want to show you that this will still work against real websites.
So I'm going to run the code.
And as you can see, every time it discovers a new directory or a new path, it's actually navigating
inside that.
So you can see, for example, when it discovered this link, it was able to go into this link and then
discover another link inside it, which is the feed again in here.
When it discovered the category, hacking and security, it was able to discover there is another link
inside it, which is network hacking.
Then it went inside network hacking and it was able to discover something called feed.
Same in here.
You can see it discovered it went back to hacking and security.
So we were already in hacking and security, but we discovered network hacking.
It discovered everything inside network hacking, but then it went back into hacking and security and
it got me web hacking.
And then inside Web hacking, it was able to discover feed.
And this program is going to keep going on and on and on, entering every possible link in my website
and extracting all links in that page and then go into each one of these links again.
And it will keep doing this until it extracts all the links from all the pages in my website.
And we manage to do all of this because we structured our function in a way so that it's recursive and
so that it calls itself every time it sees an interesting link.
Now I'm going to stop this because it's going to take some time to map my whole website.
And if I run this against me till day.
So this is where you should test it.
Again, all of these websites are actually proper websites similar to any other website, and they work
using the exact same functionality.
The only reason why Matilda is much faster than my own website is because it is hosted on my own computer.
So when I send a request, the response is received much faster than when you send it to the Internet.
And again, as you can see, the program is working and it's able to extract all the links in my target
website.
Now, this is very, very important.
It's very useful because in the future we're going to start trying to discover vulnerabilities in our
target.
But before we can discover vulnerabilities, we need to have a full map of the website so that we test
every single possible page that could be vulnerable.
Now, Crawler is such as this one have other uses than just hacking.
So there are a number of programs that will do that for you in order to generate a sitemap, for example,
for your own website.
Google uses this to archive websites so that when people search for websites, they can find what they're
looking for.
And I'm pretty sure you can think of other examples, or maybe you might need to use it in a completely
different manner.