[MUSIC].
Good day viewers, in this segment I'll talk about peer-to-peer content
distribution systems using BitTorrent as an example.
So, peer-to-peer content delivery is interesting to us, because it provides an
alternative to client server content delivery using CDNs.
And, and it's an alternative that runs without dedicated infrastructure.
That's what makes it interesting. We're going to look at these systems just
to understand a little bit about how they work using BitTorrent as an example.
And I'm really just going to give you just a quick overview of these systems,
so that you see what they look like. So, here's a little bit of context for
where peer-to-peer systems came from. We've seen delivery, where traditional
client-server methods, which was scaled up over time to become
content-distribution networks. These systems are efficient, and they're
able to scale up delivery for very popular content, and they work quite
effectively. They are also quite reliable and that's
because there are managed infrastructure. People pay for content deliveries so you
can hire people to carefully administer all of these systems.
And run them well and make sure they provide good service.
But traditional models, the CDN systems have some disadvantages too and the two
real disadvantages that come to mind first, the need for a dedicated
infrastructure. It has to be provisioned and carefully
managed. All of that costs money, so we're paying
maybe quite a lot, maybe more than we need to for this content delivery.
And second, there is an element of centralized control law oversight.
Whoever has set up the CDN is really in control, and able to make decisions about
what goes on, on that CDN. Now that may present problems for some
people for many reasons. You might not be interested, for
instance, in some large internet companies such as Google and Facebook
delivering all of your content. And having a very good idea of what's
going on. So if you care about privacy,
peer-to-peer might be more interesting. Other people have different examples of
course such as downloading illegal content.
So for peer-to-peer, the real goal of this technology is to provide delivery
without the dedicated infrastructure. All centralized control we've seen in
these other systems while still retaining the advantages that we got from the CDM.
So, we still want to efficient but deliver content at large scale when
there's very popular content. Quite reliable in providing a system
which is really very usable all of the time.
It provides good performance and always up and so forth.
This all sounds great. The key question is, just how on earth
are we going to make this work? And all we have is a pool of clients who
one would exchange it for measuring among themselves with no dedicated
infrastructure to help them. Well, the key idea here is that we're
going to have to have the participants help themselves.
The participants who are called peers in the system, that's why it's P2P
technology. Peer-to-peer took off very shortly after
some of the early systems such as Napster demonstrated a peer-to-peer system in
1999. This was really used for stealing music
or downloading illegally copied music. it's since become defunct because it was
shut down. There was a flurry of activity on
different peer-to-peer systems, including BitTorrent, was one of the fairly early
ones. It was developed around 2001 and it
quickly became very popular. It's probably the dominant peer-to-peer
system today, and it's just a substantial fraction of internet content, so it's
really, see a really wide spread use. Okay, so let's just talk about some
peer-to-peer in general, and then I'll move on to bitTorrent.
So for peer-to-peer in general for delivering content, there are a few
challenges that we're going to have to deal with.
Just because of this very different setting in client-servers.
The real problem we have here is, there are no servers on which we can rely for
anything. Uh-oh, so this means that all other
communication to deliver the content is going to be between the participants or
peers. This is why it's called a peer-to-peer
system, that's the communication pattern, not client-server.
So all of these peers are going to have to organize themselves, somehow, into
coherent architecture to accomplish the task of content delivery.
Well, you might be able to work out how to do this if you just have a group of
friends that are small scale. But we would like these peer P systems to
scale up to large scale, hundreds of thousands of clients working together to
distribute content amongst themselves, for example.
At this scale, there are a few problems. Then I'll just go through them here.
The first problem is that off limited capabilities of the participants.
So, all of the participants, the peers are just you know, they're clients that
want to download the content, too. We don't have the luxury of provisioning
highly powerful high end servers that can distribute content to many clients or in
clients. Everyone's just a client in some sense.
Well, they may range in their capabilities, we don't have the luxury
of, of deciding exactly who's going to be how powerful.
So, the question then is, how can one peer, which might not be a particularly
high-end peer, be able to deliver its content to all of the other peers in the
system if they want to download that content?
That's one issue. Another issue is that of participation
incentives and peers are really helping one another, at least that's what we want
to happen. But if as a peer all you really want to
do is download the content, you know, you want to get a copy of this file we
haven't really yet worked out. Why it is that you're going to bother
sending copies of that file to everyone else to help them out since that's not
what you're really trying to do. And the third issue is that of
decentralization. With the managed server infrastructure,
you know who to contact to find out where the file is.
With a peer-to-peer system, the set of peers is changing over time.
It's just whoever wants to be involved in the download and spreading of this
content right now. So, it's not clear in this system how you
would go about finding out who to contact to get a copy of the content.
So, we would like systems that work well when they're very decentralized.
No one's in charge and all the members of this system constantly changing.
So, these are actually three substantial challenges to deal with.
Well, there are a few, there are, there's a rough lean approach which we could use
to deal with each one. [COUGH] So, to overcome the limited
capabilities of an individual peer, we can still enable one node to distribute
its content to all of the other peers by simply making a distribution tree.
This is like multicasting, a really sort of like what a CDN is doing over time.
So, if you just imagine this source node here has content and everyone wants to
get. Well, we could get to all nodes by having
the source send only two copies to these two nodes.
And these two nodes each would only send two copies, and they're actually, I've
reached all the nodes in the network. Let me just clean that up, here it is.
You can see by following the distribution pattern no node did a lot of work, yet we
got the file to everyone in the system. Of course, with this tree, I have a small
picture here but we could scale up to a, to a very large number of nodes using
this technique. So, and, and actually we wouldn't have to
go through very many levels to get to a very large system because of the way we
are growing exponentially. So, this kind of transfer is typically
done not instantaneously, but with replicas over time.
So, each of the other nodes that you sending to, like here a replica of the
file, it started off here. A replica is being stored here.
And sometime a little later it'll be transferred to these other two nodes.
Now, this approach means that in some sense the capacity of a peer-to-peer
network is self scaling with its size. If you have more nodes in the system or
you, you have more capacity to distribute the content to all of the nodes
themselves. So it has, it certainly has the potential
to be able to work and deliver content to all of the nodes.
In terms of participation incentives I'll note that the peers here play two
different roles. The pink arrows here like these arrows.
downloads to this particular node that's shaded in gray.
Let's just call this one the peer that we're interested in looking at.
So, this is really the node helping itself.
It's downloading the content that it wanted all of the time.
At the same time, this, because it participates in a peer to peer system,
this node is doing some other work. These are the blue arrows.
Oh, I forgot a pink arrow there doesn't matter.
These blue arrows from the point of view of that same node, were uploads to other
nodes. These uploads don’t help it directly at
all. They’re really doing other nodes a favor.
So, this is the dual roles and why it’s called a peer.
It’s performing, if you like, the roles of both the client and the server.
To encourage participation in a network, we can try and couple the two roles
together. Essentially, we can try and create a
system where nodes are working together so that one of them is sort of saying.
Well, I’ll upload content to you if you upload it to me.
If they both do that, then we can create a system that works that way and will
encourage a bit of cooperation and they'll all satisfy they're goals.
Of course, we just have to come up with a protocol to do that, and that's part of
the trick. And finally, in terms of
de-centralization, nodes even though the set of nodes are changing.
The, the peers in the system need to be able to learn where to get the content.
To solve this problem, we're going to use something called DHTs, distributed hash
tables. Some of you might have heard of DHTs,
maybe not. DHTs started as, an academic line of
work. In the academic community about 2001,
with work on systems called Corn and Ken, you might have heard of.
they turn out to be all of the rage in the networking research community.
There was a huge amount of work on them, there still is.
They're some of the most highly sighted network papers of all time.
Now, I don't really have time to tell you how these new algorithms work if you like
I'm sure you can read about them a little further.
Actually, in the text there's a section on the distributed hash tables in the
same section. As peer-to-peer systems which will
explain how one of them cord works. But what I'm going to tell you now is
simply what a DHT does. So, a DHT is essentially an index.
You can look this up for their name to find out where their peers are.
That's what we want to do. Now it's a distributed index, so the
index is spread across all of the peers. And it's fully decentralized and
efficient. These are the properties we want.
So, what does all of this mean? Well well, I guess I'm repeating this a
little bit, but let me just say, the DHT's an index that's spread across all
of the peers. So, it's distributed.
The index, the provides a mapping from say, the name of a file or something you
want to get to the set of peers who currently have pieces of it that you can
contact. And any peer who's in the system is by
using the DHT algorithms able to look at this index and find out who the peers are
to contact to get a copy of this file. So, an index is a very simple thing.
it turns out to be difficult to do in a way which is fully decentralized.
Even when the peers are changing it is still very efficient and fast to look up.
That's the magic of DHTs, but I won't go into it.
Okay, so let's move on to a BitTorrent to see an example of a peer to peer system.
So, BitTorrent is the main peer-to-peer system that's in use in the internet
today. It was developed starting in 2001 by
Graham Cohen. Here he is shown on the right.
finally, we found someone younger than myself to show you a picture of, that's
great. BitTorrent after it's introduction in
2001 grew very rapidly. And it sends a very large amount of
traffic because it's used to transfer very large files, movies for instance
copies of movies are moved around. As a result today, it constitutes, a
BitTorrent traffic constitutes a significant fraction of all internet
activity, so this is one of the main kinds of traffic on the internet.
And it's used for much legal as well as illegal content.
maybe I should actually say illegal and legal content.
Depending on what you think is mostly transferred with BitTorrent.
So just to switch to some of the, a little closer to the bit torrent
terminology. Data is delivered using these things
called torrents. A torrent you might imagine is just a big
gush of data that's going to give you the file you want.
The BitTorrent system transfers files in pieces, so it's going to break up a file
into pieces. Because by using many pieces in parallel,
this parallelism is what's going to give us a high level of performance.
And lets you get a fast transfer rate even though the individual peers might
not be very fast themselves. It's notable for its treatment of
incentives as part of the protocol. What this really means is that the
protocol does actually consider incentives and try and do something about
it. We'll see that in just a minute.
And in terms of decentralization. Some of the early versions weren't really
decentralized, they used an index which was stored on just another server that
you had to find and know about. But the most recent versions have now
switched to using decentralized indexing technology using a DHT.
Okay, so here's the architecture. Oh, sorry.
Actually, before we get to the architecture, here's the steps to
download a torrent. And then, I'll show you a picture and
we'll go through these steps again. This is what you do.
Step one, start with a torrent description, you're going to know what
you want to download. Maybe someone, your friend, mailed you a
link, or a description of a torrent and says check this out, it's great.
Okay, once we've got this description, step two is to find out the peers you can
contact to get the torrent, download the file.
You can do this here by contacting this special tracker to join the set of live
peers and get the list of other peers that are currently downloading the file.
This list will contain at least a seed peer.
A special node which is initialized through the copy of the content to
bootstrap this whole process. This is the older, centralized way.
Alternatively, you can use this decentralized, fully decentralized DHT
index. Look it up, for the, under the name of
the term and find a set of peers who you can contact to get a copy of this file.
Now you know who to contact the peers, and what you begin to do is start trading
pieces with them. These peers everyone's trying to download
the same file at the same time, so everyone trades or swaps the different
blocks they have. As you contact different peers, they may
have some different blocks and you may have some blocks they want.
So, you begin to trade them until you get the complete file.
And as you do so, the rule, the incentive rule is that you're going to favor peers
that upload to you rapidly. And choke peers that don't upload to you
rapidly, by slowing your upload to them. So, somehow you're providing a little bit
of feedback. People who were doing, who were helping
the system by downloading Tu, well you'll also provide them good service.
People who aren't really helping you going to try and cut off service to them.
That way if they're not helping anyone in the system no one's going to give them
good service. I talked about that a, a little bit
through the steps, but I think they'll become clear when we just look at them.
A little more talk about how they solve the challenges.
So, here is a peer-to-peer system. You can see, I've circled one peer we're
going to look at, but all of these other nodes are peers, too.
Peer, peer, peer. So, all of these peers are trying to
download the file, receive the torrent that is, at the same time.
So that they all have the same goal in mind.
One of them here, the seed peer, already has a, a special copy of the content.
That's to get the system going. Someone has to have the content after
all. Now, in the initial versions of
BitTorrent, the way it works is once you, you have a, you have a, a description of
the torrent. It's metafile that just says what it is,
what it is you're trying to download. In this, there is information about who
the tracker node is. If you then contact the tracker node,
this line here, you can get the list of live peers.
So that will have like, one, two, three, all of these nodes.
It will tell you who they are so, you know, who the peers are.
And then, you proceed with the BotTorrent Protocol.
You contact a randomly chosen set of peers, like maybe these peers, to trade
chunks with them. You'll ask what bits of pieces of the
file they have and then you will try to download those pieces from them.
And they will also be asking you what pieces you have.
And as you're talking to multiple different peers you'll get multiple
different pieces. And you'll be able to trade them with
other pieces and this way everyone will be helping everyone.
And some peers eventually will leave this system when they have the new file, but
other peers will arrive. Especially if many peers are trying to
download at the same time. You know, there'll be a life set of
people available to help everyone else. So, let's just revisit those challenges
and see how it all works. How we've met them.
In terms of dealing with the limited capabilities of an individual peer, we've
dealt with that by dividing the file into pieces that can be downloaded in
parallel. This peer here that we're looking at is
downloading from three different other peers concurrently.
That means if these other three peers are very wimpy.
Well, we're able to get a download from all of them so the rate at which we can
download is the sum of these. It can be quite fast how infrastructure
can be providing us a fairly good rate. Because of this parallelism even though
the capabilities of each individual peer is limited.
In terms of participation in centers, I talked about this technique of favoring
the peers, which are uploading to you. You upload to them.
On the other hand, we choke peers that aren't uploading to us.
So this peer here, this unfortunate peer, is shown as choked by all the other three
peers it's talking to. That's the stops here.
This is the choking. And this other peer must be choking,
because the rotten peer here with the x's.
I'll say rotten, it's not really helping out, or unhelpful might be better.
this unhelpful peer is not providing downloads to all of the other nodes.
And so, they cut it off and they're favoring other P's.
This is going to provide an incentive for everyone who has any content to trade,
actually trade it. If you don't have anything because you're
just beginning other nodes will give you basic level of service.
But your download will take a very long time, unless you're willing to help
everyone out. Okay, and finally in terms of
decentralization. But when we switch to the fully
decentralized model, the DHT index, is spread over all of the peers.
So, a small piece of it you can see here is implemented, as part of every peer,
together they're implementing. This DHT this overall index using the DHT
technology, so any peer that confined another peer can join the system.
And look up this index and will end up carry things different parts of this
index. So, it's fully decentralized and we made
our goal of knowing how to get around without having a well known central
server that provides information. Okay, so that's a P2P systems and
BitTorrent is an example. P2P systems are interesting to us as an
alternative to CDN style client server distribution content.
They they certainly have some potential advantages if they can do away with
dedicated infostructure and decentralization.
They also have some challenges to meet to do that.
It's probably not as easy with peer-to-peer technology as it is with a
managed, carefully-provisioned infrastructure.
So if we can make it work, we're in good shape.
but you know, there are still some problems to solve to make them work very
effectively. At this stage, today, CDN technologies
are probably more mature. Nonetheless, P2P technologies are coming
along, and DHT technologies, too. And over time, we're finding that they're
more widely used in systems. And these are the technologies that are
adopted in different kinds of content distribution systems.
Actually, P2P and DHT style technologies are used as part of the Skype system and
its part of Amazon's system Amazon Cloud Computing Services.
So, I think in the future what you can expect is more and more hybrid systems
which may be used on peer-to-peer style techniques along.
With a carefully producing infrastructure for example rather than having solely
owned declines to it themselves, okay?