All the decisions are made by community assembly, and the decisions are enforced by the locally chosen officials.
When some public works need to be done, like making a new road or building a school, the townspeople work together voluntarily.
This is called tequio.
And finally, as a way to celebrate life and traditions, every town has its party or fiesta, one of the most important moments of the year.
This sign...
So I want to show you...
Well, this is part of Oaxaca, the town where the cell phone network is working.
This is very common in Oaxaca to find.
This sign is the best example that you can find in the entrance of some towns.
The private property doesn't exist in this community.
Prohibited to buy and sell community land.
Yes, it's great.
It's great to find this in some part of the world.
In 2006, there was an important social uprising led by the state's rural teachers.
More than 60,000 of them went on strike.
This strike was brutality repressed, leading to a general strike and rebellion of the population and the undertaking of the city of Oaxaca for more than six months.
Who of you know who was Brad Will?
A few people.
Okay, let me tell you.
Brad Will was a New Yorker-based indie media reporter who was assassinated by paramilitary during this rebellion as were dozens of teachers and activists.
Meanwhile, their corporate media told another story, which led over 10,000 women to march and then take control of the 12 radio stations and the state television stations, where they transmitted the truth about the movement and the injustice for more than one month.
From this point on, people in Oaxaca began thinking about radio spectrum as an integral part of the people's territory, in which the right to communicate is defended and exercised.
In fact, Oaxaca has the highest concentration of low-power community FM stations in all of Latin America and probably in the world, with over 60 unlicensed stations operating all over the state.
Mexico's own constitution, as well as international pride, allow for and legitimize the rights of fierce nations to establish and operate their own media infrastructure.
So this is a very short introduction where we are and why we are there doing this project, the cell phone network, community cell phone network.
So, Peter Bloom, please.
Okay, so, hi everybody.
Thanks for sticking around through the technical problems.
My name is Peter Bloom.
I am the general coordinator of RISEmatica, which is a group, an organization that I started some years ago, three or four years ago.
It's kind of hard to know when it started because it's not an official organization.
We're not registered anywhere.
I asked Maka to give us some context because I think it's important that we draw some lessons from what we do in Oaxaca and the story of Oaxaca.
And also in the context of many of the talks that happened this morning that I had a chance to attend.
And the first one is we need to have our own infrastructure.
I think that that's very obvious.
Everything else that we do is going to be mitigating a problem that we really cannot resolve unless we own, operate, and have full control over our own infrastructure.
In the case of Mexico, it also means being able to actually have infrastructure.
Whereas here, pretty much everybody is connected, sometimes more so than we want to be.
In Mexico, we have a different problem, which is that many people who want to be connected to the Internet, to a phone, etc., are not able to because where they live is deemed to be too rural, too isolated, etc., for the companies that are in charge of this to go and do it.
So if they don't do it themselves, no one is going to do it.
Secondly, if we want to operate and own our own infrastructure, it requires working with communities.
It requires working with people, human beings.
Otherwise, we will go back to the same sort of corporately controlled networks that we have right now.
We won't really have changed the situation.
So those two things, I think, are important, and it's a huge lesson that we as Rhizomatica, and myself personally, and Maka as well, and the others who work on the project with us, have drawn from our experience in Oaxaca, working in community where there are still community assemblies,
places where there's no private property.
It makes you think differently about how we should be building our own networks.
So a little bit about Rhizomatica.
Like I said, we started a few years ago.
It's basically four of us, two of whom are here, they're right here, Shabi and Tele, who are in charge of all the technical stuff.
And then we have a lawyer.
It's always important to have a lawyer.
I think the guys from the last presentation can speak to that as well.
Always have a good lawyer if you want to do anything.
So it's the four of us.
We've been able to do what we've been able to do, which we're quite pleased with, but there's a lot more to do.
Without any kind of external funding, we don't have the backing of any government, of any agency or anything like that.
We don't have money from USAID or any of those things.
So this is all a bunch of out-of-pocket money.
It's selling t-shirts, of which, of course, I brought three.
So we can start the bidding at $400 per t-shirt.
I honestly thought that I was going to be speaking to ten people or something.
So I'm happy that everybody's here.
Sorry I didn't bring t-shirts for everybody next time.
But literally by selling t-shirts, good old crowdfunding and stuff like that.
And then the communities help us a lot.
They feed us.
They put us up.
They pay for and maintain the infrastructure.
So that helps a lot.
But, again, you can do a lot with no money.
And I think it's really important that that lesson also sort of come out from the work that we've done.
Right now we operate three networks, three rural networks in Oaxaca.
And we sort of collaborate with another organization or another company who runs two others.
So there's five networks, five of these autonomous networks.
There's approximately 1,000 total users who use these systems every single day for their critical communications, both inside their villages and with the rest of the world.
So from that, it's fairly obvious what we do.
We install relatively low cost and easy to set up and manage networks based on open source software and many times open source hardware.
I actually want to show you what one of these looks like.
So it's not very light, but it's quite small.
This is a base station.
And not only is it a base station, it's also a BSC.
It has all the amplifiers.
This is an entire cell phone network right here, sitting here.
You put a couple of antennas on it, and you're ready to go.
This is it.
There's no other parts.
There's no other thing going on.
You connect it here through an Ethernet cable to the quote-unquote Internet, and you're off and running if you want to have long distance calls.
Otherwise, you can just set up a local network, and you're good to go.
So this is a huge step.
In how or what a cell phone network can be.
It's no longer about huge multinational corporations.
It's no longer about proprietary knowledge, code, and all of that.
There's an open source version of all of this stuff now.
It works fairly well.
It could still be better.
But this is all we need.
And one of these, without ruining our friend's business...
Maybe I won't even get there.
We'll go into that.
It costs us about $10,000 to set up the entire network.
It can sometimes cost us even less than that.
And if you look at what it costs a big company to do the same, especially in a rural area where they're putting a diesel generator, they're putting up a huge fence because they're worried about people stealing their shit.
We don't have to worry about that because this is community property.
So who's going to steal from themselves, right?
I mean, there's always somebody.
Generally speaking, it's fine.
So we don't have to put up a fence.
We don't usually have to put up a generator.
We'll just go right into grid power.
We can use solar power, et cetera.
We don't have to send some asshole technician out to fix the stuff because the people are there dealing with it themselves with our help.
So all these costs just go way, way, way, way, way down.
And now we can put up one of these networks.
There should be a picture at some point with some antennas on a piece of bamboo.
So one of the communities that we work in, they said, look, we don't have the money for a tower.
And they came out with a 12-meter piece of bamboo.
So we said, okay, great, we'll mount on that.
So there's a cell phone network somewhere in the world mounted on a bamboo pole.
Which makes me, before I go to bed, feel, I don't know, just makes me feel better.
Like I said, these networks are owned and managed by the community.
So that $10,000 approximate investment comes from the community.
We don't give anything away for free to anybody because we don't have any money ourselves.
And even if we did, that's not our philosophy.
This project isn't about donating cell phone networks to people.
It's about helping them build their own networks.
I also feel that by paying for it, people are a lot more likely to take care of it.
Generally speaking, that $10,000 amount is within the community's grasp.
There are other situations where the community can't come up with all of it.
So we try to work out some kind of plan with them so they can pay for it.
The other part of that is, again, that they manage it.
So they're in charge.
The folks in the community that we sort of train and help to understand how all this works, they sign people up for service.
They cancel people's service.
They do the credit top-offs and all of that.
So we don't have to do that.
And the community does that.
They do it in the way that they feel is best.
They have community assemblies around how they want to do that.
They get to decide how much they want it to cost, for example, per month, which every time we tell people that, they can't really quite believe that they get to decide how much they pay for their network.
It's kind of like when you're in school and the teacher says, you know, you can decide on your own grade or something.
It's like, great.
Then, so we basically set up a local network.
You know, this thing transmits locally.
It has a radius.
It really depends on all different kinds of things.
If anybody knows about RF, it's not even worth trying to explain.
But we've gotten some really good results going over mountains and stuff.
Sometimes it just gets stuck, like we were seeing.
Like you can see from the pictures, we work in the mountains.
Lots of foliage.
Lots of humidity.
So nevertheless, we get very, very good coverage from these machines here.
So we set up these local networks.
And then these local networks all participate together with my organization and others.
And then sort of an umbrella organization of which all of the communities are voting members.
They elect people from the village to be part of the organization.
And we sort of decide about and talk about things that concern the network of networks, as it were.
So there's local ownership and local control of each network.
And there's also this sort of federation, if you will, of all of these networks.
And there's another reason for it, not just because it's good practice and nice and participatory and all that.
It also helps us build a political base to defend the work that we do.
And that's another big lesson that we've learned from doing community FM, is the government is always willing and able to come and shut your network down unless you have some way to defend it.
So just finally on that, all of the networks are run in a not-for-profit fashion or non-commercially.
So there is money that changes hands, but it's very inexpensive.
So the idea is to have something that breaks even, doesn't throw off any real profit, but allows the village to recoup their investment.
It allows them to pay their ongoing monthly costs, which are things like electricity and internet, possibly pay someone in the village to help administer.
They give a small piece of the money to us.
When I say a small amount of money, we're talking about $2.50 per user per month.
That's the total what the user pays.
So that's 30 pesos per user, of which 20 pesos stay in the community and 10 pesos come to Arizomatica.
So we can eat and keep our car in working order and come and visit them.
So why did we do this?
And why in Mexico?
Or how are we able to do this?
Mexico is a particularly interesting case, I feel.
Not just because I live there and I know it, but I think sort of objectively it's interesting.
Because of this huge telecommunications monopoly that exists there, that has created the richest man in the world.
Maybe he's the second richest, I don't know, he goes up and down.
But the richest, let's call him the richest man in the world, sounds better.
Carlos Slim, who owns this huge monopoly, covers broadband, covers fixed line telephony, covers cell phones.
So the other side of that coin, having this monopoly, having the richest man in the world is there are 50,000 rural communities in Mexico and something like 20 to 30 million people who don't have a cellular signal where they work.
And this monopoly was allowed to exist, it kind of came into existence due to a bunch of neoliberal reforms that happened in Mexico in the 1990s where they basically started selling off all the public infrastructure.
And in the case of Telmex, Telso, which is his company, they didn't even really sell it off, they just kind of gave it to him.
And then after that, having forced his company or any of the other cell phone companies, there are three or four other operators, having forced them to go and cover these communities as is the case in other places where the regulatory body actually does what they're supposed to do.
And again, like I said, if the communities don't build their own infrastructure, it's not going to happen.
The infrastructure isn't coming.
We go into a lot of communities and they show us letters that they've sent to the government, to the companies, essentially begging for them to come and connect them to something, maybe a fixed line telephone, maybe cellular service.
At this point, people are more looking for cellular service.
So this is kind of the context in which we work and why we do what we do.
I mean, I should also preface this by saying I fucking hate cell phones, personally.
I find them to be really annoying.
It's like super annoying that everyone just looks at their cell phone all day.
So yeah, so I should just probably stop talking.
Yeah , no, so the idea is how do we do it?
How do we do this kind of thing without making it obnoxious like it is in New York, for example?
And we do that by working with the communities and coming up with different ways in which people... so people have already recognized that it's important to be able to have access to a telephone, which is a very separate thing from being obsessed with your telephone or getting lost inside of your apps and all that.
So people want to have access.
We don't go and work anywhere where people haven't explicitly invited us.
So we don't go and convince communities to let them work with us.
People have to come and find us, kind of like the A-team in that respect.
And we try... we do a lot of hiding since, like I said, we're only a four-man operation.
We don't really always have the capacity to, you know, attend the needs of everybody.
So people want to be connected.
But at the same time, we work in indigenous communities where they're still traditional, as you would call them, ways of life.
People maintain certain level of community unity, like there's techo community work.
Decisions are made through assembly and all these other things, which I think are really cool.
And we do our best to work together with the community so we don't ruin those things.
So we don't change them or swap them out for, oh, everybody can attend the assembly via SMS or any stupid shit like that.
We're not into that.
We're really... that's not where we're trying to go with this.
More to the point, the community tries to give us feedback.
So, like, one thing... it's really simple stuff sometimes.
The community basically said, okay, we want to have the calls.
So once you own your own network, you can do whatever you want, right?
So the community said, they said, look, you know, we want to be able to limit the length of the calls.
One, so that the network doesn't get congested.
But also so that people still go and talk to their neighbor.
They still go and visit their comadre, you know, down the road.
Because otherwise people could just get on the phone now and speak to each other for free for hours and hours and hours.
And, you know, the community life, the sort of interesting part of what's still happening in these communities would slowly begin to dissipate.
So we really don't want to be part of that at all.
And instead look for ways that we can actually strengthen that and move forward technologically without having to, like, radically alter the social relations in the place.
So we're at a hacker conference.
And I guess what we do, we look at what we do as being based on sort of three main hacks.
A technological hack, a legal and public policy hack, and a sort of social organizational hack.
So from the technological side, it's quite complicated.
But again, we set one of these things up.
It sounds really easy.
If you want to learn a lot more about this, you should come to the workshop tomorrow at 1 p.m.
on the sixth floor.
It's in the thing.
You'll find it in there.
So come to that.
It starts at 1.
We will be happy to allow you to listen to Edward Snowden speak during the workshop so that you don't leave en masse at 2 o'clock from the workshop.
So please come.
You can learn about all the software that we've written to actually make it possible for a villager or someone with very few computer skills as it were to be able to manage a cell phone network.
So we basically set up one of these.
We work with wireless ISPs.
We try to stay away from satellite because it's quite expensive and not very reliable.
Huh?
Lag, yes.
There's lag.
There's latency.
So we generally work with community wireless ISPs in the regions that we are.
We get a dedicated line, plug this guy into that directly, and do all of our outbound stuff via VoIP.
So that's the technological stuff.
I'm not going to get, like I said, I'm not going to get too far into that because you can come and hang out with us for four hours tomorrow and learn all about it if you're really into that stuff.
So I think the biggest hack or whatever or sort of our biggest success so far has been on the legal and the policy front.
We actually operate legally.
We have a license from the government.
That wasn't always the case, but they kind of were pressured into giving us a concession.
So this actually puts us on equal footing with Carlos Slim's company, with Telefonica, who's one of the largest telecommunications companies in the world.
We have the same rights and obligations as they do.
We have this concession to operate in five states in southern Mexico, which cover Oaxacans and some of the surrounding states, not Chiapas, unfortunately.
And the total population of these states is 21 million people.
So we have the right to provide service to 21 million people.
We try to stick into areas where there's no coverage, so that obviously reduces that number somewhat.
But we began, and this is another lesson that I think was important for us to reinforce, not exactly learn, because we've been doing this for a while, but basically this idea of it's much better to ask for forgiveness than to ask for permission.
When you try to deal with a national regulatory body, be it the FCC here, be it the EFETEL now in Mexico, they're very good at waiting you out.
So you come, you say, hey, you know, we want to do this thing, we think it's really great.
They say, oh, yeah, it's really great.
We'll get back to you.
Right?
And, you know, maybe they won't get back to you, maybe they won't, but years go by.
So what we did is when we went to EFETEL, we went with a signed petition from 30 communities exercising their constitutional right and their right under international treaty, the UN Treaty on the Rights of Indigenous People, ILO, Convention 169, all these other things.
And said, look, we have a legal claim that we feel is quite strong.
We also found a clause in the Mexican telecom law that had to do with redistribution of unused frequency.
So what happens with cellular frequencies is they assign frequency to a company for an entire huge area.
So, like, in our case, we asked for a specific region of one state, and they gave us five states.
Region 7.
Like, we didn't want region 7, but we're happy to have region 7.
We wanted just this one little part, right?
They said, no, we can't do it that way.
We said, okay, great.
So this is what happens with the large companies.
They get their, you know, their frequency use permission, and then they don't operate in many areas.
So there's a clause in the Mexican telecom law.
As of a few days ago, they just changed the law, so we'll see what the new one brings or what it has in store for us.
That allowed the government to reassign unused spectrum for social purposes.
So we were the first people that, I guess, found this clause or decided we were going to use it.
So we went, said we have this legal claim based on, you know, the law of this particular sector, based on the Constitution, based on international law, and we're already doing this.
So you should just let us continue.
And simultaneously, so like the fines for this are like, I don't know, hundreds of thousands of dollars.
But what we did was we tried to put the regulatory body in a really tough position, which was, number one, come and shut our network down.
Like, come into this indigenous community where they've made a decision by consensus, where they've used their own money to connect themselves due to your negligence, and come and shut down their network.
Like, that's going to look really great, right?
That's going to look really good when it's on the news.
And thankfully, we were in the news a bunch of times, kind of accidentally.
It really wasn't something that we were asking for or were particularly happy about, but it ended up being very helpful.
And so the government was like, oh, these guys are like, oh, they're in the news, they're on the BBC, they're here, they're there.
It's not a good idea for us to go and shut this down.
So we came with the legal part, you know, we came with the olive branch and we came with the stick at the same time, right?
And that kind of put them in a position where they said, okay, go for it, we're giving you a concession.
So we have a concession, which is really great.
And that allows us to pretty much do whatever we want.
We can't interfere with the other operators, but we can run the service.
One stipulation really is we have to run a non-commercial network.
So we are forbidden from generating any kind of serious profit, because that would be offensive to the large operators who have been doing this forever with no real kind of competition.
So we don't want to obviously upset them.
And the regulators are actually quite honest about them, that that's the only thing that they really care about, is having the lawyers.
Some of these guys are like married to people that work for the companies.
I literally had one guy say, if I do anything against a big company, my wife won't sleep with me.
Because she works for whoever it was.
And I said, okay, it's hard to, you know, try to compete with that, right?
Like, I'm not going to sleep with you.
The other thing that we try to do is, or that we sort of, we try to think a little bit more deeply about with the communities themselves, is about thinking about spectrum.
Because the real bottleneck to all this is spectrum.
Like, there's VoIP.
Like, you can do amazing stuff with Wi-Fi, because it's open spectrum.
I did like a really brief analysis of the way that they distribute frequencies in Mexico spectrum.
0.14% of all of the usable for telecom, telecommunications and radio and TV, et cetera, satellites, everything, 0.14% of it is open.
You know, you don't need a license.
0.14%.
It's a joke.
Everything that, you know, everyone in this room wants to do has to be crammed into 0.14% of the spectrum.
It's ridiculous.
It's stupid.
And part of what we are trying to sort of make the claim to the government is to say, look, spectrum is a territorial right.
So the issue of territorial rights, especially in these indigenous communities, is very strong.
It's in the Constitution.
You know, if you're an indigenous community, specifically an indigenous community in the case of Mexico, you have a right to your territory, to the physical land on which you live, to the things that are on that land, the flora, the fauna, the water.
And, of course, the neoliberal state in Mexico being a wonderful example of that, they contest that.
They want to privatize these things.
They want to take them away.
They want to do mining.
They now want to do fracking in Mexico of all fucking things.
Yeah, yeah.
And so there's this like circular logic that I think has been the way that, you know, generally governments, nations have expropriated land and resources from people, indigenous people and other people.
It's a circular logic.
Right.
So they say, OK, the state is the sovereign and therefore we have sovereignty over everything in this territory, including the spectrum.
Right.
And they just, yeah, this is ours.
We take it by sort of act of God, essentially.
Right.
So the indigenous communities that we work with and all throughout Mexico and in many other parts of Latin America are pushing back against us and saying, why do we not have the right to use the spectrum that's sort of present, if you will, which isn't even necessarily correct from a sort of physics point of view.
But if you think about it that way, you know, why can't we transmit on these frequencies in our own territory?
Like, how do we not have that right?
If I have the right to my land, why do I not have the right to the spectrum?
There's an interesting case of this that's actually been successful in New Zealand.
The Maori people have been able to actually claim spectrum as an ancestral right.
And we're given a concession, in fact, to operate a cell phone network, which they then, I think, farmed out to some private company.
But anyway, what are you going to do?
Right .
But it gets back to my first point.
So we're kind of moving, trying to move this idea through the national regulatory body.
We're trying to move it at the ITU, which is like made up of total, let's not even go into it, but not the most forward thinking people in terms of spectrum or all that.
But we're trying to push on this to say, look, we want to have the right to our spectrum so that we can use it how we want to use it.
Right.
And especially if it's not being used, why the hell shouldn't we be able to use it?
So that's really fundamental right as far as we're concerned.
So that's the sort of political and legal hack.
We have this concession.
We have this concession for the next couple of years.
Now with the new telecom law, apparently we can extend that concession for another 15 years.
So we're hoping to have this for the next 15 years.
The interesting part of this is that we now have exclusive frequency access rights solely, essentially.
We don't have any, quote unquote, competition from any other organizations like ourselves, which we're actually hoping happens.
We're very open about all of our knowledge.
We're releasing our software actually here at Hope.
So anybody that wants to do what we're doing is more than welcome to come and join us, do their own thing, et cetera.
So the second hack I'm going to talk about really quickly is the social part, the organizational part.
And it gets back to what I was saying at the beginning, which is, okay, we build our own infrastructure and then what?
It's really easy to just go and buy a SIM card and stick it in your thing and that's the end of it.
You don't think about it.
You don't have to worry about if the tower falls down.
You don't have to worry about if a rat gnaws through the coaxial cable and on and on and on.
But when you own the stuff, when you operate it, when it's the sole means of communication for your community, you do have to worry about that.
And you have to develop mechanisms to deal with it.
And this is so important for any kind of infrastructural project is who's going to manage it, who's going to be in charge of it, and how is that going to happen?
So we opened up the talk speaking about Oaxaca in the context.
And there's something in Oaxaca called usos y costumbres, which is essentially traditions, customary law.
It's the way that things happen, the way that decisions are made, the way that the territory, the land itself is governed and all of that.
That for us is a huge resource because there's essentially an organized base, a social, organized social base that allows us to do our work.
So we don't have to go in and say, hey, guys, you should really work together.
And people are already doing that.
They're already building their own roads.
They're operating their own schools.
In many places, they're operating their own police and auto defense forces.
So, like, we don't need to explain to anybody how to run their own business.
And in fact, we learn so much from them.
And a lot of what we do is based on what we've learned.
So having this idea of usos y costumbres, this sort of community life, is incredibly helpful when thinking about, OK, who's going to manage this infrastructure?
That system is already in place.
It's fairly strong.
It's reliable.
When people make a decision, take a decision, they do so in consensus.
And that's a decision.
There's no like, oh, you know, they make the decision and that's it.
So that's really helpful.
And we sort of come up with a really simple division of responsibilities between ourselves and the communities, which is they're in charge of purchasing, managing the network.
They buy it.
They own it.
They have to take care of it.
If someone comes and steals it, that's their problem.
If it gets hit by a lightning bolt, that's their problem.
If, like, a bunch of squirrels somehow unscrew it from the pole and carry it off, that's their problem.
And they also, of course, have input on the design of the network, the services that run on top of the network, and, like I said, the price.
Meanwhile, we at Verizon Atica, we mainly do capacity building.
So we help put up the network.
We show folks how it works, the really simple web interface, which you can learn about tomorrow at the workshop.
And then we help them maintain it.
So if anything sort of goes wrong, like if something chews through the coaxial cable, they call us and say, something chewed through the coaxial cable, can you bring another one?
Or, hey, this frickin' thing isn't working, we don't know why.
We can log in remotely, check it out.
If it's something we can't fix remotely, we drive up there and figure out what's going on.
Which we do a lot more than we wish we had to.
But things are moving along.
So those two things, sort of using existing, pre-existing community resources.
You know, for example, the tower will go to the local welder, and the local welder will build the tower, or the local bamboo farm will give us his bamboo, as the case may be.
We really try to involve local people in all this, because it makes it less expensive, and it makes it much more sustainable, not only economically, but socially and in every other aspect.
And this idea of having a really clear division of who does what, who's in charge of what, and making sure that all of that is clear from the beginning.
I'm going to wrap with that.
If there are any questions, I'd be more than happy to take them now.
I think we still have a few minutes, even with our snafu on the projector.
So, you know, however you want to do it.
Can you tell us more about the hardware that you've got there, and how that came into being?
Yeah, I mean, you can actually speak to the folks from Fairwaves who built this.
Fairwaves is a company based in Moscow and Boston.
They build open-source GSM hardware that runs open-source GSM software.
They've been around for some years.
We met.
They've been very helpful to our projects in donating hardware and thinking through things with us.
So you can look up them.
There are some other vendors that we work with.
One is called Nutac, based out of Canada.
There's a couple of options.
How many people do you have on the network, or the five networks that are running?
How many users?
Yeah.
Around 1,000 people.
1,000 people.
And our network handles, per village, it's somewhere between 4,000 and 1,000 calls per day, each network.
And, I don't know, thousands and thousands of SMS.
So since the use is unlimited, in most senses, people use it a lot.
It just gets used and used and used, which is great, because they own it, so why not use it, right?
Can you talk more about the self-governance aspect of the communities, and in particular, what might U.S.
radicals in the U.S.
learn from them?
Yeah, I kind of get asked that question a lot.
It's an interesting question.
We have a fundamental problem here, and it gets back to what we were talking about, private property.
It fucks up a lot of attempts to try to duplicate, replicate the sort of social organization that happens in these places.
It's much different organizing around property that you own communally, that literally nobody owns except for the community, than it is everyone having their own property.
It's kind of hard to explain this in, you know, a couple minutes up here.
That makes some of the lessons not totally applicable here.
You know, this idea of individualism that we have, rugged individualism that we have in the U.S., is a huge barrier to social organization.
It's not insurmountable, but I think a lot of what we see in Oaxaca, it's very difficult to try to replicate that here, because there are some really fundamental things at the base of that, that do not exist here, or that exist in very, very small sort of pockets.
So, I'm not really sure.
Huh?
Russians tried.
Russians tried , exactly.
Our Russians friends.
Russians tried.
There you go.
How about handsets?
What do they use for phones?
Okay, so this is really, you know, why do we do cellular?
Why don't we just put it up in, you know, regular landline phones?
Turns out, even though there's no coverage, most people own a handset.
And a lot of people actually have a smartphone.
Why?
Because it's useful as a camera.
It's useful as a flashlight.
You listen to music.
You watch videos.
So, like, for example, when we listen to the radio.
When you, when we went to the first community in Talea de Castro and set up the first network, nobody, like, we had come to an agreement with the local authorities, but nobody knew we were coming that day.
Sort of happened, you know, we just showed up one day and set up the network and saw over 700 phones switched on in the village of 2,500 people.
Where there's no network.
There was no network there.
700 phones were on.
And you say, okay, what the hell?
Because people were using them for other things.
And I guess they hadn't turned it into airplane mode if they had that.
Whatever.
The point is, a lot of people had phones.
And from there, we saw that about half of the population already owns a handset.
They go to the, they go into the city, they use it.
We run, essentially, an open network.
So, we're SIM agnostic.
So, it doesn't really matter what SIM you are.
We didn't want people to have to go and pay to get their phones unlocked.
Because we think the fact that the vendors lock the phones is bullshit anyway.
So, why have to spend more money to unlock your phone?
So, our phone accepts any SIM from any company.
And you just get on the network.
And you get a message with your extension.
You get a five-digit extension.
You take that to where the administrative person is, usually in the center of the village.
You say, here's my extension.
You give them 30 pesos, you know, $2.50.
And you're off and running.
Two quick questions.
Have any of the communities wanted phones to not work from, like, let's say midnight to 6 a.m.?
Like, any ideas of ours?
And second, what's the attack vector if the NSA wants to listen to their phone calls?
Do they have to crack the GSM or the VoIP?
We haven't had any requests to shut the network off ever from, you know, people at a certain time.
We do see a lot of late-night traffic between teenagers that I know, you know, people are probably concerned about.
There's a huge concern about people calling other people's wives, actually.
When the guy's, like, off in the field working, that's, like, the big question that we get.
But no, we basically don't.
We essentially, we talk about it with folks until they kind of understand that it's not really beneficial to be monitoring people shutting off the network and all of that.
So since we don't have any power constraints at night, we just leave it running at night.
It doesn't really draw very much power.
In terms of the NSA, et cetera, they would have to physically be present to capture the traffic locally, which, frankly, since we're sim-agnostic, is unencrypted.
I know, boo.
But someone would have to physically show up in these communities and listen in, which would be, I feel like, would be quite obvious to the people in the community.
And again, this gets back to this idea of community ownership.
People will defend their network.
We see this with community radio all the time.
You know, the unmarked vans pull into these communities, and what happens?
The community comes out and is like, you want to come in here?
We're going to fucking kill you, basically.
So it's your choice.
And the idea of having everybody organized in this umbrella organization is to say, you don't have to shut down one network.
You have to shut down hundreds of networks, thousands of networks.
You can close down Rhizomatica.
It doesn't matter.
You're going to have to go physically to every single community and get into a physical altercation, essentially, in every single community to shut this network down.
So if our concession expires, who cares?
Like, the communities own the stuff.
They're going to keep operating it.
And that's kind of like the real politic of it.
They already run illegal radio stations.
What's another, you know, $100,000 fine, right?
In terms of the NSA, so right, so someone would have to go there and do it.
The really vulnerable part is all the VoIP interconnection, which, unfortunately, pretty much all runs through the U.S.
Boo again, but that's how it works.
There's no exchange point in Mexico, so kind of all of the Internet traffic pretty much has to come through here, so it sucks.
But...
Have you had any resistance to connecting you to the PSTN, or are you connected?
Connecting people or the network itself?
The networks.
Should I stop?
Yeah, one minute.
One minute.
Okay, quickly.
We use a VoIP provider that connects us to the PSTN.
People choose if they want to do long distance or not.
They can receive calls always.
If they do a top-up, they pay whatever the exact price of the VoIP call is, which is like hundreds of times less expensive than a satellite phone or whatever they normally get.
So they're receiving U.S.
numbers?
No, we give them one Mexican DID number assigned to the village.
You call the village.
You get a PBX.
Answers says, hey, welcome to this village.
Who do you want to talk to?
You talk to that person.
Rings on their cell phone.
Thank you very much.
If you want to buy the T-shirts, like I said, starting bid is $400.
No, I have three T-shirts.
Come and find me.
If you don't want T-shirts, it's cool.
Workshop is tomorrow at 1 o'clock.
Edward Snowden will be available streaming in the large room on the sixth floor.
Thank you very much.
Enjoy the rest of your afternoon.
Stick around for the following workshop about a different take on community and DIY cellular.
Thank you.
Thank you, Peter and Maka.
Again, their workshop is tomorrow at 1 o'clock on the sixth floor.
A couple quick announcements.
First off, we do have openings.
Okay.
The last talk dealt with building a persistent network for community.
This talk deals with building a network for an ephemeral community.
Ladies and gentlemen, Johnny Diggs, Willow Brew.
So the thing behind us is going to be a little bit clipped, and so I hope you don't judge me on my design skills.
I promise I centered everything perfectly.
It's technical issues.
It looks really perfect right here.
It does.
It's amazing.
So we're going to talk to you today about Geeks Without Bounds, which acts as an accelerator for humanitarian projects.
And we do one to five hackathons internationally every month, which we usually see as sort of speed dating between people with technical skills or design skills and organizations that could really use their help, either in humanitarian or disaster response.
We also support some of those projects and teams through six months of mentorship in things like human rights and information security and business development, so that they are responsible not only to themselves but also to the people that they're hoping to help.
And the specific thing that we're going to focus on today is what it is to build an open source cell network at Burning Man.
Yeah, so one of the things that we focus on in addition to crisis response and humanitarian response is communications, because when I founded Geeks Without Bounds with Willow, one of the big things that I was focused on, basically based on my background,
is helping people communicate in times of crisis.
And that's one of the key things that anytime there's a disaster response team, one of the top priorities is to get comms out right away.
And right now, most of the time when comms are put up by the formal sector, it is for the official responders.
They don't consider the first responders to be the people that already live there.
And so one of the things that we really like doing is helping to also put up satellite uplinks in disaster zones.
That way, the people who live in an area can self-organize because they have better access to communications.
So, Diggs, do you want to introduce yourself?
Yes.
If it'll work.
You know, maybe.
So I'm Johnny Diggs.
That's actually my stage name.
I'm a professional dueling piano player.
And so I was encouraged to take a stage name.
And so Johnny Diggs seemed to work well.
I'm also the co-founder of several technology companies.
IRDG, which we founded in 1995, which was focused on unified messaging.
Voxeo, which is an interactive voice response company.
And most recently, Tropo, which I'm the chief evangelist.
I do all of our developer outreach.
And Tropo is an over-the-top voice application development platform.
And also Geeks Without Bounds with Willow.
And I just recently got involved with starting Orlando's first makerspace.
And I am Willow Rue.
You'll notice that both of these pictures are from Burning Man, which, of course, that makes sense.
I do things with Geeks Without Bounds.
I'm also at Center for Civic Media, the MIT Media Lab, with the Berkman Center for Internet Society.
They let me hang out and visit their libraries.
It's amazing.
I also am working with the Roddenberry Disaster Response Team, where we bring in water filtration systems to disaster zones and help set it up.
It pumps through 500 gallons of clean water a day out of any non-salinated water source.
And then we leave that stuff behind so that they can set up markets around clean water.
And also the Space Federation, which linked together a bunch of hacker and makerspaces, specifically in America.
So Black Rock City, where Burning Man happens.
It's basically in the middle of nowhere.
Has anybody been to Burning Man before?
So maybe 5%.
So the closest large airport is Reno.
About a two-hour drive north of Reno, directly into the middle of the desert.
And then... oh, hey, what's up?
Sidebar.
Don't get annoying.
And then about 75,000...
And then about 70,000 people show up to a place that nothing else is happening and they set up camp for about a week and they hang out and then they all leave with all of their stuff again.
We also set things on fire.
And we also build lots of weird art.
So, yeah, one of the things that... one of the most common misconceptions about Burning Man is that it's a music festival.
But there is music there.
It's usually a lot of...
But... dubstep.
But there's also live music, but it's not specifically a music event.
It's culture.
It's about sharing things.
It's a dirt rave.
But it's also about radical self-reliance.
It's about... they have these ten principles, guiding principles of Burning Man, which include everything from... that there's not really any exchange of money there.
It's more of a gifting economy.
You can buy ice and coffee and that's just about it.
But it's also about radical self-reliance, leaving no trace behind, being participatory, and also immediacy, which is also very important.
And it's also an experimental wonderland.
And in disaster response, it's really difficult to try out new things, because usually you have to do whatever you can with what you have right then.
And there's no space to really try something, because if it goes wrong, things are really bad.
But if you suddenly have this influx of a population, it's like a test bed, but nobody cares if you mess up.
Which is exciting.
And this is basically the population of what we call the playa.
It's a joking term about the beach, because the ground is actually usually a lake that...
It's a lake during the winter, and then it all evaporates, and it does that over and over and over, and it ends up with this really alkaline, really fine soil that eats everything, and it gets on all of your plates, and if you're even slightly compulsive,
it is the worst thing ever.
It eats all of your electronics.
And so the population of the playa tends to be zero.
There are a few festivals that happen over the summer, and then there's this huge peak that happens right around Labor Day, which is when Burning Man is.
And that means that...
Will you work?
The nearby town of Gerlach has a population of 250 people in it.
That's the closest semblance of a town.
And Gerlach has a cell phone tower, one lonely cell phone tower, that is designed to support Gerlach, but in more recent years they've pointed that thing towards the playa to support Burning Man.
But the problem is that it is one lonely cell phone tower, and when you have a peak of suddenly 70,000 people, you'll notice a decline in service very quickly.
And so we want to know what we can learn from this gap in infrastructure.
How can we play with this?
So this is...
The Burning Man layout is sort of a half circle, and this is sort of a heat map of where we put towers a couple years ago.
I don't know how well that's showing up there, but those circles are centered around cell phone towers.
We try, obviously, to get 100% coverage over Burning Man.
This year we got about 70% coverage, and obviously the more towers you put up, the more coverage you can have.
Before we get into sort of the details...
This has been floating around for a long time.
I did not create this, but if you care to understand how your fucking phone works, I suggest looking this Prezi up later on and zooming into some of these, because it's really, really funny.
I'm not going to go into all the details, but basically, your phone connects to a local BTS station, just like the one that they were showing on stage before, and that is connected to a network, and the magic of the phone network finds you and your phone rings,
and it's magic.
And that's how you get a fucking phone call.
This is what our network looked like.
I'm not going to go too much into the details.
There's a link here if you want to get into the really technical details of specifically what the connections were like in the network.
But we used almost 100% open source software, with a couple exceptions.
Open BTS, which is an open source base transceiver station software written by David Burgess and Harvin.
And then Asterisk we use for database, I'm sorry, for phone registrations.
Yates stands for yet another telephony engine.
And then of course Tropo, which we use to build some simple applications, because without an application platform like Tropo, you wouldn't really be able to do much other than send and receive phone calls.
And also Tropo provided a link to let people make outbound phone calls into what they call the default world, which is the world we're living in right now.
These are what those towers actually look like.
Pretty pictures of towers.
These were all actually set up.
We used a variety of different towers, some of them were portable towers, like the one that is right there.
That is actually designed to be set up by one person.
Believe me, it can't be set up by one person.
We used trucks, like news vans, to set up some of them, so we can get enough coverage all around in various places.
And obviously the higher and more spread out your towers are, the more coverage you're going to be able to have.
So here's some data from 2012.
I tried to get our 2013 data, but the guy who has it all is in Bucharest right now and didn't respond to me in a timely fashion.
By which we mean this morning when we were making sure everything worked okay.
I love you, David.
So 2012 we had five sites in a variety of different configurations.
The end result was we had about 1,100 phones subscribe to the network and about 8,000 phone calls made to the outside world over the course of the week.
With handset requirements, we required that you had an unlocked quad band GSM phone, which is fairly common outside the US.
We specifically blocked AT&T SIMs for no other reason than that lonely tower over at Gerlach was an AT&T service provider, so we didn't want them.
We were running a legal network, we had a license, but we didn't want any insinuation that we were interfering with the commercial network.
So we blocked AT&T phones.
And then we also provide SIM cards at our camp, so if you wanted to try it out, you could come and we'd just give out free SIM cards.
And we actually had some free burner phones.
These were like...
That was a pun.
It was actually accurate, because they're really cheap feature phones that you can get for like 10 bucks that we would pre-configure, and if you didn't want to mess with anything else, we'd just hand it to you.
Asked if you would please bring it back at the end of the week, but we only got a few of those back.
A lot of people like to keep them as souvenirs.
So inside the network, you could make phone calls to other people who are registered on the network.
So when I call that ply-to-ply voice calls, you could do ply-to-ply SMS.
We very much limited the ability to make calls inbound, and the way we did that was when your phone registered into the network, when you roamed in, you got a unique 6-digit phone number, so it didn't interfere with your normal 10-digit phone number.
But when you tried to make a phone call out of the network, we did that through Tropo, which went over VoIP, and the receiving caller would see a normal 10-digit caller ID, but it looked like a phone call coming from Reno, and then if that person answered,
you could talk to them.
If they didn't answer, they could call back to that number, and for a limited period of time, like 30 minutes, we actually would route that phone call back, based on the caller ID, back to your 6-digit phone number on the ply-to-ply.
So if you needed to get an answer from someone, they could call you back.
It also prevented your boss from bothering you, or anybody else trying to sell you something while you're trying to enjoy Burning Man.
So that's kind of how we limited it.
We said, you know, after 30 minutes, that connection is burned.
And you'll see a theme here.
And so we also did some other apps via Tropo, which included group messaging, sort of like GroupMe or WhatsApp type stuff.
We created an app for voice messaging.
So if you wanted to leave a message for other people in your camp, you could leave a message and say, hey, everybody meet at the giant steampunk octopus at 8 o'clock.
And then other people would call in and listen to the messages, just like a voice message board.
We also provided text messaging alerts for Exodus.
So one of the big challenges anybody who's ever been to Burning Man knows is getting in and getting out.
So you've got one road.
You go through Gerlach and wind your way in.
And then there's a gate where everybody has to go through, little security checkpoints.
And so you've got 70,000 people that are entering in within a short 24, 48-hour window.
There's 70,000 people also leave within a short window.
So last year we actually set up an app for what they call Exodus.
So you could subscribe to it and get a text message, send a text message and find out what the wait time at the gate was.
So if your wait time was six hours, you might as well hang out at your camp for a little while before you get in your car, unless you like sitting in your car for six hours.
And also weather alerts.
Last year was one of the first times in a long time they had severe weather come through right at the end of Burning Man.
So we were able to send a text message out to let people know that a big storm was coming.
And they don't let you drive on the Playa at all if it's raining, because it eats up the desert.
And one of those tenants was Leave No Trace, so eating up the desert would actually be a trace, I guess.
And this is really interesting.
I hope most of you were here for the previous talk, because it was amazing.
And the question of how the community decides what the social bounds to be reinforced by the technology are.
So can your cell phone call only last for an hour, ensuring that you still go and visit people?
A lot of debate happens over whether or not to get on these networks and what appropriate use of them is.
And so one of the things that we play around with is not just the technical aspect of what does it look like to set up this infrastructure, but also what is appropriate use.
So that was one of the things that bounded whether the time limit on someone being able to call you back, and what sorts of things people use the SMS for.
They decided on their own, the latter part, but it was interesting to see.
It's been... I would not say that this network, which has been the last six or seven years, been part of our camp project.
It's not without its controversy.
Some people don't think that cell phones have a place on the playa.
We actually feel that everybody is responsible for their own equipment, and they can choose to turn their cell phone off or not bring it with them.
But we find that a lot of people do actually like to have the ability to communicate.
So some of the lessons learned, specifically that opt-in policy I was just talking about, that we don't actually force your phone to connect to our network.
You really do have to set it up to do that.
So if you really want to, you can, but most people don't.
And a lot of people...
We've had some people that are really, really adamant, and one guy come screaming at us saying that we were ruining his Burning Man experience.
We just kind of sit there and go, how?
So, like I said, it's not without its controversy.
Over the course of the last few years, we've learned about the registration process.
So if you've ever roamed into a foreign country, you'll always get a text message that says, you know, welcome to the Netherlands.
Voice calls are $17 a minute.
And text messages are $5 each.
Whereas our message is, you know, welcome to Burning Man Network.
Voice calls are free, and SMS messages are free.
And if you would like to join, just respond with yes, and we'll send you your unique six-digit phone number.
And so creating that, it took us a while to figure out.
We tried with IVR systems where people had to call in, or they had to come to our camp, and we realized that the best way is to do it the way that people are used to seeing it when they roam into a new network.
One of the big lessons we learned a few years ago was one of our engineers was playing around with the network and without really consulting or talking to the rest of us, sent out a text message to everyone on every cell phone that was turned on, that included the AT&T ones,
that said, welcome to Burning Man, we're your cellular provider, we're located here.
If you want to come by and learn about our camp, come visit us.
And so I likened it to a zombie apocalypse because when the rest of us woke up in the morning, there were literally dozens of people standing there looking up at the tower at our camp with their cell phone, and they look at the tower, and they look at their cell phone,
and then I would walk up to them like, hey friend, how you doing?
Can I help you?
And they were like, I got this message on my phone, and I don't, I don't, did you hack into my phone?
And so inevitably, I'd be like, well, no, I explained a little bit of what we were doing, and they were like, well, it doesn't seem to be working, and I'm like, well, do you want it to work, or do you just need to make a phone call?
And nine times out of ten, they would be like, they'd sit there and think about it.
I kind of just wanted to make a phone call.
And so I would point over to this little handset.
That is covered in dust.
Yeah.
Like everything.
Yeah, and I'd be like, you can just use that phone, just, you know, just dial in whatever number you want to call.
And I'd be like, oh, cool.
And the next thing they would do is they'd go over and pick up the phone, and they'd put it up to their ear, and they'd hear dial tone.
And you'd be like, dial tone.
And then they would go to dial a number, and then have to pick up their cell phone again.
And look up the number that they needed to dial.
But, yeah, many of the times, people just needed to make a phone call.
And for the perfect example, one of the, a couple years ago, we had a young guy came to our camp, and he was like, I hear you guys have cell phone service, and I was like, I really need to make a phone call.
And I did my spiel, you know, do you need your cell phone to work, or do you just need to make a phone call?
And he was like, I just need to make a phone call.
And so, you know, he gets on the phone.
And it's very interesting because, you know, this is set up inside one of our tents, and the rest of us are like eating breakfast, so we hear a lot of really difficult conversations, because these are people that are at Burning Man, but usually they're making a phone call because they really need to make a phone call.
And sometimes it's just people who are just checking on their parents, or their kids, or their pets, or whatever.
But this guy, right as he was coming in, got a text message from his mom.
He was riding with some other people saying that his brother was in the hospital.
And so he wanted to call his mom to find out, hey, do I need to, like, come home right now?
Like, what is this, you know?
And it turns out his brother just had broken his arm in a motorcycle accident and was fine.
So the guy was, like, really relieved that he didn't, you know, first of all, his brother wasn't that hurt.
Obviously, he was not relieved that his brother broke his arm.
But he didn't need to actually, it wasn't that big of an emergency.
And so a lot of people came to us with logistical problems like that.
And that kind of leads to what we were talking earlier about the humanitarian aspects of why we kind of spend our time setting up this network in the middle of the desert.
Because a lot of times we learn things about people's relationship with their phone and communication.
But we get to do that in a generally safe environment that isn't an emergency.
And so one of the things about...
Wow, I totally just skipped over something.
That's okay.
So when people attend Burning Man, some of them take an entire year to plan and some of them take a few days to plan and just kind of show up.
But people, regardless of that much planning and what they're used to, they still really like the idea of being able to communicate with other people.
And so we're going to break from this for just a little bit.
And I'm going to tell you about some of the other projects that Geeks Without Bounds does, which are all in an attempt to see what mutual aid at a global scale looks like.
What can I, as a fairly pasty person, do to show up somewhere and say, I have certain skills.
Are they actually useful to the group here?
And also, what can I learn?
And what do we need to learn?
The previous talk, again, was incredible, and I love the things that we can learn from those communities and those rural networks.
It's incredible.
So I spent some time in Tanzania recently, which is in East Africa, specifically in Mfinga, which is out in southwestern, rural Tanzania.
And we were working on water infrastructure.
So a whole lot of money has been put into water points, which are basically, if you don't have enough water infrastructure, there are these giant things called gravity points, where there's a very thin pipe that sometimes trickles water into these things,
and then someone sits at the tap, and they dull out water, and they charge people for the water.
And a whole lot of these have been installed, and a whole lot of them break, and no one knows where they are, and which ones are working, and so people just keep dumping more money into installing more.
What's the statistic of...
I think it's... the Tanzanian government says that 60% of Tanzanians have clean drinking water.
And that is so not true.
And so what we did was we worked with community partners, specifically SNV, for people who had feature phones, because there is good connectivity there, to report when something was broken or fixed, which also assumes that there is... gives a general idea of where it is.
I, from coming to Hacker Cons and spending time in hacker groups for a long time, insisted that we not be able to link those reports back to the specific user, because if this ends up being used for other reporting of infrastructure, it might not be good if the individual is linked to that report.
So if someone reports something is broken, or if using an open source water sensor, like out of public labs, reports that there is no flow of water, then a water engineer would get an update, or anyone with a wrench would get an update, saying, hey, this point is broken,
you should go fix it.
They go fix it.
And then we end up with reports of what's going on, so that people, if the government is not fulfilling their obligation of fixing these water points, which they have money to fix, if they are not doing it, you at least have the documentation that something is going wrong.
And also you can see how the water engineers are doing.
Are they fulfilling their duty?
And we can start to see who is responsible for which points, when it was last broken, when it was fixed, etc.
So that's one project.
Another project is the Bachow project out of Bangalore, India, which is just a distress application.
It uses your cell phone, a hot button, it updates a list of people that you have pre-described about your last known location, and it also starts auto-uploading via any means available to the phone, to a server.
And it holds up in court, it's court admissible, and so it's used for incidents mostly of gender-based violence.
And there's also something like Mozilla emergency badges, which are... we have this really weird PTSD that's starting to happen around digital response, where people here, and also in activism, digital activism, of people getting all of these notifications of how wrong everything is,
and, oh my god, this thing is flooded, or this thing is on fire, and we don't have food, and we're trapped over here.
And you do everything within your power to help them, based on where you are, and then you don't hear anything back.
And it's really, really damaging to the psyche.
And so, the idea of this is that, while there are many people affected in disaster, and there are many organizations out there that are there to help them, and there are lots of volunteers that are there to help them, what is planned for is often not what people end up doing,
and so we have a strong emphasis at Guab, Guaborg, I like to say, of just embracing the messiness, of assuming that people are totally competent at self-organizing, so long as you get out of their way.
So we just connect people directly, so that they can say that someone actually helped them, via SMS, and the person gets the notification that they helped them, and it also sets up credentialing that is external to organizations, so that they can take that with them wherever they go,
they can just show up in an area and say, I am certified in the following four things in this standardly accepted format.
And then there's... how are we on time?
Oh, man.
We're pretty good.
Sweet.
Okay.
Yes!
Okay, and then there's... I didn't draw this one yet, because it's still emerging, funnily enough.
Emergent infrastructure challenge.
So, in the same way that there's this sudden influx of people for Burning Man, we see a similar trend only in population influx, not in basically any other way for refugee camps.
And so a large number of people have to leave their homes suddenly, and you have to set up infrastructure for a large influx of people, where there hasn't been infrastructure before.
And so there are three groups that are really interested in figuring out how to help this out.
And I want to figure out how to combine them.
So one is Doctors Without Borders has a really great...
how to set up the sanitation infrastructure in a way that is aware of gender-based violence.
So usually people just set up restrooms, and then...
even more awful things continue to happen in an already dire situation.
And so they just have a basic, like, here is how you ask questions.
You have to have a female translator.
Here are the sorts of questions you can ask.
Here's how you adhere to it.
It's very much based on community design and co-design.
And then here's how you deploy that.
So that's one section.
It's already... it's in a PDF format.
They just need to be able to transfer it between phones.
Like, this is Doctors Without Borders, and they don't have access to good technology.
They're pretty good at telemedicine, but not much else.
Good people.
And then there's also...
UNHCR realizes that in these camps they are the bottleneck for information.
People will stand outside the offices for most of a day in order to ask a simple question.
And they have realized that they're the bottleneck.
Well, everybody has feature phones, and maybe even smartphones.
And there's definitely a network in most of these places.
So can we just build a collective intelligence platform?
Like, is that a wiki?
How do we do that?
And so they're eager to do this.
And with material like what Doctors Without Borders has, we can start to seed these platforms.
And then also have... because so much of this is geographically based, and then you have to meet SPHERE standards, which is the minimum requirements to treat people like people in these situations.
Then you can have the updating mechanism through Tarifa.
So , cool projects.
We like them.
And all of them are accepting help.
And we would love your help with them.
Mostly with open source, you know, GitHub stuff, where you can go and see what they need and all that stuff.
So if you have some free time and you have the skills, all these projects could use some help.
So, again, workshop on Saturday with the crew that was just up.
Really encourage you to go to that.
Leisha, who's also from Geeks Without Bounds and is amazing, and I hope she's in the audience somewhere, she's going to give a talk on Sunday morning about cybersecurity and humanitarian projects.
I know that I love being awake at 10 a.m.
on Sunday.
Actually, I do.
I usually wake up really awfully early.
But I hope that you will join me here with enthusiasm.
And the last thing, November 21st through 23rd in Portland, we're going to be doing sort of a... we're trying something new.
We're doing a conference that's combining telephone application developers and hackers with humanitarian response professionals and digital humanitarians and also art.
So it should be an interesting conference that we're working on.
So if you want to learn more about that, I suggest you follow the Geeks Without Bounds Twitter account because we don't have a whole lot posted yet.
So that's what we've got.
And we have so much time for questions because I'm not used to doing broadcast mode.
I like interaction.
So, questions?
Yeah, so the question was to elaborate on our licensing procedure.
We operated... the FCC has a thing called an experimental license.
It puts you in a specific frequency zone and power wattage.
And as long as you stay within their parameters, and you have to obviously clearly define what you do and where you're doing it.
So part of the reason we were able to get that, consistently able to get that, is because of the remote area that we're working in.
And also, these are all professionals that everyone in our camp, in the Papa Legba camp, are professionals that work specifically in this area.
So we like to follow the rules.
Hi, I have two questions.
I think they're related.
So one is for SMS.
Can people SMS to the outside world?
And if so, is it with their own number?
And then two is, how do people discover other people's numbers within the camp?
And kind of how do they...
Because if I say I want to call my friend, who I know is on the other side, how do I know what his number is?
Good questions.
We did not enable SMS to the outside world.
We felt that... it wasn't a technical limitation.
It was sort of a community limitation.
We didn't want people to be sitting there texting their friends all the time.
It sort of went against our attempts to limit the ability for people to call back, because if you have a long-running text conversation within 30 minutes, it would just keep that connection alive.
And we really didn't want to encourage that much connectivity.
But like I said, that was not a technical limitation.
We did not have any sort of directory services.
Typically, you share that by voice.
You did have the ability to call into the message board and leave your numbers, and so if somebody else wanted to call in.
And then once you had other people's numbers, you could create group texting.
So if you wanted to text out to 15 people in your camp, you could do that.
Can you just describe how the message board works a little bit?
Say that again?
Can you describe how the message board works a little bit?
The voice message board?
Yeah.
Basically, it was a message board per camp.
So the first person to join from your camp would set up the message board.
There would be a unique message board ID.
And so it was like an IVR system.
You call in, please enter your camp ID.
It's a six-digit number.
And then it would just play like voicemail.
And there's also Center Camp, which is the center of Burning Man, and they have an opt-in registry where you can say where your camp is and that you specifically as a person are with that camp.
And so some people were leaving their phone numbers in their details with full knowledge that anyone looking them up would be able to see that phone number.
Thanks.
I jokingly refer to it as an opt-in-a-con.
Opt-in-a-con?
Opt-in-a-con.
Do you think your infrastructure could be used to build distributed sensor networks, specifically gunshot detection?
You're speaking about like, say, the Tarifa network?
I mean, basically, the Waterpoint is sort of a distributed sensor network.
Is that what you're...
Specifically, if you could use cell phones to spread out over large territory and then be able to monitor gun violence and to be able to locate...
As far as reporting, Tarifa is just a triaging platform.
You can report anything.
They've also used it for education.
If teachers have showed up today or the state of the school or whatever else.
And so people could report that they had heard something and then you could do some triage around that, or triangulation around it, but it's not running the mic at all times, and so it's not the same thing as what they've had in Oakland.
But do you think of developing an open source phone or using another firmware for an existing phone, for an existing cheap phone, to be able to use it as a reporting node continuously without human involvement?
You could.
We don't specifically have any projects on that right now, though.
All right.
Thanks.
Hi.
Great talk.
Thank you.
I had a question about data on the network.
Were there any thoughts about maybe, like the same way you did text messages out of the network, but creating maybe data that's local just to Burning Man, like some apps or anything?
So we didn't provide any data channel to date.
The Burning Man infrastructure, they have an infrastructure team, provides pretty good Wi-Fi coverage, and so if you wanted to get Wi-Fi, you can.
You have to be associated with a camp that has applied for it and jumped through all the hoops to make sure you get it.
So for that reason, plus some technical limitations and just time and effort, we didn't supply any data.
And so we may some time in the future to do some basic data, but, yeah, we focus mostly on voice and SMS.
Okay.
Thank you.
With your experimental license with the FCC, do they, you know, as a, quote, regular cell phone network, they have to do all kinds of compliance stuff with 911 and all that stuff?
Are you just completely exempt from that stuff?
Yeah.
Experimental networks are exempt from emergency E911 and regular 911.
It's... and we make that very clear that if there is an emergency, you know, find a Burning Man ranger or go to... you know, there's plenty of police presence on the playa.
So... and they're actually usually very, very good.
They're there to make sure that nobody gets hurt.
But, yeah, we made it abundantly clear that we are not providing commercial service.
This is experimental and for, you know, educational purposes.
So what happened if you dialed 911?
You had, like, an interrupt that just said, hey, go find somebody.
This isn't going to work.
Yeah, exactly.
You had, like, an automated response.
Hi.
How do you handle backhaul between the sites?
I don't really want to go too much into the detail of that, but all of that is explained.
If you go to that link that's in this... we'll post the link to this shortly, but... to the presentation.
But essentially it was, you know, point-to-point connectivity between the different towers.
Thank you.
Yeah, hi.
I have a question about one of the other projects that you mentioned briefly, Bach Chow.
And I was wondering if you could talk about how that works and maybe if you know about KiteStream, how it compares to that service.
So, Bach Chow is headed up by an amazing woman named Xin Mai who lives in Bangalore.
And I don't know any of the technical details about how her project works.
I only know how to help get it in front of the news and whatever else.
But, yes.
Yes, will you come stand at the microphone?
Leesha is in the audience.
This is amazing.
I just draw stick figures.
I use crayons mostly.
I don't know how these things work.
Okay, so...
The original Bach Chow project was created at Random Hacks of Kindness...
Excuse me.
What, a year and a half ago?
Get closer to the mic there.
The original Bach Chow was created at Random Hacks of Kindness about a year and a half ago.
It was an Android app.
It still exists.
The original Android app still exists.
That version is just an Android app with a panic button, like Willow explained.
It just uses the Android infrastructure, basically, to send text messages to up to five people that you preload...
and call the police, the local police number, 911 or 111 or whatever your local one is...
and upload audio and video according to whatever its connectivity infrastructure at that space at that moment is...
for use in courts, like you were saying.
Bach Chow has continued to expand what they're doing, though.
So now they're doing specific use tools like pendants that look like pendants...
that are panic buttons that basically have all that functionality that was on the panic button on your phone...
can now be had in a pendant.
So the other program... I don't actually know... what's it called?
KiteString, I think?
I don't know the functionality of that.
I know that there are several other panic button tools out there.
When we were doing the mentorship, I think Bach Chow graduated from our accelerator program six months ago, eight months ago.
And at that time, none of the other panic button applications that were available had the streaming piece of it...
which was actually one of the key pieces.
It wasn't just that you were asking for help, it was that you actually had evidence in case this went to court...
so that it didn't turn around where they started persecuting and prosecuting the victim.
You had proof.
Yeah, one of the cool things was when you hit that panic button, it turns on your microphone and your camera...
and immediately starts streaming audio and video to the cloud.
So even if someone took your phone, the evidence would already be up in the cloud.
I don't know where you went, but did that answer your question?
I think we're good.
Thank you all so much for your attention.
Okay.
One quick thing about Burning Man, of course.
The big motto there is, of course, piss clear here.
5-2-1 is the best thing I can suggest so you have a good conference.
For those of you who don't know what that is, those are the bare minimums we suggest.
Every night, get five hours of sleep, make sure you have two square meals a day and one shower.
A day.
How are
you?
Good.
I'm here to talk about a system similar to...
I
would like to do a 3 a.m.
bike ride because I'm going to be up at 10 a.m.
tomorrow and I will not be able to survive a 3 a.m.
bike ride.
Sorry, it's like I see the clock there and I just...
But that's not why.
I can handle the mental calculus.
What I can't handle is a.m.
and p.m.
which is why my clock says 19 on it.
It's just a...
Yeah, I know.
In case anyone in the room is not fully aware of it, the conference hashtag is hashtag H-O-P-E-X.
The official Twitter account is, of course, at HopeX as well.
With us is Jimmy O.
And he's here to talk about some of the interesting vulnerabilities that have come up through SSL.
Please welcome Jimmy O.
Thank you.
Can everybody hear me all right?
Yeah, is this house music going to keep going through the talk?
I'm cool with it.
It can.
So, yeah, hi, I'm Jim O'Leary.
Affectionately known to some as Jimmy O.
I work at Twitter and it's an honor.
I'm very humbled to have the chance to talk to you, hopefully educate and entertain with some of the things we've learned as we went from rolling out a widespread SSL deployment at Twitter.
So it's been a bit of a bumpy ride lately for SSL.
The protocol, a number of implementations, all have gotten beaten up pretty badly, even in the mainstream press.
So a few of these attacks or vulnerabilities might ring a bell to you.
There was Crime, Breach, Beast, Lucky 13, Heartbleed, 3shake.
Basically, you couldn't have an SSL vulnerability without a nifty little name.
And making basically headline news with almost all of these.
So , man, bad times all around.
But there's something else out there, way more prevalent.
It's been around for a longer time, not quite as sexy a name, but get ready for this big animated reveal.
Yell it out when you think you know what I'm going to talk about here.
Please.
Exactly, right?
HTTP.
There it is.
HTTP.
Hypertext Transport Protocol.
It's all over the internet and it's a big problem today.
So, one thing I love about hacker conferences is that you get people from all walks of life, all different levels of technical ability in one room to learn about some stuff.
So this is something for the noobs.
If you're super elite, you know this already, but let me break down for you why HTTP is a bad thing.
Here you are.
You're at Hope or you're at a coffee shop downstairs and you want to browse the internet, right?
So you're connecting via some free, unencrypted public Wi-Fi to some machine that is probably hosted in the back room or under the espresso machine of this coffee shop.
It then connects out over the internet as we know it.
So there are a whole bunch of different hops and who really knows what happens in between the coffee shop and ultimately us, your destination.
So in this case, you can call it Twitter.
As we all know, there are a lot of these folks probably here today.
I think I saw a guy wearing this hat in the elevator earlier.
But there are people out there that might be interested in looking at what you're doing online.
So surveillance is a big topic of this conference.
And also maybe mucking around with it, having a good time at your expense.
The key thing to note is that every hop in this node presents an opportunity for somebody to sniff your traffic, manipulate your traffic, or do something else nasty.
Man, this is super funky.
But don't just take my word for it, right?
So this is a screenshot that has been terribly distorted.
It looks fine on my machine.
But of Wireshark, right?
And what I'm showing here, this is me in a coffee shop capturing traffic with a promiscuous network card.
And looking at the ones and zeros as they fly through the air, assembling them into TCP and then HTTP, and then being able to click in and jump into what's actually being sent on the wire.
So, again, this is all super fuzzed and mutated because of the display.
But what you'll see here, if you dig in, is that, hmm, this is interesting.
It's an HTTP post to a login endpoint with a username, say, my username, and a password that I just put, yeah, right.
It's not my actual password.
But it's flying over the network in clear text.
So, we've been logging in over SSL for a long time now.
A lot of sites, even the worst sites on the internet, don't accept your username and password over HTTP.
It was interesting to see the EFF talk not too long ago where there were some sites that still take username and password over HTTP.
But, for the most part, we kind of got that solved.
We're using HTTPS for username and passwords because a lot of people reuse passwords on different sites.
So, that's good.
What is not good is if you log in over HTTPS and then you downgrade people to HTTP, these session cookies are flying around the network in clear text.
And so, you might not steal anybody's username and password, but you'll be able to sniff out, steal, and playback authentication cookies that are used for logging into a site.
So, as you probably know, when you log into twitter.com, you type your username and password once, we give you a cookie, your browser holds onto it and sends it up to us every single time you make a request so that we know who you are.
In the world of plain text, internet traffic, things like this come about.
So, this is an old screenshot.
2010.
This is super old Twitter.
I think, for the sake of everybody's eyeballs, I'm going to unplug this and plug it back in.
And then pray.
Oh, alas.
Is that even worse than it was before?
Sure.
But, anyways, this is FireSheep.
Again, super clunked out FireSheep.
But it's an HTTP sniffing extension that you can add to your Firefox browser.
It'll allow you to, again, smell the ones and zeros as they fly through the air, see who is logging into different websites with HTTP-supporting traffic.
So, this is back in the day, like I said, 2010.
If you could see here, there's Twitter, there's Facebook, there's Yahoo Mail, I think there was Hotmail, there's Amazon.
And you, as a kind of nasty hacker bad guy, can just select whichever person you want to log in as.
You don't even need to know their username, you don't need to know their password, you just look around.
You say, that guy looks like an interesting guy to log in as.
And you click his name and you're signed in to whatever service you want.
So, again, that was 2010.
That was back in the day.
This is sort of pre-Snowden revelation.
But what was nice about FireSheep is that it shone a light on an issue that existed forever.
Like , you don't need FireSheep to do this, you can just use Wireshark and write a script yourself.
But FireSheep was kind of the first mainstream adoption and realization that this is a problem, and it's really easy to man-in-the-middle HTTP traffic.
So, as a result, at Twitter, that was like December 2010 that FireSheep came out.
January 2011, we rolled out an option to use HTTPS all the time on your Twitter.com sessions.
So, what we had here is you would dig into your settings page.
We announced it, but in order to use this feature, you had to go into settings and click it, and then from that point forward, you'd be safe while you were logged in from FireSheep-type attacks.
Once you were authenticated, it was SSL from that point forward and nothing else.
That was great.
People used it.
We ironed out a few bugs in the process.
But what kind of sucks is that you only get the people that dig deeply down into their settings and really care about security.
Ideally, we'd be able to protect everybody all the time, and you'd have to go out of your way to do something insecure.
So, over the course of a few months, we gradually turned up a little knob to make this default behavior opt out for HTTPS all the time.
So, you log in now, and unless you go in and uncheck the box, you get SSL, kind of whether you like it or not.
After cranking that up to 100% and seeing only a few people opt out, we just took away the option altogether.
So, we said, alright, if you want to use Twitter.com, SSL, tough luck, you're going to get it in 2012.
So, I mentioned that surveillance is something that we all care deeply about here, as is censorship.
And so, I'm going to sprinkle in a few wins and good stories that we saw as a result.
That was better, at least.
And so, it's tough to read.
What I want to highlight down here is that there was a situation in India some years ago where a little political discontent stuff going on, and the government wanted to censor certain Twitter accounts from being accessed.
And they did so in a pretty naive way.
They gave all the ISPs in the country URLs to say, block these URLs, basically.
Don't let anybody visit Twitter.com slash activist1.
What's great about HTTPS, and this has been mentioned before too, is that after the handshake and you make your SSL connection, everything in the URL, so whether it be the path or the query string, is totally blind, or anybody in the middle is totally blind to what that could be.
So, what people realized is that if they went to a profile for somebody that had been blocked by the government, all they had to do was actually sign in, and then they would access the site over HTTPS, and then hurrah, you get to see the stuff that the government had tried to block.
So, that was a win, it was really easy, and it made the government look kind of dumb, so it's even better.
So, one thing to note, and if I don't say this, somebody is going to yell at me from the crowd and say it for me, so I'll get in front of that and mention this now.
Even if you have sessions that are 100% SSL all the time, this guy still has a few tricks that he could use to try to steal your cookies.
One gift he might have for you is an HTTP embed on one of his pages, it's not on twitter.com, he can send you an email that can contain this HTML tag, and the way browsers work, again, is that once you get a cookie from a domain, if you don't do anything special with it,
it's going to send that cookie to the domain every time you make a request there.
So again, if you're signed into twitter.com, you're going to make this fetch, and you're going to send your cookie along, who knows, it's over HTTP, he'd be able to steal it and send it back to us.
Unless, of course, you mark your cookie secure, so this is, again, kind of old news, but if I didn't say it, somebody would call out and yell at me.
So, if only security were always this easy, where you just pass down an option that says, yeah, make it secure.
But for a set cookie, that's really the case.
Mark it secure, and it won't fly out over the network in clear text.
So, it was a very gradual process towards getting everything on SSL all the time.
For a while, we'd allow you to load HTTP twitter.com.
Like I said, all of our authenticated sessions were SSL.
Our unauthenticated sessions, we still let you browse a site not logged in over HTTP.
So what that means is that you load HTTP twitter.com, and you end up with a page like this.
This was mentioned, again, in some of the earlier talks, but how do you know where this post is going to land up when you click sign in, right?
Like, you put your username and password in there, but there's nothing indicating to you that this is going to go over a secure channel or to some other website that Twitter doesn't even own.
So, there's an attack that Markstein and Marlinspike popularized along with SSL SNP, which basically takes the page that has been served up over HTTP and substitutes HTTP links for every HTTPS link that it sees.
So, in this case, I'll kind of go to exhibit A, which was this first screenshot, and exhibit B, which is the second screenshot.
One of these has been SSL stripped, the other hasn't, and they look exactly the same, right?
So, what the point of this slide is, is that even though we moved everybody over to SSL once they logged in, there was still this window of opportunity for attackers to get you before we get you into a safe space with SSL.
So, what do we do about that?
This is a bit of a motivational image.
Again, it would be fantastic if this thing would actually look good when I showed it to you, but to kind of represent the next level shift that we've done that I'm going to talk about here.
And, you know, most good tech talks have a disclaimer of some sort.
You know, I'm not a lawyer, and the disclaimer here is that we did all this stuff, and I'm up here because you can too, and I would like you to do this.
So, let's take a step back a little bit, a little more handcrafted slide work from me.
But, say that you are just a Joe user.
You hear about this Twitter thing.
Ellen tweeted some funny selfie, or Geraldo's taking pictures of himself in the bathroom mirror.
You want to find out what Twitter's all about.
So you go into your browser, and you open your location bar, and you just type Twitter.
This is kind of a standard interview question for some, like, networking jobs and stuff like that.
So, you know, what happens when you type Twitter into the location bar?
In this case, the user didn't qualify with, like, a .com or a .net.
So, actually, the default behavior you're going to get is you're going to end up on some browser or some search engine, right?
So, maybe you use Bing.
Maybe you use Yahoo.
Maybe you use Google.
And so, I'll go back a few slides here.
One of these things is not like the other.
So, first result for Bing, Twitter.
Yahoo, Twitter.com.
But Google's URL is HTTPS, Twitter.com.
Again, this is before we moved everything over to SSL all the time, logged in or logged out.
So what's this crazy magic?
How do we pull this one off?
You might not be surprised to know that there's a lot of name calling on the Internet.
And this is, like, bullies and meanies, or it's DNS and C names.
So, we serve up content when you make a request to our servers, and it's possible for people to actually put their domain names in front of our servers, and we can't really do too much about it.
So, we can detect it when it happens, but there's nothing really stopping you from saying, I'm going to register, you know, HopeTwitter.com, and I'm going to point it over here, so that when you make a request to HopeTwitter.com, it gets looked up and ends up getting content served from our servers.
And so, this does all sorts of bad things for, like, the business and for the website.
It lets your content get indexed on other domains, it makes people feel comfortable potentially logging into other domains that you don't own, and it's a generally thing that you don't want to have happen.
So, Google supports this link rel canonical element that you can embed in the HTML on your page, and it lets you, as a server that's rendering the content, tell a crawler or a search engine bot where this content is authoritatively being served from.
So, again, we had HTTP Twitter.com links out there, we had HTTP Twitter.wacky domain stuff out there, but we always served up the content with this hidden tag in there that told the crawlers that no matter how you got to this site, index it as if it came from HTTPS Twitter.com.
So, as a result of that, we saw the HTTPS URL kind of jump off the chain in the search engine results, and what's nice, the big defense against SSL strip is never letting people try to access your site over HTTP in the first place.
So, if you do an encrypted web search, you're on HTTPS Google.com, and then the link that you click is to HTTPS Twitter.com, no HTTP request is happening there, no SSL strip, no SSL sniff, great win.
So, that was fantastic.
I mentioned this...
again, this is kind of old news for us, but not a lot of people do it, so that's why I want to tell people about it.
It's one line of code, and you can index your content with a secure link.
There was a blog post that somebody pointed out to me from some of these search engine optimization folks that said this is actually a bad thing to do.
You're going to confuse the crawlers, and you're going to end up with negative results when it comes to your URL getting indexed.
And so, I think that we saw this maybe that it took a little while for the HTTPS URL to creep up to the top, but it's not like we lost credibility in the process.
And so, one little funny aside again, is that any web hackers in the audience know what this URL encoded value is?
That's right.
So, I was doing something where I was, you know, maybe some directory traversal attack where I was like, yeah, what was the URL encoding for slash again?
So, you hit up Google, and you just do q equals percent 2f slash, because you're like, ah, let me check this out.
Funny enough, the number one search result for slash is, like, slash on Twitter, and it's an HTTPS site.
So, rock and roll, everybody wins, secure, and all the better.
So, Bing and Yahoo, these dudes are still loading the page over HTTP.
What gives?
They say that in order to be indexed over HTTPS, you need to respond with a 301, which is an HTTP response code telling the crawler or anybody that visits this site, no, we permanently move this resource to this HTTPS URL over here.
So, we were like, okay, 301 redirects are coming up, freshly served from our front end to you, the user.
Anytime you load HTTP Twitter.com, you're going over to HTTPS.
It took a little while for them to catch up.
Like, literally a pretty long time.
Our search results are still HTTP Twitter.com on Bing.
Still HTTP Twitter.com on Yahoo.
Or HTTP Twitter.com on Yahoo.
What's the deal here?
So, eventually, after some time, I'd given versions of this exact talk at smaller, like B-sized conferences in Seattle and San Francisco.
And before I updated my slides for Hope, I was like, let me just go back, hopefully by now we're indexed as HTTPS somewhere.
So, Bing caught up.
Now we're HTTPS Twitter.com on the front page of Bing when you look.
All right, we're almost there.
Yahoo was still HTTP.
So, this was a bummer.
What I did, actually, recently while I was preparing for this talk, is email the chief security officer of Yahoo and was like, this isn't a threat.
I'm about to give this talk in a few weeks, and I've noticed that we've been serving up this 301 redirect for, like, a year plus now, and we're still indexed as HTTP.
As a result, now we're indexed as HTTPS Twitter.com.
And so, that was an effective way to get us basically linked as the number one search result for Twitter with HTTPS URLs.
So, that's nice.
If you're trying to find Twitter through a search engine because you don't know that you should probably just do this, then you're protected.
Which brings me to this next point, is if you're a little more tech savvy, and you type Twitter.com into your browser because you think that's where Twitter might live, are you actually less secure than you might be if you were coming to us via a search engine?
And you would think, yes, right?
Like, by default, the browser behavior is to try HTTP first.
So, you type Twitter.com.
It's going to make an outbound HTTP Twitter.com request.
It'll get the 301, and then it'll send you to HTTPS.
But, that gives the attackers that are man-in-the-middling us an opportunity to basically never send us the SSL version of Twitter at all, and to basically perform SSL strip attacks and steal our cookies and steal our passwords and all that stuff.
So, I wouldn't be telling you about this if we didn't do something to address this issue as well.
And so, an opportunity for you to play along at home, if you want, is type Twitter.com into Chrome or Firefox.
And you'll notice this behavior.
So, open up Wireshark, if you want, or open up the network tab in the developer tools, and you'll see that you get a redirect.
Oh, my God, it looks terrible up there.
But, you'll get a redirect that happens locally before ever hitting the network.
And so, if you could see this, there's this arrow pointing down here that says Twitter.com pending, and then a redirect over there to HTTPS Twitter.com.
So, the browser did try to load the resource over HTTP, but it itself knew, no, no, no, no, no, we're going to go SSL before we actually make this outbound request.
And what is this all about?
The HSTS header has been mentioned a few times today, but HTTP strict transport security is a header that you can set down when you visit a site over SSL that tells the browser, kind of like, oh, phew, you made it here over SSL, let's never go back to that scary place of plain text HTTP traffic ever again.
So, from this point forward, until this max age expires, which is basically 20 years in this case, if you ever get an HTTP Twitter.com reference, turn it to HTTPS locally on the client before you do anything else.
So this is fantastic.
It actually defeats SSLstrip, basically.
And so, it's funny, you know, Moxie, the author of SSLstrip, he worked at Twitter for a little while, super legit dude, he was my boss, he's a really cool guy, but it's not as if we got defense against SSLstrip by hiring the author and then telling him to make it not work against Twitter anymore,
which some people thought we did.
We actually just used the HSTS header.
So you too can protect against SSLstrip without hiring the author and just send down this header, it's one line in your Apache config or whatever and you're all set.
Even better than HSTS is the ability to preload HSTS into the browsers themselves.
So, let's say that you've never been to Twitter.com and you never got redirected to SSL and you never got the HSTS header, it's basically physically impossible, and that's because, similar to emailing the CSO at Yahoo and being like, hey dude, let's get the SSL over here,
you can email Adam Langley, he's probably here somewhere maybe, or some developers on the Chrome team and kind of pre-cook HSTS into the browser.
So, Chrome started doing this first, Firefox has picked up their list and started doing it as well.
All you have to do is tell them that, yeah, we don't want people to ever visit HTTP resources using your browser ever.
So, if you load Twitter.com, like I said, in a fresh install that's never been on the network before, you type in HTTP Twitter.com, it's going out over SSL, it's getting redirected locally before it hits the network.
Again, no SSL strip, cooked into the clients.
This is awesome.
So, we rolled all this stuff out gradually and measured some things, but didn't really keep a super close ID on it and just flip the switch all at once.
We were pretty gradual in rolling out both opt-out SSL and basically removing the ability to opt-out SSL.
All these things happened kind of gradually.
We had the luxury then, after we knew that it worked, to flip the switch on some of our subdomains that are not on the Twitter.com domain and then monitor what happens.
So, this particular graph shows kind of the trend.
It looks like somebody's dying here or something, but what we're counting is the number of HTTP to HTTPS redirects we're doing when you load web.tweetdeck.com.
So, what would happen is you would type in, you know, I want to use the TweetDeck web client, and we would say, oh, you hit us over HTTP, you need to go to SSL instead.
What's apparent here is when we started setting the HSTS header because the number of HTTP requests we got to this domain like dropped off almost entirely.
Granted, like about a third of the traffic, maybe still, came to us over HTTP, and that's because this happened back in the day when not every browser supported HSTS.
I think IE still doesn't, but is going to do it soon.
But so this is not like people stopped using the site, they just stopped using the site in a way that would make them susceptible to SSL strip attacks.
So, this is great to see in practice, where you're like, yeah, it worked, awesome, and kind of gave us this mantra that we abide by inside Twitter.
So, if you join Twitter Engineering, and on your third day you hear me or somebody from my team give a talk about writing secure code at Twitter and all the things we do, and this is a slide we pull from like the New Hire Orientation Deck.
Like, we jam it down your throat on day one, and it's like a fantastic law to abide by.
And again, I'm here to tell more people to do the same thing.
There's an earlier kind of jab, it's like, come on everybody, it's 2014, we can use HTTPS.
But when you do, once you start living by this mantra, it's still possible for you to make mistakes, or other people to kind of make mistakes on your behalf and affect the integrity of your SSL implementation.
So, this little dude, the yellow lock, the broken lock, is an indicator that somebody's done something wrong with your SSL content that's been loaded up, and so what we use as a little trick is content security policy.
So , this is something that I'm not aware of many other people doing, and again, I hope more people take from this talk and start doing.
Get ready for a super busy slide that is going to be super tough to read.
But this is a CSP header that we send down with some of the content you load, and I'll walk through it a little bit to tell you what's happening here.
So, similar to how HSTS tells the browser that, if you ever go back to HTTP again, CSP tells the browser, okay, fine, here's all these resources that you load.
Now, here's some stuff not to do with the code that we just gave you.
So, it starts off, you know, the HTTP header, content security policy.
After that, we use a whole bunch of different directives, but you can tell what we're trying to accomplish here, is like, if you want to load some content, it needs to start with the HTTPS protocol.
The iframes that we load on our site.
Okay, they need to start with HTTPS, whatever.
We're not enforcing that all the content needs to come from Twitter.com.
You can see there's a couple examples here, where, in order to be backwards compatible with some stuff, we allow a few HTTP connections.
But, what is nice about this is that, if you have it in just default mode, which is content security policy, the browser's actually going to block any case where we messed up.
Like, it's going to say, oh, looks like the developers included an HTTP link, or HTTP, not a link, but HTTP resource in this page.
I'm just not going to load it.
And what's great is that it will notice it as developers, because things break when we do the wrong thing.
Even better than that, is that CSP has a support for a report URI directive.
And so, when one of these violations happen, the browser is actually going to make a post back to a URL that we specify in the header, and it's going to tell us, like, we did something wrong.
When we tried to load this page, and something that we told it to not do was done.
And so, we have some, basically, like, real-time aggregation stuff going on.
And we can basically tell, if we shift a bug to production, oh no, this thing slipped out, and we have the HTTP resource.
Let's fix that up in the next deploy, and get everything back to the nice green lock.
So, use CSP for a number of things.
It's traditionally thought of as, like, cross-site scripting mitigation, but you can use it for getting and detecting any usage of HTTP on your HTTPS site.
So, it's great.
It's also tough to apply uniformly across all your different properties in the same way all the time.
And we have HSTS, we have content security policy, there are all these other headers that you should send down as well, to tell the browser not to mess up this stuff that you're trying to display.
And so, we created a library that we've open-sourced.
It's up on GitHub.
It basically does the right thing by default.
So, I mentioned earlier, with the SSL all the time on Twitter.com, how it was crucial for us to make the right thing happen by default, and not rely on our users going in and checking the checkbox.
Same thing with our developers at Twitter.
We want them to use this library.
It's been ported to, like, Go, and all these different languages.
If you visit this site, we have links to the other languages it's been ported to.
But for developers, too, we just want you to do the right thing without having to think about it.
So, if this library is out there, please contribute to it.
Please use it.
Check it out.
It'll do all this stuff for you without thinking about it.
Another good way to detect when you might make a mistake is just to make things break hard when you do.
So, after we moved the whole HTTP Twitter.com site to HTTPS, we still had the platform to worry about, right?
Like, our clients connect to us over api.twitter.com.
A lot of our third-party partners connect to us over api.twitter.com, which was sort of a longer tail to fix, because we have partners that, like, we can't just say, okay, boom, HTTPS now for everybody, and get ready for it.
So, what we did there is kind of gradually tell people about it, tell the developers that we're going to move api.twitter.com to SSL all the time, and get ready for it.
And we ran a series of, like, blackout tests, you might call it, where the way you might notice that we've made this change is that your app just breaks, and it stops working.
And so, in order to be kind of nice to people, we say, okay, we're going to turn off HTTP for an hour on this day.
Everything's going to break if you're not over on SSL yet.
And people that aren't paying attention to, like, our mailing list and stuff are going to be like, oh, what the heck happened?
My thing stopped working.
And then they're going to yell at us about it, and we're going to say, yeah, we were running this test, so just add an S to your HTTP api.twitter.com URL, and then you'll be all set.
So we did this over the course of, like, a few months, and then now the api.twitter.com endpoint will basically return this error if you try to use HTTP.
So this is great because not everybody uses HTTPS Twitter.com.
Like I said, we have third-party clients.
We've got our own native clients.
But everybody there now is using SSL all the time because the server enforces it.
So this is great.
There's all the ways you can mess up.
There are a number of documented situations where somebody else messes up SSL for you.
So there is the TLS protocol itself, and there's a dependency on the certificate authorities, and trust needs to come from somewhere.
So there's this paper, there's this incriminating slide that names a number of folks that have been involved in one way or another in subverting the trust model that TLS depends on.
But basically, you need to get the trust that you're talking to, who you think you're talking to, from somewhere, and that's where certificate pinning comes in.
So again, this is something that not everybody is doing.
I think Trevor is in the audience somewhere.
This isn't actually a certificate pinning image.
I stole it from the TAC website.
But the way pinning works is that you rely on the operating system, so the code beneath you, to do all the things that normally happen during an SSL handshake.
So check out the certificate that you got.
Does it match the host name?
Is it expired?
Has it been revoked?
Do all that kind of stuff that you normally do.
Then, when you're in app land, check out the certificate that was actually used as part of this SSL handshake.
So what you can do if you own the client code, and we do do this in Twitter for iOS and Twitter for Android, Chrome actually does it for us too, is that we check against the pinned routes.
So we say, we know that we're only ever going to buy certificates from these three CAs.
And if you get a certificate that came from TurkTrust or the Indian government, something's up.
Even though this is a certificate you would normally trust, how about you don't trust it in this Twitter.com session?
And after that, everything else works.
It's just kind of like on bootstrapping the SSL connection that you implement the certificate pinning, and nothing else really needs to change in your application.
So OWASP, I think, has a great wiki page on implementing cert pinning in your own applications, and so go check that out.
It's a little more detailed than I can get to in slides, for sure.
Of course, security is a cat and mouse game.
We rolled out certificate pinning, and then next year at Black Hat, this talk comes out about getting around certificate pinning.
And so the thing I want to stress when it comes to pinning is that the threat we're trying to protect against and the adversary in this situation isn't you, yourself, with your jailbroken iPhone.
So these attacks, while they work, they basically rooted your device, hooked some DLLs, and then modified the way in which the code was running on the client so that it could get around pinning.
So this is good if you want to pen test Twitter or you're scared that we're sending some information off your device that you don't want us to send and you want to man-in-the-middle it.
That's cool.
Do your thing.
Intercept the way.
But what we don't want to have happen is basically on that first diagram where we have all these hops along the way in between you and us, we don't want anybody there messing with the traffic.
So if you want to mess with your own local instance of the application running, go to town, go nuts.
That's not the threat we're defending against with pinning.
What we want to stop is anybody that's got their hands on a certificate that wasn't us and detecting that.
Unfortunately, step one there was rely on the operating system for the first part of the handshake.
And this is a slide that I did update this year.
This might ring a bell to some folks, but Apple had a kind of nasty bug that came from this second conditional, not really being conditional, and always jumping down here.
Adam Langley, again, being a super legit and super smart guy, has a way better write-up than I could talk through on stage at his blog.
Check that out.
But basically, if you're relying on the OS for some part of the handshake and you're not doing it yourself, you can be susceptible to the underlying problems in that code.
So Twitter for iOS was affected by GoToFail, but we also inherited the patch as soon as iOS patched.
So we could go many steps further than that, but you need to take a reliance on the operating system at some point.
We're not going to implement our own full HTTP stack and write all our own TLS code.
At some point, you need to draw the line.
Pinning is going a lot deeper than most people do, but it still opens up to this kind of problem.
So perfect forward secrecy has been getting a lot of attention lately, and rightfully so.
I prefer the term forward secrecy.
I've been in security for too long to call anything perfect, but it can be super easy for you to implement if you're just running your own web server and you're not trying to do everything to massive scale and TLS sessions.
Basically, you can go to Mozilla's page for setting up a secure web server, copy and paste the recommended Cypher suite list that they have for you to implement, and just put that into your HTTP config and restart your server, and boom, you've got PFS Cypher suites to the top of the list,
and everything is negotiated from that point forward.
We'll use perfect forward secrecy.
We'll use ECDHE and knock yourself out.
If somebody steals your keys, they can't go back and encrypt all the traffic that they may have captured in their mass surveillance efforts.
Forward secrecy is great.
We have this blog post because, again, Adam Langley has a blog post about how you can botch forward secrecy if you're using a TLS session or something, which you might want to use for performance reasons.
We have this post about how you can encrypt your tickets and share them all.
Again, check it out, because it's a lot more detailed than I can get into here.
I think Jacob, who wrote this post and is now at the EFF, is here somewhere.
He's a cool dude.
I think he might be talking somewhere.
But if you see him, say what's up and say thanks for the forward secrecy of Twitter.
So, Heartbleed has this problem.
Sometimes you're good and sometimes you're lucky.
Twitter, in this case, got kind of lucky.
Our stack is such that there's code that runs before you get to the OpenSSL library.
It was essentially too dumb to pass along the Heartbeat request.
And so, even if we were running, which we were, vulnerable versions of OpenSSL, the exploit couldn't get to the OpenSSL code.
And we were like, oh my god, we got super lucky.
But, all that being said, we rekeyed, we patched the OpenSSL version and all that.
The thing that I would like to take away from Heartbleed and pass along to all of you is that it's really good to get good at patching OpenSSL these days from an operational perspective, right?
CCS injection happened, I don't know, like a month and a half later or something?
And we learned a whole bunch during Heartbleed.
It's like, oh my god, we have servers over there?
They're running what version of SSL?
And, you know, over the course of like a few days, we kind of got our act together in...
well, I mean, that's harsh.
Our entire act together.
Stuff was patched super quickly on anything that really, really mattered.
But we got to operationalize patching OpenSSL really pretty good by the time the CCS injection came around.
And so, it would be silly to assume that that was the last OpenSSL bug, and now we've got it all figured out.
So you should, again, practice.
Even if there's no CDE that came out against OpenSSL, why don't you go ahead and just pretend there were.
Restart all your servers, make sure everything's cool, and you can still resume your sessions, nothing breaks, and just go ahead and practice as if Heartbleed 2 is right around the corner, because it probably is.
And with that, I know I've covered a whole bunch.
I've got time for questions.
And when I give this talk, this is normally the first question, so I wanted to get in front of it, and also bring back one of my favorite memes, which has kind of died off on the internet.
But what about performance?
And so, what I can say, and this has been documented in other places now too, is that Twitter is faster now, like significantly faster, once we move to HTTPS all the time.
And there's no way that your code, up and down the stack on the client and on the server, is so finely tuned that the one thing that you can do to shave off a few milliseconds here and there is use HTTP.
So, you kind of just treat it as a non-compromised issue and say, that's it, we're using transport layer security all the time.
And find other places for you to speed things up, and you will, and you'll get much better gains than you might if you just focus on, even just like tweaking your TLS implementation the whole time.
You'll find more lucrative spots in your code base to fix things up.
And with that, I know I ended a little early, I think I've got 10 or 20 minutes left, but this wasn't just the whole pitch to say, oh, come work at Twitter, we're awesome.
But you can, if you want, because we've got a lot of security jobs open.
But basically, like I said, this is a whole bunch of stuff we did that happened under the hood, and unless you're super into security and you're sniffing all your traffic all the time, you might not notice that we did it.
And that in itself is kind of a good thing, because we changed a lot of what was happening underneath the hood, and people didn't notice, things didn't break.
And so, if we can do it at Twitter, which is humongous, we've got a ton of traffic coming in and out of this place, you can do it on your site too, and you should do so.
And that's it.
Thank you.
So you mentioned a couple things that involve revocation.
Yeah.
Of certificates.
So like, Harp is a good example.
What are you guys doing for revocation?
Are you using OCSP stapling?
Are you using live OCSP?
Or are you just relying on CRL sets in Chrome?
Yeah, so, revocation, we basically...
we don't do anything fancy for revocation.
There was talk about OCSP stapling, and a lot of people pushed that.
We've experimented with it, and just haven't implemented it fully.
So, basically, same thing for pinning and all that.
We just rely on the OS.
We rely on the browsers.
And kind of have it be up to them.
So if they want to use CRLs, they use CRLs.
If they want to use OCSP, use OCSP.
But we don't do anything special.
Light's killing me.
I'll be around.
There are a bunch of other tweets here.
Jan up in the front row.
Isabella.
JJ, you're relatively new, right?
I've been on Skype with you before.
Yeah, what's up?
Like I said, I can't see anything.
I'm completely blind.
Yep.
Yeah.
Economics of what?
Oh!
Yeah, yeah, yeah.
So...
The question, yeah, the question was that...
I mentioned that we pin to basically like three different root certificates.
And what are the economic implications of doing so?
So you're right that there can be.
There's a little bit of a good faith in there that says like, Okay, now that we've pinned to VeriSign, VeriSign is going to say, Oh, time to renew the certs, huh?
400 million dollars.
Because we're pinned in that way.
Dr.
Eagle and whatnot.
But so the reason we have multiple is basically in the case where somebody does stick it to us, potentially.
We can have a little bit of agility.
We can also upgrade the clients over time to change out any roots if we need to do that.
And then the multiple certs are there as well.
It's kind of like defense in depth.
So say that we rely on VeriSign, VeriSign issues a bad certificate and we need to revoke that.
We can fall back to one of the other two.
One thing to note about pinning, and we've seen bad pinning implementations happen for sure, where people pin against the leaf certificate.
So this is a certificate that expires every year.
And when April rolls around, all of a sudden, all of these clients, I won't name any names, stop working because they pinned to basically a SHA hash of the leaf cert when you should pin to the root.
So it's a little gotcha for anybody implementing their own pinning.
Hey, follow up to that one.
Why do you pin to roots rather than using an intermediate offline signing key for yourself?
Yeah, so it's kind of just the ease of implementation, I think.
You're right.
We could just roll out our own entire, basically, API endpoint that's signed by ourselves.
We could run our own CA.
This is sort of easy because you get a bit of the best of both worlds.
You can serve up normal traffic to normal people with their own root store.
And they can trust us because they trust the CAs versus doing an intermediate cert on the side.
So there's a tech proposal which is out there which allows for this kind of thing.
It's been a while since I've looked at the proposal, but it was something that Moxie and Trevor worked on.
We're for it.
We just need more browsers to pick it up, too.
It takes two to tango kind of thing.
You need the client and the server to agree on it.
And so we just need some more adoption.
And we look at it, but without something like that, it's just kind of like using...