You know enough to be involved and be in the right places.
Hello everyone.
I'm here today to share with you a little story, a case history, about a company I was engaged with.
That, in my opinion, is very important to tell, because it describes something that is happening every day.
We are used to reading about nations, cyber warfare, huge distributed denial of services against huge companies, multinationals and so on.
But in this story, the victim is a real small company, a company like dozens of others exist.
And the attack was quite strange, very targeted and, in my opinion, is a cool story that needs to be told.
It's a story that has, in a good way, we were lucky about that.
I will talk about this because I work for a company that usually takes care about security of companies.
But security, in my opinion, is not only something that permits me to have money at the end of the month, but it's something that I believe so much that I use all my free time inside a lot of non-profit organizations, whose common aim is to share knowledge about this new technology that are supposed to help us and have our life more simpler,
but often hide some strange things that could be not so safe.
So discussing it and choosing, knowing what we are choosing, can make these technologies useful.
So the background is that everything starts with a phone call from a number that I didn't know.
And on the other side was an unknown guy that was telling me, we have a problem, please help us because we don't know what to do.
The guy on the other side of the telephone was the network manager of this company that is a little company that has its business all web-based.
It's something like a dashboard where you can exchange messages.
For example, I'm selling this, I'm renting this, I'm searching for that, and so on.
As you can see, it's a normal company, completely private held, that is a normal company in my country.
So what is going to happen is happening and is going to happen and happened to a lot of companies like this.
The problem was that someone was flooding their web servers and making them unavailable for customers.
And this was their infrastructure in the moment in which they were calling me.
As you can see, it's very common infrastructure.
There is no bleeding edge technology, nor things that are so old and bad.
It's normal.
And the problem was that the firewall was a quite good firewall, but was not able to handle all the traffic that was arriving.
The problem, as we will see, will be also that was not able not to manage the bandwidth, but that packet rate that is quite different.
I have also some graphs about how the traffic went during the attack.
I don't have all graphs because, as you can understand, during emergency, our urgency was to make the network work and not to save graphs for a conference.
Well, we were able to reach a disclosure agreement with a customer to tell you all these things.
And because of this, I'm forced to say two or three things.
I want to be clear that this attack was a denial-of-service.
So the website was not reachable, but no data was still or modified, and all users' data were still safe.
No one was able to introduce into their servers.
Even because during the attack we were looking before, one of our suspect was that someone were using the denial-of-service as a cover activity for trying to own the servers, but they were not trying to own the servers, only denial-of-service.
For curious guys, how you can run a denial-of-service attack, you must create your botnet, you must infect as many hosts as you can, and make them all create traffic towards a single target, but the fastest way is to find someone that will take money to do that.
The funny thing is that paying for that is cheaper than paying a consultant that is defending your network usually.
It makes me really laugh to see how flexibility there is in buying the attack from someone else.
You know you can rent a botnet to run the attack by your own, or you can rent the attack directly, obviously it's quite more expensive.
As you know, this kind of attack is very common and is not focused on any nation, and the number of hosts involved in each attack of this kind is really huge.
It's no more like when we were used about one huge server sending a lot of traffic to the target, and so you can ask your internet provider to filter that out.
Now there are a lot of home computers that have not so much bandwidth as common home connections, but all together can do a lot of damages.
These graphs represent one of the recent attacks of root DNS servers, and all autonomous systems are colored in a different way, it doesn't quite look good.
One other problem I had during this attack was that many newspapers in Italy were talking about this attack, and as usual media tend to use the wrong word.
So I had also to fight a war in communication to make them understand that it was not so complicated and technological rocket science attack, but was something like any vandals can do, you know.
Even one of the more dumb questions at the end of the talk is, you know who were running the attack?
The answer is no, for two reasons.
The customer had no more money to pay to investigate about that, policemen were involved to search something about that, but no results at all were found, you know, about this scheme of managing botnet that make really hard to come to the real attack manager.
And obviously a lot of people asked me things like, okay, if you had no police collaboration to analyze any of the infected host, could you own one of the host?
Obviously yes, but was not legal in Italy, yeah, is the correct answer, yeah.
Also, one of the biggest problem when dealing with this kind of attack is that if the network is not designed from the beginning to deal with attacks, with this kind of attacks, you had to follow emergency in every moment and do whatever you can do with what you have in that moment.
For example, having an appliance don't have often enough memory to manage some basic blacklists for the huge number of hosts involved in the attack or are not enough flexible to be configured to follow the attack.
So, coming to the real context.
As I said before, the attack turned out that was a sinful attack.
It make me laugh.
I was reading about this 20 years ago.
It's a really common attack, but in that moment was quite a huge problem.
So, our task was to filter out all SIM packets that was not giving the returning acknowledge.
And there are a lot of way to do that.
The fate was that I was giving a lecture in a university about OpenBSD in that moment when I received the call.
And that day, there were other OpenBSD users talking about, oh, you don't know which strange kind of phone calls I just received.
We decided to use OpenBSD to filter all this connection because OpenBSD has in its firewall, as many other technologies, the instruction to filter out SIM fluid and make something like a SIM proxy.
So, from that moment, just talking, this was the emergency response team that turned out.
So, I was in charge about making decision.
I had a colleague working from another city.
Roberto, instead, is the network manager of the customer.
And he was in another city.
And there was always someone in the data center in another city.
So, everything was managed by the phone.
I saw in face the customer for the first time two months after the attack was finished.
So, deciding the PIX was not okay, we decided to implement OpenBSD.
Even here, it was quite funny because they never heard about OpenBSD.
They know something about Linux.
And we start installing it by phone.
And the first installation was quite long.
Because, imagine, press this, and now read me on the screen, yeah.
Third option, press yes, okay, and go on.
Imagine this during a conference like this, while you are outside the room waiting for your talk.
And then after the talk, where you are on your car, obviously with a headset, and talking with them, and so on.
Obviously, the new firewall was able, in first instance, to filter out everything.
Imagine how much I was happy.
I received a phone call, and by phone I was able to solve the problem in some hours.
The time needed to make the customer install OpenBSD on a new rack server that they had in the data center.
And everything was working.
So, really happy winner, I was thinking.
This is the syntax to activate the SYNPROXY feature on OpenBSD PF.
Obviously, this was not the only option to manage an attack like this.
There are plenty of products and different approaches that can be used.
The first problem was about the internet provider.
OpenBSD is one of the biggest internet providers in Italy.
But when we asked them to use some approach, root-based, they told us, oh, no, no, too complex, we don't do things like that.
And you will see later on other slides, their support was zero.
We were really in difficulty.
Imagine when you call the manager in charge in the middle of the night, asking to, for example, reboot one of your servers, and they told you, okay, I will do that in half an hour.
Half an hour?
That's why so long.
I'm how to eat a pizza.
Oh, okay.
Very bad.
So, using our own servers to filter out sync fluid was okay.
It was a good approach.
We were so comfortable, everyone was driving back to his location.
So Fabio went back to Venice, where he lives, and I was going to another conference.
It happened that the attacker didn't appreciate our work to filter out his attack.
So, we started using more and more bandwidth to generate more and more sync fluid.
And that was okay.
The server was responding without any problem, until we were obviously forced to buy more bandwidth from the data center.
We were so lucky to be in a data center, because having more bandwidth was only about money.
We send money, sign something, and more bandwidth.
If all the servers were on customer site, there were no way to... or better.
Everything would be very longer and more expensive.
So, we saw the traffic growing, growing, growing, but feel comfortable, because the servers were responding very good, until there were too much year queue.
And to imagine this, if you went to the firewall console and start pressing keys on the console keyboard, no letters were appearing on the monitor.
And this was quite difficult to manage.
So, for the first thing, we asked the provider to do the only thing they were able to do for us.
To filter out international traffic.
The company business is Italian-based, so cutting out international traffic seems okay, because no customers outside Italy.
Until you live in Milan, and you buy your connection from a company called Swisscom, that is Swiss-based, and so you are international traffic in that moment.
So, some customers were cut out.
But they are so few, that was acceptable while we tried to arrange something.
This was not the final solution, cannot be.
So, we decided to split the problem to solve it.
We were lucky, because they had quite a bunch of new rack servers in the data center, already Linux-installed, to be the new web front-end.
And we told them, okay, take some of those servers, put one in the rack, another one, and install OpenBSD on that.
The incredible thing at this point was that the network manager, his second installation of OpenBSD was completely autonomous.
He installed by himself, very good.
The other problem was that OpenBSD, as every firewall, provides a mechanism to synchronize information, such as states, for example.
But we didn't want to use any of this mechanism, because having a virtual IP means firewall exchanging traffic.
Exchanging states mean create more traffic.
And our problem was we have too much traffic.
So, we installed two different independent firewalls.
Again, maybe installing a dedicated Ethernet with CPU on it could solve the problem about managing all that ERQs.
But finding it on Sunday was quite difficult.
Think that all other phases of the attacks usually happens during night or require hardware that is not normally available in any store street.
So, we had to do what was available in that moment.
Obviously, also managing more hosts was not a problem, because we started with modifying all files by hand.
Then we created some scripts that modifying one file on one host was replicating on every host.
So, we continued with PF and with the hardware that was available.
The first problem about having two hosts was to send traffic to these two hosts.
And it's quite simple.
You have to provide, please, route the traffic with a round-robin algorithm to both our IP.
Sure, it's not possible.
Oh, you are Internet provider?
It's not possible?
No, no, we don't create such strange configurations.
It was quite a problem.
So, we start looking inside the traffic and realized that we were lucky again, because the traffic was coming DNS-based.
So, they were resolving the name and it was good, because that allowed us to configure inside the DNS two different IP for the website and balancing and round-robin the traffic on two hosts.
The attacker never realized that or never took care about we doing that and never switched to attacking directly on IP or we had to change strategy.
So, this was working until the end.
The other problem to face was about the states, because if you send traffic...
Okay, I hope... I need to see the animation while talking on it.
I feel a rock singer.
All traffic coming from users reach, obviously, the customer infrastructure.
The problem is that traffic that come back...
And here, decide what is the default gateway.
That come back from the same path is authorized, because it exists on the firewall, the state.
But the traffic that is asymmetrically routed is blocked, because no states exist for that traffic.
And this was quite a problem.
So, how to solve this?
As usual, when you have a problem and look like there is no simple solution, you can try to use a different technology, talk for something different and use it to do that.
For example, NAT is used often to NAT the traffic that is going out from your network.
But we implemented that for the traffic coming in.
So, the result is that all traffic coming from users to the infrastructure look like coming from those two hosts, only those two hosts.
For this reason, no default gateway was needed, and all traffic was routed back to the correct firewall that was denoting the traffic.
The only con was that we were losing a lot of information about statistics of IP addresses accessing the data center, but we need statistics when your network is working and you're facing an attack.
So, we decided it's a good solution.
While doing all this, we also worked on CCTL and all system parameters to find out something to optimize buffers and so on to have those two firewalls running more smoothly and faster.
So, we asked the provider to take off the filters on international traffic and everything was working fine.
And the bandwidth was more managed, was more than before, and was correctly managed.
And we saw the traffic raising, raising, raising again.
And again, both firewalls got unresponsive.
So, as I said before, we had a lot of rack servers there in a corner Replicate!
So, imagine all those people putting firewalls inside the rack and this network manager from Turing configuring through a remote connection on another connection because the first one was full of traffic installing OpenBSD asking the provider to provide more bandwidth.
And again, with all those firewalls, we were able to manage the traffic.
But look at us, imagine about us in front of the monitor looking at CACTI graphs going up, up, up, up until it arrives at 850 megabytes.
It stays there, it stays there.
They are joking, maybe they have finished money, have finished hosts, who knows?
Still there, still there.
We won, we are ready, we were so cool!
Imagine that all this can be told in some minutes like I'm doing now but imagine it all managed by phone because the Turing guy was phoning me, I was phoning the Milan guy that was phoning the Venice guy, the Venice guy was calling me back then I was calling the Milan guy back again and so 24 hours round.
Those are some of the graphs that we were able to collect during the attack.
Then we were, again, so comfortable.
We were smarter, we outfit everything.
Wonderful.
And then the traffic starts going down, down, down.
I finished money, I hope so.
Then another some kind of traffic starts rising.
It's no more SynFluid, it's normal HTTP connection.
So it was basic, I get fluid.
Again, not a technological problem because on OpenBSD we can rate limit connection.
It's not a problem until you know how to rate limit because the syntax is very simple, as you can see.
You find every host that is exceeding the rate limiting and you put it on blacklist.
The real problem is having this number.
What is the border of rate limit that cut out the attacker but keep all the customers inside?
The task here is not to cut out customers, not to rate limit there because it's very...
Okay, you understood.
So we were able to find that parameters.
Again, the attack was mitigated and then we saw that another trend of get fluid was arriving but was no more many hosts continuously refreshing the home page but was many hosts refreshing at a rate lower than the rate limit, only one page.
That was the page that had two characteristics.
Was the worst query on the database and produced the longest page.
In that moment, their page were not divided for, for example, 10 or 20 results a page.
Was a single page full of results.
Imagine people renting flats on Milan.
Huge query.
And here was quite a problem because...
At layer 3, you cannot handle with this.
Because you cannot rate limit, you have to go at layer 7 and look inside the traffic and decide to filter out this in some way.
But it's more applicative.
So we start talking with the software house that were managing the balancers and keeping statistics and so on, asking them if they had any idea.
And their manager told me, Yeah, yeah, we have an idea, but we need to develop it.
So try to do something.
Manage the situation for one or two days.
We will come with something.
Then we start thinking about what to do.
And the owner of Bakeka turned out to have some friends that were really suggesting him to buy a filtering service from some company outside.
Usually, I tend to disagree with people that want to send their traffic to someone else, even for privacy.
But it was an approach wanted by the customer and we followed them and we helped them.
Remember that we were managing a lot of hosts that were constantly changed.
Our blacklists were very long.
Having those servers being taught to be web servers, so had gigabytes and gigabytes of RAM, helped a lot in managing all huge blacklists.
So, as I said before, the software house needs more time.
And we started dealing with this traffic laundry that promised that you send us all your traffic, we will clean the traffic and send you back only clean traffic.
Well, the first one was Cayman-based support, only through Skype.
No phone numbers, only a fax number to send a contract signed.
We modified the DNS, sent them the traffic, they phoned back on Skype, telling us, please, please take your traffic back, we don't want to handle nothing for you.
Okay, that was a strange company, but there are a lot of professional companies in this field, you know.
And again, the customer chose another company that received all the traffic after we turned on the DNS.
They called us back, said this was a serious company with email support, phone support, and so on.
They phoned back saying, oh, you're really under attack.
Oh, fuck.
Yeah, and then, well, the filters that we usually provide are not taught to deal with this kind of huge attack.
So you need our plus contract that involves a lot of rocket science technology that can...
And if you send money, okay, okay, send money back to us, no problem.
So it was a really bad experience.
I don't want to say that does not exist on the market any company able to manage this kind of attack on their data centers, but our experience was really, really bad.
Those are the graphs that one of these laundry sent us.
We are really facing an attack.
So while we were playing with DNS to send them traffic, the software house was able to identify inside the GetFluid some parameters to cut out at balancer layers the GetFluid on that specific page.
They use, obviously, some application technology.
Again, nothing so complex or incredible, but the risks is always that when you apply these, you cut out customers.
And if you do that, you're losing money.
While doing that, the other thing we did was to install a second MySQL server as database to avoid a single point of failure, but also to host a load balancer to obtain more performance, and this helped a lot in mitigating the attack.
And again, the infrastructure was up and running really smooth.
We were really, really tired.
At this point, two weeks were passed, and I think that if my wife didn't decide to broke with me during this situation, we'll never do.
Because imagine that I was sleeping with a headset in my hair and a phone under the pillow.
And I went to the restaurant with laptop with me, and in the middle of the dinner, phone call, so open the laptop and connect, and so on.
Every day, every time, every hour.
But finally, the infrastructure was up, running really smooth.
We are the winner.
Party.
Yeah, party a lot until in the middle of the night, ring your phone call saying, Oh, I forgot to tell you one thing, told me the network manager.
The DNS servers are not in the same data center, are in another place, and they have a very few bandwidth.
So they are flooding the DNS, so no problem.
We install bind on, or some other DNS, I don't remember, inside the host in the infrastructure, and had two reliable DNS with one gigabit bandwidth and some firewall in front of them.
And everything was running smooth again.
And again, we thought when everything was working good, we thought we can have a party, and they had finished.
Well, one of the satisfaction of all this is that I had no sleep at all.
But the guy on the other side has no too, because his reaction time was often about two or three hours after our configuration.
So on the other side, we were not sleeping too.
Another little detail about how the site works.
For private users, the site is for free, and everyone can go on the site and create a new announce.
You only need to insert this email.
You receive an email saying that you just insert this announce with these details and so on.
If you confirm the email, your announce is published.
And then the attacker created a script that inserted a lot of announce.
The problem about that was not having all the announce on the queue.
Because with some scripting, you can easily find them and cut them out.
The problem was that every announce was generating an email, and the email server's queue was full.
And there are not enough details inside the email to search in the queue those emails to be deleted.
So the only way was to find a really fast solution to get all that emails to go away.
I'll tell you a secret.
A very bad thing.
It's more than 10 years that I complain with other vendors, with customers, and so on, that say, Did you look at my new wonderful device?
I built this new firewall that acts also as a fax server, file server, email server, and so on.
Okay, that's not a firewall, it's a sheet, but...
I think you will agree with me, a firewall is a firewall until its only task is to filter packets.
It cannot do anything else.
But having their hacked servers that were only filtering traffic was too tempting.
And we activated SendMail on every host and configured the mail servers to use all eight hosts to relay emails.
In this way, we were able to split the queue and have everything going out, even because those SendMails are OpenBSD, so it's really different from the normal one.
It's quite more secure.
We're configured to be very aggressive in timing, even because, as you can imagine, most of the email had non-existent domains, non-existent users, and so on, so we're generating a lot of bounces, and so on.
And this was working very good.
After all these emails flow out, we lived for many days thinking, and now, if I would be the attacker, I would do this, I would do that, another option could be doing that.
Lucky ones that maybe they have finished money or something else, nothing happens in the day after this.
So this was the last, in the middle of the night, call I received.
About all this situation, the thing that really I found amazing was the approach of the customer with the problem.
I said before, they are a marketing company.
Maybe they know nothing about IT or too much.
Usually you know when your company is under attack, the approach is, OK, let's hide everything.
Let's say, attack?
Which attack?
No one is attacking.
Oh, it's just a hardware fault or something like that.
He decided to approach the problem in a completely different way.
He started advertising the attack, buying advertisement on newspapers, talking with bloggers, creating a media campaign saying, our site is free for everyone.
Someone don't want our site to be available to everyone.
Someone don't want freedom on Internet.
Fight with us.
It was a real cool approach.
They paid a lot of newspapers to interview me as the expert to describe the attack, to explain what was happening, and even some technical details.
Asking for the customers to support them because they were in difficult because of their believing in freedom.
This was a really cool move because not only they didn't lose any customer, but they get new customers because of all those advertisements.
So, in my opinion, this approach was really cool.
I think that in the first month, during the last two weeks of the attack and other two weeks, 200 people wrote on the site they opened about the attack.
We support you, we are with you, Internet is for freedom, and so on.
200 people that decided to use their time to support this company was a really amazing example of incident management, in my opinion.
So, we are near conclusion.
You remember that the infrastructure when we start.
When we were at this point, this was the new infrastructure.
Now, everything is changed from this point because when the emergency finished, we took the time to sit down and design a network that was told to afford this kind of problems.
So, they are no more in that data center where people don't know what to do.
They have a different infrastructure, so all these details are now obsolete.
And, as you see, a lot of traffic, but mainly a lot of hosts were involved in this.
It was quite a big botnet that we saw during this attack.
As I said before, we were lucky from different point of view.
From a technical point of view, because the attack was DNS-based, we were in data center and so on.
But also, because we were talking with a company that had enough money to protect their business.
If they didn't have enough money to pay one gigabit bandwidth, the attack was successful and no one can do anything else.
They were able to pay consultants.
Often, there are a lot of companies that prefer to stay down than to pay something to manage the situation.
And something that I really appreciated was that all people I was working with, except for the ESP people, were really smart.
From the network manager that became an OpenBSD guru in a week.
And all people that were like me, responding to the phone every hour, every moment, always, always, always.
Obviously, when you have to manage with this kind of attack, it's always a nightmare.
When you are leaving that on your skin, it's very painful, because in many moments...
Now, talking about that, I gave you what we did.
But when you are there, alone, in the middle of the night, thinking, and now, what can I do now?
It's quite painful.
But everyone involved in this adventure had the hand, was really smiling and happy to say we were able to defeat the evil.
Another interesting aspect of this is, as I said before, I went to a lot of conferences during those weeks.
And you know, often, you don't remember everything about syntax by memory.
And so, imagine me, by phone, outside here, walking, talking with the customers, and then asking whoever passed, do you remember that option of TCP dump that helped me in finding that particular...
So, a lot of people were helping out only with a suggestion or because they were reminding about something.
A lot of people, friends, reading on newspapers what's happening, emailed me, phoned me, do you need any help?
I have something that can help you.
A friend that is the... I don't know if you know NTOP, is the program to network the monitor, yeah?
Luca Derry is a friend of mine.
And he phoned me to ask, oh, I'm testing some 10 gigabytes network card with CPU on it.
If you want, I have one here, I can send it.
So, a lot of support, even morale, from a lot of people.
And those two guys get a lot of rants because they are colleague that I was calling during night, telling them, oh, and now what can we do?
And so on.
Also, I want to finish with a little photo that I really love, that is a speech as Emperor Palpatine that say, don't be evil.
So, I try to be quite faster to have some more time about question.
Please.
I'm recorded, yeah?
Well, look like some... we have no proof at all.
It's just my imagination.
But some weeks before the attack start, the owner of the company received an offer from a competitor to be bought.
And he, in a real rogue way, told them, well, wait, we are too small now.
When we will be more expensive to buy, then come back again.
I don't know if those two things are related, but was the only option we had in our mind.
Standard, standard.
As I said before, they just bought 10 of them to put in the rack as new web servers, so we're legacy servers.
As I said, we did with what was available in that moment.
Please.
We tried...
No.
We tried to figure out the only certain data we were able to track was the contemporary hosts generating traffic were 20,000.
But they were rotating in something like 15, 20 minutes.
If a host that was appearing in the morning was coming back in the afternoon, we were not able to have a matrix to track that.
But in some moments, the blacklists were over 200,000 lines, hosts.
But we were cleaning that because of tests.
Often customers went in.
So no trusted data.
Please.
It seems that in this case...
Louder.
It seems that in
this case...
No.
From a low point of view, the only information they gave us was if these things go too far, we will cut you out because maybe you will damage other customers in the data center.
It was the only thing they say about the contract.
From a technical point of view, their only answer were...
Or we don't do that.
Or what are you talking about?
And it's one of the...
It's not the biggest, but one of the top five data center in Italy.
Please.
Can you repeat slowly, please?
It's
traffic.
We didn't make any tests.
Legacy machine we saw at 180 megabytes.
Drop connections.
But we didn't have time to put more processor or change something.
Just a moment.
There was a guy from before.
Let's say yes.
I already had in the past some experience with them.
I give them some consultants for the new data center that they were opening.
But they had never opened that data center.
Just built the walls and designed the network.
So we already know each other.
But at the end of the story, nothing go farther.
I offered, but you know, if I help them, I want money back.
And they decided it's not okay.
So...
OpenBSD
conference helped me a lot in deciding about OpenBSD.
Even commercial.
My company, in my company I decided to use OpenBSD because I find it more flexible from this point of view.
Because usually if you want to install a firewall, for example, a normal distribution you have to install and then take away things.
In OpenBSD usually you install and add whatever you need.
Because it's more clean.
I prefer it, but it's like religion.
It's not true, it's my preference.
Bakeka was a customer of my company.
Here I decided to take away my company's signs and so on.
No, no, no.
They had, during the attack, they are in Turin, in Italy.
And in Turin one of my best friends, that is Raul Chiesa, that was friend of the owner of this.
And he called Raul asking, Raul, I know that your company deal with security.
Can you do something for me?
No, our company do other things, but I have a friend that is in this.
And gave them my phone number.
So we were engaged with my phone.
Yeah, yeah.
Okay, if there are no more questions, I can close, so next speaker can arrange the podium.
Thank you very much.
MW2, feel free to write me or follow me on Twitter or add me on Facebook and so on.