Here we go.
So now we're going to set up one broker, one machine, and then we'll set up a cluster in the next
lecture.
Okay.
So to set up one broker, by the way, just a quick note.
If you're having network issues or whatever and you're just wondering, well, don't ask a question
right now.
Just go to the lecture called Can I Connect from the outside and Understanding Communication with Kafka,
They will help you figure out most of the network issues.
Okay, We shouldn't.
If you've been following all I've been doing from the very start, exactly like I've been doing, you
will not have network issues.
If you've been trying to do your own thing.
You may have network issues.
Okay.
But I'll tell you what you may experience in the next lecture.
Okay, so we want one broker, and first we're going to augment the file handle limits because Kafka
opens many, many files.
And you definitely want Kafka to not be shut down because it has too many files open.
Okay, that's a very common error for Kafka to have.
So we're going to set up Kafka.
Such as?
It can handle 100,000 files, which should be more than enough for your use case.
Then we're going to launch Kafka on one machine and set up Kafka as a service the same way we did set
up Zookeeper as a service so we can do sudo service.
Kafka start.
Okay, so that's about it.
So let's get started with our three machines, and we go to set up six Kafka single.
So as you can see, we're halfway through the the setup.
It's been a pretty long setup and I hope you take your time to do it.
So we do Echo and these basically settings says you can open 100,000 files if you want to.
Okay.
And we append this to this file called CTC Security limits dot conf So we copy this, go on all our
servers, paste it, enter and we're good.
That's what we needed to do.
So we just added the security for it to be taken into effect, you have to do a reboot.
So let's just go ahead and reboot the machines.
And again, for every reboot, you need to wait about two minutes for the instance to reboot.
So I'll just pause the video up until this is done.
Okay, so it's been two minutes for me and I'm just going to switch back into my instances.
And here I am.
I'm back.
All right, cool.
So.
Well, good.
Now I'm going to stop doing things on all machines simultaneously.
Okay.
That was cool.
Now I'm going to stop and I'm going to put the two brokers, three and four, just.
Just maybe right here or something.
Uh, you see, I'm just going to reduce them because we won't need them right now, so might as well
put them on the side.
Here we go.
All right.
So we have one broker open right here, and I'm going to increase the font size.
Okay, Excellent.
So.
So somewhere we want to go now.
Is basically configuring the servers.
Okay, so we're going to edit the server properties and we need to add the server properties that we
saw before, which are all the properties right here in this one.
Okay.
So the best way to do this is just remove the server properties and then create a new file.
I that's the easiest.
Okay, so let's do it.
Here it is.
We do remove CD Kafka.
Very important.
Then we do remove config server.properties.
Then we do nano config Server.properties proper.
Make sure you don't make any typo, obviously.
Then you paste Ctrl X?
Yes, enter.
So remember, that was just like my first broker.
The broker number one.
So that was it.
That was enough.
Um, and the other brokers, when we do this, we'll have to set up two and three.
Okay.
So that's.
That's enough to get started.
And next, we have to launch the server.
Okay, so here we go to launch the server with the custom settings.
We do bin Kafka server start dash config dot server.properties.
All right, let's try this out.
Okay, let's see how we go.
We paste this.
And something's not happy.
File Not found.
Server.properties.
So what did I do?
I obviously named it with a wrong naming, and it's good to make errors, so I'm not going to delete
this.
It's servers dot properties that I named it and it should have been server.properties.
So if we do move config server dot to config server with no s at the end a dot properties.
Excellent.
You can see how many things can go wrong when you set up Kafka.
Just naming files and stuff.
It's pretty horrible.
Anyway, if we go back and and relaunch our Kafka server start.
Fingers crossed, things should be fine.
Enter.
And it's shutting down.
It's still not happy.
It says permission denied.
So what's happening?
All right.
So I figured out what was happening.
It was a permission issue on the Kafka folder.
It's because I've mounted the thing after giving permissions, and I should have give permissions after
mounting.
Anyway, I fixed it in the script in the course, so it's fine.
You'll get the fix.
But just so you know what's happening, you do sudo clone dash r ubuntu ubuntu.
And then directory is data Kafka enter and we're good.
Okay, that fixes the problem.
So I'm don't need these comments anymore, so I'll just put them again, fold them a little bit or actually
we're fine.
And then what we do is we run our bin commands to start Kafka.
So I have to go up a little bit to find it.
Here we go.
Bin Kafka Server Start Config dot Server Properties Press Enter.
And the Kafka server is now started.
Awesome.
All right, so it says Kafka Server one started and I think it still complains a little beginning about
some, some some file not found exception or permission denied.
It's pretty odd because the files are created, so I think you can just forget about those for now.
Okay.
And as you can see, the log gives you the Kafka configuration values.
So we have advertised listeners right here and we have advertised port and and all of these settings.
Okay.
But the most important one right here is that the advertised listeners is set up to Kafka 19092.
The broker ID is one and then for example, auto topic create.
Let's check it out.
Enable equals true.
So every every setting we set was was set and then some of them other all of the others are defaults
basically.
Okay so it started it's connecting to the zookeeper server and it says cool.
It started.
It connected to it.
It ran a lot of Java file.
And then at the very, very bottom you can read through this log.
It's pretty interesting.
But at the very end it says Kafka Server one started.
Okay, so we have a Kafka server started.
That's pretty awesome already.
But let's, let's dig deeper and see what happens.
Remember we had our zoo navigator, so let's just refresh the children list.
And it says session was lost, so we have to reconnect.
But that's okay.
Zookeeper 12181.
We connect, and now we have the Kafka node.
We can actually remove my nodes.
It's not needed.
Okay.
So we have the Kafka node, and if we click on Kafka, it created many, many nodes in Zookeeper.
The cluster, the controller, the controller epoch, the brokers, the admin, and so on.
Something that's really cool that I want to show you is brokers.
IDs and we have broken number one.
And you see it's pretty awesome.
There was a broker registered in this node and it says the endpoint is Kafka 19092.
So very important for you to understand that what Kafka did is creates a bunch of stuff in Zookeeper.
Not for you to worry about, not for you to change, but good for you to know about.
Okay?
When you boot up Kafka, it creates a bunch of stuff in Zookeeper, and it registers all the brokers
into the brokers IDs.
That's why Zookeeper is so important, because it holds Kafka data.
So let's get back to it.
So now that we have our server set up with the custom settings, we can basically, um, we can basically
relaunch Kafka and then install Kafka Bootstrap.
So let's just do it.
We're going to install Kafka Bootstraps in order to have Kafka as a service.
Okay, so now we just have to install the Kafka server scripts.
I'm going to do it on the three terminals at the same time.
Um, again, if you can't do it, do it one by one.
I know it's tedious, but you kind of have to.
Okay, until you automate everything.
So very easy.
Um, what you do is right here you do sudo nano init Kafka.
So let's do it.
And it opens text editors.
The file we have to copy is this one in the Kafka directory?
Kafka.
So we're just copy it all and paste control X, y, enter.
And that basically created that init script for Kafka.
Then we have to make it executable and make it owned by root.
So let's do it.
We're making it executable.
Excellent.
And then owned by Root.
Excellent as well.
It worked.
And then we update the services list using update RC and you can again ignore any warning they give
you.
That's okay.
So that's.
That's about it.
I'm going to stop doing the.
All the servers at the same time.
So I'm just worrying about server one.
But what we gained out of it is now we can do pseudo service.
Kafka start and stop.
So pretty awesome.
So let's do, let's do this right.
Um, first we do bin, we do pseudo service.
Kafka status.
And it's not running right now.
Okay.
It's inactive and start and it started.
If we do status again, we see it says Active.
It's running.
Cool.
So let's just stop this and how to make sure it's working.
We'll do localhost and localhost 9092, which is the Port of Kafka enter and it says connection to port
succeeded.
So we have one broker currently working as a service on our machine and if we do sudo service, Kafka
stop.
It's going to stop the broker gracefully and we do it again, the NC minus localhost 9002 and it says
connection refused.
So pretty awesome right?
So let's, let's keep it started for now.
And one last thing you may want to do is maybe view the logs, which is cool.
So to view the logs, it's pretty easy.
You do cat home Umbutu Kafka logs server dot log.
So let's just do it, clear the screen, copy the logs.
And as you can see now, we're going to be all fine.
There's going to be no more errors, which is what we need basically every time we restart the server.
So that was the last shutdown.
It basically depends.
Here we start at the server and it says, Look at the Kafka config, Here it is, Here's the Kafka config.
Da da da da da.
Starting connecting to zookeeper, loading the logs, blah blah blah, blah.
And then it starts, start, start, starts.
And at the very end it says Kafka Server one started.
So here we go.
We have Zookeeper and Kafka on this machine that are started, okay, which is super awesome.
So we have a cluster of one broker right now and we can, in the next lecture test how to create a topic
producer to a topic.
Read the data and so on.
Okay, and list topics.
So let's do this in the next lecture.