Hi, guys.
In this lesson, I'll explain that and how Kafka's data is taught and how to write a message to Kafka.
Kafka stores data using topics.
Each topic has its own name.
These topics are stored in brokers.
Topics are used for reading, as well as for writing data to Kafka.
To write the data, we use producers producers publish data to the topics and topics, received data
and stored that if we dive deep into the topics, we encounter partitions.
In fact, what we call a topic is a structure consisting of partitions.
In fact, we write the data to the partition, not the topic.
We can specify the number of partitions for each topic according to our needs.
We can specify it according to our events load.
Each partition has a unique number must be called.
Partition actually uses the log principle.
So the things we write are constantly added to the back of the partitions so we cannot edit to the front
of the partition.
Data is right, and to do partition in the order in which we sent it into item data cannot be changed.
So they are immutable.
The data is stored in the hard disk.
Data is not stored forever.
Kafka has two different storage configurations.
One of them is time based storage configuration.
These stores the data for seven days, so data older than seven days will be automatically deleted.
Of course, we can change it.
The second one is data size based storage configuration.
For example, if we set it as 50 gigabyte, the data starts to be truncated, then the sum of the data
exists 50 gigabytes.
This method is not preferred one because the results are unpredictable and all set is assigned to the
each message in the partition.
We can read these data using this offset.
Actually, offset is used to determine the position of the data.
Now, let's see how we can use partitions in writing.
Let's assume that we have three different partitions and two, we want to send data.
OK, but how do we decide which to write?
Here we can design the partitions as we want with the decision we made.
For example, if we don't specify anything, then sending a message.
Kafka uses the round robin method, so it divides the incoming message into all partitions in order.
So it sends the first incoming message to the first partition and the second message to the second partition
and the third message to the third partition and goes on like that.
The other partition design method is record key.
We can give a key to the message that we sent.
Kafka uses these key values and writes the same key witness to the same partition.
For example, let's assume that we have different partitions for payment events, and these partitions
expects different payment types, such as credit card, PayPal or other.
Then we sent the message with the credit card it is sent to the first partition.
The PayPal message to the second partition and the message with the other key to the third partition.
We actually aggregate the data using these methods so we can say that we can aggregate, write and data
according to certain characteristics in partitions.
Let's continue with the partition design methods with record key.
We can provide sorting or event sourcing in the partitions.
For example, we can send customer ID is a record key, and we can store each user's events sequentially
in the same partition.
So here we are, actually doing events sourcing.
So with using can record key.
We can use partitions according to our needs.
Another benefit of partitions is parallelism.
Kafka topics are divided into a number of partitions.
Partitions allow us to paralyze a topic by splitting the data.
We can create multiple partitions and we can increase the performance with parallelism.
Consumers can also be paralyzed so that multiple consumers can read from multiple partitions in a topic
allowing for a very high message processing throughput.
OK, now we have learned the writing mechanism in the Kafka.
From now on, we can check the reading mechanism in Kafka, but we can handle it in the next lesson.
That's all for this lesson.
Thank you.