Let's talk about schemer evolution with afro.
So our taxi booking application is alive and quite successful but we've had a business requirement changed
to add a field to our complaints schema.
We want to represent the number of rides taken by our customers whenever they complain.
We want to know how many rides they've already taken on the system.
Perhaps this is important for customer loyalty.
Perhaps we went to ensure that our loyal customers who use the taxi booking application in the past
have their complaints.
Expedia dated if they've actually book with us many times before.
Let's talk about our user Ed.
Ed is complaining he's received a journey in a dirty car.
He's not a new customer.
In fact he's taken 22 journeys with our taxi booking system.
Perhaps we want to have a deep dive into what's making Ed unhappy with our service.
But the problem is that we've actually had this stream up and running and we've got a light live operational
system.
We can't just change our schemas or can we actually because we're using afro.
We have the ability to change our schemas to evolve our data structures on the fly.
So let's talk through how we're going to introduce change introduce an additional field to a stream
that's already up and running but before we jump down to the command line let's take you through some
of the components of the complement platform and specifically we want to introduce complement control
center sometimes just called CCC.
Now we've been dealing with a whole bunch of components but it's worth calling out how each component
has been licensed.
Firstly let's go over some of the Apache license components that you've been familiar with Apache ZooKeeper
and Apache Kafka.
As part of the complement platform before so I've been dealing with some community license components
such as schema registry and case equal itself but let's also introduce the confluence platform enterprise
license.
So confluence control center is a super useful way of visualizing schemas and schema evolution.
It's worth stressing it is optional for these exercises but it's a super valuable way of being able
to visualize some schema revolution that we're going to be exploring in this exercise.
It's also worth pointing out that it's covered by the developer license which gives you free use as
long as you're using a single broker Kafka Costa as is the case in this exercise.
Let's get into it
although optional.
It's gonna be super useful to use consequent control center for this activity.
It gives us the ability to visualize some of the schemas and be able to understand the effects of schema
and schema evolution.
But before we get started let's just remind ourselves what we have running at the moment let's just
type confluence local status to remind ourselves of the components that are up and running at the moment.
You'll see that we've got zookeeper Kafka schema registry in case call up and running but you'll notice
that control center is currently down.
That's bring that up.
So what we're gonna do is we're going to say confluence local start this is just the last step to bring
up the remaining components.
That's the Kafka risk proxy Kafka Kinect case SQL Server is already running and finally control center
Okay that looks like it's done.
Let's just remind ourselves what's finally running.
We can just type confluence local status and you'll see that in fact all of the components are up including
inclusive of control center
now the control center is up and running.
We can just fire it up by navigating to our local host and then port 1921 now where you've hosted your
conflict and control center will be quite specific to your environment if you've installed everything
locally it'll be local host 1921 but if you're using a remote machine make sure you put the IP address
of that remote machine regardless you'll most likely find that conflict and control center is brought
up on Port 1921 now you'll find that the confluence control center is pretty straightforward to navigate
around on this landing screen you'll see pretty much an overview of number of clusters you've got available.
In this case we've just got one cluster we can dive into that cluster and have a quick glance at some
of the statistics on that system that's production rate consumption rate uptime and some other great
attributes.
One of the great things that we might want to initially look at is topics we can actually look at the
topic list here.
For example you can jump into our user profile topic and see some of the messages being created
but I'm going to now have a look at a recently created complaints approach topic.
Now if I jump into this topic I can then actually look at the Associated schema because this was encoded
at approached the console and control center gives me a great way of just navigating both into the data
and the schema that comprises that afro schema so you can see here if I go into complaints ever I've
got both the key schema and the value schema.
Let's just look at the value schema here.
You'll see that in fact I can visualize the schema that is associated with this data of in fact what
the four fields visible against it and just for note.
Now we've got version 1 of that schema which has those four fields.
Let's just put that off to the side for the time being so my goal now is to write some new Arrow serialized
data using a new schema that is a schema with the additional attribute of number of right now.
Again I'm going to be using Catco every console producer to produce every serialized data into the existing
Kafka topic of complaints ever so you will see that this is a syntax that we've used before but this
time I'm specifying a slightly different schema that is a schema that has this fifth attribute of number
of rights.
So I'm just going to write some new data in of a new form that is with that fifth field so the customer
name is Ed.
His complaint is he was complaining about a dirty car the trip cost the new customer flag but importantly
there is an additional field of number of rides 22 so Kafka every console producer has taken this Fifth
Element and attached it to the new schema.
But let's go and check how this appears in control center
okay.
So remember we can just fire up control center and let's now go and look at that recently changed topic
so I'm again going to jump into the topic browser going to complaints ever go look at the schema and
this time you'll notice and now looking at version 2 of the schema and lo and behold we've now got one
two three four five fields against that schema.
So we've actually seen that number of rides has been included as that fifth field.
What's it really useful about this user interface is we've actually got the history available to us.
So if I click this button Virgin history you'll see that I've got this little dropdown now so I can
go from Virgin to back to version 1 version one has four fields version two has five fields what's even
easier is I can actually turn on Virgin diff and I can see two versions side by side you'll see that
the change between version one and version 2 was the inclusion of the fifth field number of rights.
This is the great way of being able to see the evolution of a schema the data is still there but we
can actually see that the shape in the schema has evolved over time.
Let's just put that off to one side for the time being so let's remind ourselves what we've done so
far.
We've written in some data initially into the Kafka topic using the first version of that schema which
had four fields and then we wrote an additional record into that same Kafka topic with five fields and
essentially generated a new schema.
So how does this look in case equal we can just say select star from complaints
Afro to see the existing data.
So here we see Carol's record and remember that came in with four fields against schema version one
and it does in fact have one two three four data fields.
Now remember Ed came through with five fields it was schema version two but this is surprising the complaints
Afro stream only shows for fields against its record.
So what's going on.
So what actually happened here is that when we created the complaints Afro stream we did it against
version one of the schema we did it back when we only had Carole's information and we did it when we
only had one version of the schema in the schema registry if we'd just simply describe the stream we'll
see that in fact it only specifies those four attributes customer name type cost customer there's no
number of trips attribute here.
So when we defined the complaints rose stream we did it against version one of the schema where it only
had four attributes so this is great.
It keeps consistency for our consumers.
We've introduced a data change but we haven't broken anything anything that we're still expecting the
ad stream to remain unchanged is continuing to operate.
But we obviously did this for a reason.
We want to introduce that change to additional consumers and we can actually support this side by side.
We can do this by creating ourselves a new stream.
We'll just call it complaints at Roe the two against exactly the same underlying Kafka topic and again
specifying that the value format is average.
And now we have two streams running side by side.
If we say live streams we'll have the initial version which we know has the initial schema with 4 attributes
and the new version called the two let's describe that so if we say describe complaints Abra the two
we in fact have that new attribute the number of rights.
So we've now got two streams pinned to the same Kafka topic and we can support the old version and the
new version.
Side by side.
But what's even better is our new version can also select the same data even though we've now got a
mixture of serialization against it so if I say select start from complaints Abra the two will have
all of the old data and the new data set the new data does have the additional attribute of number of
rights 22 and when we wrote Caro's record in at a time where that attribute didn't even exist we just
get a null back.
And that makes sense.
So before we let this lesson let's just have one quick additional look at Confluence control center.
If I go into the K sequel tab you'll see that there's a whole bunch of additional things that we can
do right in the user interface.
For example you can use the interactive command line interface in the case equal Ed. You can browse
streams you can look at and define tables and you can even run and query running queries.
So this gives you a bit of an idea of what's available in Confluence control center.
So before we close out this long lesson let's just have a bit of a recap about the things that we've
discovered.
We talked about how to introduce AB row and how to introduce changes to existing average topics.
That is we talked about schemas and schema evolutions.
We saw both how we could introduce changes to data and how to visualize those changes using the com
flowing control center schema registry.
Hope you enjoyed this lesson.