So we've had a go at building our complaints data set using comma delimited and we hit some troubles
with that data representation and then we tried again using Jason which got us a little bit further.
But again we had some troubles understanding where data were data errors were creeping into our system.
So now let's look at representing our complaints data set in the taxi application using average.
Let's remind ourselves the characteristics of average data represented at Afro always relies on a schema
and each schema has a version or data has an escape associated schema that's stored alongside the data
that's serialized Casey call relies on the confluence schema registry has the backing store for schemas
going in and out of Casey call ever.
Looks like Jason format is actually stored as binary that is AB restores the data definition in Jason
format making it easy to read and interpret but the data itself is stored on disk in a binary format
making it very compact and efficient when accurate data is read the scheme used when writing it is used
to a serialize it and schemas can help support change.
That is data that evolves over time because each schema has a version and changes create new versions
of the schema
so let's remind ourselves when we're talking about the data set the the complaints for our taxi application
we're going to have both the combination of Jason format alongside the schema that represents the data
so we've always got data and schema coupled together when we're dealing with data represented as average.
Let's see how this applies with our Kafka based case sequel taxi application.
I've already created myself an empty Kafka topic called complaints afro and now I want to start writing
some Afro information into this Kafka topic.
Now I can't just use Kafka console producer this time I'm going to be using Kafka Afro console producer
to write Afro encoded data into the complaints Afro topic not the syntax is a little more complicated
than just writing into Jason so I'm going to just walk you through this.
So firstly I'm going to be using Kafka every console consumer with some fairly boilerplate options here.
Now I do need to specify the schema that's going to be used to serialize the data into the aggro topic.
Now this can be specified on the command line and for clarity that's what I'm doing here.
So I haven't written the data yet I am specifying the schema that's going to be used to encode this
information so I'm going to be always specifying the data that I'm about to write into the topic is
to be serialized as an arrow using this schema in front of me.
Press return again it starts reading from standard in so my first data is going to be fairly straightforward
I'm going to be writing customer name of Carol.
Her complaint type is late arrival for her trip cost was nineteen dollars sixty and she is not a new
customer and that record will have been written and encoded as Arrow because of already special specified
the schema.
So I'm going to jump down to my case equal prompt now and just check that it indeed have been available.
Topic NSA these topics and I can see that my complaints approach topic is available.
It's just print the raw contents of that topic so I'm going to say print complaints I borrowed from
the keening
that's good.
So I came.
Even though this information has been encoded as an arrow I can still use a print command to peek into
the information held in the topic.
Let's go ahead and create a stream around this topic.
Now what's a little more interesting this time is I'm going to be creating a stream on that Kafka topic
but I'm not going to be describing the schema.
You'll notice that I've just written the words create stream complaints Afro specified the name of the
underlying topic specified it's an afro but at no point have I had to say what the columns are that
make up this stream because this stream was encoded as Afro and because Afro already has a schema against
it.
I when I create that stream do not need to specify the columns they're already bundled with the data.
That's pretty clever
but we didn't come this far not to have a little peek into the data.
So let's query our stream.
So let's say select start from complaints ever
that's pretty good.
It looks like our record really has turned up.
Now an interesting side note here is how this float has been represented.
We sent the value nineteen decimal point six zero in as a float and when we queried it again it's not
a precise equivalence.
It's actually come up with a slight rounding error.
So when we build up our application a bit later we might need to take that into account.
This is not a true numeric field but that's a distraction for the time being.
What we really want to do now is inject some incorrectly form data and see what happens.
Remember when we put incorrectly form data in previously as either delimited or Jason.
The only way we could discover what was going on was to interrogate the case equals server logs because
we're using afro.
I suspect what's going to happen is we're going to see that error much sooner.
We're going to actually see it at encoding time rather than seeing errors arriving in a downstream server
log.
So let's try that out.
I've already got my my producer running up here that's going inject some incorrectly formed data so
this time I'm going to insert customer name called bad data and his complaint is bad driver.
It's a sensible trip cost but I'm not going to be passing in a boolean and passing in a very wrong field
and see what's happened that's immediately being caught by the average producer.
It's given me a sensible error but this time it's happened right up at the producer's side which is
much closer to the application which means that when we're building our application the error is going
to be noticed right upfront rather than being delayed and being interpreted in case equals server log.
So in short when we've got validation errors it's much better to notice them and build an exception
path around them much earlier in the data flow and using average.
We've had the advantages of being able to do schema conformance right up front right up in our producer.