Welcome back. 1
In the last few lectures, we learned how to work with DynamoDB Streams. 2
In this lecture, I'm going to introduce you to Amazon Kinesis… 3
Kinesis allows you to process streaming data… 4
With Kinesis, we can collect, process, and analyze large streams of data records in real 5
time… 6
This can include log data from various applications, Audio-Video Streaming data, social media feeds, 7
financial market data feeds, and so on… 8
We'll build a simple streaming data processing application to understand how to work with 9
Kinesis. 10
So first, we'll create a Kinesis Stream… this stream will receive Notes created by 11
different users… and then we'll write a simple Lambda function to read from this Kinesis 12
stream and load the stream data into our Serverless Notes Backend table in DynamoDB. 13
So, the Lambda function will act as a serverless consumer of the Kinesis stream… 14
We'll also create a simple producer application that generates streaming data as input to 15
our Kinesis stream. 16
This will allow us to simulate several users adding creating notes simultaneously… and 17
we should be able to see Kinesis Streaming data processing in action. 18
So, let's begin by creating a new Kinesis stream… 19
Open the Kinesis console… 20
Click on Get Started… 21
Here, we have a couple of options… we can create a data stream, delivery stream, video 22
stream or an analytics application… 23
We'll go with the first option… to create a data stream… 24
Let's name our stream, as Serverless Notes Stream, for example… 25
We have to specify the number of shards to use. 26
Shards simply define the processing power or resources allocated to the stream. 27
One shard can ingest about 1000 records per seconds up to about 1MB in total size… 28
For our purpose, we can go with just ONE shard… and that should be sufficient for our demo. 29
Click Create Kinesis Stream… 30
And that's about it… it might take a few seconds to create the stream… and then we 31
can start using it. 32
And now, our stream is active… let's open it… 33
The monitoring tab will show us the read and write metrics for this stream… 34
Since wehave just created this stream, there is nothing to see here as yet… 35
As we write data records to this stream, or read from it… we should be able to see the 36
metrics here… 37
Before, we add records to this stream… let's first setup the consumer application. 38
We're going to use a serverless approach… so we'll create a simple Lambda function and 39
configure it to fetch records from this Kinesis Stream. 40
So, let's do that in the very next lecture.