Let's take a look at what a data lake is really doing in.
Let's say an AWS environment here.
S3 is an object storage system that
has a high degree of concurrency available to
essentially an infinite or near infinite amount of
concurrency and also processing power.
The reason that this is set up in the way it is,
is that you can do multiple things at the same time.
You have the ability to process
both structured data and unstructured data.
Let's say data that's in
a key-value store format versus just raw text.
You can also work on analytics problems,
there are tools inside of AWS like AWS Athena that lets
you do queries across
the unstructured data and create reports.
At the same time, you can do machine learning and use
the infinite disk IO and storage to store models,
version those models, and even
pull training data and actually
create new models that get put back onto the S3 system.
You can also synchronize a on-prem system.
Let's say you have a big cluster of
GPUs in a specialized data center,
you could go and synchronize data back and forth,
maybe train a model on your on-prem and
then push that model and push that
the training data back to S3 to
keep snapshots of it and finally,
you can work on things with real time
and that's a huge aspect of using a data lake.
A couple of other things to think about
would be that because of the fact that
the data is able to be processed
with this essentially infinite capacity,
you don't have to move the data all over the place.
This is a huge win in terms of
building new managed services on top and
that's why things like AWS Sage Maker or the AWS Athena,
the query system, are able to be built on top of it,
is that the data doesn't have to
be moved and then finally,
it also has to be a low-cost storage system.
All of these things combine to
create a value proposition for a data lake