- Let's take a look at our first business question.
Here it is.
Now as with all these scenarios where
we're getting information and trying
to figure out whether and which NoSQL database
would be useful to help answer
this business question, after formulating
the question as clearly as possible,
the next step is to get information
about the Source Data.
Now what I've done for the purposes
of the scenarios is I've just provided the
answers to the questions that are most
commonly asked so I'll show you
what that looks like.
So for this particular case,
the Source Data is in Log files,
and the questions that are great to ask are
do you keep all of the data.
And in this case the client did,
and all the data is kept and archived
because one of the first situations around
these data projects is sometimes clients
are throwing away data because
it's too costly to save it,
and that alone can be a driver towards NoSQL
because it's often much cheaper to store
larger amounts of data than
a traditional relational system.
The next answer is what format is your data in,
and in this case it's a text format,
it's a CSV,
and it's also structured in that
it's an identifier and an event.
So it's important to consider the structure
of the data when you're figuring out whether or not
to use a relational or non-relational solution.
The next step, the next V, you might
recognize here, the volume.
So in this case this client had 1 TB per year,
which in terms of big data is small big data.
It is not in the mini TB, it's not in the petabytes,
and it realistically could be stored
even in some relational systems
depending on what the infrastructure is.
The next thing is the velocity,
and here we have how frequently this is produced.
It is produced each minute.
Now notice I'm not saying how
frequently it's analyzed,
and in this case it's not being analyzed
at all other than on an exception basis,
and this is a very common kind of situation.
And then the last question is can the data
be stored on the public cloud.
Some companies are not either comfortable
or not permitted to store data on the public cloud,
and this can impact your choice of data solution.
So, here's a potential solution, including NoSQL.
Now the Justifications are,
upon further questioning we really only need
to have simple aggregate queries
going against this log data.
And what I mean by that is we need min, max, count
in order to answer that business question of
how's my hardware being utilized
because we can figure out by aggregates
against those logs when hardware is not utilized.
We don't need to do any predictive queries
or anything around machine learning.
Nothing really fancy.
Just simple queries.
We have predictable volumes of data.
We don't have spikes in our hardware usage
because we're an enterprise
that basically runs 9 to 5,
and we have a regular sort of business flow,
we have a moderate volume velocity,
and we have no variety in the data at all.
So hopefully this starts to help you
to think about those V's from big data
in relationship to gathering information
so you can make a decision.
So what kind of implementation is recommended
for this kind of situation?
A NoSQL Key-value or Wide column store,
and the reason for this is the storage of larger
amounts of data is much, much cheaper than
storing it on a relational system,
and even though the customer only has
1 TB of data right now, the idea is that you want
to build a system that can grow over time.
And as they start to make use of this data,
and query it more frequently,
they're going to need a system that's performant
as data volumes increase,
and as query intensity increases.
Now notice my last bullet point here.
Selection of a Key-value or a Wide column
depends on query complexity.
Now this is a subtle, but important point
in that as you start running through these
minimum viable report cycles you'll understand
if simple aggregation can answer
that business question
or if you need complex grouping and sub-queries
and complexity around the queries.
And the reason that you have either Key-value
or Wide column is because Wide column allows
for more indexing in general,
which facilitates more complex query types.
Another consideration in this scenario is
do you have on your Development Team,
or your DBA Team,
any team members who are already familiar
with either Key-value or Wide column.
Common Key value stores that are in wide use
are Redis, Memcache,
and then there's the new player
on the landscape, Aerospike.
Wide columns Cassandra, HBase,
and there are many cloud Wide columns as well.
So in this particular scenario,
NoSQL is the recommended solution because
it meets the needs,
it's able to answer the business question,
and it's much cheaper than relational.