[Music]
Hi, thanks for tuning into Singularity
Prosperity. In this video, we'll be
discussing big data -
more specifically, what big data is, the
exponential rate of growth of data, how
we can utilize the vast quantities of
data being generated as well as the
implications of linked data on big data.
[Music]
Data has been an essential part of human
evolution for thousands of years, we as a
species are pattern solvers by default
and use data as a tool, in other words, we
gather information of our surroundings
to optimize our lives. The earliest
humans passed on information just
through word of mouth, but as our brains
developed we began to record this
information. The first examples of this
data preservation come from over
35,000 years ago, with cave paintings -
this was the hunter-gatherer age, so it
makes sense that most of the paintings
were of animals, a way to pass down
knowledge of the types of animals to
hunt and the weapons used. Rather than a
long game of broken telephone, recording
information was a much more efficient
way to pass down information and allowed
us to do three important things that life
was unable to do before: 1) Expand
knowledge rapidly. 2) Preserve the
knowledge gained over generations. 3)
Build on past knowledge to gain deeper
insights. This accumulation of knowledge
continued for thousands of years, as our
ancestors started to become more
advanced and develop more powerful tools
and survival instincts. Around 3,000 to
5,000 years ago, we reach another
milestone in human data collection.
Whereas previously only information that
correlated with our survival was
obtained and transferred, at this point,
all across the world, in various
civilizations, history and stories began
to be recorded more frequently. Primarily
due to the fact that, we began to become
better at communicating ideas and
expressing our thoughts, as spoken
and more importantly written languages
became more sophisticated. Fast-forward
to the year 300, and we arrive at the
Great Library of Alexandria. Rarely
before or since has a government
allocated so much of its gross national
product, for the acquisition of knowledge.
Every ship entering its harbour was
searched, not for treasure but for
books that could be copied and stored there.
Estimates on the number of scrolls
contained in the library before it was
destroyed range from half a million up
to 1 million, and they contain many of
the core pillars of mathematics, science,
geography, philosophy and more from some
of the greatest human minds in history
such as: Euclid, Pythagoras, and Socrates
to list a few. Derived from the knowledge
in the scrolls came astounding
conclusions: from how to calculate the
mass of the earth to mathematical
formulas that we still use today to map
out the universe and everything in
between, Alexandria contained unbounded
wisdom and knowledge. The Great Library
of Alexandria, demonstrated the
importance of the accumulation of vast
amounts of information and how
deciphering the information could
produce meaningful results that further
contribute to our ability to solve
problems. Other examples of datasets over
the years that have opened our eyes to
the world around us include: The
astronomical data set really began to
take off in the 1500s with Nicholas
Copernicus, Galileo and Johannes Kepler.
By using years of previous data on the
movement of celestial bodies obtained by
Greek astronomer Ptolemy in the year 100,
and by tracking the motion of the
planets over the course of years, there
combined contributions were able to
derive that, instead of celestial bodies
orbiting Earth how many thought, it was
in fact Earth and the planets that
orbited the Sun. The microscopic realm
was opened to our eyes in the mid-1600s
with Anton van Leewenhoek and Robert
Hooke. Through the use of the first
microscopes, we were finally able to see
and make observations past the
macroscopic and bacterial, cells and even
microscopic life were observed. Without
the microscopic dataset, we wouldn't
have modern medicine and the majority of
technologies we see today. Past the
microscopic, the quantum dataset began
to be added to starting in the 1900s,
from the theories of great minds such as:
Einstein, Max Planck, Niels Bohr,
Ernest Rutherford and many others. The
ability to interact with individual
atoms through the use of the tunneling
electron microscope is behind the
significant advances in technology today,
allowing us to fit more smaller
transistors on chips. This dataset is
still in its infancy, giving birth to new
jobs such as nanoengineering, and
there is still so much to explore, which
we'll discuss in future videos. These
datasets are just a small subset of all
the knowledge acquired in the last 2,000
years, and all the information since the
cave paintings but a small blip compared
to the information currently at our
fingertips, our own
personal libraries of Alexandria. 1990
corresponds with the birth of the
worldwide web and the start of the
information age, a data revolution. In
1986, approximately 2.6 exabytes of data
existed, by 1993, in the span of seven
years and three years after the web went
public, that number increased over six
times to 15.8 exabytes.
For a visual representation, before we
move on, an exabyte is one billion
gigabytes and a zettabyte is one
thousand exabytes. The real before big
data, after big data moment came in the
mid-90s with the creation of search, a
way to sift through the already vast amounts
of data on the young web. Another seven
years from 1993, in 2000, as search
engines matured and web 1.0 started to
go mainstream, the amount of data nearly
quadrupled to 55 exabytes, and
seven years from 2000, in 2007, the rate
of growth increased to six times again,
with the total data in existence growing to
300 exabytes. Information added to the
web up to this point was primarily just
digitizing all human knowledge: written,
spoken and experiential. However, 2007 was
a pivotal turning point for big data
as sensor data began to populate more
space on the web and web 2.0 started to
go mainstream. The iPhone not
surprisingly was released in 2007, packed
with sensors such as the accelerometer,
gyroscope and more and started the
transformation to our now digitized
mobile world. In less than seven years
since 2007, by 2013 the rate of growth of
data had increased over 15 times with
4,500 exabytes or 4.5 zettabytes of
data produced, with total data officially
passing the zettabyte mark sometime in
2009 or 2010. The worldwide adoption of
mobile phones has sparked an exponential
explosion in data creation, and has been
the catalyst for the field of big data
to evolve. Beyond just mobile phones,
internet connected devices are evolving
into a field of their own with the
internet of things, which in turn
correlates to the increasing number of
sensors that can digitize the physical
world into data even further. For example,
over the past few years, a big trend in
big data we've all seen is the health
community, with the digitizing and
tracking of ourselves through the use of
smartwatches and other health tracking
devices. We have over 6 billion data
points sitting in our genomes alone! On
top of this, if you've watched my global
connectivity series, there are countless
initiatives to bring the rest of the
world online to the internet, with
right now approximately 48% connected. As
this number of users connected increases,
the rate of growth of this exponential
data production will continue to
increase as well. In fact, as seen in
this picture, we are currently at an
inflection point between current big
data markets and emerging markets.
Onwards from 2017, emerging markets will
begin to produce larger and larger
portions of the data the world produces.
By 2020, conservative estimates pin
total data production at 44 zettabytes
and by 2025 at 200 zettabytes! As seen in
this exponential graph of data
production over the years, we are primed
to reach the yottabyte era by 2030. As
reference, 1 yottabyte is equal to 1000
zettabytes. As an interesting note, beyond
yottabyte nothing officially exists, although
the tech industry is leaning towards
brontobyte to represent 1000 yotta
bytes and geopbytes ti represent 1000
brontobytes. Another interesting trend in itself
to look at is the increase in data produced
per day, from less than a 40th of an
exabyte in 2002 to 3 exabytes in 2013 and
now we are on pace to reach 5 exabytes
per day in 2018. At this rate of growth,
we'll be producing nearly 120 exabytes of
data per day like 2025! To put these
numbers in perspective, in 2013, 90% of
all data we as a human race had ever
accumulated was produced between 2010 to
2013. At our current rate of growth, in
2020, the same quantity of data produced
up until 2013 will have been created in
just that year alone. Our data production
is on an accelerating exponential trend
with no end in sight.
[Music]
So, what is big data you ask, it is
simply large sets of data. For a better
understanding of big data, let's look at
the three Vs, its core pillars: volume,
variety and velocity. We looked at volume
in the previous section, with how much
big data is being produced and the rate
of growth of data generation. Variety in
big data means having either structured
or unstructured data. Structured data is a
tool we've been optimizing since the birth
of the web, data that is organized in
tables within databases and mostly
easily understandable. However, the vast
majority of data on the web is
unstructured, and constantly morphing,
expanding and evolving. Most unstructured
data to us at this point is essentially
static noise, but this is beginning to
rapidly change as we'll later explore. The
real power of big data is being able to
sift through various datasets both
structured and unstructured,
understand them and derive correlations
and conclusions from them. You can
consider big data to be the microscope we
use to peer deeper into the world
around us. For example, let's take a
look at the astronomical data set we
talked about earlier. All that data on the
motion of the planets by itself would
have been useless, but the conclusion
derived from it; that we and all the
other planets orbit the Sun, validated
the dataset. So first the data had to be
accumulated, then a conclusion in the
dataset drawn and finally the
conclusion had to be verified by
gathering more data and other
astronomers verifying it as well.
Verifying the conclusions and
correlations drawn for big data as
accurate representations of the world is
the hard part. Depending on how you view
data, you will draw different
correlations from it, not all of them are
correct or have real-world applications.
As discussed in the previous section,
with sensors almost everything is
quantifiable, from the liquidity in the
soil to radiation in the atmosphere to the
heartbeat and breathing of a newborn. Yet
even now, much our world remains to be
digitized: from mapping of the human body
on a cellular level to better
understanding the genome to in-depth
mapping of the oceans - the list can go on.
The increasing number of internet
connected devices due to mainstream
adoption, smart cities and more will help
with this digitization. Individually these
products are great and help contribute
to our own personal well-being and life,
but the real power is when hundreds of
millions of people contribute data and
we can begin to see patterns emerge from
across the world. However, at this point,
data generation or volume isn't our
issue anymore, it is understanding the
variety in data and datasets and being
able to utilize them by deriving
verifiable conclusions and correlations
that have pratical
real-world applications. When working
with structured data, languages such as SQL
have been around for nearly two
decades to assist in recognizing
patterns in datasets. As big data volume
has grown, additional data frameworks
such as Hadoop have been introduced. For
a simplistic example of a structured
database, let's take a look at something
an ecommerce website, let's say, Amazon,
would use: Let's imagine a structured
database with the columns: gender of users,
categories of products bought and rating
given to the purchased product. By
analyzing the data, they could derive
what categories each gender favors based
on rating given to the products they've
purchased in those categories. To
validate their findings, these results would
be plotted over years, and by doing so even
more patterns would be observed, like the
categories changing with the seasons and
the same repetition year after year. If
only life was as simple as our example,
working with structured data when it
comes to transactions is typically
fairly simple, the real difficulty
arrives when working with multiple
sensors or other ambiguous data sets.
Even more convoluted is unstructured
data, most of the data on the web is
unstructured and correlations cannot be
easily seen or derived. Some examples of
unstructured data include: photos, videos,
social media data, satellite images,
scientific data and more. These data
types and sets are extremely difficult
to convert into actionable insights. To
derive any conclusion from unstructured
data is a long, manual, painstakingly
difficult process, also there is a major
shortage of skills in analyzing
unstructured data. Not to mention, that
working with structured data can be just
as difficult, as humans we can only see
patterns at such a deep level before
things stop making sense to us. With
the increasing popularity of machine
learning
this is starting to rapidly change, as
we'll explore in our upcoming AI video
series. As stated earlier, big data acts
as more of a microscope, allowing us to
peer deeper into the world and draw
correlations and conclusions from the
patterns we see through the use of
powerful algorithms and machine learning.
The role of data scientists is to be
able to derive these conclusions, by
taking data from various sources and
deciphering them by shifting and
transforming the data to understand it.
In other words, to recontextualize the
data and put it in formats we can
perceive, think and talk about and then
taking action on them. Machines can
derive the actionable insights but it's
up to us to actually take action on them.
[Music]
The role of data engineers is to get
data into a structured or usable format,
whether from sensors or converting the
unstructured data constantly being added
to the web. It is to be noted that new
types of data are constantly being
uploaded to the web, further increasing
the complexity of unstructured data. Also,
to be noted is the fact that data engineers
and scientists were jobs that did not
exist not too long ago,
further exemplifying how fast this field
is growing. The percentage of data on the
web that was in a useable analytical
format in 2013 was a 22%, by 2020
this is expected to reach 37%. What this
means is that most data on the web is
simply unusable, but through the efforts
of data engineers this is beginning to
change. We can also help identify data
through the use of tags, hashtags and
metadata such as: video descriptions and
transcriptions, allowing the data to be
more understandable to us and computers
for big data analytics. What many of the
statistics mentioned in this video don't
take into consideration is a huge
paradigm shift data is currently
undergoing, linked data. When we
transitioned from spoken to written
language and then to digital, each time a
revolution was needed in the way we
communicate to further advance our
species and technology. Linked data
does this once again for us and our
devices, the next step in the data
revolution. If you watched my video on the
evolution of the web, you'll know that
link that is critical for web 3.0, the
Semantic Web, in actually giving context
and meaning to all the information we
see. Linked data is also critical for
laying the groundwork for future
technologies to come with web 4.0. So,
what is that data and what does it fix?
Linked data is simply a way of structuring
data, so when you're ready you can easily
share with the world and aims to solve
the data isolation and hoarding issue, as
well as reduce the data clutter on the
web.
Let me explain: If you publish a document
say an excel file on the web, that is a
fine way to get your information out, if
the purpose is just for reading, say in a
blog. However, the data behind such a
document is locked away and inaccessible
to the public. If a change is made to the
data in the document, that previous post
and all other posts that relate to the
previous dataset are essentially
rendered useless, thereby increasing
useless data clutter on the web. Linked
data is about applying the core
principles of the web, sharing
information, on a much deeper level than
just presenting the information. Linked
data gives each entity on the web its
own Uniform Resource Identifier, URI,
just like the URI aka URL given to
websites. This will dramatically change
the way information is shared
stored on the web. For example, say there is
a post on a scientific journal
discussing the correlation between
atmospheric conditions and the growth of
tomatoes, with data obtained through
atmospheric sensors over the past two
years. Without linked data, while the
article would be useful in assisting
farmers and gardeners to grow better
quality tomatoes, the data would only be
viewable not interactable. Also, the
atmospheric data set would have to be
continually updated through new articles
as more data was obtained and new
correlations derived, further increasing
data clutter on the web. With linked data,
the latest captured sensor data could
be continuously updated to the article
and bring deeper insights into the best
conditions for tomato growth. Also, other
farmers and gardeners could begin to
post their own articles on the growth of
other crops, with additional information
such as soil moisture and more, all
linking back to the original dataset.
So instead of the article just being a
static web page showing the best
atmospheric conditions to grow tomatoes
based of past data,
this simple article now evolves into a
platform of its own, becoming a guide to
growing various crops based on
atmospheric, soil saturation and other
conditions. As more people come to this
hypothetical platform, it would
continually evolve and because of the
structure of linked data, big data
scientists or machine learning
algorithms could go through the data and
begin to find correlations that
weren't previously known. This video clip
should provide a better representation
of the power of linked data: An app
developer has an idea, she uses the City
Council open dataset to find every drop
curve location in the city and creates a
route finder specifically who will
change. The council learned about that
and felt inspired to Commission a
website to viusalize this data on
upcoming roadways, helping commuters to plan
their travel. This web developer is big
into nature, so in his spare time he
creates a similar websites with an
open data set from a natural
conservation charity. His impressive site
puts up a ton of shares on social media,
earning the charity free boosts in
exposure and donations which they use to
fund a new data set. Which quickly gets
linked to a university data set
being studied by a team of researchers,
the link to your data reveals patterns
in thier data leading to a game-changing new
discovery. With linked open data, a whole
new world of opportunity is at our
fingertips. An example of simplistic
linked data we can see in our everyday
lives already, is the increasing complexity
of Google search. When searching for a movie
for example, Google will pull information
from: IMDB, Rotten Tomatoes, Wikipedia,
YouTube and other trusted sites as well
as even display show times from your
local theater. Along with reducing data
clutter since data updated in one
location can be updated across all
locations that it is linked to, linked data
will also reduced data hoarding and isolation.
Much of the data on the web is stored
in what many like to call, data vaults or
silos. While sometimes these are necessary
for protecting sensitive information,
often useful information is stored away
as well due to the vast volumes of data
some companies generate. Also, due to
internal office politics, regulatory and
a variety of other reasons often data
between departments in the same company
will even be isolated from each other.
One person's useless data is another
person's key to success, as linked data
becomes more prevalent, information
stored away in these silos can begin to
be unlocked and networked together so
more insightful correlations between
various sets of data can be drawn. Linked
data will not only be critical for the
transfer human knowledge, but a way for
machines to efficiently communicate with
each other as well. For example, say a
sensor in your back yard shows the soil
is dry and the sensors the weather
network are using are predicting rain
fall. Instead of wasting water in your
garden, your home personal assistant will
tell you not to water the plants because
of a thunderstorm later, as the two
senses were able to seamlessly and
efficiently communicate between each
other.
This machine readability linked data is
oriented around is a key feature for big
data analytics, providing a structured
format for data added to the web and a
way to convert current data on the web
into a structured format. This added
structure will allow for machine
learning algorithms and big data
scientists to better navigate through
the data and use more of the data
available on the web. The information on
the web will begin to communicate amongst
itself and even draw conclusions from
itself like a living organism. Linked
data sets are beginning to grow
exponentially, from none in 2006,
30 in 2007, 300 in 2011 and so far 1150 this
year. At this exponential rate of growth,
we'll hit 5,000 by 2020 and 25,000 by
2025! These numbers may appear small but
each data set could have anywhere from
tens,
hundreds, thousands or even millions of
entries. For example, DBPedia has over 4.5
million. Not all data added to the web
has to be uploaded in a linked format and
it is nearly impossible that it ever
will. However, it is critical that we get
the most important pieces of data linked
together, such as: medical research,
traffic maps, environmental data and
other very important datasets. As time
progresses, as with any technology,
exponential adoption will bring change
faster than many expect. Linked data will
open up a new era of innovation, paving
the path for future technologies and
better utilization of current technologies
and data. Blockchain is aimed to
radically transform linked data as well,
but that is a topic best left for
another video. The third V of big data
is velocity. While earlier in this video
we looked at one aspect of velocity, the
rate of growth of the volume of big data,
velocity also means the rate of analysis
of big data. In today's growing mobile
society people demand real-time results.
I'm sure many of you have experienced
frustrations when a webpage takes longer
than a few milliseconds to load. Linked
data assists in increasing the velocity
of data processing, since as discussed
earlier, changes made to data in one
location will be propagated across the
entire network. Unfortunately, it's not
always as simple as a change to an
existing analyzed dataset. With the size
and complexity of big data sets yet to
be analyzed, even a supercomputer would
take days, months or even years to derive
any conclusions from data let alone
deliver it real time to devices across
the world. As we'll explore in the next
video in this digital infrastructure
series, cloud computing will play a
pivotal role in big data analytics, allowing
everyone to access big data insights
from individuals, companies and startups
in real time. Also in the next few videos
in our big data series, we'll explore
some of the use cases of big data as
well as the issues big data poses to
society. At this point the video has come
to a conclusion, I'd like to thank you
for taking the time to watch it. If you
enjoyed it please leave a thumbs up and
if you want me to elaborate on any of
the topics discussed or have any topic
suggestions, please leave them in the
comments below.
Consider subscribing to my channel for
more content, follow my medium
publication for accompanying blogs and
like my Facebook page for more
bite-sized chunks of content!
This has been Ankur, you've been watching
Singularity Prosperity, and I'll see you
again soon!
[Music]