KARIN GICHUHI: Good afternoon.
My name is Karin Gichuhi.
I am the team lead for Strategic Information in
the Office of HIV/AIDS at USAID.
I wanted to start this presentation by
acknowledging three other co-presenters couldn't
join me today, and their names are listed below.
They're all members of the division in which I work:
Aaron Chafetz, who is one of my Data Visualization
Specialists; Ramona Godbole, who is one our
Expenditure Analysis advisors; and, Catherine
Nichols, who is one of the data analysts on my team.
So I didn't put all of these slides together
myself and I wanted to acknowledge them before I
get started.
Today I want to talk a little bit about being a
critical consumer of information.
And I titled this presentation: Evidence: It
is all about interpretation?
And I want you to think about that as we go
through today's presentation.
When we talk about being a consumer of information,
what do I mean by that.
Many of you in the audience today are in
school, and you may be working with data sets or
a task of data analysis; others of you are in the
workplace and you might be having to make
multimillion dollar decisions based off of
information you have ten minutes to review, and so
I wanted to spend a little bit of time talking about
the responsibility we have when reviewing data or
putting data together.
Increasingly, we have access to more data, or
what we call strategic information, and even
reading the newspaper or you Facebook account
presents you with a challenge on whether or
not to reference the information shared with
you as you go about you day.
It becomes increasingly difficult to discern if
you are in the presence of useful or useable or
validated data, or what I like to call data vomit.
I put my information in two categories of data vomit.
There's the situation where you have more
information than you even know what to do with,
that's one form of data vomit, and then the other
group of information that I like to call data vomit
is where people have pieced pieces of
information together to tell the story that they
want to be told.
I boiled down a critical consume of information
into three pieces.
It's about being curious, it's about being clear
with the question that you want to answer, and
there's also a responsibility about
maintaining data integrity.
So how I apply these concepts to work, and what
I encourage everyone on my team and the people I work
with to do, really revolves around the fact
that when reviewing information it's important
to ask questions about the data source, about the
methodology, and understand how the
summarizing statement was determined.
Not all data sets or summary information is
even appropriate to talk about related to the
question somebody wants answered, and maintaining
data integrity means don't cherry pick the
information to tell your story.
I wanted to share a few recent headlines that I
reviewed last September and they all have to do
with real life examples of this.
In the first slide, you'll notice that I mentioned a
number of individual decisions we make
throughout the day.
Whether you're grocery shopping or determining
which doctor to go to, a lot of information is used
to make financial based decisions: what should I
fund, what should I buy, but we also need to
consider the responsibility of
distilling down the results of a study to a
single headline.
And I wanted to highlight a few things that came up
recently.
The first, in September of last year, the Census
Bureau released their latest report indicating
that household median income grew in urban
areas, but declined in rural areas.
This was in five of the major newspapers, yet the
data set, released separately and in a
separate report days later, showed that this
was not the case.
We hear over and over that baby boomers and
millennials ruin everything.
In a VOX article they quoted that "baby boomers
and millennials also ruin the census bureau data
set," because when you looked at the inputs into
this data set and who was monitored over time and
the impact on the median income as it was
calculated, baby boomers and millennials played a
bill role but that wasn't able to be captured here.
How we defined urban and rural also had work to be
done.
But you wouldn't know this because in many of the
larger released studies they don't associate the
data set with what's being reported.
The sugar industry.
As universities and research organizations
rely on funding to initiate and continue
research, there are also responsibilities within
research organizations to ensure that conflicts of
interest are not present even prior to research
being funded.
So, in this issue, we have the sugar industry who
hired two Harvard nutritionists back in the
day and explained the outcome that they wanted.
So they went and did a literature review and
downplayed the role that sugar played on coronary
and heart disease and came up with the answer that it
was all about fat and cholesterol, and you know
the rest of that story.
But, more recently, HHS issued new rules to open
up data from clinical trials.
This has become such a glaring issue, we're
making a lot of monetary decisions about what to
fund and in which direction to go to based
on the results of studies without access
to the date set.
So while the rule has been in place, it hasn't been
enforced and so it's encouraging to see that
that's moving forward.
I list these examples because I really want to
highlight that it's not mission impossible to
understand what you're looking at, but we have a
responsibility, if we're working with information,
to select the appropriate data visualization when
presenting our analysis, and I'm going to talk
today about the same data different messages when
using visualization and, when presenting data, be
clear about what the data does and does not tell you.
Be honest.
A lot of times, when we hook at information, it's
not going to answer the questions that we have.
It may encourage us to ask additional questions and
that's okay and that's what we want to see happen.
So here I have pulled a graphic from The Sun, and
this is a recent article.
And what I want to draw your attention to here is
this data visualization piece in the middle.
There's a benefit in making it easier to look
at the results of an election poll, but if you
can see, from the front page here, you'll notice
that UKIP, the purple that's got a red ring
around it, received zero seats, which is greater in
donut chart than the other section which received 22
seats and throws off the rest of the proportions.
So what?
That was just a silly mistake.
Maybe.
But I would argue that oftentimes there are
people and businesses who are using visualization
principles to influence you and sell you on things
every day.
So we can look at this wall of sauces here in the
grocery store which may look innocent and someone
in the store simply stocked these in random
order, but we know that companies pay store
premiums to place their products in the middle of
the aisle so that you'll see them first, and as
you're going down the aisle, you're more likely
to pick those items.
So we're being manipulated into what we see and
purchase without recognizing the forces at
play here, but that's why it's important for all of
us to be critical consumers of data.
So what's the value in visualizing data?
First, it helps us to explore and understand the
data that we're working with, and secondly, it
helps communicate our message and our idea to a
broader audience.
So this is where this interplay of using your
visualization for good versus evil comes into
play because you can take the same information and
display it in multiple different ways to ensure
that the decision you want to have happen, happens.
I don't know how many of you have heard of
Anscombe's quartet.
This is what I'm showing on the board right here.
This is 1973, Francis Anscombe showed that while
the data sets here appear to be pretty similar, when
we plot the data sets on an XY coordinate plane we
get the following results.
So, looking at summary statistics, this looks the
same in the chart, but now, in section 1, I've
got a rough linear relationship with
variance, and section 2, it fits a neat curve, but
it's not linear.
In section 3 it's tight linear with one outlier,
and in the fourth example here, X remains a constant
but there's one outlier.
And if I just looked at the summary statistics - I
don't know if they display easily enough, but they're
very similar values, and I wouldn't be able to
understand the relationship of these
values without plotting them.
Visualizing data can reveal trends and
relationships, so if I give a decision maker the
raw data on the left, that will take them a lot of
time to go through and also be very frustrating,
but now I can communicate a message to my audience
by plotting the information in a way
that's more digestible.
These are all common sense pieces, but I wanted to
highlight that right now many of use Excel, and we
have macro buttons that we can pick and choose the
different ways you want to visualize data, but there
are also lots of resources out there called "visual
vocabulary tabs" or lists.
This is one of my favorite from the Financial Times.
Ann Emory, Stephanie Everett, a lot of people
give you options and little tutorials on how to
use the best graph or chart with the data that
you have.
As I mentioned in the opening of the
presentation, data and statistics, in particular,
are getting a bad rap these days, and I would
argue that, yes, methodology can be flawed,
but decisions are being made by individuals who
are not critical consumers of information.
What if the methododology was sound and the data was
just presented in a way that made the decision
maker, who had ten minutes of free time, enabled them
to make a quick decision and a decision that was
made easy based upon stellar visualization?
As consumers of data, we should try to understand
the purpose and biases of the data.
You should be thinking about accuracy, data
sources, uses, data quality and message, but I
do want to highlight some of the common data
visualization lies that we often come across.
Some are well intentioned and some are done on
purpose.
So in the abstract, these questions seem difficult
for us to answer but let's start simple and look at a
handful of common examples.
Right here I have a slide with the title Impressive
growth in testing volume over the past four
quarters.
So on X axis, I'm looking at time periods of
quarters, three months at a time, and on the Y axis,
I'm looking at the absolute number of
individuals tested.
And the summary statement here is saying: we've done
really well.
We've improved the number of people we've tested
over the quarters.
But, in actuality, there's one thing wrong with the
way this graph was set up.
Does anyone have an idea what would improve this
graph?
Yeah.
AUDIENCE MEMBER: The Y axis needs to start at zero.
KARIN GICHUHI: That's correct.
She said the Y axis needs to start at zero.
Because what you're really seeing in absolute numbers
here is a six percent change over four quarters.
If I take the same graph and start my Y axis at
zero, now the percent change is still six
percent, but the visual is not as dramatic.
Pie charts.
So, pie charts, there's two pieces here a want to
highlight.
One is a little more about us and humans and the
other about our dear friends at Excel.
The default in Excel placed a color legend next
to the graph and the reader has to go back and
forth to figure out what they're actually looking
at, but another issue with pie charts is they often
contain too many slices, depending on your data
set, and it's too cluttered to make sense of.
But on the more fundamental level, pie
charts are difficult for readers due to the way or
eyes and brains are structured.
We're good at judging length and have a
difficult time interpreting areas, so how
large is this piece of pie number 2 over time?
When pictured next to each other I have a hard time
determining if there's an illusion here or is
FY17Q2, pie slice number 2, actually larger than
any of the others?
When I change this to a bar chart, our brains are
better able to compare the length and it's easier to
use this to convey the same information.
There are a number of issues to be wary of when
looking at maps.
One big issue that can occur, as seen in this
comic, is that not scaling variables can lead to
wrong conclusions.
So, in this scenario, does the business owner adjust
their content or advertising to Martha
Stewart and Furry Porn fans, or is the reason
that these maps look the same just happen to match
the population concentration, and the
answer is, they happen to match the population
concentration.
There's no significant relationship between
geolocation and the mentioned subgroups.
And lastly, this map shows color coding associated
with percent of individuals below 185
percent of federal poverty threshold.
This is another census map, but the color moves
from a lighter to a darker red as poverty rates
increase and, as a result, the map makes poverty look
like a gaping wound.
So with those few examples, I did just want
to walk through the practical application of
how we use information when informing a policy maker.
And just like every super hero has been origin
story, our graphs also need to have an origin story.
We need to start looking at data with a purpose and
have a specific question in mind that we're trying
to answer with the information we have.
One of my biggest pet peeves right now is I get
people all the time coming and saying, "Karin, can
you or your team tell us how we're doing?
Can you be more specific?
How are you doing with what?
With identifying individuals for testing?
With capturing certain age groups among people living
with HIV who need to be enrolled into care?"
There are many different ways to answer the
question: how are we doing?
And one of the biggest things you can do in any
organization or implementing any project
is have an agreed upon set of goals so that you are
all on the same page about what you're hoping to do
and then you can agree on what you want to monitor
over time.
So once I have my question, I want to review
some of the information I have that can answer that
question and I like to go about collecting and
tidying up the data, plot it on a graph that's most
appropriate, and simplify it using font, color, and
size to my advantage.
So it's critical that everything in our graph is
done with a purpose, even the font, even the color
and the size to strategically convey our
message.
So this is when that power of data visualization
comes into play because how you incorporate your
colors and your fonts and your size are how you're
going to influence the viewer.
It's also really important to go ahead and spell out
what your message is in the graph.
So here, I started out with data points, and
these points represent the volume of testing and the
percent of individuals tested who are positive,
and I've divided up each of these dots into a
quadrant and essentially, the lower quadrant where
you see C1, C2, and C3, are those countries that
have high testing volume and low positivity.
So how might I use this information?
We actually make funding and program implementation
decisions off of this type of information to see
where we're doing more effective testing.
Effective defined as, are you finding the most
positives.
So when we go back to this idea of a critical
consumer, I really do like to emphasize that only you
can prevent the perpetuation of data
visualization lies.
You have the ability to ask questions, and so ask
questions in any situation.
Ask for the data set.
Thinking critically about global health metrics is
also becoming more important than ever.
We have access to a lot of health data.
Here's an example of sources of global health
data on the left and, on the right, we've listed a
number of topics.
And so this is because it's important to have a
specific research question in mind, or general
question about program implementation, prior to
sifting through the global health data that we have
access to in order to have an appropriate approach.
Well, we're going to focus on, for some examples in
PEPFAR is HIV/AIDS as the global health topic, and
the open source of global health data are the PEPFAR
Dashboards.
So first, I just wanted to see who in the room has
heard of PEPFAR?
Yay, okay!
So I won't bore you with the information that's on
the slide, but what's been interesting for me as a
data person is when we started out we had about
50,000 people on ART and now we talk not just about
the 11.5 million people we have on ART, but we also
talk more specifically about reduced new
pediatric infections which have reduced by 70 percent
since 2000 and, for the first time, we've been
able to show validated declines in adult HIV
incidents in Malawi, Zambia, and Zimbabwe to
the tune of 51 to 76 percent decline.
So we're getting more sophisticated with the
surveys we're using and the type of information
we're collecting to be able to talk about new
outcome reports related to incidents or viral load
suppression, so that's exciting.
Monitoring is what we really focused on a lot
for the general audience, for all of our partners,
for Congress, and for general reports.
You're all familiar with this definition, but it's
really that routine process of collecting
performance based indicators that we've all
agreed upon in the global arena to monitor how we're
doing.
We consider this important in PEPFAR because it, we
think, and we know, that it drives greater impact
transparency and accountability and, more
importantly, we are able to improve or partner
performance and increase program efficiency and
effectiveness.
And I'll talk a little bit later about how we're
defining efficiency and effectiveness.
You can see these Dashboards online at
data.pepfar.net/global, and you can review lots of
information there including the results and
targets for PEPFAR's monitoring.
But I wanted to use a very specific example, and one
of the most generic ones, about how we critically
consume information as it relates to results towards
targets.
So one of the things we do with many of our global
health implementation programs, is we set targets.
We say we're giving you this amount of money, we
expect you to reach this many individuals with this
intervention, and then we're going to see how you
did at each quarter, at each semiannual time
point, or at the end of the year.
One PEPFAR indicator used to monitor treatment
coverage is Current on Treatment, so it's the
number of adults and children currently
receiving ART.
And one way that PEPFAR monitors performance is we
say percent achievement.
You set your target at this, and you reached how
many?
So a specific example in Tanzania, and this is a
made-up partner name, Superior Health Services
provides direct service delivery of ART for adults
and children in 1500 sites.
So, at the end of the fiscal year, they put over
78,000 individuals on ART which means they achieved
93 percent of their target.
Now, if I want to determine if I should
continue to fund this partner or if they did a
good job, would this be enough to answer that
question?
I ask you.
I am looking for no.
But what other information might I want to know here?
AUDIENCE MEMBER: Is the target right?
KARIN GICHUHI: Is the target right.
What do you mean by "is it the target right?"
AUDIENCE MEMBER: Are the people there?
KARIN GICHUHI: Are the people there.
So, she's bringing up a really good point as we
set targets based on Epi data, people living with
HIV.
Do we have recent survey data that suggests we know
that denominator, so that's one.
What was their target last year?
Sometimes I see partners who didn't achieve very
much but their target was five times more than it
was the year before.
And so looking at percent is not enough.
I often want to go back and look at the trajectory
over time and see that they're improving, not
necessarily that their percent achievement rose
dramatically.
Just little things, but the context matters.
This is an example of looking at treatment
current within a Cascade.
It's another way we monitor the PEPFAR program
on the pepfar.net Dashboards, and one of the
things we've done to make it easily digestible for
the consumer is we've color coded it.
So I can focus in on any one of these indicators
and you'll see that the CURR on treatment is the
second from the bottom, and I see green green,
yellow.
So I achieved my target at 91 percent at the end of
last year.
At the end of this quarter I'm still doing well, but
I think projected by the end of the year I'm going
to fall into the yellow.
So this is a really easy way to digest information
and I can just focus in on these colors.
But, if I am being a critical consumer of the
information, is it all about the color and the
percent?
No.
I want to go back and see some of this additional
information.
And so you can see, to the right of it, I can also
see the actual number of results and I would want
to go back to the country and ask about other issues
for underperformance.
Was there election violence?
Were the nurses on strike?
Was there a drought?
Did the partner just start are have they been there
for 15 years?
All of the context is really important before I
make any type of funding decision.
So Context and Methodology Matter.
I think probably one of the hottest topics right
now, and I had a couple of sessions of this earlier
today, is about costing, expenditure analysis, and
what we do and don't know when we look at that type
of data.
So health economic data, as you know, presents a
unique set of interpretation challenges.
It's more about context and methodology than many
of the other data sets we've looked at.
There's an old joke: You ask a mathematician, an
accountant, and an economist what's two plus
two.
And the mathematician says, "Four."
And the accountant says, "About four, more or less
on average," and the economist smiles and says,
"What do you want it to be?"
And this is really easy to do when there are so many
unknowns with the inputs and when you can also lay
on data visualization.
So context and methodology matter tremendously no
matter what data you look at, but this is, perhaps
exacerbated when you're talking about health
economic data specifically.
So why it is so important?
The decisions you're making aren't just about
who or how many people you'll target, but it's
about how much money is allocated to everything.
So not just related to the economy itself right now,
but more so than ever, global stakeholders,
Global Fund, PEPFAR, WHO, UNAIDS, we're all looking
at the same information together to make decisions
about how we fund moving forward, and one of the
biggest questions we have to be able to answer is,
hey, country A, when we leave, this is how much
it's going to cost for you to maintain and sustain
these programs.
We don't know how much some of these things cost
and it's something we're taking a look at, and it's
something that we're monitoring very closely,
but things get real when you're talking about
allocating billions of dollars from one health
program to another and deciding whether to budget
for certain things or how funds are distributed.
So without context and methodology, we run the
risk of misinterpretation and misuse, so we rarely
have the perfect data.
Everyone admits that, but particularly economic data
available for decision making and health right
when it's needed.
Nevertheless, with the limited resources
available, it's imperative that we ensure we're
maximizing outputs at a given cost, ensuring that
resources are allocated optimally, and budgeting
and planning prudently.
So whenever we design a health cost analysis,
there's quite a few things we need to think about.
The type, the perspective, the purpose, the data
collection.
The figure here summarizes just a few of these
things, and when you look at any health economic
data, these are exactly what types of things that
you should consider in order to interpret it
correctly.
But the kicker: almost certainly, none of these
things are going to be on a graph, a table, or in
the data that you see.
You have to be a critical customer of this data,
especially if you want to be a responsible decision
maker.
So what is expenditure analysis in the way that
we use it?
It's collected annually, we look retrospectively at
the money spent by all the organizations we give
money to and these organizations report, from
their perspective, all the expenditures they incurred
in the past fiscal year, even if they bought
investments that will last for 10 years.
When we've got related results data, we link
these expenditures to the PEPFAR beneficiaries to
calculate a spend per person or unit
expenditure.
This data can be tremendously helpful from
a resource tracking perspective and it helps
us better understand the expenses that the United
States government incurs to provide a range of HIV
services.
To improve accountability and oversight of PEPFAR
efforts by tracking the spending and
accomplishments over time, we have a sense of getting
to that question: how much will it cost to continue
to offer the services and get these programs in a
position where they can be sustained by the host
country government.
Before we get into PEPFAR-specific examples,
I had Ramona put together an explanation of how she
would like to demonstrate EA-like data.
This graphic here shows her personal expenditures
for the last month.
She's grouped it into some broad categories and she's
noted a couple additional contextual details like
her travel expenditures might have been high
because going to a friend's wedding.
Based on this data, would you feel comfortable
making the following decisions?
Your budget for the next month?
Does this reflect the full cost of the support of her
family when there are two income earners and only
one budget here being shown?
This comes up a lot in the countries we work in
because I can look at what PEPFAR is doing, but I
don't know what Global Fund is contributing or
other stakeholders.
So when we look at the overall budget as an
input, those are things to consider.
Would you expect other families to have similar
spending patterns?
Should you benchmark your spending against others?
This is an illustrative example of how we might
budget using historical unit expenditure by
multiplying the spend per person times the target
for the upcoming year.
So you can see, for instance, in prevention it
says: My unit expenditure for the last year was $25
per person and if I wanted to reach one million
people then it's going to cost 25 million dollars.
Are you prepared to say to the Ministry of Health
that $25 per person is the cost of the program?
What would you say?
Is there any information here that's missing?
AUDIENCE MEMBER: What is the Ministry contributing?
KARIN GICHUHI: What is the ministry contributing.
What is anyone else contributing?
What are the inputs here, are those agreed upon?
So this is at the highest, highest level, a very
rudimentary way of approaching cost per
person.
This is one of my favorite graphs to talk about
because it drives me crazy.
This graph shows the allocation of resources
based on expenditure analysis and other data
and the explanation of this graph says:
expenditure per PLHIV, so person living with HIV and
percent of PLHIV by SNU.
SNU is sub national unit.
So what I'm looking at here is the provinces are
listed across my X axis, the dollar amount is
listed on the Y axis, and you can get a sense of the
total expenditures per PLHIV.
PLHIV is that aqua colored dash.
And so one of the conclusions that you might
come to in looking at this, is that Province G
is underfunded where you see a very low amount of
dollar spend, and a very high number of PLHIV being
reached, while Province E and F are overfunded
relatively.
But having more data or context here could
significantly change the way in which I interpret
this data.
So what else would I want to know in this instance?
Yeah.
AUDIENCE MEMBER: Prior year funding and PLHIV
rates.
KARIN GICHUHI: Prior year funding and PLHIV rates?
AUDIENCE MEMBER: And how they have changed.
KARIN GICHUHI: And how they've changed.
Yeah.
The other thing is, when I see Province G and I see
that they spent so little money but reached so many
people, I ask myself who else is there providing
funding and what other support is occurring in
that Province?
Or, when I see the cost is very high and I'm not
reaching very many people, is it the case that my
partner there just started and they're opening up a
new project in a place we've never worked before?
The title alone says PLHIV, but when if I'm
looking at the cost of testing?
When I test people they're not PLHIV.
Some of them are and some of them aren't.
So we have to be really careful with not only
explaining the inputs into this analysis, but also
how we want to explain what this graph says as we
look across the bars, and also what title we should
be using if the people being touched by these
activities are and are not PLHIV?
Efficiency is a very hot topic right now.
This graph shows the estimated per person spend
for each partner.
Partners are the blue bars.
And the number of beneficiaries reached,
which are these, I guess, they're yellow circles.
At first glance, you'll see two different partners
here with very different expenditure per person to
reach approximately the same number of results.
You might assume that the one on the left is more
efficient, reaching the same number of people at a
much lower cost to PEPFAR, but knowing the
methodology of EA and context should show that
this is a false assumption.
For example, in this case, one is primarily funded by
Ministry of Health while the other is primarily
funded by PEPFAR and only appears more expensive.
So what we're showing here, HTC is how we - an
acronym we use for testing, and so when we
look at how much it costs to test in a hospital,
which you see on the left compared to home base
testing intervention, at the hospital, all of the
staff are paid for, the electricity is paid for,
the facility is paid for and you're not traveling.
In the community, you're having to travel, you're
having to hire workers, you're having to buy the
stock, and there are a number of different costs
associated with community-based testing
that are not associated with facility-based
testing.
So, if I'm trying to determine the best way to
spend my dollars and I see that that this graph shows
me home-based testing is more expensive, how should
a choose to spend my dollars?
Is this enough to make a decision on?
So this is showing I'm reaching the same number
of individuals with both services.
Do I only go to facility-based testing?
I'm seeing some head shakes.
AUDIENCE MEMBER: Who are you reaching?
KARIN GICHUHI: Who are you reaching at the facility?
The people who are going to the facility, and so
one of the things we're doing right now at this
stage in the epidemic is, we can still see that a
percentage of the population is infected and
we're not reaching them, and that percentage of the
population is not showing up to the clinic and
asking to be tested.
So context matters, and context matters a lot when
you're talking about funding decisions because
epidemics continue and they change and they
evolve, and if you decide in a 10-minute discussion
that you're no longer going to fund one
intervention based on a look at a stellar graph,
then you've lost a lot of footwork that you've
already made in trying to reduce the epidemic.
I wanted to go back to this comment about
critical consumer of information.
And I love this cartoon, it's one of my favorite,
and I like to ask people to not be a Dogbert
because there are a lot of discussions right now
about the importance of Dashboard, so we have all
this information available to us, and if you could
just compile it in one place and put it on a
Dashboard that would be really easy for people to
read, then we could just go to that one place and
make a lot of decisions.
I won't read this for you, but it really highlights
the fact that sometimes we ignore the data that's in
front of us and make decisions based on
politics.
And I hope that, as more of you are fed information
in the forms of graphs and charts and data sets, and
as more of you work with them yourselves, that you
will keep in mind the importance of what we mean
when we say: be a critical customer of information.
Be curious.
Ask lots of questions, and it's okay to ask questions
because if the data can be defended people will
answer those questions.
When you're asking for information, please be
clear about the question that you want answered.
It's really frustrating for those people who are
responsible for providing the information to figure
out what you actually want.
And lastly, maintain data integrity.
Don't cherry pick your data.
You may be really good at data visualization, use it
for good and not evil.
And lastly, I'll go back to the title of the
presentation: Evidence: Is it all about
interpretation?
And I would posit that sometimes it is.
So, depending on who's reviewing the information
and who's putting the information together,
everyone has a responsibility here to be
a critical consumer and producer of that
information.
On this last slide, I've just listed notes and
attribution about where some of these pictures and
references came from.
If you want additional information, I'm happy to
share links to other resources for you, but at
this time I wanted to open it up to see if there are
any questions.
And they've asked if you have a question, that you
physically walk down the stairs in front of
everyone and stand in front of the microphone
and use the microphone.
Okay I hope you're all inspired to ask questions
the next time someone hands you a graph.
Did that happen?
Yes.
Okay.
Wonderful.
Thank you so much for your time.
[Applause]