ad-free
Channel / Source:
TEDx Talks
Published: 2017-05-30
Source: https://www.youtube.com/watch?v=Y2y7BzjbSNo
my husband John and I are philanthropists that would means different things to different people to us it means aggressive investments in a number of issue areas from evidence based policy to criminal justice education and health care a pension reform public accountability and many more I asked a lot of questions about her work why did you choose issue areas that you did how did you know once
I always feel that when responding to that question people are expecting you to tell a story but some transformative event that happened in my life for John's life that led us to embark on this path maybe they want to hear that we invest in health care because we had family members who unfortunately have had cancer meet anyone here story about my life growing up in Porter
eco and how that fueled my interest in in passion for fighting social justice issues and addressing poverty it's terrible we've come to expect that every talk will start with a story like something intensely moving and personal business starts with when I was a kid but I'm not going to tell a story like that although I could I think personal stories are so important they are what
fuels are passed for many of us through the reason why we do the work we do but I believe that they're precisely the wrong place to look for insight in fact I believe that we as policy makers government officials philanthropists even as individuals spend entirely too much time on anecdotes and not enough time on evidence I'm here today to argue that we need to change that
that we routinely making all kinds of decisions based on incomplete inconsistent far even nonexistent data based on anecdotes but this is harming all of us in ways that were not appreciated and that we can and should do better now I just need a bunch of provocative statements so let me get to work at convincing we'll start with that one example from the ten global stage in
twenty twelve the second most viewed Ted talk of all time Amy cutting psychologist Harvard Business School professor famous in large part for her Ted talk for her work on power posing so she basically argues that standing in a power pose position like Wonder Woman like this or like this you know whatever makes you feel super powerful changes the hormone levels in your body and might even
be conducive to greater success it's amazing right I mean that's so cool I should just family this entire time Ted community ate this up over thirty eight million views over the last four and a half years the talk is the subtitled into forty languages and counting here's how the Ted websites today describes her because her panties work it says that because if it is work reveals
the benefit power posing and it makes a reference to this formal level issue that I just talked about national news outlets gushed about the work and many of us rush to buy her best selling book now need to be a Downer but I think worlds it paraphrasing doesn't work but at least not as advertised look at the myriad studies have been conducted on par opposing ever
since professor Connie released her findings the point the glaring errors in her methodology the question the entire theory of power posing some researchers try to reproduce her work including others try to reproduce her work and not exactly the opposite result the part of losing reduces feelings of power it gets worse even professor can he's cool water who herself is a prominent it's completely disavow the study
as she did so in no uncertain terms you know she said I do not believe the powerprs affects a real if she does find in there she confesses to glaring errors in the way that they conducted their study tiny sample sizes flimsy data selectively reported findings even professor cutting herself it continues to stand by a power pose now has sort of amended her story and now
she says that she's agnostic about power opposes effects on hormone levels not static that was the whole reason she went on the tent staging did this and did this to make us all do this and feel powerful you may think this is a trend and it's by no means meant to be personal who cares if I stand in front of you like this it makes me
feel good or not and what does it have to do with solving the country's problems in the world's problems which is why we're all here well allotted because this example is endemic a something that we see throughout academic research and what we Surgeon General virtually everywhere you look you'll find researchers many of them prominent and most of them well intentioned actively misleading us into believing that
bad research has proven fact we've seen this time and again work I would not I sort of the foundation we set out to solve the country's problems by taking root causes so we're pretty new at this so we figured that the best place to start was to try to get our head around what we did and didn't know as a community collectively as a country what
do we know about what does and doesn't work after all philanthropists is been trying to save the country and make the country better for many many years and has spent billions of dollars doing it governments state governments local government suspend exponentially more on health care programs social programs anti poverty programs job training you name it so will soon but that there had to be a massive
body of evidence that could help quatre investment decision what we found was really alarming we saw a bad research everywhere it didn't matter where we looked for example we turn to nutrition as potentially an Avenue to address our health epidemics we wanted to get smart about what factors in foods might be conducive to chronic diseases obesity heart disease diabetes we found some very good research but
we also found an abundance of studies like these chocolate makes you skinny well beer helps you work out kids you know costly prevents Alzheimer's super catchy conclusions but researcher notices a people understudy who are eating chocolate are also skinny is merely come to the conclusion or least tells us that they're skinny because the chocolate same thing with her Alzheimer's city Shari research bad methodologies small sample
sizes correlations touted as because Asians selectively reported findings just like the iniquity study Miley there's a lot of these one might say a lot of alternative facts in twenty twelve a group of researchers randomly chose fifteen greetings from a couple normal stuff like milk banks and they took a look at the body of research relating to these ingredients to see what they could learn about whether
these ingredients did or did not because he's seems like a worthwhile exercise here's what they found for everybody that you see listed on the slide wine tomatoes milk and scorn coffee butter there was at least one research study that argued that the ingredient cause cancer and at least one research study that argued that the ingredient prevented cancer what is this to do with this information we
turn to health care same thing we saw that the authors of the vast majority of clinical trials reported in top medical journals silently change the outcomes of their reporting so they said they witnessed any one thing but they reported on another now well they do that but I don't claim to be an expert scientific research is enormously complicated certainly clinical research is as well but one
theory might be that's the original studies didn't pan out the way they wanted so the cherry pits positive findings from those same studies on secondary outcomes and reported on those instead so they can get published now either have to have a peach seeds are wondering whether Mimi there could be something fishy going on here and the signs have been everywhere take a recent project that we
did the reproducibility prod so we asked researchers to reproduce one hundred psychology experiments that have been published in top psychology journals in two thousand and eight so good to me if you do them again will you find the same results as we wanted to you know how often they could find the same results one third to one half of the time now I'm not claiming that
scientists and researchers are actively in intentionally committing fraud I'm saying there's something broken in our system which scientists are feeling the need to report only positive findings at the expense of the whole story how they do that what what's ways so let's say that a researcher wants to prove that there's a relationship between eating lemons and prostate cancer because that would be newsworthy so he designs
conducts experiments there's a fine relationship okay so he does it again same experiment does it again no relationship it is a seventeen more times no relationship but on the twentieth time he does find a relationship film he can publish boon ends because prostate cancer new study shows he doesn't tell us about the nineteen other times that he conducted that seems fitting failed he just told us
about the one time that he succeeded that's called a file drawer effect researcher can also tweak the statistical analysis until he gets the results that he wants ask a P. hacking or he could report is still very narrowly defined group when in fact the original research said he was meant to address a much much larger group what is happening well there could be lots of reasons
but I would argue that a reason is it the incentive system in science and in research is broken in an ideal world scientists and researchers were motivated by one thing the pursuit of truth but in the real world scientists and researchers are equally motivated by the desire to publish because that's the vehicle for achieving tenure that's a vehicle for getting funding in for achieving notoriety scientific
journals are clamoring for articles that report flashy results so those articles can get cited even if the truth of this is called the impact factor then of course there's the media and we as consumers we all want to hear those flashy results as a researcher's deliver even if it's at the expense of full transparency in rigor and while the worse off for it no foundation we
try to break the cycle and perform the system by funding organizations are promoting transparency in good practices and collaboration and data sharing the center for urban science the center for evidence based medicine at Oxford the metric centers Stanford health news organizations that are holding the media's feet to the fire as to what the report on research studies and they're all doing terrific work but we need
to do so much more we can become philanthropists to hang out with academics and roam the hallowed halls of universities we became fond of the cities the world but how can you even think about what to change if we don't know what works and I was personal works if we can't trust research so this is our problem this is our issue discussed the core of who
we are and what we do we can't function if we don't have a healthy research because problem of bad research in bad science isn't limited to academia this is just as bad in public policy I don't remember the scared straight program you know at risk youth would go maximum security prisons or meet prisoners of the prisoners would yell at them and tell them what they're bad
choices and how they have to live their life on the street narrow we spent millions of dollars on these for in the mean intuitive sense a separate then work research showed that these firms actually increase the likelihood but these kids would commit criminal acts we should've known this much earlier we would save millions of dollars and more importantly maybe we would have saved some of these
kids three federal departments department of labor HHS in the department of education have funded randomized control trials on it social programs have either been administered by the government or by the private sector and when it was control trials are the gold standard in research here's what they found seventy percent of the employment and job training programs of the department of labor looked at had weaker no
positive of the twenty eight team pregnancy prevention programs at HHS looked at only three were worthwhile and eighty ninety percent of the education programs that the department education looked at had weaker no positive effects relative to the screens were doing already so we know so little about what works in education and in job training and in policy in general even when we think we do take
the federal government's what works clearinghouse which was established as a resource for practitioners to determine what works in education it purports to rely on rigorous research let's look at this research there is a study that had only a few dozen participants there's another study that was conducted over the course of around twelve weeks then there's a bunch of positive labels on secondary outcomes ultimately don't matter
for example the website labels as positive a reading preparation program because research showed that after going through the program kids were able to recognize letters of the alphabet well you might say that's a stepping stone to reading so that sounds pretty reasonable to me and it's ours except the website doesn't tell us that those same researchers studied that same program and found that the program had
no effects on kids ultimate ability to read well is no we cannot so we're spending millions and millions of dollars on programs that and we're still work and a bass we don't know we don't have sufficient data we don't how reliable the so this is a change we need to stop as a philanthropic community as a policy making community we need to stop funding these programs
and relying on this ecosystem that isn't giving us the results that we need and I got some ideas on how to do that how to reform the system first more randomized control trials those of us who work with governments for governments collaborate with governments need to demand more randomized control trials so that we can get evidence and understand what does and doesn't work second we all
need to follow the evidence is on all of us governments and philanthropist need to stop funding what doesn't work and start planning what does work we need to hold our own selves accountable for these results and for this problem this is a man that was imposed upon us it's something that we've created and this on us to fix and third we need better data we can't
do research if we don't have healthy data systems that are speaking to each other better harmonize that researchers can access to give us the answers that we need and here to there are excellent organizations both within and outside the government that are working on these issues within the government the commission on evidence based policy making the social behavioral sciences team are conducting randomized control trials pushing
an evidence based policy making agenda within government and they're doing terrific work in collaboration with state and local governments in the nonprofit sector the word island policy division louts J. pal results for America are all contributing to reforming this ecosystem bite either conducting minimize control trials were working acts promoting evidence and thanks to the work of these organizations and many others we now have some answers
we now know which programs doing the needle on things that we care about like child welfare and education and job training and recidivism armed with that information or we need a better spot as people who care is concerned citizens as philanthropists as policy makers to make good decisions we mean to follow data we need to focus more on research only to demand that of governments and
