<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p></p>
<div>Hi Todd,</div>
<div><br>
</div>
<div>I hope it is OK for me to comment on this thread as a bit of an outsider. </div>
<div><br>
</div>
<div>> What I meant to say was that a p-value tells you how likely your data are given the null hypothesis, it doesn’t really say anything about the probability of the null hypothesis. So a SMALL p-value means that my data are unlikely given the null hypothesis,
and a LARGE p-value means my data are likely given the null-hypothesis... but they could be even more compatible with some other hypothesis!</div>
<div><br>
</div>
<div>That is a nice summary. I would narrow it down even further: the p-value tells you the probability under the null hypothesis of getting the value of your chosen test statistic on the data, or any greater value (values which of course do not correspond
to your data).</div>
<div><br>
</div>
<div>The data are always more compatible with some other hypothesis; the question is whether they are compatible with some other plausible hypothesis; but then perhaps we should weight these hypotheses according to their plausibility...starts to sound familiar...</div>
<div><br>
</div>
<div>Ian.</div>
<div><br>
</div>
<br>
<p></p>
<p><br>
</p>
<div id="Signature">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-size:13px">
<p class="MsoPlainText">--------------</p>
<p class="MsoPlainText">Ian H. Jermyn</p>
<p class="MsoPlainText">E: <a href="mailto:i.h.jermyn@durham.ac.uk" id="LPNoLP">i.h.jermyn@durham.ac.uk</a>
</p>
<p class="MsoPlainText">--------------</p>
<p class="MsoPlainText">Department of Mathematical Sciences</p>
<p class="MsoPlainText">Durham University</p>
<p class="MsoPlainText">Science Laboratories</p>
<p class="MsoPlainText">South Road</p>
<p class="MsoPlainText">Durham DH1 3LE</p>
<p class="MsoPlainText">United Kingdom</p>
</div>
</div>
</div>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> visionlist <visionlist-bounces@visionscience.com> on behalf of Horowitz, Todd (NIH/NCI) [E] <todd.horowitz@nih.gov><br>
<b>Sent:</b> 16 November 2017 20:45<br>
<b>To:</b> visionlist@visionscience.com<br>
<b>Subject:</b> Re: [visionlist] Statistical Criticism is Easy; I Need to Remember That Real People are Involved</font>
<div> </div>
</div>
<div lang="EN-US" link="blue" vlink="purple">
<div class="x_WordSection1">
<p class="x_MsoNormal">Oops!</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">What I meant to say was that a p-value tells you how likely your data are given the null hypothesis, it doesn’t really say anything about the probability of the null hypothesis. So a SMALL p-value means that my data are unlikely given
the null hypothesis, and a LARGE p-value means my data are likely given the null-hypothesis... but they could be even more compatible with some other hypothesis!</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">thanks</p>
<p class="x_MsoNormal">Todd</p>
<p class="x_MsoNormal"> </p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b>From:</b> Horowitz, Todd (NIH/NCI) [E] [mailto:todd.horowitz@nih.gov]
<br>
<b>Sent:</b> Thursday, November 16, 2017 2:00 PM<br>
<b>To:</b> Pam Pallett <ppallett@gmail.com>; visionlist@visionscience.com<br>
<b>Subject:</b> Re: [visionlist] Statistical Criticism is Easy; I Need to Remember That Real People are Involved</p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I thought that a a large p-value simply meant that my data were unlikely given the null-hypothesis, a statement which yields no evidence about either the null- or alternative hypotheses.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><b>From:</b> Pam Pallett [<a href="mailto:ppallett@gmail.com">mailto:ppallett@gmail.com</a>]
<br>
<b>Sent:</b> Thursday, November 16, 2017 10:29 AM<br>
<b>To:</b> <a href="mailto:visionlist@visionscience.com">visionlist@visionscience.com</a><br>
<b>Subject:</b> [visionlist] Statistical Criticism is Easy; I Need to Remember That Real People are Involved</p>
<p class="x_MsoNormal"> </p>
<div>
<p class="x_MsoNormal">Hi All,</p>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal">I came across a blog today by Frank Harrell, Professor of Biostatistics and Founding Chair at Vanderbilt. His most recent post is the title of this email. But as I'm reading through his blog, I'm hearing a lot that has been discussed
and experienced by professors and postdocs subscribed to this list. We are often very separated from our neighboring departments, and I actually found some comfort in the fact that these problems seem spread across the board (misery loves company). Even if
we have been echoing these problems for over a decade with little effective change. </p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal">In his most recent post he says, "<span style="font-size:10.0pt; font-family:"Arial",sans-serif; color:#444444">There are several ways to improve the system that I believe would foster clinical research and make peer review more objective
and productive." I'm curious about what the people in the vision community think of these suggestions and whether they are realistic to implement in our field. His list is at the bottom of the entry. </span><span style="font-family:"Arial",sans-serif; color:#444444"><a href="http://www.fharrell.com/2017/11/">http://www.fharrell.com/2017/11/</a></span></p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:10.0pt; font-family:"Arial",sans-serif; color:#444444">For those experiencing TL;DR, here is the shortlist:</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-right:0in; margin-bottom:3.0pt; margin-left:0in; text-indent:-.25in">
<span style="font-size:10.0pt; font-family:Symbol; color:#444444"><span style="">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><span style="font-size:10.0pt; font-family:"Arial",sans-serif; color:#444444">Have journals conduct reviews of background and methods without knowledge of results.</span></p>
<p class="x_MsoNormal" style="margin-right:0in; margin-bottom:3.0pt; margin-left:0in; text-indent:-.25in">
<span style="font-size:10.0pt; font-family:Symbol; color:#444444"><span style="">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><span style="font-size:10.0pt; font-family:"Arial",sans-serif; color:#444444">Abandon journals and use researcher-led online systems that invite open post-"publication" peer review and give researchers the opportunities to improve their
"paper" in an ongoing fashion.</span></p>
<p class="x_MsoNormal" style="margin-right:0in; margin-bottom:3.0pt; margin-left:0in; text-indent:-.25in">
<span style="font-size:10.0pt; font-family:Symbol; color:#444444"><span style="">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><span style="font-size:10.0pt; font-family:"Arial",sans-serif; color:#444444">If not publishing the entire paper online, deposit the background and methods sections for open pre-journal submission review.</span></p>
<p class="x_MsoNormal" style="margin-right:0in; margin-bottom:3.0pt; margin-left:0in; text-indent:-.25in">
<span style="font-size:10.0pt; font-family:Symbol; color:#444444"><span style="">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><span style="font-size:10.0pt; font-family:"Arial",sans-serif; color:#444444">Abandon null hypothesis testing and p-values. Before that, always keep in mind that a large p-value means nothing more than "we don't yet have evidence against
the null hypothesis", and emphasize confidence limits.</span></p>
<p class="x_MsoNormal" style="margin-right:0in; margin-bottom:3.0pt; margin-left:0in; text-indent:-.25in">
<span style="font-size:10.0pt; font-family:Symbol; color:#444444"><span style="">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><span style="font-size:10.0pt; font-family:"Arial",sans-serif; color:#444444">Embrace Bayesian methods that provide safer and more actionable evidence, including measures that quantify clinical significance. And if one is trying to amass
evidence that the effects of two treatments are similar, compute the direct probability of similarity using a Bayesian model.</span></p>
<p class="x_MsoNormal" style="margin-right:0in; margin-bottom:3.0pt; margin-left:0in; text-indent:-.25in">
<span style="font-size:10.0pt; font-family:Symbol; color:#444444"><span style="">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><span style="font-size:10.0pt; font-family:"Arial",sans-serif; color:#444444">Improve statistical education of researchers, referees, and journal editors, and strengthen statistical review for journals.</span></p>
<p class="x_MsoNormal" style="margin-right:0in; margin-bottom:3.0pt; margin-left:0in; text-indent:-.25in">
<span style="font-size:10.0pt; font-family:Symbol; color:#444444"><span style="">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><span style="font-size:10.0pt; font-family:"Arial",sans-serif; color:#444444">Until everyone understands the most important statistical concepts, better educate researchers and peer reviewers on <a href="http://biostat.mc.vanderbilt.edu/ManuscriptChecklist"><span style="color:#4D469C">statistical
problems to avoid</span></a>.</span></p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal">Best,</p>
</div>
<div>
<p class="x_MsoNormal">Pam Pallett</p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>