PAPERS
How To Do Good ScienceAbstract
In his book - The Ecology of Ecologists - Jeremy Fox asserted that one of the great strengths of ecology is our diversity of approaches. However, it seems indisputable to me that a diversity of approaches must share a common ontogeny that is the foundation of ‘good’ science. But I have never read a description of what those shared core values are – what the common DNA is. I propose 12 philosophical principles of good science and explain why I believe they are foundational principles for my discipline - ecology. Further, I discuss a set of technical characteristics that are necessary for doing science well. Every practicing scientist should be able to write a paper with this title. I suspect that many, if not most, would be very different from this paper, but every scientist should be able to articulate what scientific "best practices" are. This paper isn't definitive, exhaustive or perhaps even insightful. But I do hope it opens a useful discussion about what it means to do science well. How To Do Good Science Optimal α 2.0: A new approach to interpreting p-values in clinical trialsAbstract
Null hypothesis significance testing (NHST) is widely used in clinical trials despite widespread understanding of the weaknesses associated with NHST including that (1) p-values provide little information about effect sizes and (2) traditional rejection thresholds are arbitrary. Optimal α 1.0, introduced in 2012, was an improved approach to approach to setting rejection thresholds because it had an explicit and defensible rationale, minimizing the probability of making Type I and II errors. However, it still used observed p-values to make binary decisions (reject or fail to reject the null) and it required the arbitrary selection of ‘critical’ effect sizes. Here we describe an extension of optimal α 1.0, optimal α 2.0, that mitigates many of the weaknesses associated with optimal α 1.0. Optimal α 2.0 uses observed p-values to estimate the maximum effect size for which the data provide evidence and the range of effect sizes against which the data provide evidence. Using optimal α 2.0, p-values are no longer used to reject or fail to reject the null but are used instead to make inferences about the range of effects sizes for which there is evidence. Here, I show that relationship between observed p-values and the hazard ratios for which the data provide evidence, for log-rank analyses of simulated clinical trials data. I also apply optimal α 2.0 to three clinical trials that used a traditional NHST approach and demonstrate how the results obtained using optimal α 2.0, allow more nuanced and useful interpretation than the traditional NHST interpretation. Optimal α 2.0: A new approach to interpreting p-values in clinical trials Using optimal alpha rather than traditional NHST thresholds reverses conclusions for 14-34% of cancer clinical trials.
Abstract
Clinical trials are conducted to test treatment efficacy and safety and to provide high quality data for healthcare decision making. Most phase III cancer clinical trials continue to use traditional Null Hypothesis Significance Testing (NHST), which has been criticized for using an arbitrary rejection threshold, sensitivity to sample size, and ignoring Type II error or the effect size. Optimal alpha is a method for setting the rejection threshold that is explicitly designed to address the key limitations of NHST. This study re-analyzed 2,197 statistical tests from published phase III cancer clinical trials using optimal α, and compared conclusions that were originally reached using traditional NHST thresholds to those reached using optimal α. Our results show that in 23.6% of the tests, using optimal alpha would have resulted in a different conclusion than was reached using traditional thresholds. One Sentence Summary: We conclude that adopting optimal α for setting rejection thresholds would improve our ability to make correct inferences from cancer clinical trials. Using optimal alpha rather than traditional NHST thresholds reverses conclusions for 14-34% of cancer clinical trials. Rescuing null hypothesis significance testing (NHST) using optimal alpha.Abstract
For decades, NHST was the dominant approach to inferential statistics and, despite the increasing popularity of alternative approaches such as, information-theoretic techniques it continues to be widely used. However, the list of criticisms of NHST is long and varied. Most criticisms rest on three fundamental properties of NHST and p-values – (1) that effect size is not an explicit consideration, (2) that Type II errors are not considered and (3) that p-values don’t provide what researchers want - error probability statistics (i.e. estimates of the probability of making a mistake when the null is rejected or not). Mudge et al (2012) developed an approach to setting the statistical threshold for NHST – optimal alpha - that rather than controlling long-run probability of Type I errors, minimizes the cost or probability of making a Type I or II error for a specific test. These preliminary pilot studies suggest that using optimal alpha rather than α= 0.05 would result in different conclusions in about 22-28.5 % of tests of null hypotheses. We interpret this as meaning that in 22-28% of tests researchers made the wrong decision in accepting or rejecting the null. : If the objective of null hypothesis significance testing is to maximize the probability of making the correct decision when choosing between the null and the alternative then not only is optimal alpha superior to traditional NHST approaches, it addresses most of the concerns that have been raised about traditional NHST. But, we can also use optimal alpha so that all decisions about accepting or rejecting the null identify the inflection effect size where the evidence changes from support for the null to support against the null. Rescuing null hypothesis significance testing (NHST) using optimal alpha. Original Mudge et al (2012) paper How I learned to stop worrying about chaos and love the Gompertz (updated version February 6, 2026)Abstract
In 1974 Robert May demonstrated than simple deterministic logistic population dynamics could produce chaotic fluctuations at high intrinsic growth rates (r). This work implied that some medium to long-term population fluctuations might be impossible to predict because even tiny measurement errors on initial conditions would lead to enormous prediction errors. This led to several decades of discussion about how common or rare chaotic dynamics might be in natural systems. I conclude that chaotic dynamics are rare in nature because the deterministic model that best fits most empirical data is the Gompertz rather than the logistic and unlike the logistic the Gompertz does not produce chaotic dynamics for even implausibly high intrinsic growth rates. Further, none of the estimated intrinsic growth rates for more than 20000 population time series in nature were large enough to produce chaotic dynamics using the logistic model. The more likely reason that ecologists have been unable to produce population models that make good predictions is because population dynamics are complex. That is, population fluctuations are caused by many, nonlinear and interacting drivers. I conclude that the focus on chaotic dynamics in ecology is and has been an interesting dead end. - Jeff Houlahan |