Statistical significance and other complementary measures for the interpretation of the research results
Article Sidebar
Main Article Content
Abstract
The contrast hypothesis constitutes the most used method in the scientific research to estimate the statistical significance of any find. However, nowadays its use is questionable because it did not have other statistical criteria that make possible the credibility and reproducibility of the studies. From this condition, this study review how the use of the null hypothesis significance testing has been and the recommendations made regarding the application of other complementary statistical criteria for the interpretation of the results. It is described the main controversy of only use the probability value to reject or accept a hypothesis. The interpretation of a non significant value, as prove of effect absence or a significant value as existence of it, is a frequent mistake in scientific researchers, according to the reviewed literature. It is suggested to make a rigorous assessment of the obtained data in a research and include in the study reports other statistical tests, as the test power and the effect size of the intercession, to offer a complete interpretation and increase the results quality. Specifically, it is recommended to the editors of scientific journals to consider the report of the mentioned statisticians in the papers who required, as part of the criteria to take into account for their evaluation.
Key words: null hypothesis significance testing, probability value, statistical power, effect size
Article Details
Those authors that have publications with this journal accept the following terms:
1. They will retain their copyright and guarantee the journal the right of first publication of their work, which will be simultaneously subject to the License Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) that allows third parties to share the work whenever its author is indicated and its first publication this journal. Under this license the author will be free of:
- Share — copy and redistribute the material in any medium or format
- Adapt — remix, transform, and build upon the material
- The licensor cannot revoke these freedoms as long as you follow the license terms.
Under the following terms:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- NonCommercial — You may not use the material for commercial purposes.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
2. The authors may adopt other non-exclusive license agreements to distribute the published version of the work (e.g., deposit it in an institutional telematics file or publish it in a monographic volume) whenever the initial publication is indicated in this journal.
3. The authors are allowed and recommended disseminating their work through the Internet (e.g. in institutional telematics archives or on their website) before and during the submission process, which can produce interesting exchanges and increase the citations of the published work. (See the Effect of open access).
References
Abelson, R.P. 1997. "On the surprising longevity of flogged horses: Why there is a case for the significance test". Psychological Science, 8(1): 12-15, ISSN: 1467-9280. https://doi.org/10.1111/j.1467-9280.1997.tb00536.x.
American Psychological Association. 1994. Manual of the American Psychological Association, 4th ed., Washington D.C, United States: American Psychological Association, 368p. ISBN: 9781557982414, Available: <https://apastyle.apa.org>, [Consulted: April 10, 2022].
American Psychological Association. 2001. Manual of the American Psychological Association, 5th ed., Washington D.C, United States: American Psychological Association, 439p. ISBN: 9781557987901, Available: <https://apastyle.apa.org>, [Consulted: June 14, 2022].
American Psychological Association. 2010. Manual of the American Psychological Association, 6th ed., Washington D.C, United States: American Psychological Association, 272p. ISBN: 9781433805615, Available: <https://apastyle.apa.org>, [Consulted: June 16, 2022].
Antúnez, P., Rubio, E.A. & Kleinn, C. 2021. "Hypothesis testing in forestry, agriculture and ecology: Use and overuse of the 0.05 and 0.01". Ecosistemas y Recursos Agropecuarios, 8(1): 1-5, ISSN: 2007-901X. https://doi.org/10.19136/era.a8n1.2616.
Bakan, D. 1966. "The effect of significance testing in psychological research". Psychological Bulletin, 66(6): 423-437, ISSN: 1939-1455. https://doi.org/10.1037/h0020412.
Bakker, M. & Wicherts, J.M. 2011. "The (mis) reporting of statistical results in psychology journals". Behavior Research Methods, 43(3): 666–678, ISSN: 1554-3528. https://doi.org/10.3758/s13428-011-0089-5.
Bologna, E. 2014. "Estimación por intervalo del tamaño del efecto expresado como proporción de varianza explicada". Evaluar, 14(1): 43-46, ISSN: 1667-4545. https://doi.org/10.35670/1667-4545.v14.n1.11521.
Bono, R. & Arnau Gras, J. 1995. "Consideraciones generales en torno a los estudios de potencia". Anales de Psicología, 11(2): 193-202, ISSN: 1695-2294.
Borges, A., San Luis, C., Sánchez, J.A. & Cañadas, I. 2001. "El juicio contra la hipótesis nula: muchos testigos y una sentencia virtuosa". Psicothema, 13(1): 174-178, ISSN: 0214-9915. https://doi.org/10.7334/psicothema2001.14462.025.
Botella, J. & Zamora, A. 2017. "El meta-análisis: una metodología para la investigación en educación". Educación XXI, 20(2): 17-38, ISSN: 1139-613X. https://doi.org/10.5944/educXXI.18241.
Caballero, A. 1979. "Tamaños de muestras en diseños completamente aleatorizados y bloques al azar donde la unidad experimental esté formada por grupos de animales". Cuban Journal of Agricultural Science, 13 (3): 225-235, ISSN: 2079-3480.
Caperos, J.M. & Pardo, A. 2013. "Consistency errors in p-values reported in Spanish psychology journals". Psicothema, 25(3): 408-414, ISSN: 0214-9915. https://doi.org/10.7334/psicothema2012.207.
Carver, R.P. 1978. "The case against statistical significance testing". Harvard Educational Review, 48(3): 378-399, ISSN: 0017-8055. https://doi.org/10.17763/haer.48.3t49026164281841.
Carver, R.P. 1993. "The case against statistical significance testing revisited". Journal of Experimental Education, 61(4): 287-292, ISSN: 0022-0973. https://doi.org/10.1080/00220973.1993.10806591.
Chow, S.L. 1988. "Significance test or effect size? " Psychological Bulletin, 103(1): 105-110, ISSN: 1939-1455. https://doi.org/10.1037/0033-2909.103.1.105.
Clark-Carter, D. 1997. "The account taken of statistical power in research published in the British Journal of Psychology". British Journal of Psychology, 88(1): 71-83, ISSN: 2044-8295. https://doi.org/10.1111/j.2044-8295.1997.tb02621.x.
Cochran W. y Cox, G. 1999. Diseños experimentales. 2nd ed., México: Editorial Trillas, S.A. 75p., ISBN: 968-24-3669-9. Available: <https://www.urbe.edu/UDWLibrary/InfoBook.do?id=5068>, [Consulted: August 3, 2022].
Cohen, J. 1988. Statistical power analysis for the behavioral sciences. 2nd ed., New York, United States: Routledge, 590p., ISBN: 9780805802832, Available: <https://www.routledge.com/books/Statistical-power-analysis-for-the-behavioral-sciences>, [Consulted: August 8, 2022].
Cohen, J. 1990. "Things I have learned (so far) ". American Psychologist, 45(12): 1304-1312, ISSN: 1935-990X. https://doi.org/10.1037/0003-066X.45.12.1304.
Cohen, J. 1992. "A power primer". Psychological Bulletin, 112(1): 155-159, ISSN: 1939-1455. https://doi.org/10.1037/0033-2909.112.1.155.
Cohen, J. 1994. "The earth is round (p < 0.05) ". American Psychologist, 49(12): 997-1003, ISSN: 1935-990X. https://doi.org/10.1037/0003-0.66X.49.12.997.
Cohen, J. 1997. Much ado about nothing. Conference presented at the annual meeting of the American Psychological Association, Chicago, United States.
Cortina, J.M., & Dunlap, W.P. 1997. "Logic and purpose of significance testing". Psychological Methods, 2(2): 161-172, ISSN: 1939-1463. https://doi.org/10.1037/1082-989X.2.2.161.
Craig, J.R., Eison, C.L. & Metze, L.P. 1976. "Significance tests and their interpretation: An example utilizing published research and omega-squared". Bulletin of the Psychonomic Society, 7(3): 280-282, ISSN: 0090-5054. https://doi.org/10.375/bf03337189.
De la Fuente, E.I. & Díaz-Batanero, C. 2004. "Controversias en el uso de la inferencia en la investigación experimental". Metodología de las Ciencias del Comportamiento, 5(1): 161-167, ISSN: 1575-9105.
Díaz-Batanero, C., Lozano-Rojas, O.M. & Fernández-Calderón, F. 2019. La controversia sobre el contraste de hipótesis: Situación actual en psicología y recomendaciones didácticas. En: Contreras, J.M., Gea, M.M., López M.M. & Molina E. (eds.), Actas del Tercer Congreso Internacional Virtual de Educación Estadística. España, Available: , [Consulted: July 12, 2022]
Falk, R., & Greenbaum, C. W. 1995. "Significance tests die hard: the amazing persistence of a probabilistic misconception". Theory and Psychology, 5(1): 75-98, ISSN: 1461-7447. https://doi.org/10.1177/0959354395051004.
Faulkenberry, T.J. 2022. Psychological statistics, the basics. 1st ed., New York, United States: Routledge, 122p., ISBN: 97811032020952, Available: <https://www.routledge.com/books/Psychological-statistics,-the-basics>, [Consulted: October 18, 2022].
Fisher, R.A. 1925. Statistical methods for research workers. 1st ed., Escocia: Genesis Publishing, 269p., ISBN: 4444000761336. Available: <https://www.iberlibro.com/buscar-libro/titulo/statistical-methods-research-workers/autor/sir-ronald >, [Consulted: May 18, 2022].
Fisher, R.A. 1935. The design of experiments. 1st ed., London: Oliver and Boyd, 256p., ISBN: 0028446909. Available: <https://www.iberlibro.com/buscar-libro/titulo/statistical-methods-research-workers/autor/sir-ronald >, [Consulted: June 5, 2022].
Fisher, R.A. 1950. Contributions to mathematical statistics. New York, United States: John Wiley & Son, 600p., ISBN: 9780678008898. Available: Rothamsted Research, https://repository.rothamsted.ac.uk, [Consulted: September 10, 2022].
Fisher, R.A. 1955. "Statistical methods and scientific induction". Journal of the Royal Statistical Society, Series B, 17(1): 245-251, ISSN: 1369-7412.
Frías Navarro, M.D., Pascual Llobel, J. & García Pérez, J.F. 2000. "Tamaño del efecto del tratamiento y significación estadística". Psicothema, 12(Suplemento): 236-240, ISSN: 0214 - 9915.
Frías, M.D., Pascual, J. & García, J.F. 2002. "La hipótesis nula y la significación práctica". Metodología de las Ciencias del Comportamiento, 4(1): 181-185, ISSN: 1575-9105.
Fritz, R.W. 1995. "Accepting the null hypothesis". Memory & Cognition, 23(1): 132-138, ISSN: 0090-502X. https://doi.org/10.3758/BF03210562.
Fritz, R.W. 1996. "The appropriate use of null hypothesis testing". Psychological Methods, 1(4): 379-390, ISSN: 1939-1463. https://doi.org/10.1037/1082-989X.1.379.
Funder, D.C. & Ozer, D.J. 2019. "Evaluating effect size in psychological research: Sense and nonsense". Advances in Methods and Practices in Psychological Science, 2(2): 156–168, ISSN: 251-2467. https://doi.org/10.1177/2515245919847202.
Greenwald, A.G., Gonzalez, R., Harris, R.J. & Guthrie, D. 1996. "Effect size and p-values: What should be reported and what should be replicated? " Psychophysiology, 33(2): 175-183, ISSN: 1469-8986. https://doi.org/10.1111/j.1469-8986.1996.tb02121.x.
Guerra, W.C., Herrera, M., Fernández, L. & Rodríguez, N. 2019. "Modelo de regresión categórica para el análisis e interpretación de la potencia estadística". Cuban Journal of Agricultural Science, 53(1): 13-20, ISSN: 2079-3480.
Hagen, R.L. 1997. "In praise of the null hypothesis statistical test". American Psychologist, 52(1): 15-24, ISSN: 1935-990X. https://doi.org/10.1037/0003-066X.52.1.1.
Harlow, L.L., Mulaik, S.A. & Steiger, J.H. 2016. What if there were no significance tests? 2nd ed., New York, United States: Routledge, 444p., ISBN: 9781317242857, Available: https://www.routledge.com/books/ What-if-there-were-no-significance-tests?, [Consulted: August 8, 2022].
Hickey, G.L., Grant, S.W., Dunning, J. & Siepe, M. 2018. "Statistical primer: Sample size and power calculations-why, when and how? " European Journal of Cardio-Thoracic Surgery, 54(1): 4-9, ISSN: 1873-734X. https://doi.org/10.1093/ejcts/ezy169.
Ioannidis, J.P.A. 2018. "The Proposal to Lower P Value Thresholds to .005". Journal of the American Medical Association, 319(14): 1429-1430, ISSN: 0098-7484. https://doi.org/10.1001/jama.2018.1536.
Kuffner, T.A. & Walker, S.G. 2019. "Why are p-Values Controversial? " The American Statistician, 73(1): 1-3, ISSN: 1537-2731. https://doi.org/10.1080/00031305.2016.1277161.
Levin, J.R. 1993. "Statistical significance testing from three perspectives". Journal of Experimental Education, 61(4): 378-382, ISSN: 1940-0683. https://doi.org/10.1080/00220973.1993.10806597.
Manzano, V. 1997. "Usos y abusos del error de Tipo I". Psicológica: Revista de metodología y psicología experimental, 18(2): 153-169, ISSN: 1576-8597.
Marín, L. & Paredes, D. 2020. Valor p, correcta e incorrecta interpretación. Revista Clínica de la Escuela de Medicina de la Universidad de Costa Rica, 10(1): 45-52, ISSN: 2215-2741.
McMillan, J,H. & Foley, J. 2011. "Reporting and discussing effect size: Still the road less treveled". Practical Assessment Research Evaluation, 16(14): 1-12, ISSN:1531-7714. https://doi.org/10.7275/b6pz-ws55.
Menchaca, M.A. 1974. "Tablas útiles para determinar tamaños de muestras en diseño de Clasificación Simple y de Bloques al Azar". Cuban Journal of Agricultural Science, 8 (1): 111-116, ISSN: 2079-3480
Menchaca, M.A. 1975. "Determinación de tamaños de muestra en diseños Cuadrados Latinos". Cuban Journal of Agricultural Science, 9 (1): 1-3, ISSN: 2079-3480.
Menchaca, M.A. & Torres V. 1985. Tablas de uso frecuente en la Bioestadística. Instituto de Ciencia Animal. Cuba.
Morrison, D.E. & Henkel, R.E. 2006. The significance test controversy: a reader. 1st ed., Chicago, United States: Aldine, 352p., ISBN: 9780202300689, Available: https://www.abebooks.com/The-significance-test-controversy:-a-reader, [Consulted: August 6, 2022].
Neyman, J. & Pearson, E.S. 1928. "On the use and interpretation of certain test criteria for purposes of statistical inference". Biometrika, 20A: 175-240, ISSN: 0006-3444. https://doi.org/10.1093/biomet/20A.3-4.263.
Nickerson, R.S. 2000. "Null hypothesis significance testing: a review of an old and continuing controversy". Psychological methods, 5(2): 241-301. ISSN: 1939-1463. https://doi.org/10.1037/1082-989x.5.2.241.
Ochoa, C., Molina, M. & Ortega, E. 2019. "Inferencia estadística: probabilidad, variables aleato-rias y distribuciones de probabilidad". Evidencias en Pediatría, 15(2): 27-32, ISSN: 1885-7388.
Ochoa, C., Molina, M. & Ortega, E. 2020. "Inferencia estadística: contraste de hipótesis". Evidencias en Pediatría, 16(1): 11-18, ISSN: 1885-7388.
Odgaard, E.C. & Fowler, R L. 2010. "Confidence intervals for effect sizes: compliance and clinical significance in the Journal of Consulting and Clinical Psychology". Journal of Consulting and Clinical Psychology, 78(3): 287–297, ISSN: 0022-006X. https://doi.org/10.1037/a0019294.
Ponce, H.F., Cervantes, D.I. &Anguiano, B. 2021. "Análisis de calidad de artículos educativos con diseños experimentales". Revista Iberoamericana para la Investigación y el Desarrollo Educativo. 12(23): 49-79, ISSN: 2007-7467. https://doi.org/10.23913/ride.v12i23.981.
Rendón, M.E, Zarco, I.S. & Villasís, M.A. 2021. "Métodos estadísticos para el análisis del tamaño del efecto". Revista Alergia de México, 68(2): 128-136, ISSN: 2448-9190. https://doi.org/10.29262/ram.v658i2.949.
Rivera, F. 2017. Convivencia del nivel de significación y tamaño del efecto y otros retos de la práctica basada en la evidencia. Boletín Psicoevidencias, No. 48. Junta de Andalucía y Consejería de Salud, Andalucía, España, ISSN: 2254-4046.
Rothman, J. 1978. A show of confidence. New England Journal of Medicine, 299(24): 1362-1363, ISSN: 0028-4793. http://dx.doi. org/10.1056/NEJM197812142992410.
Scheffé, H. 1959. The Analysis of Varianza. New York, United States: John Wiley & Sons, Inc, 477p., ISBN: 0-471-75834-5, Available: https://www.abebooks.com/The-significance-test-controversy:-a-reader, [Consulted: January 6, 2023].
Schmidt, F.L. 1996. "Statistical significance testing and cumulative knowledge in psychology: implications for training of researchers". Psychological Methods, 1(2): 115-129. ISSN: 1082-989X. https://doi.org/10.1037/1082-989X.1.2.115.
Serdar, C.C., Cihan, M., Yücel, D. & Serdar, M.A. 2021. "Sample size, power and effect size revisited: simplified and practical approaches in pre-clinical, clinical and laboratory studies". Biochemia Medica Journal, 31(1): 1-27, ISSN: 1330-0962. https://doi.org/10.11613/BM.2021.010502.
Sesé, A. & Palmer, A. 2012. "El uso de la estadística en psicología clínica y de la salud a revisión". Clínica y Salud, 23(1): 97-108, ISSN: 2174-0550.
Sun, S., Pan, W. & Wang, L.L. 2010. "A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology". Journal of Educational Psychology, 102(4): 989-1004, ISSN: 1939-2176. https://doi.org/10.1037/a0019507.
Thompson, B. 1988. "A note about significance testing". Measurement and Evaluation in Counseling and Development, 20(4): 146-148, ISSN: 1947-6302. https://doi.org/10.1080/07481756.1988.12022864.
Thompson, B. 1989. "Asking «what if» questions about significance tests". Measurement and Evaluation in Counseling and Development, 22(2): 66-68, ISSN: 1947-6302. https://doi.org/10.1080/07481756.1989.12022912.
Thompson, B. 1996. "AERA editorial policies regarding statistical significance testing: Three suggested reforms". Educational Researcher, 25(2): 26-30, ISSN: 0013-189X. https://doi.org/10.2307/1176337.
Thompson, B. 1997. If statistical significance tests are broken/misused, what practices should supplement or replace them? Conference presented at the annual meeting of the American Psychological Association, Chicago, United States.
Thompson, B. 1999. "If statistical significance tests are broken/misused, what practices should supplement or replace them? " Theory and Psychology, 9(2): 165-181, ISSN: 1461-7447. https://doi.org/10.1177/095935439992.
Valera, S., Sánchez, J. & Marín, F. 2000. "Contraste de hipótesis e investigación psicológica española: Análisis y propuestas". Psicothema, 12(2): 549-582, ISSN: 0214-9915.
Venereo, A. 1976. "Número de réplicas en diseños cuadrados latinos balanceados para la estimación de efectos residuales". Cuban Journal of Agricultural Science, 10(3): 237-246, ISSN: 2079-3480.
Ventura, J. 2018. "Otras formas de entender la d de Cohen". Revista Evaluar. 18(3):73-78, ISSN: 1667-4545. https://doi.org/10.35670/1667-4545.v18.n3.22305.
Verdam, M.G., Oort, F.J. & Sprangers, M.A. 2014. "Significance, truth and proof of p values: reminders about common misconceptions regarding null hypothesis significance testing". Quality of Life Research, 23(1): 5-7, ISSN: 1573-2649. https://doi.org/10.1007/s11136-013-0437-2.
Wasserstein, R.L. & Lazar, N.A. 2016. "The ASA's Statement on p-Values: Context, Process, and Purpose". The American Statistician, 70(2): 129-133, ISSN: 1537-2731. https://doi.org/10.1080/00031305.2016.1154108.
Wilkinson, L., & TFSI - Task Force on Statistical Inference. 1999. "Statistical methods in psychology journals: Guidelines and explanations". American Psychologist, 54(8): 594-604, ISSN: 0003-066X. https://doi.org/10.1037/0003-066X.54.8.59.