An investigation of two widely used health datasets reveals poor data provenance and potential reliability issues, raising concerns about the clinical prediction models built from them and prompting calls for stricter research standards.
Study: Evidence of unreliable data and poor data provenance in clinical prediction model research and clinical practice. Image credit: Andrey_Popov/Shutterstock.com
A new study published in BMC Medicine conducted exploratory analyses to examine the quality of data and reporting in two large, publicly available datasets on stroke and diabetes, widely used in clinical prediction models.
Fast-churn research raises concerns over data quality
By 2024, researchers had published an estimated 250,000 clinical prediction models to help clinicians diagnose disease, estimate prognosis, and guide treatment decisions. Because these models can directly influence patient care, their reliability depends on both robust analytical methods and high-quality underlying data.
To improve transparency, the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines were introduced in 2015, providing a framework for reporting prediction model research. The 2024 TRIPOD+AI update expanded these recommendations to include both traditional regression and machine learning models, placing even greater emphasis on documenting data provenance, the metadata that records where data came from, how they were collected, and whether they can be trusted and reused.
The growing availability of large, routinely collected health datasets has accelerated the development of clinical prediction models. However, it has also fuelled what researchers describe as "fast-churn" research: rapid, formulaic studies that prioritize publication volume over meaningful scientific advances. According to the authors, this approach can increase the risk of false findings and waste valuable research resources.
Concerns about data quality have already prompted some publishers and journals to tighten their editorial policies following the misuse of widely used datasets, including the Global Burden of Disease and National Health and Nutrition Examination Survey databases. Other examples, such as unverifiable cancer cell lines that ultimately led to article retractions, have further highlighted the consequences of poor data provenance.
Although initiatives such as the Findable, Accessible, Interoperable and Reusable (FAIR) principles encourage better data stewardship, adoption remains inconsistent. Likewise, while repositories such as Kaggle make datasets widely accessible, they do not require users to provide comprehensive provenance information. The authors argue that without stronger standards for verifying data provenance, unreliable datasets can continue to circulate through the scientific literature, potentially undermining evidence-based medicine.
Study assessed provenance using TRIPOD+AI standards
Two popular, publicly available health datasets with likely poor data provenance were selected for their high download counts and relevance to clinical prediction model research. One dataset focused on stroke and the other on diabetes, with both accessed from Kaggle on August 27, 2025. The current study aimed to highlight data provenance issues in clinical prediction models using these datasets.
Each dataset was evaluated using nine TRIPOD+AI data provenance items, and exploratory analyses were conducted to assess authenticity, including checks for simulated data, unexpected correlations among variables, abnormal distributions, and duplicate rows. Public Kaggle discussions about data provenance were also reviewed, and concerns were raised with Kaggle.
Google Scholar was searched to identify peer-reviewed articles that used these datasets for model development or validation, and full texts were screened for inclusion. Exclusions included non-peer-reviewed works and non-English articles. The authors noted that this search strategy likely underestimated the use of these datasets because studies that did not include direct Kaggle links would not have been captured.
Inconsistencies in reporting were documented, along with checks for dataset origin disclosures and reviews of statements about ethical approval and potential clinical use. Policy uptake was examined via Altmetric and Overton. Author affiliations by country were analyzed, and research volume over time was plotted using OpenAlex.
Questionable datasets underpin more than 125 studies
An assessment of two widely used Kaggle health datasets for clinical prediction model research uncovered major concerns about data provenance and authenticity. Out of 653 research outputs identified, 125 published articles developed or validated clinical prediction models using these datasets.
Evaluation using nine TRIPOD+AI items revealed serious deficiencies in both datasets: neither provided information on when, where, why, or how the data were collected, nor could authenticity be independently verified. Both datasets failed all nine TRIPOD+AI data provenance assessment items.
The stroke dataset included 5,110 cases, which contained irregular patient IDs, improbable blood glucose and age distributions, and unrealistically little missing data. Similarly, the diabetes dataset comprised 100,000 cases that contained repetitive and unnatural values, artificial correlations, and many duplicate entries. Together, these findings indicate that both datasets are likely synthetic, fabricated, or otherwise unreliable and therefore unsuitable for research or clinical application.
The 125 included articles originated from 32 countries. However, reporting on ethical approval was rare, and most articles lacked sufficient information about data provenance. Only a small number described their data sources, and the majority did not meet basic transparency standards.
Nevertheless, these datasets were widely cited and frequently used to make recommendations for clinical care. Of the 125 studies, three models showed evidence of potential use in practice, one was cited in a medical device patent, and 86 review articles referenced these models.
Some articles described actual or potential use in clinical settings, and 11 studies developed web- or app-based prediction tools with graphical user interfaces, two of which were publicly accessible. None of the studies were referenced in policy documents.
The number of publications using these datasets has continued to rise, despite ongoing concerns about the quality and authenticity of the underlying data.
Improving dataset transparency to protect patient care
The current study highlights the urgent need to address the use of unreliable data in clinical prediction model research. Reliable data and transparent methods are essential to ensure trustworthy clinical decisions and safeguard patient care. The authors recommend action by journals, publishers, data repositories, researchers, and clinicians to improve standards and promote responsible research practices.
The authors also emphasized that this study examined only two publicly available Kaggle datasets and that it remains unclear how widespread similar data provenance issues are across other datasets and repositories.
Download your PDF copy by clicking here.
Study finds growing burden of diabetes-related chronic kidney disease