Addressing Data Leakage and Reproducibility Crises in Machine-Learning-Based Biomechanics
DOI:
https://doi.org/10.29207/z4sxww09Keywords:
dataset audit, data provenance, gait event detection, feature engineering, reproducibilityAbstract
Gait-event features dataset intended to support machine-learning-based locomotion analysis, but delivered files are rarely checked against their own documentation or against standard values from the gait-analysis literature. We apply a six-stage, three-layer audit (provenance, structural/physical plausibility, signal validity) to “Robust Gait Event Detection Using GRF and EMG Signals,” a 208-row, 12-feature dataset describing four stance-phase gait events (Heel Strike, Loading Response, Mid-Stance, Heel Off) from 52 participants. Row count, feature count, and participant count all match the hosting page’s Instructions section. However, direct inspection finds two of the twelve documented features are exact duplicates under different names: the column labeled COV (coefficient of variation) is byte-identical to VAR (variance) rather than the standard STD/mean ratio, and the column labeled mode is byte-identical to minimum across all 208 rows, consistent with a computation error rather than genuine central-tendency estimation. An eight-row contiguous block is duplicated verbatim at a fixed 32-row offset, indicating the true unique sample is 50 participants (200 rows), not the documented 52 (208 rows). The file also provides no field distinguishing GRF-derived from EMG-derived features, despite both being named as contributing sources in the dataset’s title and abstract. Finally, while all twelve features discriminate strongly between the four event classes by one-way ANOVA, the percentage feature is exactly 100.000000 with zero variance for every Heel-Off row, consistent with a windowing artifact of the class definition rather than an independently varying measurement, and its per-class averages are inconsistent with standard stance-phase timing reported in the clinical gait-analysis literature.References
[1] D. Alvarado-Rivera, P. A. Niño-Suárez, and L. G. Corona-Ramírez, “Robust Gait Event Detection Using GRF and EMG Signals,” IEEE DataPort, 2025, doi: 10.21227/0mch-sc46.
[2] J. Perry and J. M. Burnfield, Gait Analysis: Normal and Pathological Function, 2nd ed. Thorofare, NJ: SLACK Incorporated, 2010.
[3] J. A. Zeni Jr., J. G. Richards, and J. S. Higginson, “Two Simple Methods for Determining Gait Events during Treadmill and
Overground Walking Using Kinematic Data,” Gait & Posture, vol. 27, no. 4, pp. 710–714, 2008, doi: 10.1016/j.gaitpost.2007.07.007.
[4] H. J. Hermens, B. Freriks, C. Disselhorst-Klug, and G. Rau, “Development of Recommendations for SEMG Sensors and Sensor Placement Procedures,” Journal of Electromyography and Kinesiology, vol. 10, no. 5, pp. 361–374, 2000, doi: 10.1016/S1050-6411(00)00027-4.
[5] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumé III, and K. Crawford, “Datasheets for Datasets,” Communications of the ACM, vol. 64, no. 12, pp. 86–92, 2021, doi: 10.1145/3458723.
[6] R. Y. Wang and D. M. Strong, “Beyond Accuracy: What Data Quality Means to Data Consumers,” Journal of Management Information Systems, vol. 12, no. 4, pp. 5–33, 1996, doi: 10.1080/07421222.1996.11518099.
[7] S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman, “Leakage in Data Mining: Formulation, Detection, and Avoidance,” ACM Transactions on Knowledge Discovery from Data, vol. 6, no. 4, article 15, 2012, doi: 10.1145/2382577.2382579.
[8] S. Kapoor and A. Narayanan, “Leakage and the Reproducibility Crisis in Machine-Learning-Based Science,” Patterns, vol. 4, no. 9, article 100804, 2023, doi: 10.1016/j.patter.2023.100804.
[9] M. Baker, “1,500 Scientists Lift the Lid on Reproducibility,” Nature, vol. 533, no. 7604, pp. 452–454, 2016, doi: 10.1038/533452a.
[10] M. R. Munafò, B. A. Nosek, D. V. M. Bishop, K. S. Button, C. D. Chambers, N. Percie du Sert, U. Simonsohn, E.-J. Wagenmakers, J. J. Ware, and J. P. A. Ioannidis, “A Manifesto for Reproducible Science,” Nature Human Behaviour, vol. 1, article 0021, 2017, doi: 10.1038/s41562-016-0021.
[11] J. P. A. Ioannidis, “Why Most Published Research Findings Are False,” PLoS Medicine, vol. 2, no. 8, e124, 2005, doi: 10.1371/journal.pmed.0020124.
[12] M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru, “Model Cards for Model Reporting,” in Proc. Conf. on Fairness, Accountability, and Transparency (FAT* ’19), Atlanta, GA, USA, 2019, pp. 220–229, doi: 10.1145/3287560.3287596.
[13] J. M. Hausdorff, “Gait Variability: Methods, Modeling and Meaning,” Journal of NeuroEngineering and Rehabilitation, vol. 2, article 19, 2005, doi: 10.1186/1743-0003-2-19.
[14] D. A. Winter, Biomechanics and Motor Control of Human Movement, 4th ed. Hoboken, NJ: John Wiley & Sons, 2009.
[15] IEEE, “IEEE Editorial Style Manual for Authors,” IEEE Editorial, 2019.
[16] O. Tur, A. Tur, V. Shabunina, and E. Chernaia, “The IEEE Style: Peculiarities of the Format and Application Prospects,” in Proc. 25th IEEE Int’l Conf. on Problems of Automated Electric Drive. Theory and Practice (PAEP), 2020, doi: 10.1109/PAEP49887.2020.9240790.
[17] R. Kohavi, “A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection,” in Proc. 14th International Joint Conference on Artificial Intelligence (IJCAI), Montreal, Canada, vol. 2, 1995, pp. 1137–1143.
[18] T. Chen and C. Guestrin, “XGBoost: A Scalable Tree Boosting System,” in Proc. 22nd ACM SIGKDD Int’l Conf. Knowledge Discovery and Data Mining, San Francisco, CA, USA, 2016, pp. 785–794, doi: 10.1145/2939672.2939785.
[19] H. He and E. A. Garcia, “Learning from Imbalanced Data,” IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 9, pp. 1263–1284, 2009, doi: 10.1109/TKDE.2008.239.
[20] T. Saito and M. Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets,” PLOS ONE, vol. 10, no. 3, e0118432, 2015, doi: 10.1371/journal.pone.0118432.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Journal of Systems Engineering and Information Technology (JOSEIT)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
