Toward Statistical Risk Analysis for Aerospace Systems with AI/ML Components
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Abstract
Aerospace systems increasingly rely on machine learning (ML) and AI components. Such safety-critical applications require careful verification and validation (V&V), and certification. A central task in safety certification is risk assessment: for a large number of off-nominal failure conditions, it must be demonstrated that they do not pose undue risk during operation. Events with major, hazardous, or catastrophic consequences must only occur extremely infrequently.
This paper addresses the statistical modeling and analysis of rare events in AI/ML-based components at both the systemand component-level within the SYSAI analysis framework. Here, we focus on typical perception-related functional tasks that are found in unmanned aerial vehicles (UAVs) or supporting aircraft operations, specifically an Autonomous Centerline Tracking (ACT) system and an Autonomous Emergency Braking System (AEBS). We provide an overview of the SYSAI statistical learning and analysis framework and discuss challenges for the analysis of failure modes and rare events in systems with AI/ML components. We describe how SYSAI can efficiently generate and evaluate scenarios that are likely to contain rare events. These experiments support the estimation of reliable probability distributions for such events and the analysis of temporal dependencies between individual input images.
How to Cite
##plugins.themes.bootstrap3.article.details##
verification and validation, AI/ML systems, Aerospace systems
Au, S.-K., & Beck, J. L. (2001). Estimation of small failureprobabilities in high dimensions by subset simulation.In Probabilistic engineering mechanics (Vol. 16,pp. 263–277). doi: 10.1016/S0266-8920(01)00019-4
Bucklew, J. A. (2004). Introduction to rare event simulation.Springer. doi: 10.1007/b97424
Cohn, D. A. (1996). Neural network exploration using optimalexperimental design. Advances in Neural InformationProcessing Systems, 6(9), 679–686.
Concepts of design assurance for neural networks (CoDANN)(Tech. Rep.). (2020). European Aviation SafetyAgency.
de Koning, M., Cai, W., Sadigh, B., Oppelstrup, T., Kalos, M., & Bulatov, V. (2005, 03). Adaptive importancesampling monte carlo simulation of rare transitionevents. The Journal of chemical physics, 122,074103. doi: 10.1063/1.1844352
Dmitriev, K., Rhein, J., Beller, L., Br¨ocker, J., Huber, E.,Schumann, J., & Holzapfel, F. (2024). Safety assessmentof a machine learning-based aircraft emergencybraking system: A case study. In 2024 aiaa datc/ieee43rd digital avionics systems conference (dasc) (p. 1-10). doi: 10.1109/DASC62030.2024.10749696
Dmitriev, K., Rhein, J., Bender, J., Beller, L., He, Y., Schumann,J., & Holzapfel, F. (2025). Certifying machinelearning in aviation: An end-to-end dal c case study. In 2025 aiaa datc/ieee 44rd digital avionics systems conference(dasc).
Do-178c: Software considerations in airborne systems andequipment certification (No. RTCA DO-178C). (2011). Washington DC: RTCA, Inc.
EASA concept paper: First usable guidance for level 1&2machine learning applications (Tech. Rep.). (2023). European Aviation Safety Agency.
EASA concept paper: First usable guidance for level 1 machinelearning applications (Tech. Rep.). (2021). European Aviation Safety Agency.
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel,W., Bethge, M., & Wichmann, F. A. (2020). Shortcutlearning in deep neural networks. Nature MachineIntelligence, 2, 665–673. doi: 10.1038/s42256-020-00257-z
Goodfellow, I., Shlens, J., & Szegedy, C. (2014, 12). Explainingand harnessing adversarial examples.
Gramacy, R., & Polson, N. (2011). Particle learning ofGaussian process models for sequential design and optimization.Journal of Computational and GraphicalStatistics, 20(1), 467–478.
He, Y. (2012). Variable-length functional output predictionand boundary detection for an adaptive flight controlsimulator (Unpublished doctoral dissertation). Universityof California at Santa Cruz.
He, Y. (2015). Online detection and modeling of safetyboundaries for aerospace applications using activelearning and bayesian statistics. In 2015 internationaljoint conference on neural networks, IJCNN2015, killarney, ireland, july 12-17, 2015 (pp. 1–8). IEEE. Retrieved from https://doi.org/10.1109/IJCNN.2015.7280595 doi: 10.1109/IJCNN.2015.7280595
He, Y., & Schumann, J. (2020). A framework for the analysisof deep neural networks in aerospace applicationsusing bayesian statistics. In Proc. ijcnn, wcci.
He, Y., & Schumann, J. (2024). Statistical analysis and runtimemonitor- ing for an ai-based autonomous centerlinetracking system. IJPHM, 15(3).
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F.,Dorundo, E., . . . Gilmer, J. (2021). The many facesof robustness: A critical analysis of out-of-distributiongeneralization. Retrieved from https://arxiv.org/abs/2006.16241
Janai, J., G¨uney, F., Behl, A., & Geiger, A. (2020). Computervision for autonomous vehicles: Problems, datasetsand state of the art. Foundations and Trends in ComputerGraphics and Vision, 12(1–3), 1–308. doi: 10.1561/0600000079
Jones, D., Schonlau, M., & Welch, W. J. (1998). Efficientglobal optimization of expensive black box functions.Journal of Global Optimization, 13, 455–492.
Juneja, S., & Shahabuddin, P. (2006). Rare-event simulationtechniques: An introduction and recent advances. In S. Henderson & B. L. Nelson (Eds.), Handbooks inoperations research and management science: Simulation(Vol. 13, pp. 291–350). Elsevier. doi: 10.1016/S0927-0507(06)13009-3
Kalra, N., & Paddock, S. (2016). Driving to safety: Howmany miles of driving would it take to demonstrateautonomous vehicle reliability? (Tech. Rep.). RANDCorporation. doi: 10.7249/RR1478
Leveson, N. G. (2012). Engineering a safer world: Systemsthinking applied to safety. MIT Press.
MacKay, D. J. C. (1992). Information–based objective functionsfor active data selection. Neural Computation,4(4), 589–603.
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C.(2020). Hidden stratification causes clinically meaningfulfailures in machine learning for medical imaging. Proceedings of the ACM Conference on Health, Inference,and Learning (CHIL), 151–159. doi: 10.1145/3368555.3384468
O’Connor, P. D. T., & Kleyner, A. (2012). Practical reliabilityengineering (5th ed.). Wiley.
Ranjan, P., Bingham, D., & Michailidis, G. (2008). Sequentialexperiment design for contour estimation fromcomplex computer codes. Technometrics, 50(4), 527–541.
Redmon, J., & Farhadi, A. (2018). Yolov3: An incrementalimprovement. arXiv preprint arXiv:1804.02767.Retrieved from https://arxiv.org/abs/1804.02767
Rubino, G., & Tuffin, B. (2009). Rare event simulation usingmonte carlo methods. John Wiley & Sons. doi: 10.1002/9780470745403
Taddy, M. A., Gramacy, R. B., & Polson, N. G. (2011). Dynamictrees for learning and design. Journal of theAmerican Statistical Association, 106(493), 109-123.
Vesely, W. (2002). Fault tree handbook with aerospace applications.Washington, D. C.: NASA Office of Safetyand Mission Assurance.
Yadron, D., & Tynan, D. (2016). Tesla driver diesin first fatal crash while using autopilot mode.The Guardian. Retrieved from https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
The Prognostic and Health Management Society advocates open-access to scientific data and uses a Creative Commons license for publishing and distributing any papers. A Creative Commons license does not relinquish the author’s copyright; rather it allows them to share some of their rights with any member of the public under certain conditions whilst enjoying full legal protection. By submitting an article to the International Conference of the Prognostics and Health Management Society, the authors agree to be bound by the associated terms and conditions including the following:
As the author, you retain the copyright to your Work. By submitting your Work, you are granting anybody the right to copy, distribute and transmit your Work and to adapt your Work with proper attribution under the terms of the Creative Commons Attribution 3.0 United States license. You assign rights to the Prognostics and Health Management Society to publish and disseminate your Work through electronic and print media if it is accepted for publication. A license note citing the Creative Commons Attribution 3.0 United States License as shown below needs to be placed in the footnote on the first page of the article.
First Author et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 United States License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.