Operational Metrics to Assess Performances of a Prognosis Function. Application to Lubricant of a Turbofan Engine Over- Consumption Prognosis



Published Jul 8, 2014


In the aeronautical field, one of the major concerns is the availability of systems. To ensure availability, Prognosis and Health Management algorithms are developed. The aim of these algorithms is twofold. The first one is to detect and locate degradation premise of “no go” condition occurrence. The second one is to predict the health state of the system at a given time horizon. Before introducing PHM algorithms in operation, it is necessary to assess their performances. This is accomplished thank to a “maturation” phase. This phase consists in defining the performance metrics from an operational relevance point of view, in estimating this performance indicator and finally in proposing improvements to meet the airline companies requirements. We consider that the maturation of the detection function has already been completed and that we are interested in the maturation of the prognosis function. This paper deals with the performance assessment of a prognosis function using two operational metrics. A performance estimation procedure is developed. It is applied to the prognosis of turbofan engine lubricant over-consumption. The considered prognosis function is the probability to cross “no go” condition threshold at a given time horizon. This prediction is made thanks to an indicator of the health state of the system. Then it is compared with a threshold in order to trigger an alarm and give rise to a removal if necessary. Within this framework, we have defined two operational metrics for assessing the performance of this prognosis function. These metrics are the “ratio of justified removals” (P(Alarm|Crossing)) and the “ratio of not justified removals” (P(No-crossing|Alarm)). These metrics require the availability of observed lubricant over-consumption to compare the prediction results to the observed values. In the absence of lubricant over-consumption values in operation, a way is to simulate values.
This communication describes the procedure to estimate the performance of the prognosis function and presents the obtained results. The performances estimations trigger improvements. It appears that we have to enhance the precision of the considered health indicator before continuing to assess the performance of the considered prognosis function.

How to Cite

HMAD, O., MASSE, J.-R., GRALL-MAËS, E. ., BEAUSEROY, P., & MATHEVET, A. (2014). Operational Metrics to Assess Performances of a Prognosis Function. Application to Lubricant of a Turbofan Engine Over- Consumption Prognosis. PHM Society European Conference, 2(1). https://doi.org/10.36001/phme.2014.v2i1.1497
Abstract 103 | PDF Downloads 85



maturation, performance evaluation, PHM, failure prognosis, Turbofan engine, Engine Health Monitoring, Prognostic Evaluation

Brier G.W. (1950). Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, pp 1-3.
Bröcker J. et Smith L.A. (2007). Increasing the reliability of reliability diagrams. Wea. Forecasting, 22, pp 651-661.
Candille G. et Talagrand O. (2005). Evaluation of probabilistic prediction systems for a scalar variable. Royal Meteorological Society, pp 1-20.
Demaison F., Flandrois X., Massé J.R., Massot G., Hmad O. & Ricordeau J. (2011). Method and system for monitoring the level of oil contained in a tank of an aircraft engine. Patent # WO/2011/131892.
Dragomir O.E., Gouriveau R. et Zerhouni, N. (2008). Adaptive neuro-fuzzy inference system for midterm prognostic error stabilization. International Journal of Computers, Communications and Control, pp 271-276.
Ebert B. Forecast Verification - Issues, Methods and FAQ. Last access 2013, url: http://www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html.
Hmad O., Grall-Maes E., Beauseroy P., Masse J.R. & Mathevet A. (2012). A Maturation Procedure for Prognosis and Health Monitoring Algorithms. PSAM11 & ESREL12 conference, June 25-29, Helsinki, Finland.
Jardine A.K.S., Lin D. et Banjevic D. (2006). A review on machinery diagnostics and prognostics implementing condition based maintenance. Mechanical Systems and Signal Processing, 20, pp 1483-1510.
Letot C. et Dehombreux P. (2009). Modèles de dégradation pour l’estimation de la fiabilité et de la durée de vie résiduelle : applications à la fissuration. Actes du 4ème congrès PENTOM, Autrans, France.
Murphy A.H. (1973). A new vector partition of the probability score. J. Appl. Meteor., 12, pp 595-600.
Nikulin M.S., Limnios N., Balakrishnan N., Kahle W. et Huber-Carol C. (2010). Advances in Degradation Modeling Applications to Reliability, Survival Analysis, and Finance. Birkhauser, Boston.
Roussignol M. (2009). Gamma stochastic process and application to maintenance. Cours de l’université Paris Est – Marne la vallée.
Saxena A., Celaya J., Balaban E., Goebel K., Saha B., Saha S. et Schwabacher M. (2008). Metrics for Evaluating Performance of Prognostics Techniques. 1st International Conference on Prognostics and Health Management, Denver, USA.
Saxena A., Celaya J., Goebel K., Saha B. et Saha S. (2009). Evaluating Algorithm Performance Metrics Tailored for Prognostics. IEEE Aerospace Conference, Big Sky Montana, pp 1-13.
Si X.S, Wang W., Hu C.H., Zhou D.H. (2011). Remaining useful life estimation – A review on the statistical data driven approaches. European Journal of Operational Research, Volume 213, Issue 1, pp 1-14. ISSN 0377-2217.
Sikorska J.Z., Hodkiewicz M., Ma L.. (2011). Prognostic modelling options for remaining useful life estimation by industry. Mechanical Systems and Signal Processing, Volume 25, Issue 5, pp 1803-1836. ISSN 0888-3270.
Vachtsevanos G., Lewis F.L., Roemer M., Hess A. et Wu B. (2006). Intelligent Fault Diagnosis and Prognosis for Engineering Systems. First edition, John Wiley & Sons, Hoboken, New Jersey.
Van Noortwijk J.M. (2009). A survey of the application of gamma processes in maintenance. Reliability Engineering and System Safety, 94(1), pp 2–21.
Technical Papers