Rigorous model validation is an ongoing challenge in the Department of Defense (DOD) and is only becoming more important as the DOD furthers adoption of digital engineering and model-based systems engineering (MBSE). The STAT COE is actively researching improvements to model validation methods and published several best practices in FY22. In the area of digital engineering, the STAT COE is developing Model Validation Levels (MVLs), which provide an objective, automatable framework for quantifying how much trust can be placed in the results of a model to represent the real world. The STAT COE is currently looking to partner with DOD programs in this area to demonstrate MVL application to real-word systems. Additionally, in MBSE, the STAT COE advocates for a Platform Independent Model (PIM) approach with a verification and validation (V&V)-driven system decomposition to increase opportunities for early testing.

Digital Engineering uses models as a continuum throughout the system lifecycle, and those models must be validated to be considered trustworthy for use. Traditionally, validation has been a onetime process which designates a model as either valid or not valid. However, as the DOD shifts toward digital engineering, a new paradigm is needed which allows model validity to evolve over the system lifecycle. The STAT COE developed MVLs to meet this need and recently published, “Elements of a Mathematical Framework for Model Validation Levels,” authored by STAT COE contractors Mr. Kyle Provost, Ms. Corinne Weeks, and Mr. Nicholas Jones, and STAT COE Deputy Director, Maj Victoria Sieck. The paper expands upon previous work to detail the mathematical methods which factor into an MVL. A second best practice, “Constructing a Metric for Fidelity in Model Validation,” authored by Ms. Weeks, Mr. Jones, and Dr. Melissa Key, describes the construction and mathematical behavior of the MVL fidelity metric. The team is currently working to develop an automated tool for computing MVLs as well as a comprehensive guide for using and understanding MVLs.

MBSE formalizes the application of models to the systems engineering process and is key in the shift toward digital engineering. The systems engineering process, however, typically excludes V&V planning up front, resulting in late discovery of risk. Mr. Chuck Rogal, STAT COE contractor, authored the STAT COE best practice, “Model Based Systems Engineering for Developmental System Verification and Validation.” The paper illustrates the use of a Platform Independent Model (PIM) which facilitates V&V activities earlier in the lifecycle. Additionally, the paper defines a hierarchal set of models which map to the traditional V-model of systems engineering. Each modeling layer generates opportunities for V&V activities early in the development cycle, creating artifacts which can be passed forward to guide test and evaluation activities later in the lifecycle. Applying this approach can help produce more mature systems, promote artifact reuse, identify risk earlier, and expedite risk mitigation.

These best practices can be found at https://www.afit.edu/STAT/. Additionally, if your program would be interested in collaborating with STAT COE in applying MVLs, please reach out to AFIT.ENS.STATCOE@us.af.mil.

Authored By:
Ms. Corinne Weeks, ctr; Mr. Nicholas Jones, ctr; Mr. Kyle Provost, ctr
STAT Experts