Title: F-statistic for model validation over experimental regions using least squares response surfaces

Authors: Scott M. Storm; Raymond R. Hill; Joseph J. Pignatiello; Edward D. White; G. Geoffrey Vining

Addresses: AFIT/ENS, 2950 Hobson Way, Bld 641, Wright Patterson AFB, OHIO 45433, USA ' AFIT/ENS, 2950 Hobson Way, Bld 641, Wright Patterson AFB, OHIO 45433, USA ' AFIT/ENS, 2950 Hobson Way, Bld 641, Wright Patterson AFB, OHIO 45433, USA ' AFIT/ENS, 2950 Hobson Way, Bld 641, Wright Patterson AFB, OHIO 45433, USA ' Virginia Polytechnic Institute, Department of Statistics, 250 Drillfield Drive, Blacksburg, VA 24061, USA

Abstract: When a simulation operates over an array of input settings, it is critical that its validity across all settings is considered. This paper proposes an F-statistic, based on a ratio of mean square errors, to assess model validity. System data is fit to an Ordinary Least Squares response surface and the mean square errors between both system and model data and the fitted surface are calculated. The hypothesis under test is that system and model data that possess the same distributional properties will produce similar measures of disagreement with regard to the fitted surface. If the F-statistic indicates the mean square errors are statistically different, it is determined that the system and model data do not share distributional properties and the model is assessed as invalid. A notional example demonstrates the methodology and considerations to the method's confidence with respect to the model sample size is discussed.

Keywords: experimental design; F-statistic; linear regression; mean square error; response surface; simulation; stochastic model validation.

DOI: 10.1504/IJEDPO.2017.087588

International Journal of Experimental Design and Process Optimisation, 2017 Vol.5 No.3, pp.133 - 150

Received: 04 May 2017
Accepted: 08 Jun 2017

Published online: 21 Oct 2017 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article