When is a model good enough? Deriving the expected value of model improvement via specifying internal model discrepancies

A “law-driven” or “mechanistic” computer model is a representation of judgments about the functional relationship between one set of quantities (the model inputs) and another set of target quantities (the model outputs). We recognize that we can rarely define with certainty a “true” model for a particular problem. Building an “incorrect” model will result in an uncertain prediction error, which we denote “structural uncertainty.” Structural uncertainty can be quantified within a Bayesian framework via the specification of a series of internal discrepancy terms, each representing at a subfunction level within the model the difference between the subfunction output and the true value of the intermediate parameter implied by the subfunction. By using value of information analysis we can then determine the expected value of learning the discrepancy terms, which we loosely interpret as an upper bound on the “expected value of model improvement.” We illustrate the method using a case study model drawn from the health economics literature.