Bayesian Model Specification: Some problems related to model choice and calibration

In the development of Bayesian model specification for inference and prediction we focus on the conditional distributions p([theta],[beta]) and p(D[theta],[beta]), with data D and background assumptions [beta], and consider calibration (an assessment of how often we get the right answers) as an important integral step of the model development. We compare several predictive model-choice criteria and present related calibration results. In particular, we have implemented a simulation study to compare predictive model-choice criteria LS[cv] , a log-score based on cross-validation, LS[fs], a full-sample log score, with deviance information criterion, DIC. We show that for several classes of models DIC and LS[cv] are (strongly) negatively correlated; that LS[fs] has better small-sample model discrimination performance than either DIC, or LS[cv]; we further demonstrate that when validating the model-choice results, a standard use of posterior predictive tail-area for hypothesis testing can be poorly calibrated and present a method for its proper calibration.