Model Assessment

Astro 497: Week 4, Monday

Logistics

• Lab 4 link

Overview

1. Validate methods using simulated data

2. Validate computation

3. Compare Likelihood to expected distribution

4. Sensitivity to individual data points

5. Sensitivity to assumptions (e.g., prior distributions)

6. Predictive distribution

7. Cross validation

Validate methods using simulated data

• Generate simulated data sets that satisfy all model assumptions.

• Apply model and computational methods to simulated data

• Compare results to true parameter values.

Checks to perform based on simulated data:

• Are estimated model parameters close to true values?

• Do estimated parameter uncertainties accurately characterize the dispersion from the true values?

• Is the distribution of parameter estimates centered on their true values?

• How many iterations are required for algorithms to converge?

• What can we learn reliably?

• Are there regions of parameter space that result in low quality inference?

Validating methods on simulated data is a key step
(too often overlooked)

This approach is very useful for diagnosing many of the most common problems:

• Not enough data

• Measurement uncertainties too large

• Spatial or temporal sampling leads to degeneracies

• Model has too many unknown parameters

• Iterative algorithms don't converge reliably

• Iterative algorithms take longer than expected to converge

Real data is (almost always) more complicated than our idealizd model, but...

• If our method doesn't work reliabily for idealized data, then we shouldn't trust them applied to real data.

• If our method requires $10^6$ iterations to converge for idealized data, then we know we'll need to run it for at least that long for real data (and potentially longer).

Validate the computation

(This step applies regardless of whether analyzing simulated or real data.)

• Are the results pausible?

  • E.g., Magnitude of best-fit value for each model parameter

• For algorithms that have stochastic element, are results robust to:

  • Different (reasonable) initial guesses?

  • Different draws from psuedo-random number generator?

• For itterative algorithms:

  • Why did algorithm stop?

  • Any warning signs of non-convergence?

• For algorithms that return an uncertainty estimate, is the reported precission:

  • Robust?

  • Scientifically useful?

Compare Likelihood to expected distribution

Likelihood:

$$\begin{eqnarray} \mathcal{L} & = & p( \mathrm{data} \; | \; \mathrm{model\; parameters}, \; \mathrm{model} ) \\ & = & p (y|θ,M) = p(y|\theta) \end{eqnarray}$$

Often, we have a complex physical model, but a relatively simple model of the measurement process. In these cases, if we knew the full model was correct and the model parameters ($\theta$), then we can compute the distribution of the likelihood.

For simple measurement models, this can often be done analytically. E.g.,

• Poisson distribution for sum of counts (if counts are from a Poisson process)

• χ² distribution for sum of squared residuals (if measurement errors are Gaussian)

• F-distribution for ratio of two variables drawn from χ² distribution

For more complex measurement models, this is often done via simulation.

Sensitivity to individual data points

In practice, there are usually complications omitted from the model being used for inference.

• Could be missing astrophysics (e.g., sample of targets is contaminated with a small fraction of objects that different from intended sample)

• Could be incomplete model for measurement process (e.g., cosmic ray, observer points telescope at wrong object, ...)

To procted against making poor decissions/scientific inferences:

• Ask would our decisions/inferences change if a small fraction of measurements had be omitted from analysis?

Sensitivity to prior distribution/assumptions

• In a Bayesian analysis, we specify prior assumptions quantitatively by specifying a prior distribution for unknown model parameters.

• There are assumptions in any model (even if they're hard to find, e.g., in a frequentist test).

• Usually, reasonable scientists could make different reasonable assumptions.

• The precise quantiative results will differ depending on the assumptions.

• The differences in the results under different assumptions might be:

  • So small they are scientifically unininteresting,

  • So large that more detailed analysis isn't a good use of your time,

  • Somewhere in the middle.

• We need to know where this analysis falls along that spectrum before we start drawing scientific conclusions.

Predictive distribution

• Does model make good "predictions" for the observed data?

Numerical experiments

Simualted dataset & best-fit model

Order of polynominal to generate data:
a₀: a₁: a₂: a₃:
Number of observations: Number of outliers:
Measurement uncertainty:

Now let's repeat that analysis many times and compare the MLE estimates for each parameter with the true values.

Number of simulated datasets:

What would have happend if we included only the intercept and slope terms (and not $x^2$ or $x^3$ terms)?

Cross validation

• Previously, we fit a model based on all of our observations and then asked if the resulting model made good "predictions". This is a relatively weak test.

• A stronger test would be to see if the model can make good predictions for new data.

• Often, acquiring new data is very time consuming.

• This motivates a more powerful test, cross validation.

Cross validation of a single model

• Divide data into two subsets (e.g., 75% "training", 25% "validation")

• Fit data ("train model") using only training set

• Use resulting parameters to make predictions for validation set

• How well did the model do? (evaluate "loss function")

Cross validation when considering multiple models

• Divide data into three subsets (e.g., 60% "training", 20% "validation", 20% "test")

• For each model you want to consider:

  • Fit/Train using only data from training set

  • Validate using only data from validation set

• Pick which model you'll use for decissions/publication:

• Evaluate predictions for test data set.

• Report results on test data.

• Avoid temptation to make further "improvements" at the point.

Helper Code