Effect-interval coverage and model-selection recovery
Source:vignettes/validation.Rmd
validation.RmdThis page is the precomputed report for two practical validation
questions: do simulated effect intervals cover known effects at the
advertised rate, and does model comparison recover the true DAG from a
small candidate set? The report will show cached live-drmTMB results
once inst/validation/generate.R has produced
inst/validation/validation-results.rds. Until that cache
exists, this article renders the study design only and makes no
validation claim. This document never fits models while it is
rendered.
What is being tested
These are simulation studies where the true effects or true graph are
known before fitting; the question is whether drmSEM’s intervals and
model-selection tools recover that known truth. The validation matrix
(docs/design/11-validation-matrix.md) established
point recovery: the machinery returns the right
number on a known-answer data-generating process. This page
(docs/design/12-coverage-calibration.md) is designed to
establish the operating characteristics – does the
uncertainty have the advertised behaviour, and do the
inferential procedures calibrate? These are many-replicate
studies, so the full grid runs in the live lane and is cached, like the
d-separation calibration page.
Two studies are tabulated here:
-
Effect-interval coverage. Does
uncertainty = "parametric"produce intervals that cover the true effect at the nominal rate? -
Model-selection recovery. Does
[
best()] select the true DAG under the chosen information criterion, and how much model weight lands on the truth?
Two related checks live elsewhere and are not
reimplemented here: d-separation Type-I / power is the calibration study
on its own page, and sampler dispersion vs
drmTMB::simulate() is the standalone probe
inst/validation/sampler-dispersion-probe.R.
The study design (shown, never run)
SEED <- 20260607
GRID <- list(
R = 300, # replicate datasets per cell
n_grid = c(300, 1000), # sample sizes
B = 400, # parametric-bootstrap draws per fit
level = 0.95 # nominal CI level
)Effect-interval coverage
A linear identity-link Gaussian chain x -> m -> y
with a direct x -> y edge, whose true total /
direct / indirect effects are known in closed form – products
of the data-generating coefficients times the contrast width
sd(x):
# data-generating coefficients
b_xm <- 0.6; b_my <- 0.5; b_xy <- 0.3
m <- rnorm(n, b_xm * x, 1)
y <- rnorm(n, b_xy * x + b_my * m, 1)
# known truth (response units of y, per sd(x))
indirect <- b_xm * b_my * sd(x)
direct <- b_xy * sd(x)
total <- (b_xy + b_xm * b_my) * sd(x)
sem <- drm_sem(
m = drm_node(drmTMB::bf(m ~ x), family = gaussian()),
y = drm_node(drmTMB::bf(y ~ x + m), family = gaussian()),
data = dat
)
tot <- total_effects(sem, "x", "y", method = "simulate",
uncertainty = "parametric", B = 400)
dec <- indirect_effects(sem, "x", "y", uncertainty = "parametric", B = 400)
# record whether conf.low/conf.high bracket the known truth, per quantityModel-selection recovery
A true DAG (x -> m -> y, with the direct
x -> y edge omitted) versus a candidate set that
includes the truth, an over-fitted rival, and a missing-edge rival. Over
replicates we record how often [best()] returns the truth
and the mean model weight on it. The default CBIC criterion
uses a BIC-style penalty for extra paths; CICc is still
reported as the phylopath-style support criterion and can retain
over-fitted rivals.
models <- drm_model_set(
truth = drm_dag(m ~ x, y ~ m), # data-generating DAG
overfit = drm_dag(m ~ x, y ~ x + m), # spurious x -> y edge
missing_edge = drm_dag(m ~ x, y ~ x) # true m -> y arrow left out
)
cmp <- compare(models, data = dat,
family = list(m = gaussian(), y = gaussian()))
selected <- cmp$model[which.min(cmp$CBIC)] # best() defaults to CBIC
selected_cicc <- cmp$model[which.min(cmp$CICc)] # CICc support rankingThis study code is shown for transparency only and is never evaluated when the vignette is built.
Acceptance criteria
The cached study object stores these checks in
val$acceptance:
- Every
(n, quantity)empirical coverage lies inside nominal +/- 2*SE. - Every
nhas truth selection rate >= 0.80 and missing-edge selection rate <= 0.05. - The truth selection rate is nondecreasing in
n(within 0.05 Monte-Carlo jitter).
| Criterion | Rule | Pass |
|---|---|---|
| effect-CI coverage | Every (n, quantity) empirical coverage lies inside nominal +/- 2*SE. | TRUE |
| model-selection recovery (CICc) | Every n has truth selection rate >= 0.80 and missing-edge selection rate <= 0.05 under CICc. | FALSE |
| model-selection recovery (CBIC) | Every n has truth selection rate >= 0.80 and missing-edge selection rate <= 0.05 under CBIC. | TRUE |
| selection rate increasing in n, by criterion | Truth selection rate is nondecreasing in n for each criterion (within 0.05 Monte-Carlo jitter). | TRUE |
Effect-interval coverage
Empirical coverage of the nominal 0.95 parametric interval per quantity and sample size, with the Monte-Carlo acceptance band (nominal +/- 2*SE) and mean interval width as a secondary efficiency diagnostic.
| quantity | n | coverage | MC lower | MC upper | mean width | reps | pass |
|---|---|---|---|---|---|---|---|
| direct | 300 | 0.957 | 0.925 | 0.975 | 0.261 | 300 | TRUE |
| direct | 1000 | 0.950 | 0.925 | 0.975 | 0.143 | 300 | TRUE |
| indirect | 300 | 0.950 | 0.925 | 0.975 | 0.176 | 300 | TRUE |
| indirect | 1000 | 0.943 | 0.925 | 0.975 | 0.098 | 300 | TRUE |
| total | 300 | 0.943 | 0.925 | 0.975 | 0.251 | 300 | TRUE |
| total | 1000 | 0.963 | 0.925 | 0.975 | 0.138 | 300 | TRUE |
Model-selection recovery
Selection rate of the true DAG and mean criterion weight on the truth
per sample size, with the rate at which the under-fitted (missing-edge)
rival is wrongly selected. CBIC is now the default ranking
because it is the parsimonious true-DAG recovery criterion.
CICc (Shipley 2013) remains available as a
phylopath-style support measure (Bijl 2018) for a small over-fitted
candidate set; it is an AIC-style criterion and so, as predicted by the
classical theory (Schwarz 1978; Burnham and Anderson 2002), does not
asymptote to 100% true-DAG selection in this grid.
| criterion | n | selection rate | mean truth weight | missing-edge rate | reps | pass |
|---|---|---|---|---|---|---|
| CBIC | 300 | 0.927 | 0.817 | 0 | 300 | TRUE |
| CBIC | 1000 | 0.973 | 0.883 | 0 | 300 | TRUE |
| CICc | 300 | 0.610 | 0.502 | 0 | 300 | FALSE |
| CICc | 1000 | 0.573 | 0.495 | 0 | 300 | FALSE |
How this was generated
The cache backing this page is built by
inst/validation/generate.R, run in the
live-drmTMB lane:
That script guards on requireNamespace("drmTMB"), sweeps
the grids above, collects only data (coverage flags, widths, selection
labels, model weights – never fitted objects), and writes a compact
summary list to inst/validation/validation-results.rds. The
cache carries a provenance stamp recording the
drmTMB version, date, R version, git SHA, replicate count, seed,
and n-grid so every table on this page is traceable to the
exact engine that produced it. If a cache was assembled from a retained
coverage block plus a fresh model-selection block, the scope note below
records that explicitly.
#> Cache provenance:
#> generated: 2026-06-09 04:32:15 UTC
#> drmTMB version: 0.1.3.9000
#> drmTMB SHA: 17b132119eeef40ac6a9655a7372281844007ae6
#> drmSEM version: 0.5.0
#> R version: R version 4.5.2 (2025-10-31)
#> git SHA: d1e91f9
#> replicates: 300
#> seed: 20260607
#> n grid: 300, 1000
#>
#> Scope note: C3 model selection regenerated from current source; C1 coverage retained from previous cache because full coverage rerun exceeded the live-lane wall-time budget.
#>
#> Study blocks:
#> C1 coverage generated: 2026-06-09 00:24:49 UTC (drmSEM 0.2.0.9000, git d1e91f9)
#> C3 model selection generated: 2026-06-09 04:32:15 UTC (drmSEM 0.5.0, git d1e91f9)