Run a simulation benchmark for pigauto — simulate

Generates trait data under various evolutionary models, introduces missing data, fits the Brownian motion baseline and a pigauto model, and compares performance. GNN training in the pigauto model is controlled by the gnn argument. Use it for exploratory comparisons and workflow checks. Its default of three replicates, and the one-replicate example, do not by themselves establish performance. For consequential performance claims, pre-specify a design, use replication suited to the question, and account for every requested attempt, including failures. The result summary alone does not preserve that denominator. The baseline is fitted first with fit_baseline()'s defaults: estimated continuous-family and discrete-trait lambda, joint_solver = "inhouse", and automatic exact/per-column routing where the joint path is eligible. Arguments in ... are forwarded only to fit_pigauto() after the baseline is fitted; they do not change its settings. To compare another baseline configuration, construct that baseline explicitly in a custom evaluation workflow.

Usage

simulate_benchmark(
  n_species = 100L,
  n_traits = 4L,
  scenarios = c("BM", "OU", "regime_shift", "nonlinear", "mixed"),
  missing_frac = 0.25,
  n_reps = 3L,
  epochs = 500L,
  verbose = TRUE,
  seed = NULL,
  gnn = TRUE,
  ...
)

# S3 method for class 'pigauto_benchmark'
print(x, ...)

# S3 method for class 'pigauto_benchmark'
summary(object, ...)

Arguments

n_species: integer. Number of tips in the simulated tree (default 100).
n_traits: integer. Number of continuous traits (default 4). Ignored for scenario = "mixed", which generates a fixed trait set.
scenarios: character vector. Subset of c("BM", "OU", "regime_shift", "nonlinear", "mixed"). Default runs all.
missing_frac: numeric. Fraction of observed cells held out (default 0.25).
n_reps: integer. Number of replicate trees per scenario (default 3).
epochs: integer. Maximum GNN training epochs (default 500).
verbose: logical. Print progress (default TRUE).
seed: optional integer. When supplied, makes the generated benchmark replicates reproducible.
gnn: logical. Whether the pigauto arm trains the GNN. Default TRUE, so this function keeps comparing the baseline with the GNN-corrected model even though fit_pigauto and impute default to gnn = FALSE.
...: additional arguments passed to fit_pigauto.
x: A benchmark object.
object: A benchmark object to summarize.

Value

An object of class "pigauto_benchmark" with:

results: data.frame with columns: scenario, rep, method, trait, type, metric, value, n_test. Failed attempts are caught and omitted; their messages are suppressed when verbose = FALSE. There is no returned attempt/failure table.
summary: data.frame averaged over surviving metric rows. Its n_reps column counts non-missing values for that metric, rather than all requested attempts.
scenarios: character vector of scenarios run.
n_reps: integer: requested replicates per scenario.
n_species: integer.

For scientific method comparisons, retain a separate attempt/failure table and reconcile every requested replicate before interpreting the returned summaries; this function alone does not preserve that denominator.

The print method returns x invisibly.

The summary method returns object$summary invisibly.

Details

Available scenarios:

"BM": Independent-trait Brownian-motion control. Each trait is generated separately; no cross-trait covariance is imposed. The fitted baseline estimates parameters rather than using known generating values. Finite-sample changes in GNN performance are measured outcomes, not a promised benefit from cross-trait correlation.
"OU": Ornstein-Uhlenbeck traits with a specified stabilising-selection parameterisation. This differs from the BM control; the scenario definition alone does not establish which method predicts better.
"regime_shift": Two-regime BM with opposite additive offsets for two clades. The resulting distribution depends on the tree and offset; bimodality is not guaranteed.
"nonlinear": Traits are generated with non-linear inter-trait relationships. Any difference in predictive performance is an empirical outcome, not a guaranteed advantage for either method.
"mixed": Mixed trait types: 2 continuous + 1 binary + 1 categorical trait (3 levels). This scenario exercises these three types together; it does not cover every supported trait type.

Printing

Prints requested replicates and aggregate metrics from surviving result rows. The ... argument is currently unused.

Summary method

summary() prints scenario metrics and relative RMSE changes using the surviving rows. Relative changes can be undefined for a zero baseline RMSE. Its ... argument is unused.

Examples

# \donttest{
bench <- simulate_benchmark(n_species = 20L, n_traits = 2L,
                            scenarios = "BM", epochs = 5L, n_reps = 1L,
                            verbose = FALSE, gnn = FALSE)
#> Warning: Small validation set for 2 trait(s): trait1 (n=1), trait2 (n=1). Calibrated gate and conformal scores will be noisy for these trait(s). 95% split-conformal coverage is NOT achievable for 2 trait(s) with fewer than 19 validation cells (trait1 (n=1), trait2 (n=1)): the achievable ceiling is n_val / (n_val + 1), which only reaches 0.95 at n_val >= 19. See `?fit_pigauto` under 'Calibration at small n' for smoothing options.
bench$summary
#>    scenario   method  trait       type                metric      mean sd
#> 1        BM baseline trait1 continuous                   mae 0.5508262 NA
#> 2        BM baseline trait1 continuous             pearson_r 0.8952109 NA
#> 3        BM baseline trait1 continuous                  rmse 0.6322628 NA
#> 4        BM baseline trait2 continuous                   mae 0.2679830 NA
#> 5        BM baseline trait2 continuous             pearson_r 0.9930443 NA
#> 6        BM baseline trait2 continuous                  rmse 0.3211798 NA
#> 7        BM  pigauto trait1 continuous conformal_coverage_95 0.0000000 NA
#> 8        BM  pigauto trait1 continuous                   mae 0.5508262 NA
#> 9        BM  pigauto trait1 continuous             pearson_r 0.8952109 NA
#> 10       BM  pigauto trait1 continuous                  rmse 0.6322628 NA
#> 11       BM  pigauto trait2 continuous conformal_coverage_95 0.2500000 NA
#> 12       BM  pigauto trait2 continuous                   mae 0.2679830 NA
#> 13       BM  pigauto trait2 continuous             pearson_r 0.9930443 NA
#> 14       BM  pigauto trait2 continuous                  rmse 0.3211798 NA
#>    n_reps
#> 1       1
#> 2       1
#> 3       1
#> 4       1
#> 5       1
#> 6       1
#> 7       1
#> 8       1
#> 9       1
#> 10      1
#> 11      1
#> 12      1
#> 13      1
#> 14      1
plot(bench)

# }