Compare BM baseline and pigauto methods across replicates

Runs a full comparison of the BM baseline and pigauto (with attention + calibration) over multiple random seeds. Returns a tidy data.frame suitable for plotting or downstream analysis.

Usage

compare_methods(
  data,
  tree,
  splits = NULL,
  seeds = NULL,
  epochs = 500L,
  verbose = TRUE,
  gnn = TRUE,
  ...
)

Arguments

data: pigauto_data object.
tree: phylo object.
splits: pre-computed splits (applied to all reps) or NULL to create fresh splits per seed.
seeds: optional integer vector of random seeds for replication. The default NULL performs one run using the current RNG stream.
epochs: number of training epochs.
verbose: logical.
gnn: logical. Whether the pigauto arm trains the GNN. Default TRUE, so this function keeps comparing the baseline with the GNN-corrected model even though fit_pigauto and impute default to gnn = FALSE.
...: additional arguments passed to fit_pigauto.

Value

A data.frame with columns: method, trait, type, metric, value, rep.

Details

For each seed the function:

Creates train/val/test splits.
Fits the phylogenetic BM baseline.
Fits pigauto (with attention and calibration enabled by default).
Evaluates both methods on the test split.
Collects results into a single data.frame.

The default comparison trains the optional torch GNN. The example checks that torch can create a tensor before fitting; it reports a skip when the Lantern runtime is unavailable.

Examples

# \donttest{
data(avonet300, tree300)
tree <- ape::keep.tip(tree300, tree300$tip.label[seq_len(30L)])
traits <- avonet300[match(tree$tip.label, avonet300$Species_Key),
                     c("Mass", "Wing.Length"), drop = FALSE]
rownames(traits) <- tree$tip.label
torch_ready <- requireNamespace("torch", quietly = TRUE) &&
  isTRUE(tryCatch(torch::torch_is_installed(), error = function(e) FALSE)) &&
  isTRUE(tryCatch({
    probe <- torch::torch_tensor(1)
    as.numeric(probe$item()) == 1
  }, error = function(e) FALSE))
if (torch_ready) {
  cmp <- compare_methods(preprocess_traits(traits, tree), tree, seeds = 1L,
                         epochs = 5L, verbose = FALSE, gnn = TRUE)
  aggregate(value ~ method + trait + metric, data = cmp, FUN = mean)
} else {
  message("Skipping GNN comparison: install torch and its Lantern runtime.")
}
#> Warning: Small validation set for 2 trait(s): Mass (n=1), Wing.Length (n=2). Calibrated gate and conformal scores will be noisy for these trait(s). 95% split-conformal coverage is NOT achievable for 2 trait(s) with fewer than 19 validation cells (Mass (n=1), Wing.Length (n=2)): the achievable ceiling is n_val / (n_val + 1), which only reaches 0.95 at n_val >= 19. See `?fit_pigauto` under 'Calibration at small n' for smoothing options.
#>      method       trait                metric      value
#> 1   pigauto        Mass conformal_coverage_95 0.00000000
#> 2   pigauto Wing.Length conformal_coverage_95 0.28571429
#> 3  baseline        Mass                   mae 0.09312548
#> 4   pigauto        Mass                   mae 0.55488154
#> 5  baseline Wing.Length                   mae 0.42005270
#> 6   pigauto Wing.Length                   mae 0.42995965
#> 7  baseline        Mass             pearson_r 0.99645716
#> 8   pigauto        Mass             pearson_r 0.99487251
#> 9  baseline Wing.Length             pearson_r 0.86823896
#> 10  pigauto Wing.Length             pearson_r 0.86833454
#> 11 baseline        Mass                  rmse 0.11534613
#> 12  pigauto        Mass                  rmse 0.62598823
#> 13 baseline Wing.Length                  rmse 0.51055379
#> 14  pigauto Wing.Length                  rmse 0.52436554
# }