Evaluate a fitted pigauto model on its test set

Computes type-specific performance metrics for each trait. By default the model is evaluated on the test split stored in the fit object. An alternative data can provide the truth matrix, and alternative splits can select compatible held-out cells from the original fit; evaluation does not refit the baseline or GNN for a new holdout design.

Usage

evaluate(fit, data = NULL, splits = NULL)

Arguments

fit

pigauto_fit object.

data

pigauto_data object (default: NULL, uses the training data stored in the fit via predict()).

splits

splits object (default: NULL, uses fit$splits). Alternative indices must be a compatible subset of the cells held out when the fit was made; this argument does not refit for a new holdout design.

Value

A data.frame with columns: method, trait, type, metric, value, n_test.

Details

Metrics by trait type:

continuous

RMSE, Pearson r, and MAE on the z-scored latent scale

count

RMSE, MAE, and Pearson r on the z-scored log1p latent scale

binary

Accuracy, Brier score

categorical

Accuracy (overall)

ordinal

RMSE on the z-scored latent scale, Spearman rho

proportion

RMSE and MAE on the transformed latent scale, Pearson r

zi_count

RMSE, MAE, and Pearson r on the original count scale, comparing observed counts with predicted expected counts; zero-accuracy and Brier score are computed for the zero/nonzero gate

multi_proportion

Aitchison distance in CLR space, z-scored CLR RMSE, and decoded simplex MAE

When conformal scores are present in the fit, conformal_coverage_95 is reported for continuous, count, ordinal, and proportion traits. This is the empirical fraction of test predictions inside the same trait-level conformal radius around each prediction. It uses one score per trait, not cell-specific or Mondrian bounds. Cross-validation and model reports can also report coverage of their actual per-cell bounds.

When the fit includes a baseline, baseline metrics are appended with method = "baseline" for direct comparison. This uses the baseline already stored in the fit and its selected defaults; evaluate() has no argument for alternative baseline settings. To compare another baseline, construct it explicitly in a separate evaluation workflow.

Examples

# \donttest{
data(avonet300, tree300)
tree <- ape::keep.tip(tree300, tree300$tip.label[seq_len(30L)])
traits <- avonet300[match(tree$tip.label, avonet300$Species_Key),
                     c("Mass", "Wing.Length"), drop = FALSE]
rownames(traits) <- tree$tip.label
pd <- preprocess_traits(traits, tree)
splits <- make_missing_splits(pd$X_scaled, trait_map = pd$trait_map)
fit <- fit_pigauto(pd, tree, splits = splits, epochs = 5L,
                   verbose = FALSE)
#> Warning: Small validation set for 2 trait(s): Mass (n=1), Wing.Length (n=2). Calibrated gate and conformal scores will be noisy for these trait(s). 95% split-conformal coverage is NOT achievable for 2 trait(s) with fewer than 19 validation cells (Mass (n=1), Wing.Length (n=2)): the achievable ceiling is n_val / (n_val + 1), which only reaches 0.95 at n_val >= 19. See `?fit_pigauto` under 'Calibration at small n' for smoothing options.
eval_df <- evaluate(fit)
eval_df[eval_df$metric == "rmse", ]
#>      method       trait       type metric     value n_test
#> 1   pigauto        Mass continuous   rmse 0.2115063      8
#> 5   pigauto Wing.Length continuous   rmse 0.4305777      4
#> 9  baseline        Mass continuous   rmse 0.2115063      8
#> 12 baseline Wing.Length continuous   rmse 0.4305777      4
# }