Evaluate a fitted pigauto model on its test set

Computes type-specific performance metrics for each trait. By default the model is evaluated on the test split stored in the fit object, but an alternative data / splits can be supplied.

Usage

evaluate(fit, data = NULL, splits = NULL)

Arguments

fit

pigauto_fit object.

data

pigauto_data object (default: NULL, uses the training data stored in the fit via predict()).

splits

splits object (default: NULL, uses fit$splits).

Value

A data.frame with columns: method, trait, type, metric, value, n_test.

Details

Metrics by trait type:

continuous

RMSE, Pearson r, MAE

count

RMSE, MAE, Pearson r

binary

Accuracy, Brier score

categorical

Accuracy (overall)

ordinal

RMSE (on integer scale), Spearman rho

When conformal scores are present in the fit, conformal coverage at the 95\ ordinal traits.

When the fit includes a baseline, baseline metrics are appended with method = "baseline" for direct comparison.

Examples

# \donttest{
data(avonet300, tree300)
tree <- ape::keep.tip(tree300, tree300$tip.label[seq_len(30L)])
traits <- avonet300[match(tree$tip.label, avonet300$Species_Key),
                     c("Mass", "Wing.Length"), drop = FALSE]
rownames(traits) <- tree$tip.label
pd <- preprocess_traits(traits, tree)
splits <- make_missing_splits(pd$X_scaled, trait_map = pd$trait_map)
fit <- fit_pigauto(pd, tree, splits = splits, epochs = 5L,
                   verbose = FALSE)
#> Warning: phylo_signal_gate requires the 'phytools' package; returning NA for all traits.
#> Warning: Small validation set for 2 trait(s): Mass (n=1), Wing.Length (n=2). Calibrated gate and conformal scores will be noisy for these trait(s). 95% split-conformal coverage is NOT achievable for 2 trait(s) with fewer than 19 validation cells (Mass (n=1), Wing.Length (n=2)): the achievable ceiling is n_val / (n_val + 1), which only reaches 0.95 at n_val >= 19. See `?fit_pigauto` under 'Calibration at small n' for smoothing options.
eval_df <- evaluate(fit, data = pd)
eval_df[eval_df$metric == "rmse", ]
#>      method       trait       type metric     value n_test
#> 1   pigauto        Mass continuous   rmse 0.2251785      8
#> 5   pigauto Wing.Length continuous   rmse 0.5190160      4
#> 9  baseline        Mass continuous   rmse 0.2115063      8
#> 12 baseline Wing.Length continuous   rmse 0.4305777      4
# }