Evaluates trait-imputation performance by rotating held-out values through k test folds. Hold out trait cells, whole species, or rooted clades. Returns per-fold and aggregated metrics.
Displays the number of folds and replicates, then reports each trait's
mean and standard deviation for every metric. Returns x invisibly.
Additional arguments are ignored by this print method.
Arguments
- data
pigauto_data object (output of
preprocess_traits).- tree
phylo object.
- k
integer. Number of folds (default 5).
- seeds
optional integer vector. Seeds for replicate runs. The default
NULLperforms one run using the current RNG stream.- epochs
integer. Training epochs per fold (default 500).
- verbose
logical. Print progress (default TRUE).
- holdout
One of
"cell"(default),"species", or"clade". Species and clade modes mask every trait and every observation of each held-out species. They require at least three folds and retain the full tree, including held-out tips.- ...
Additional arguments forwarded by
cross_validate()tofit_pigauto(); ignored by the print and summary methods.- x
A
pigauto_cvobject returned bycross_validate().- object
A
pigauto_cvobject returned bycross_validate().
Value
A list of class "pigauto_cv" with:
- results
Data.frame with columns: fold, rep, trait, type, metric, value.
- summary
Data.frame with mean and sd across folds/reps for each trait + metric.
- conformal_coverage
Data.frame of coverage per trait across folds (if available).
- k
Number of folds.
- n_reps
Number of replicates.
- holdout
The chosen holdout mode.
- fold_assignments
For grouped modes, a data frame identifying the held-out species for each fold and replicate; NULL for cell mode.
Invisibly returns the summary data frame stored in
object$summary.
Details
cross_validate() precomputes one graph shared across folds and uses
k_eigen = 8L for that graph when k_eigen is omitted. An
explicit k_eigen is used for the shared graph and forwarded to
fit_pigauto(). Cell folds divide observed cells within each trait.
Species folds assign entire species at random; clade folds partition the
rooted tree into k disjoint rooted subtrees by repeatedly splitting the
largest eligible subtree. A rooted ape::phylo representation is
required, as determined by ape::is.rooted(); a root polytomy may
require a root edge. Clade folds can be unequal in size. Multifurcating
trees can defeat this greedy partition even when a partition exists. None
of the modes are
stratified by outcome class. The next fold is used for validation, and the
rest for training. Grouped folds require at least two training observations
and two training species for each trait.
In species and clade modes, trait centering, scaling, observed ranges, and
other trait-map statistics are recalculated using the training rows only.
The trait types, factor levels and log-transform choices from the supplied
preprocessed data remain fixed across folds. Specify these in advance when
they would otherwise be inferred from held-out values. The phylogeny and
any supplied covariates remain available for held-out species. Grouped
modes require data freshly constructed by the current preprocess_traits().
Cell-mode
preprocessing retains its existing whole-data behaviour.
Interpret each metric on its documented scale in
evaluate_imputation; continuous and count RMSE use latent
scales. Conformal coverage is calculated separately after predictions are
decoded. Clade-structured holdouts need not be exchangeable with the
validation cells, so nominal conformal coverage is not guaranteed.
Printing
print.pigauto_cv() reports k folds and the replicate count,
followed by each trait's metric means and SDs. Displayed n is the
number of represented fold-by-replicate metric rows, not just the fold
count. This count includes rows with missing metric values; means and
SDs omit those values. When available, a separate conformal block shows
mean coverage and its SD against a nominal 95% target; the target is
not a guarantee. Empty summaries print “No results.” The method
ignores ... and returns x invisibly.
Summary method
summary.pigauto_cv() prints a compact per-trait table with RMSE,
Pearson \(r\) for continuous traits, Spearman \(\rho\) for ordinal
traits, and accuracy when returned (binary/categorical accuracy or
multi-proportion dominant-component accuracy). A dash means the metric
row is absent. A present but nonfinite metric may display NA or
NaN. An empty summary prints “No results.” The full
object$summary data frame, which can contain metrics omitted from
this display, is returned invisibly. The method ignores ....
Examples
# \donttest{
data(avonet300, tree300, package = "pigauto")
tree <- ape::keep.tip(tree300, tree300$tip.label[seq_len(12L)])
traits <- avonet300[match(tree$tip.label, avonet300$Species_Key),
c("Mass", "Wing.Length"), drop = FALSE]
rownames(traits) <- tree$tip.label
cv <- cross_validate(preprocess_traits(traits, tree), tree, k = 3L,
seeds = 1L, epochs = 1L, verbose = FALSE,
k_eigen = 4L, hidden_dim = 8L,
use_transformer_blocks = FALSE)
#> Warning: Small validation set for 2 trait(s): Mass (n=4), Wing.Length (n=4). Calibrated gate and conformal scores will be noisy for these trait(s). 95% split-conformal coverage is NOT achievable for 2 trait(s) with fewer than 19 validation cells (Mass (n=4), Wing.Length (n=4)): the achievable ceiling is n_val / (n_val + 1), which only reaches 0.95 at n_val >= 19. See `?fit_pigauto` under 'Calibration at small n' for smoothing options.
#> Warning: Small validation set for 2 trait(s): Mass (n=4), Wing.Length (n=4). Calibrated gate and conformal scores will be noisy for these trait(s). 95% split-conformal coverage is NOT achievable for 2 trait(s) with fewer than 19 validation cells (Mass (n=4), Wing.Length (n=4)): the achievable ceiling is n_val / (n_val + 1), which only reaches 0.95 at n_val >= 19. See `?fit_pigauto` under 'Calibration at small n' for smoothing options.
#> Warning: Small validation set for 2 trait(s): Mass (n=4), Wing.Length (n=4). Calibrated gate and conformal scores will be noisy for these trait(s). 95% split-conformal coverage is NOT achievable for 2 trait(s) with fewer than 19 validation cells (Mass (n=4), Wing.Length (n=4)): the achievable ceiling is n_val / (n_val + 1), which only reaches 0.95 at n_val >= 19. See `?fit_pigauto` under 'Calibration at small n' for smoothing options.
print(cv)
#> 3-fold cross-validation (1 replicate)
#> ──────────────────────────────────────────────────
#> Mass [continuous]
#> rmse 0.9445 +/- 0.3935 (n=3)
#> pearson_r -0.2631 +/- 0.7741 (n=3)
#> coverage_95 0.8333 +/- 0.1443 (n=3)
#> Wing.Length [continuous]
#> rmse 1.1610 +/- 0.6086 (n=3)
#> pearson_r 0.0371 +/- 0.4233 (n=3)
#> coverage_95 0.8333 +/- 0.2887 (n=3)
#>
#> Conformal coverage (95% target):
#> Mass 0.833 +/- 0.289
#> Wing.Length 0.750 +/- 0.250
summary(cv)
#> 3-fold cross-validation (1 replicate)
#> ──────────────────────────────────────────────────────────────────────
#> Trait Type RMSE (mean+/-sd) r (mean+/-sd) Acc (mean+/-sd)
#> ─────────────────────────────────────────────────────────────────────────────────────
#> Mass continuous 0.945 +/- 0.393 -0.263 +/- 0.774 ─
#> Wing.Length continuous 1.161 +/- 0.609 0.037 +/- 0.423 ─
# Whole-species validation, with the full tree retained:
cv_species <- cross_validate(preprocess_traits(traits, tree), tree,
k = 3L, seeds = 1L, holdout = "species",
gnn = FALSE, verbose = FALSE)
#> Warning: Small validation set for 2 trait(s): Mass (n=4), Wing.Length (n=4). Calibrated gate and conformal scores will be noisy for these trait(s). 95% split-conformal coverage is NOT achievable for 2 trait(s) with fewer than 19 validation cells (Mass (n=4), Wing.Length (n=4)): the achievable ceiling is n_val / (n_val + 1), which only reaches 0.95 at n_val >= 19. See `?fit_pigauto` under 'Calibration at small n' for smoothing options.
#> Warning: Small validation set for 2 trait(s): Mass (n=4), Wing.Length (n=4). Calibrated gate and conformal scores will be noisy for these trait(s). 95% split-conformal coverage is NOT achievable for 2 trait(s) with fewer than 19 validation cells (Mass (n=4), Wing.Length (n=4)): the achievable ceiling is n_val / (n_val + 1), which only reaches 0.95 at n_val >= 19. See `?fit_pigauto` under 'Calibration at small n' for smoothing options.
#> Warning: Small validation set for 2 trait(s): Mass (n=4), Wing.Length (n=4). Calibrated gate and conformal scores will be noisy for these trait(s). 95% split-conformal coverage is NOT achievable for 2 trait(s) with fewer than 19 validation cells (Mass (n=4), Wing.Length (n=4)): the achievable ceiling is n_val / (n_val + 1), which only reaches 0.95 at n_val >= 19. See `?fit_pigauto` under 'Calibration at small n' for smoothing options.
cv_species$fold_assignments
#> rep fold species
#> 1 1 1 Eudromia_formosa
#> 2 1 1 Rhea_americana
#> 3 1 1 Ptilopachus_petrosus
#> 4 1 1 Megapodius_nicobariensis
#> 5 1 2 Bambusicola_fytchii
#> 6 1 2 Francolinus_psilolaemus
#> 7 1 2 Dendragapus_obscurus
#> 8 1 2 Tragopan_temminckii
#> 9 1 3 Nothoprocta_pentlandii
#> 10 1 3 Meleagris_ocellata
#> 11 1 3 Tympanuchus_phasianellus
#> 12 1 3 Arborophila_campbelli
# }