Impute missing traits using a fitted pigauto model
Source:R/predict_pigauto.R
predict.pigauto_fit.RdReturns imputed trait values on the original scale for data used to fit the model. Supports continuous, binary, categorical, ordinal, count, proportion, zero-inflated count, and multi-proportion traits.
Multiple draws are diagnostic only. With n_imputations > 1,
returned draws are not supported for downstream inference. For
multi-proportion traits, pooled point values and simplex probabilities are
omitted; use a single-run prediction when you need the simplex output.
Usage
# S3 method for class 'pigauto_fit'
predict(
object,
newdata = NULL,
return_se = TRUE,
n_imputations = 1L,
baseline_override = NULL,
pool_method = c("median", "mean", "mode"),
clamp_outliers = FALSE,
clamp_factor = 5,
match_observed = c("none", "pmm"),
pmm_K = 5L,
evaluation = FALSE,
...
)
# S3 method for class 'pigauto_pred'
print(x, ...)Arguments
- object
object of class
"pigauto_fit".- newdata
Must be
NULL: predictions use the fitted data. Any non-NULL value is rejected; new-species prediction is not implemented.- return_se
logical. Return type-dependent uncertainty summaries? Default
TRUE; see thesereturn description below.- n_imputations
integer. Number of stochastic prediction-diagnostic draws (default
1L). With multiple draws, independent Gaussian perturbations are added per latent cell using the selected baseline SD; cells with zero selected baseline SD receive no perturbation. GNN dropout also varies across runs when the GNN is enabled. These are not joint phylogenetic BM posterior draws or validated analysis-aware imputations.- baseline_override
optional
list(mu, se)with the same shape asobject$baseline. When supplied, predictions use this baseline instead of the one saved in the fit. Used internally bymulti_impute_trees()to reuse a trained GNN across posterior trees. Most users can ignore this. DefaultNULL(use the fit's own baseline).- pool_method
character. One of
"median"(default),"mean", or"mode". Controls pooling whenn_imputations > 1; see "Pooling decoded draws" below. Some types use the same rule for every setting.- clamp_outliers
logical. When
TRUE, cap back-transformed predictions for log-transformed continuous, count, and zi_count magnitude traits attm$obs_max * clamp_factor, wheretm$obs_maxis the observed maximum on the original scale, recorded during preprocessing. This optional cap limits extrapolation afterexp()orexpm1(); it is not an uncertainty bound. DefaultFALSEleaves these predictions uncapped.- clamp_factor
numeric scalar (>= 1). User-selected multiplier of the observed maximum for the upper cap when clamping is enabled. Default
5. Ignored whenclamp_outliers = FALSE.- match_observed
character, one of
c("none", "pmm"). When"pmm", uses predictive Mean Matching (Little 1988; see References) on the at-risk types (log-transformed continuous, count, zi_count magnitude, proportion). For each missing cell, finds thepmm_Kobserved cells whose own predictions are closest to the missing cell's prediction, samples one, and returns its observed value. Imputed values are guaranteed to lie in the observed data range – no extrapolation is possible by construction.When to use: PMM is a niche feature in pigauto. The package already provides conformal prediction intervals (calibrated against held-out residuals) and stochastic conformal-width and mc_dropout prediction-diagnostic draws. PMM is only worth enabling for: (a) methodological comparison against mice / equivalent packages, or (b) workflows that specifically require imputed values to come from the observed data pool. For tail safety on single-imputation point estimates, prefer
clamp_outliers = TRUE. PMM failed the downstream fixed-effect redesign pilot and is unsupported for inference.Donor matching can reduce extrapolation but also replaces accurate predictions with donor values, so it is not a general RMSE improvement.
Discrete-class types (binary / categorical / ordinal / multi_proportion) and un-log continuous: no-op. Default
"none"leaves predictions unmatched.- pmm_K
integer scalar (>= 1). Donor pool size for PMM. Default
5L(mice convention). Ignored whenmatch_observed = "none".- evaluation
Logical. Use
TRUEto predict validation and test cells using the fit's held-out baseline and masked input context. Requires stored splits andnewdata = NULL. DefaultFALSEuses all observed data for production imputation. Useevaluation = TRUEbefore passing predictions toevaluate_imputation.- ...
Internal compatibility arguments. Unrecognised arguments are an error, including misspelled argument names.
- x
A pigauto prediction object.
Value
A list of class "pigauto_pred" with:
- imputed
data.frame of model predictions for every fitted row (observed and missing cells), on the original scale with proper R types (numeric, integer, factor, ordered). Observed values are not retained in this component. Use
impute()and its$completedcomponent for a data.frame that preserves observed values and fills missing cells when predictions are available.- imputed_latent
Numeric matrix (n x p_latent) of predictions in latent scale.
- se
Numeric matrix (n x n_original_traits) of per-cell uncertainty. See the "What
$semeans per trait type" section below. Single-run and multi-draw constructions differ. These are diagnostic summaries, not calibrated total predictive SDs or valid inputs to downstream pooling.NULLifreturn_se = FALSE.- se_latent
Numeric matrix on the latent scale, with one column per latent trait column. Returned when
return_se = TRUEfor current trait-map fits; otherwiseNULL. Binary and categorical entries areNA; forzi_count, only the magnitude column has a summary.- probabilities
Named list. Binary traits: numeric probability vector; categorical traits: n x K class-probability matrix; zi_count traits: probability of nonzero. A single-run multi_proportion prediction includes its n x K simplex matrix. Compositional entries are absent from pooled multi-draw output.
- conformal_lower, conformal_upper
Matrices of lower and upper conformal limits when the fitted model has conformal scores. Values are on the original trait scale except for ordinal traits, whose bounds are rounded zero-based indices. The lower limit is floored at 0 and the upper limit capped at K - 1; these one-sided limits do not guarantee both endpoints are in range or that the interval is non-empty. Check bounds before mapping an index to a label using the corresponding ordered factor levels (index + 1 in R). Entries can be
NAfor unsupported traits or unavailable scores; both areNULLwhen scores are absent.- conformal_scores
Named numeric vector of the scores stored in the fitted model, or
NULLwhen the fit has no scores.- imputed_datasets
List of M stochastic prediction data.frames when
n_imputations > 1;NULLotherwise. Not supported for downstream inference.- trait_map
Trait map from the fitted model.
- species_names
Character vector.
- trait_names
Character vector.
- n_imputations
Integer, number of imputations performed.
- evaluation_safe_idx
Latent indices excluded from fitting and prediction input, used to check held-out scoring.
The print method returns x invisibly.
Details
With n_imputations > 1, predictions include stochastic diagnostic
draws, not joint phylogenetic BM posterior draws. On each run, independent
Gaussian perturbations are added per latent cell to the selected baseline
component, scaled by that component's standard deviation (the second
parameter in N(mu, sd) is an SD, not a variance). A route that
supplies zero uncertainty placeholders adds no Gaussian perturbation for
those cells; a zero placeholder does not indicate certainty. When the GNN
is enabled, dropout also varies across runs. Because predictions use the
phylogenetic graph, predictions across species need not be independent.
The calibrated latent prediction combines the selected baseline, GNN
prediction and optional mean baseline using their calibrated weights;
an applicable direct covariate contribution is then added. In stochastic
runs the selected baseline term uses its perturbed value.
A zero GNN weight removes only the neural contribution. The output equals
the baseline draw only when the baseline weight is one and every other
contribution is absent. Point predictions and these stochastic diagnostic
draws are distinct outputs.
Point estimates follow the type-specific "Pooling decoded draws" table
below. The M complete datasets are returned in
imputed_datasets for prediction diagnostics. These datasets are not
supported for downstream inference. Use multi_impute_analysis() for
the narrow analysis-aware backend after consulting its supported-model and
lifecycle documentation.
gnn = FALSE fits (traditional-stats mode): when the fit was
produced with gnn = FALSE, no GNN was ever trained, so this
function makes no torch:: calls at all – prediction is the
calibrated baseline blend r_cal_bm * baseline + r_cal_mean *
mean_baseline_per_col (r_cal_gnn is always zero) computed on
plain R matrices. Two baselines are available: in production mode (the
default, evaluation = FALSE) predictions use baseline_full
– fit with splits = NULL, so every observed cell can inform every
other cell; in evaluation mode (evaluation = TRUE) predictions use the held-out
baseline instead, so scored cells never see their own value.
n_imputations > 1 adds independent Gaussian perturbations per
latent cell, scaled by the selected baseline standard deviation. Cells
with zero selected baseline SD receive no Gaussian perturbation. These
are stochastic prediction diagnostics, not joint phylogenetic BM posterior
draws. The gnn = FALSE path has no GNN-dropout contribution.
Decoding per type:
- continuous
reverse z-score, then
exp()if log-transformed- binary
sigmoid(latent)to probability, round to 0/1- count
reverse z-score of log1p,
expm1(), round, clip >= 0- ordinal
reverse z-score, round to nearest valid integer level
- categorical
softmax()over K latent columns, argmax
Conformal intervals: conformal_lower / conformal_upper
use whatever conformal_method the fit was trained with
(fit_pigauto). For "mondrian", the half-width is not
constant across rows: each target cell's phylogenetic locality (mean
cophenetic distance to its 5 nearest training-observed species for that
trait) picks the near- or far-stratum score stored on the fit.
These are stratum-specific intervals; the far-stratum score is not
necessarily larger than the near-stratum score. See
fit_pigauto's conformal_method argument for details.
zi_count conformal intervals are conditional on non-zero.
conformal_lower[, nm] / conformal_upper[, nm] for a
zi_count trait back-transform the magnitude latent (log1p-z of the
count given it is non-zero) plus/minus the conformal half-width
that was calibrated on that same magnitude column
(fit_pigauto's compute_conformal_scores() scores
zi_count on latent_cols[2], never on the decoded E[X]).
They are not an interval for the expected value
E[X] = P(nonzero) * E[count | nonzero] – constructing a naive
\(\pm\)quantile interval directly on E[X] would conflate gate
and magnitude uncertainty into one number with no valid coverage
interpretation. The gate probability P(nonzero) is available
separately in pred$probabilities[[nm]]; combine it with the
conditional interval yourself if you need an E[X]-scale bound.
Pooling decoded draws
These rules apply to n_imputations > 1, on the decoded scale.
| Type | median | mean | mode |
|---|---|---|---|
| continuous, unlogged | mean | mean | mean |
| continuous, logged | median | median | median |
| count / proportion / zi_count | median | mean | median |
| ordinal | rounded mean class index | rounded mean class index | modal class |
| binary | mean probability, then threshold 0.5 | same | same |
| categorical | mean class-probability vector, then argmax | same | same |
| multi_proportion | pooled output unavailable | unavailable | unavailable |
imputed_datasets. Do not interpret this omission as simplex averaging.
What $se means per trait type
For fits with a trait_map, pred$se is an
n x n_original_traits matrix of type-dependent diagnostic summaries.
Its construction also depends on n_imputations. It is not a
uniformly Gaussian standard error, a calibrated total predictive SD, or
an input to Rubin pooling.
| Type | Single run | More than one draw |
|---|---|---|
| continuous / count / ordinal / proportion | Selected-baseline latent uncertainty scaled to the original scale, using delta-method back-transformation where applicable. | Square root of transformed baseline variance plus the between-decoded-draw variance. Logged continuous, count, and proportion transformations use the first latent run as their anchor. |
| binary | min(p, 1-p) | The same score using the probability averaged across draws. |
| categorical | 1 - max(p_k) | The same score using the class-probability vector averaged across draws. |
| zi_count | A delta-method product summary for P(nonzero) * E[count | nonzero], combining gate and magnitude terms. | Between-draw SD of decoded expected counts, without an additional baseline-variance term. |
| multi_proportion | NA | NA |
For a BM Gaussian baseline, the single-run continuous-family source is a
model-dependent BM/joint conditional SD. It is not total final
blended-prediction uncertainty: it describes uncertainty under the
selected baseline, not prediction error in the final baseline–GNN blend.
Other route summaries do not inherit that interpretation automatically.
Discrete scores measure distance from a
certain class prediction, not a Gaussian SE. For compositions, inspect
the separate CLR latent summary in se_latent. The multi-draw
construction can contain baseline-draw variation with gnn = FALSE;
it should not generally be described as an MC-dropout SE.
Older saved fits without a trait_map use a separate continuous-only
legacy prediction path and are not described by this table. For supported
downstream multiple-imputation routes, consult
multi_impute_analysis or multi_impute with
draws_method = "posterior", then use
with_imputations and pool_mi under their
documented eligibility conditions.
Printing
Prints dimensions, trait types, draw count and available probability/bounds components. Component presence does not establish interval coverage. The ... argument is currently unused.
References
Little RJA (1988). "Missing-Data Adjustments in Large Surveys." Journal of Business & Economic Statistics 6(3): 287-296. doi:10.1080/07350015.1988.10509663 . This is a source for predictive mean matching, not validation of pigauto's implementation.
van Buuren S, Groothuis-Oudshoorn K (2011). "mice: Multivariate Imputation by Chained Equations in R." Journal of Statistical Software 45(3), 1-67. doi:10.18637/jss.v045.i03 . The official mice PMM documentation describes that software's method; it does not validate pigauto's PMM.
Examples
# \donttest{
data(avonet300, tree300)
tree <- ape::keep.tip(tree300, tree300$tip.label[seq_len(30L)])
traits <- avonet300[match(tree$tip.label, avonet300$Species_Key),
c("Mass", "Wing.Length"), drop = FALSE]
rownames(traits) <- tree$tip.label
data <- preprocess_traits(traits, tree)
splits <- make_missing_splits(data$X_scaled, trait_map = data$trait_map)
fit <- fit_pigauto(data, tree, splits = splits, epochs = 5L,
verbose = FALSE)
#> Warning: Small validation set for 1 trait(s): Wing.Length (n=3). Calibrated gate and conformal scores will be noisy for these trait(s). 95% split-conformal coverage is NOT achievable for 1 trait(s) with fewer than 19 validation cells (Wing.Length (n=3)): the achievable ceiling is n_val / (n_val + 1), which only reaches 0.95 at n_val >= 19. See `?fit_pigauto` under 'Calibration at small n' for smoothing options.
pred <- predict(fit, return_se = TRUE)
pred$imputed
#> Mass Wing.Length
#> Nothoprocta_pentlandii 300.7 148.2
#> Eudromia_formosa 772.9 223.9
#> Rhea_americana 23000.0 604.5
#> Ptilopachus_petrosus 193.0 117.2
#> Bambusicola_fytchii 313.7 142.8
#> Francolinus_psilolaemus 478.3 167.3
#> Meleagris_ocellata 5525.0 362.5
#> Dendragapus_obscurus 1047.3 226.5
#> Tympanuchus_phasianellus 882.4 200.6
#> Tragopan_temminckii 1163.4 236.8
#> Arborophila_campbelli 336.0 129.5
#> Megapodius_nicobariensis 836.7 226.8
#> Dendrocygna_guttata 800.0 206.6
#> Chloephaga_rubidiceps 2086.0 325.6
#> Somateria_spectabilis 1616.7 266.3
#> Rhynochetos_jubatus 860.0 264.3
#> Caprimulgus_longirostris 45.6 151.6
#> Caprimulgus_saturatus 52.0 144.5
#> Aegotheles_cristatus 42.8 135.8
#> Lampornis_clemenciae 7.6 73.7
#> Lampornis_calolaemus 5.1 62.6
#> Stellula_calliope 2.6 41.4
#> Hylocharis_sapphirina 4.4 49.5
#> Hylocharis_chrysura 4.5 51.8
#> Amazilia_cyanifrons 5.0 55.6
#> Campylopterus_phainopeplus 7.7 68.1
#> Metallura_iracunda 3.9 58.1
#> Anthracothorax_viridis 6.6 62.8
#> Polytmus_theresiae 3.8 56.6
#> Phaethornis_striigularis 3.0 39.1
pred$se
#> Mass Wing.Length
#> Nothoprocta_pentlandii 0 0
#> Eudromia_formosa 0 0
#> Rhea_americana 0 0
#> Ptilopachus_petrosus 0 0
#> Bambusicola_fytchii 0 0
#> Francolinus_psilolaemus 0 0
#> Meleagris_ocellata 0 0
#> Dendragapus_obscurus 0 0
#> Tympanuchus_phasianellus 0 0
#> Tragopan_temminckii 0 0
#> Arborophila_campbelli 0 0
#> Megapodius_nicobariensis 0 0
#> Dendrocygna_guttata 0 0
#> Chloephaga_rubidiceps 0 0
#> Somateria_spectabilis 0 0
#> Rhynochetos_jubatus 0 0
#> Caprimulgus_longirostris 0 0
#> Caprimulgus_saturatus 0 0
#> Aegotheles_cristatus 0 0
#> Lampornis_clemenciae 0 0
#> Lampornis_calolaemus 0 0
#> Stellula_calliope 0 0
#> Hylocharis_sapphirina 0 0
#> Hylocharis_chrysura 0 0
#> Amazilia_cyanifrons 0 0
#> Campylopterus_phainopeplus 0 0
#> Metallura_iracunda 0 0
#> Anthracothorax_viridis 0 0
#> Polytmus_theresiae 0 0
#> Phaethornis_striigularis 0 0
pred$probabilities
#> list()
pred2 <- predict(fit, n_imputations = 2L)
pred2$imputed_datasets
#> [[1]]
#> Mass Wing.Length
#> Nothoprocta_pentlandii 300.7 148.2
#> Eudromia_formosa 772.9 223.9
#> Rhea_americana 23000.0 604.5
#> Ptilopachus_petrosus 193.0 117.2
#> Bambusicola_fytchii 313.7 142.8
#> Francolinus_psilolaemus 478.3 167.3
#> Meleagris_ocellata 5525.0 362.5
#> Dendragapus_obscurus 1047.3 226.5
#> Tympanuchus_phasianellus 882.4 200.6
#> Tragopan_temminckii 1163.4 236.8
#> Arborophila_campbelli 336.0 129.5
#> Megapodius_nicobariensis 836.7 226.8
#> Dendrocygna_guttata 800.0 206.6
#> Chloephaga_rubidiceps 2086.0 325.6
#> Somateria_spectabilis 1616.7 266.3
#> Rhynochetos_jubatus 860.0 264.3
#> Caprimulgus_longirostris 45.6 151.6
#> Caprimulgus_saturatus 52.0 144.5
#> Aegotheles_cristatus 42.8 135.8
#> Lampornis_clemenciae 7.6 73.7
#> Lampornis_calolaemus 5.1 62.6
#> Stellula_calliope 2.6 41.4
#> Hylocharis_sapphirina 4.4 49.5
#> Hylocharis_chrysura 4.5 51.8
#> Amazilia_cyanifrons 5.0 55.6
#> Campylopterus_phainopeplus 7.7 68.1
#> Metallura_iracunda 3.9 58.1
#> Anthracothorax_viridis 6.6 62.8
#> Polytmus_theresiae 3.8 56.6
#> Phaethornis_striigularis 3.0 39.1
#>
#> [[2]]
#> Mass Wing.Length
#> Nothoprocta_pentlandii 300.7 148.2
#> Eudromia_formosa 772.9 223.9
#> Rhea_americana 23000.0 604.5
#> Ptilopachus_petrosus 193.0 117.2
#> Bambusicola_fytchii 313.7 142.8
#> Francolinus_psilolaemus 478.3 167.3
#> Meleagris_ocellata 5525.0 362.5
#> Dendragapus_obscurus 1047.3 226.5
#> Tympanuchus_phasianellus 882.4 200.6
#> Tragopan_temminckii 1163.4 236.8
#> Arborophila_campbelli 336.0 129.5
#> Megapodius_nicobariensis 836.7 226.8
#> Dendrocygna_guttata 800.0 206.6
#> Chloephaga_rubidiceps 2086.0 325.6
#> Somateria_spectabilis 1616.7 266.3
#> Rhynochetos_jubatus 860.0 264.3
#> Caprimulgus_longirostris 45.6 151.6
#> Caprimulgus_saturatus 52.0 144.5
#> Aegotheles_cristatus 42.8 135.8
#> Lampornis_clemenciae 7.6 73.7
#> Lampornis_calolaemus 5.1 62.6
#> Stellula_calliope 2.6 41.4
#> Hylocharis_sapphirina 4.4 49.5
#> Hylocharis_chrysura 4.5 51.8
#> Amazilia_cyanifrons 5.0 55.6
#> Campylopterus_phainopeplus 7.7 68.1
#> Metallura_iracunda 3.9 58.1
#> Anthracothorax_viridis 6.6 62.8
#> Polytmus_theresiae 3.8 56.6
#> Phaethornis_striigularis 3.0 39.1
#>
# }