Multi-observation imputation benchmark

pigauto v0.6.0 — observation-level covariates · 200 species · 5 obs/species · 2 replicates · 44.0 min compute · Generated 2026-05-11 10:58

1. The problem

Comparative datasets increasingly contain multiple data points per species measured under different experimental or environmental conditions: critical thermal maximum (CT_max) at different acclimation temperatures, metabolic rate at different body temperatures, performance at different substrate concentrations. Missing data is ubiquitous in these datasets, and different species may be missing measurements at different condition levels.

The challenge: can imputation methods use observation-level covariates (the experimental condition under which each measurement was taken) to produce covariate-conditional predictions? A species measured at 20°C acclimation should receive a different CT_max imputation than the same species measured at 30°C. Standard phylogenetic imputation methods that operate at the species level cannot make this distinction.

2. Simulation design

The data-generating process simulates a thermal physiology scenario:

CTmax_ij = mu_i + beta * acclim_temp_j + epsilon_ij

where:
  mu_i        ~ phylogenetic BM (species-level intercept)
  acclim_temp ~ experimental condition (observation-level covariate)
  beta        = within-species slope (swept: 0, 0.5, 1.0, 1.5)
  epsilon_ij  ~ N(0, sigma^2) residual noise

Each species has multiple observations at different acclimation temperatures. The parameter beta controls the strength of the within-species covariate effect. When beta = 0, there is no within-species variation driven by the covariate; when beta > 0, the covariate contains information that should improve imputation.

Methods compared

3. Results

Observation-level RMSE

Average RMSE on held-out cells across replicates. Bold green marks the best method per scenario.

Pearson r (observed vs predicted)

Higher is better. Bold green marks the best method per scenario.

Covariate lift

RMSE ratio of pigauto + covariates relative to pigauto (no covariates). Values < 1 indicate observation-level covariates help.

4. Key findings

• Species-level phylogenetic imputation is necessary but not sufficient. The BM baseline captures inter-species variation driven by shared evolutionary history. But when within-species variation is structured by an experimental condition, species-level imputation leaves that structure unexploited.
• Observation-level covariates fill the gap. By conditioning on the experimental condition (acclimation temperature in this simulation), pigauto can produce different imputed values for different observations of the same species, matching the true data-generating process.
• Safety when covariates are uninformative. The gated architecture can fall back to the phylogenetic baseline when the covariate has no predictive value in this simulation (beta = 0). Treat covariates as a validation-checked addition, not an automatic improvement.

5. Architecture: observation-level refinement

Standard phylogenetic imputation operates at the species level: one prediction per species per trait. To handle multiple observations per species, pigauto uses a two-stage architecture:

1. Species-level message passing. The GNN aggregates observations within each species (scatter_mean), performs phylogenetic message passing on the species graph, then broadcasts the species-level representation back to observations (index_select). This captures inter-species structure from the phylogenetic tree.
2. Observation-level refinement MLP. A small feedforward network takes the species-level representation and concatenates it with the observation-level covariates. This allows the model to learn within-species adjustments: the same species gets different predictions at different covariate values.

# Schematic of the multi-obs + covariate pipeline
#
#   observations          species level          observations
#   (n_obs x p)           (n_species x d)        (n_obs x p)
#       |                      |                      |
#   scatter_mean  --->  GNN message passing  --->  broadcast
#                                                     |
#                                              concat obs covariates
#                                                     |
#                                              refinement MLP
#                                                     |
#                                                  delta_obs

The final prediction is still the gated blend:

pred = (1 - r_cal) * baseline + r_cal * delta_obs

where baseline is the phylogenetic BM prediction (species-level, broadcast to observations) and delta_obs is the observation-level GNN output that incorporates both phylogenetic structure and covariate information.

6. Reproducibility

Driver: script/bench_multi_obs.R. Tree: 200 species. Observations per species: 5. Training: 200 epochs. To reproduce:

Rscript script/bench_multi_obs.R
Rscript script/make_bench_multi_obs_html.R

References

• Goolsby, E.W., Bruggeman, J. & Ané, C. (2017). Rphylopars: fast multivariate phylogenetic comparative methods for missing data and within-species variation. Methods in Ecology and Evolution, 8, 22–27.
• Rubin, D.B. (1987). Multiple Imputation for Nonresponse in Surveys. Wiley.
• Nakagawa, S. & Freckleton, R.P. (2008). Missing inaction: the dangers of ignoring missing data. Trends in Ecology & Evolution, 23, 592–596.

Generated 2026-05-11 10:58 by script/make_bench_multi_obs_html.R from pre-computed results in script/bench_multi_obs.rds.