pigauto benchmark: Covariate effectiveness simulation

Covariate effectiveness simulation

This benchmark simulates traits with varying levels of phylogenetic signal (λ) and environmental effect (β) to demonstrate when environmental covariates improve imputation accuracy. Covariates help most when phylogenetic signal is low and environmental effects are strong.

Setup

Tree: 300 species (tree300, BirdTree Hackett MCC)

Traits: 3 continuous (simulated per scenario)

Covariates: 3 (uncorrelated normal, fully observed)

Missingness: 20%, 40%, 60% MCAR

Replicates: 3 per cell

Epochs: 300

Wall time: 119.6 min

Key finding: Covariates reduce RMSE by up to 11% in the Low phylo, strong env (λ=0.1, β=1.5) scenario ( λ=0.1, β=1.5, 40% missing ). When phylogenetic signal is high and environmental effects are absent, covariates provide zero lift (ratio ≈ 1.0), confirming the gated safety.

RMSE by scenario, missingness, and method

Scenario	λ	β	Miss %	Method	RMSE	Pearson r
Low phylo, strong env (λ=0.1, β=1.5)	0.1	1.5	20%	pigauto	2.7136	0.4208
Low phylo, strong env (λ=0.1, β=1.5)	0.1	1.5	20%	pigauto + covs	2.5046	0.4964
Low phylo, strong env (λ=0.1, β=1.5)	0.1	1.5	40%	pigauto	2.8370	0.3365
Low phylo, strong env (λ=0.1, β=1.5)	0.1	1.5	40%	pigauto + covs	2.5128	0.4887
Low phylo, strong env (λ=0.1, β=1.5)	0.1	1.5	60%	pigauto	2.8860	0.3274
Low phylo, strong env (λ=0.1, β=1.5)	0.1	1.5	60%	pigauto + covs	2.5846	0.4782
Moderate phylo, strong env (λ=0.3, β=1.0)	0.3	1.0	20%	pigauto	1.5049	0.2497
Moderate phylo, strong env (λ=0.3, β=1.0)	0.3	1.0	20%	pigauto + covs	1.4635	0.3010
Moderate phylo, strong env (λ=0.3, β=1.0)	0.3	1.0	40%	pigauto	1.5298	0.2268
Moderate phylo, strong env (λ=0.3, β=1.0)	0.3	1.0	40%	pigauto + covs	1.4739	0.2782
Moderate phylo, strong env (λ=0.3, β=1.0)	0.3	1.0	60%	pigauto	1.6484	0.1464
Moderate phylo, strong env (λ=0.3, β=1.0)	0.3	1.0	60%	pigauto + covs	1.6004	0.2008
High phylo, moderate env (λ=0.7, β=0.5)	0.7	0.5	20%	pigauto	1.4887	0.2104
High phylo, moderate env (λ=0.7, β=0.5)	0.7	0.5	20%	pigauto + covs	1.4369	0.2773
High phylo, moderate env (λ=0.7, β=0.5)	0.7	0.5	40%	pigauto	1.4657	0.2189
High phylo, moderate env (λ=0.7, β=0.5)	0.7	0.5	40%	pigauto + covs	1.3757	0.2858
High phylo, moderate env (λ=0.7, β=0.5)	0.7	0.5	60%	pigauto	1.4837	0.1329
High phylo, moderate env (λ=0.7, β=0.5)	0.7	0.5	60%	pigauto + covs	1.4317	0.2012
High phylo, no env (λ=0.9, β=0)	0.9	0.0	20%	pigauto	0.5716	0.5804
High phylo, no env (λ=0.9, β=0)	0.9	0.0	20%	pigauto + covs	0.5695	0.5809
High phylo, no env (λ=0.9, β=0)	0.9	0.0	40%	pigauto	0.6105	0.6280
High phylo, no env (λ=0.9, β=0)	0.9	0.0	40%	pigauto + covs	0.6085	0.6284
High phylo, no env (λ=0.9, β=0)	0.9	0.0	60%	pigauto	0.6476	0.5886
High phylo, no env (λ=0.9, β=0)	0.9	0.0	60%	pigauto + covs	0.6459	0.5888

Scenario

Miss %

Method

RMSE

Pearson r

Low phylo, strong env (λ=0.1, β=1.5)

0.1

1.5

20%

pigauto

2.7136

0.4208

Low phylo, strong env (λ=0.1, β=1.5)

0.1

1.5

20%

pigauto + covs

2.5046

0.4964

Low phylo, strong env (λ=0.1, β=1.5)

0.1

1.5

40%

pigauto

2.8370

0.3365

Low phylo, strong env (λ=0.1, β=1.5)

0.1

1.5

40%

pigauto + covs

2.5128

0.4887

Low phylo, strong env (λ=0.1, β=1.5)

0.1

1.5

60%

pigauto

2.8860

0.3274

Low phylo, strong env (λ=0.1, β=1.5)

0.1

1.5

60%

pigauto + covs

2.5846

0.4782

Moderate phylo, strong env (λ=0.3, β=1.0)

0.3

1.0

20%

pigauto

1.5049

0.2497

Moderate phylo, strong env (λ=0.3, β=1.0)

0.3

1.0

20%

pigauto + covs

1.4635

0.3010

Moderate phylo, strong env (λ=0.3, β=1.0)

0.3

1.0

40%

pigauto

1.5298

0.2268

Moderate phylo, strong env (λ=0.3, β=1.0)

0.3

1.0

40%

pigauto + covs

1.4739

0.2782

Moderate phylo, strong env (λ=0.3, β=1.0)

0.3

1.0

60%

pigauto

1.6484

0.1464

Moderate phylo, strong env (λ=0.3, β=1.0)

0.3

1.0

60%

pigauto + covs

1.6004

0.2008

High phylo, moderate env (λ=0.7, β=0.5)

0.7

0.5

20%

pigauto

1.4887

0.2104

High phylo, moderate env (λ=0.7, β=0.5)

0.7

0.5

20%

pigauto + covs

1.4369

0.2773

High phylo, moderate env (λ=0.7, β=0.5)

0.7

0.5

40%

pigauto

1.4657

0.2189

High phylo, moderate env (λ=0.7, β=0.5)

0.7

0.5

40%

pigauto + covs

1.3757

0.2858

High phylo, moderate env (λ=0.7, β=0.5)

0.7

0.5

60%

pigauto

1.4837

0.1329

High phylo, moderate env (λ=0.7, β=0.5)

0.7

0.5

60%

pigauto + covs

1.4317

0.2012

High phylo, no env (λ=0.9, β=0)

0.9

0.0

20%

pigauto

0.5716

0.5804

High phylo, no env (λ=0.9, β=0)

0.9

0.0

20%

pigauto + covs

0.5695

0.5809

High phylo, no env (λ=0.9, β=0)

0.9

0.0

40%

pigauto

0.6105

0.6280

High phylo, no env (λ=0.9, β=0)

0.9

0.0

40%

pigauto + covs

0.6085

0.6284

High phylo, no env (λ=0.9, β=0)

0.9

0.0

60%

pigauto

0.6476

0.5886

High phylo, no env (λ=0.9, β=0)

0.9

0.0

60%

pigauto + covs

0.6459

0.5888

Covariate lift (RMSE ratio: with covariates / without)

Ratio < 1.0 means covariates improve imputation; ratio ≈ 1.0 means no effect.

Scenario	λ	β	Miss %	RMSE (no cov)	RMSE (cov)	Ratio	Improvement
Low phylo, strong env (λ=0.1, β=1.5)	0.1	1.5	20%	2.7136	2.5046	0.923	7.7%
Low phylo, strong env (λ=0.1, β=1.5)	0.1	1.5	40%	2.8370	2.5128	0.886	11.4%
Low phylo, strong env (λ=0.1, β=1.5)	0.1	1.5	60%	2.8860	2.5846	0.896	10.4%
Moderate phylo, strong env (λ=0.3, β=1.0)	0.3	1.0	20%	1.5049	1.4635	0.972	2.8%
Moderate phylo, strong env (λ=0.3, β=1.0)	0.3	1.0	40%	1.5298	1.4739	0.963	3.7%
Moderate phylo, strong env (λ=0.3, β=1.0)	0.3	1.0	60%	1.6484	1.6004	0.971	2.9%
High phylo, moderate env (λ=0.7, β=0.5)	0.7	0.5	20%	1.4887	1.4369	0.965	3.5%
High phylo, moderate env (λ=0.7, β=0.5)	0.7	0.5	40%	1.4657	1.3757	0.939	6.1%
High phylo, moderate env (λ=0.7, β=0.5)	0.7	0.5	60%	1.4837	1.4317	0.965	3.5%
High phylo, no env (λ=0.9, β=0)	0.9	0.0	20%	0.5716	0.5695	0.996	0.4%
High phylo, no env (λ=0.9, β=0)	0.9	0.0	40%	0.6105	0.6085	0.997	0.3%
High phylo, no env (λ=0.9, β=0)	0.9	0.0	60%	0.6476	0.6459	0.997	0.3%

Scenario

Miss %

RMSE (no cov)

RMSE (cov)

Ratio

Improvement

Low phylo, strong env (λ=0.1, β=1.5)

0.1

1.5

20%

2.7136

2.5046

0.923

7.7%

Low phylo, strong env (λ=0.1, β=1.5)

0.1

1.5

40%

2.8370

2.5128

0.886

11.4%

Low phylo, strong env (λ=0.1, β=1.5)

0.1

1.5

60%

2.8860

2.5846

0.896

10.4%

Moderate phylo, strong env (λ=0.3, β=1.0)

0.3

1.0

20%

1.5049

1.4635

0.972

2.8%

Moderate phylo, strong env (λ=0.3, β=1.0)

0.3

1.0

40%

1.5298

1.4739

0.963

3.7%

Moderate phylo, strong env (λ=0.3, β=1.0)

0.3

1.0

60%

1.6484

1.6004

0.971

2.9%

High phylo, moderate env (λ=0.7, β=0.5)

0.7

0.5

20%

1.4887

1.4369

0.965

3.5%

High phylo, moderate env (λ=0.7, β=0.5)

0.7

0.5

40%

1.4657

1.3757

0.939

6.1%

High phylo, moderate env (λ=0.7, β=0.5)

0.7

0.5

60%

1.4837

1.4317

0.965

3.5%

High phylo, no env (λ=0.9, β=0)

0.9

0.0

20%

0.5716

0.5695

0.996

0.4%

High phylo, no env (λ=0.9, β=0)

0.9

0.0

40%

0.6105

0.6085

0.997

0.3%

High phylo, no env (λ=0.9, β=0)

0.9

0.0

60%

0.6476

0.6459

0.997

0.3%

Interpretation

Low phylogenetic signal + strong environmental effects (λ=0.1, β=1.5): Covariates provide substantial RMSE reduction. The GNN learns to use environmental information to predict trait values that phylogeny alone cannot explain.

High phylogenetic signal, no environmental effects (λ=0.9, β=0): Covariates provide ~zero improvement. In this simulation the gated fit does not gain meaningful accuracy from uninformative covariates; compare the exact RMSE ratio in the table. This is consistent with the Delhey 2019 real-data result.

Crossover: The benefit appears when the environmental component is strong relative to phylogenetic signal. Moderate phylo + strong env (λ=0.3, β=1.0) already shows clear improvement.

Generated 2026-05-11 10:58 by script/make_bench_covariate_sim_html.R