Changelog
Unreleased
Authorship: added co-authors Yefeng Yang (ORCID 0000-0002-8610-4016) and Szymon Drobniak (ORCID 0000-0001-8101-6247) in
Project.toml,CITATION.cff, and Documenterauthors; Shinichi Nakagawa remains primary contact. No version or capability change.Identifiability warning #416: the fit-free rank pre-check that
fit_two_effect_remlandfit_multi_effect_remlalready ran now also runs infit_sparse_multi_effect_aireml,fit_multi_effect_mc_reml,fit_repeatability_reml(one record per animal makessigma_pe2andsigma_e2inseparable) andfit_direct_maternal_reml. The numerical rank check warns of possible weak identification. Nonzero covariance columns are now normalized before SVD, so the unchanged1e-8relative tolerance applies to their directions rather than their raw scales; this changes numerical warning decisions in the existing dense routes as well. Rescaling invariance assumes the computed columns remain finite and nonzero; overflow or underflow during covariance construction can lose information, and nonfinite columns skip the check. No warning is not evidence of identification or practical precision. A warning does not prove structural non-identifiability or identify which components are confounded. The check only logs: estimates,convergedand result fields are unchanged. The sparse and matrix-free routes skip it above 1e7 stacked cells.Genetic GLLVM (#50, owner decision 25 Oct 2026):
fit_gllvm_laplace_reml, theGeneticGLLVMFitfit object and a newgenetic_gllvm_result_payload(fit)are now exported so the R lane can bridge them. Both are labelled partial and experimental (the payload carriesstatus = "partial",experimental = true,scale = "latent"). The payload reports only rotation-invariant functionals ofG_lat(never raw loadings) and no heritability.GeneticGLLVMFitgains a trailingfamiliesfield (per-trait response family tokens). No capability status, row count or version changes.Changed #413: the dense
fit_two_effect_remlandfit_repeatability_remlfitters (andtwo_effect_ratio_interval,repeatability_interval, which call them) default to scale-aware starting variances set from the response variance:sigma_e2 = var(y)/2and the two random-effect variancesvar(y)/4each, the same rule asinitial = :autoon the sparse multi-effect fitter. The default start was(1, 1, 1), which is blind to the unit ofy: on a response with variance near 1e7 (milk yield in kg x 1000) NelderMead stopped at a wrong near-zero-heritability point withconverged = true, whiley / 1000gave the correct REML answer. The optimum now no longer depends on the unit of the response. A constant response falls back to(1, 1, 1). An explicitinitialis used as given, and passing(1, 1, 1)explicitly withvar(y)far from 1 still emits the unit-scale start warning (#447). The default start changes the NelderMead path, so estimates on unit-variance data can differ from earlier releases in the last digits and the reported optimum can differ where the old start stopped early; thesim/recovery harnesses pass the unit start explicitly and are unchanged.convergedstill means only that the simplex tolerance was met, not that the REML score is small (superseded by #424 below).Changed #424: the eleven NelderMead fitters (
fit_variance_components,fit_sparse_reml,fit_two_effect_reml,fit_multi_effect_reml,fit_direct_maternal_reml,fit_repeatability_reml,fit_random_regression_reml,fit_multivariate_reml,fit_multivariate_repeatability_reml,fit_gllvm_laplace_reml, and the NelderMead branches offit_laplace_reml) no longer takeconvergedfromOptim.convergedalone. One shared helper insrc/likelihood.jlalso requires, at the returned optimum, a numerically zero gradient (Newton decrement at most 1e-3 log-likelihood units) and curvature in every direction (smallest eigenvalue of the finite-difference Hessian in correlation form above 1e-3). Coordinates already at a boundary (a variance share at or below 1e-6, a covariance that lost rank or a variance in it, an FA uniqueness at its floor, a #327 safety rail) are left out of the check, and the rotation directions of free loadings are set aside. A flat REML ridge, such as Va and Ve with one record per unrelated animal, or a rank-1 factor-analytic G_lat for two traits infit_gllvm_laplace_reml, now reportsconverged = false, and so does a simplex that stopped short of the optimum. The check costs2d^2 + 4dextra objective evaluations fordchecked parameters. Point estimates of a single fit are unchanged; the multivariate FA two-start search now prefers a start that passes the check, and bootstrap refits that fail it count as not converged.
None of the entries below change any capability status, row count, or package version. public_covered_count stays 7, and no entry below touches Project.toml. The package version is 0.9.0; its released section is recorded below.
Part of #408: new
variance_component_uncertainty(fit; level = 0.95)returns the variance-component covariance, its standard errors, theh²standard error and the logit-deltah²interval of a converged REMLAnimalModelFitfrom one average-information matrix. Every field is bit-identical to the standalone call, which each build and factorise their own matrix, so the bundle saves two of three factorisations. The R bridge does not call it yet. No estimand, interval or coverage claim changes.#411 and #409 (no result change; outputs are bitwise identical to before on the test fixtures). The refitting forms of
repeatability_interval,two_effect_ratio_interval,direct_maternal_intervalandmulti_effect_ratio_intervalnow reuse the denseA = inv(Ainv)(and denseX,Z,y) that their fit already built, instead of inverting eachAinva second time for the finite-difference information. The profile intervals (heritability_interval(fit; method = :profile)andvariance_component_interval) keep one mixed-model-equation workspace per interval: the cross products and the sparsity pattern are formed once, each evaluation refills the coefficient matrix in place and refactors it with the cached symbolic analysis (cholesky!), where before every evaluation re-formed the cross products, concatenated the blocks and reordered. The number of evaluations is unchanged.Part of #411:
repeatability_interval,two_effect_ratio_intervalanddirect_maternal_intervaleach gain a method that takes an already-fitted result (repeatability_interval(fit, y, X, Z, Ainv; ...), and likewise for the other two) and computes only the finite-difference information atfit.variance_components, so a caller that has just fitted no longer repeats the whole dense Nelder-Mead fit inside the interval. The existing refitting methods now call the same code and return identical values.repeatability_intervalalso acceptsmax_dense_cellsand forwards it to its fit; before, a cap raised forfit_repeatability_remlwas ignored by the interval, which refused any model over the default 1,000,000 cells. Still open: the R bridge calls these functions with the refitting form, the sparsemulti_effect_*route is not yet used for the repeatability and direct-maternal bridge targets, and each interval still forms one denseinv(Ainv).reliability's denominatordiag(inv(Ainv))comes from1 .+ Fwhen Julia buildsAinvfrom pedigree rows:pedigree_inversealready computes the inbreeding coefficientsF(Meuwissen & Luo) for Henderson's rules, so the spec now carries them (animal_model_spec(...; relationship_diag)) andmethod = :autoreads them instead of a selected inverse ofAinv. The payload-v2 route carries this diagonal when it builds a pedigree block in Julia; callers that pass onlyAinv, including the current R formula bridge, still obtain it through the selected inverse. Values are unchanged to 1e-10; explicit:selinvand:densekeep their own paths. Szymon Drobniak.Fixed #350:
prediction_error_varianceandreliabilitydefault tomethod = :auto, which takes the sparse Takahashi selected-inverse path for a sparseAinvand the dense path for a dense genomicGinv(:selinvand:densestay explicit; the recursion over a dense factor is 67x slower thaninv), andreliabilityreadsdiag(inv(Ainv))through the selected inverse ofAinvinstead of forming the dense inverse, soresult_payloadandbreeding_values_plot_datano longer form any dense inverse for a sparseAinv. The original q = 5,000 timing was removed because its pedigree generator, call script, and hardware were not recorded; it was not a reproducible package benchmark. Numerics are unchanged to machine precision. Szymon Drobniak (finding). The selected-inverse path refuses a numerically singular coefficient matrix (e.g. a duplicated column ofX) with a namedArgumentError, using a relative-pivot test (L_ii² / C_ii) that is invariant to the units of a covariate: a well-posed fit with a covariate stored at magnitude 10⁶ or more is not refused, and neither is a degenerate-but-well-posed boundary fit (the guard's floor iseps-level, 1e-15; a duplicated column ofXmeasures 7.4e-18).Fixed #370/#371: K-effect post-fit uncertainty can call
multi_effect_uncertaintyonce and reuse one covariance calculation for component standard errors, ratio standard errors, and sum-ratio intervals. This applies only to the multi-effect K-effect route; the univariate animal-model route is unchanged. Performance figures from an undistributed great-tit dataset are intentionally omitted here because readers cannot reproduce them from this repository.Fixed #334:
tools/write_validation_status_page.jlno longer stamps a<!-- regenerated: <timestamp> -->comment intodocs/src/validation-status.md. The stamp came fromnow(UTC)on every call and was rewritten unconditionally, so a plaindocs/make.jlbuild dirtied the tracked page even when the generated table body was unchanged. The BEGIN/END generated-block markers are unchanged. Docs-build hygiene only; no table content changed.Clarified #333:
fit_snp_blup's docstring no longer states the GBLUP↔SNP-BLUP equivalence unconditionally. The equivalence is exact for the unregularizedG; throughfit_gblup's requiredGinv = inv(G + ridge·I)— the package's only GBLUP route — the two differ byO(ridge), about 1–2 % ofsd(gebv)at the package defaultridge = 0.01, because the ridge changes the covariance kernel. Docstring-only; no arithmetic changed; the per-cell measurement is in issue #333;docs/src/genomic-models.mdgives the reason, not the numbers.Fixed hsquared#210 (Julia half): the Willham total-heritability docstrings and the stored
conventionstring indirect_maternal_intervalwroteσ_Pfor a variance. Changed every such user-visible string insrc/toσ²_P, consistently, with no change to any arithmetic (src/likelihood.jl,direct_heritability/maternal_ratio/total_heritabilitydocstrings and the returnedconventionfield).Fixed hsquared#212 (Julia half):
fit_payload_v2accepts and forwardsinitial/iterationson the:multi_effect(dense) and:direct_maternalarms, which previously hardcoded the engine call and discarded both;fit_gblup_reml,fit_single_step_remlandfit_metafounder_single_step_remlgain aniterationskeyword. Defaults unchanged (nothingomits the keyword entirely). The R half of #212 is separate and does not land here.Fixed #331:
covariance_structure_lrt'sdfcounts identified parameters —_mv_nparamsremoves ther(r−1)/2rotational indeterminacy ofΛfor:lowrankand:factor_analytic, so a Ledermann-valid comparison is no longer refused withdf = 0. The function now always uses the plain χ²dfreference and never the 50:50 chi-bar mixture: a:factor_analyticnull is a regular submanifold and reportsboundary = false, while a:lowranknull staysboundary = truewith the naive tail and its direction explicitly unknown — the word "conservative" is gone. A newreferencefield names which distribution producedpvalue. Behaviour change: a structured comparison withdf = 1(reachable at:lowrank,t = 2,rank = 1) now reports twice its former p-value.Fixed hsquared#214/#217 (Julia half):
max_dense_cellsis a keyword on the dense-validation fitters, andfit_repeatability_remlnow runs the dense-cell guard.Fixed #327:
NonGaussianFitgains aboundaryfield;nongaussian_three_field_payloadrefuses boundary-riding fits; an opt-inrestart_check(withrestart_estimate) is added; the:nbinomand:gaussianbranches gain the same log-unit rail the gamma branch already has. No capability-status or count changes. Defaults are unchanged for the seven families that already had a bound; for:gaussianand:nbinomthe new rail applies on the default path, so a fit whose optimum lies outsideexp(log(initial) ± 8)now stops on the rail and setsboundary = truewhere it previously searched unbounded.Fixed #343 (follow-on to hsquared#212/#337):
fit_payload_v2's:multi_effectdispatch now forwardsinitial/iterationson the opt-inscale_method = :autoroute too (previously forwarded on:denseonly), viafit_multi_effect's ownkwargs...to whichever engine:autoselects. Default calls on either route are unaffected (nothingomits the keyword entirely).Clarified #347 (follow-on to #327):
nongaussian_three_field_payload's boundary-refusalArgumentError, and the correspondingfit_laplace_remldocstring paragraph, no longer adviserestart_check = trueas a way to clear a flagged boundary — that check is one-directional and can only makeboundarystricter, never clear it. Both now name the actual lever: a differentinitial, which recentres the log-scale search bracket/rail. Inside the payload builder the family gate now runs before the boundary gate, so the message's ± 6 bracket is exact for every family that can reach it (the jointly-estimated families stop on a ±8-log-unit rail instead); a boundary-flagged fit of an unsupported family is now refused by the family message rather than the boundary message. Wording plus that one refusal-order change; no numerical behaviour change.
0.9.0 (experimental)
This release retained the experimental package status and did not authorize a new covered-capability count. It consolidated the Julia engine and R-bridge contracts present at the 0.9.0 tag; subsequent development remains under Unreleased above.
0.8.0 (experimental)
Experimental numbered bump after both 0.8 engine pillars are covered (FA V4-FA + single-step V2-SSHINV). Version tracks the engine pillar pair, not R-public count. public_covered_count stays 7. Experimental label retained. Not production, not Julia General. R FA stays planned. R single_step() stays opt-in partial.
0.7.0 (experimental)
Experimental numbered bump after R G10. Version tracks R-public covered capability (public_covered_count 6 → 7). Engine V4-MV-REML was already covered; this is not an engine covered flip. Experimental label retained. Not production, not Julia General. k≥3 / diagonal multivariate and interval calibration stay out of the covered claim.
0.6.0 (experimental; historical)
This historical experimental marker followed the R-public unstructured multivariate route, which changed public_covered_count from 5 to 6. V4-MV-REML was already engine-covered, so this was not an engine covered-row promotion. It did not establish production readiness, a Julia General registration, interval calibration, or a 0.9 release. The dated coordination record is 2026-09-02.
0.5.0 (experimental)
Experimental numbering change (0.0.1 → 0.5.0). The engine remains experimental. This is not a covered flip, not a production-ready release, and not a Julia General registration. public_covered_count stays 5. Install from the GitHub URL until a General PR merges.
Corrected the Julia #44/R-bridge status wording after hsquared PR #96. The R twin already has an opt-in
target = "nongaussian"bridge for Poisson and Binomial LA/VA fits plus the PR #95 normalizer fixture; the remaining bridge gap is per-record varying-trial formula/bridge activation and deeper validation/comparator/calibration evidence, not wholesale absence of R non-Gaussian bridge fitting.Recorded hsquared PR #95 consumption of the
test/fixtures/non_gaussian_parity/target in Julia status ledgers. This banks R Julia-freeNonGaussianFitnormalizer parity only; per-record varying-trial activation, interval calibration, external comparator evidence, and covered-status promotion remain open.Added
test/fixtures/non_gaussian_parity/, a Julia-native #44 bridge payload target fornongaussian_result_payload(::NonGaussianFit). The fixture pins Poisson Laplace and per-record Binomial variational payloads for R normalizer tests; it is not per-record varying-trial R activation, interval calibration, or external comparator evidence.Added pedigree normalization and sparse
Ainvconstruction utilities.Added low-level animal-model specification validation.
Added dense Gaussian ML/REML log-likelihood evaluation at supplied variance components.
Added sparse REML log-likelihood evaluation at supplied variance components using the Henderson MME determinant identity.
Added experimental
fit_sparse_reml()andfit_animal_model(...; target = :sparse_reml)dispatch for REML-only sparse validation optimization.Added an experimental sparse prediction-error-variance / reliability path:
prediction_error_variance(...; method = :selinv)andreliability(...; method = :selinv)use a Takahashi selected inverse of the sparse Henderson MME coefficient matrix (kernel adapted from DRM.jl, MIT). The default stays:denseand matches it to machine precision on tiny fixtures.Added an experimental average-information REML estimator:
fit_ai_reml()andfit_animal_model(...; target = :ai_reml)estimate the two variance components by AI/Newton steps on the sparse Henderson MME (score from the selected inverse, AI matrix from working-variate re-solves). Validated to recover the same optimum as the NelderMead optimizers; REML-only, Gaussian, experimental.Added
genomic_relationship_matrix()— the VanRaden (2008) genomic relationship matrixGfrom a 0/1/2 (or dosage) marker matrix. Construction utility only (Phase 2 start); no genomic fitting yet.Added
genomic_relationship_inverse()— the ridge-regularized dense inverseinv(G + ridge·I)of a genomic relationship matrix, intended for later GBLUP use. Construction utility only; not wired into model fitting, and no single-step (H-matrix) blending.Added
fit_gblup()— genomic BLUP at supplied variance components: feeds a genomicGinvinto the existing Henderson MME (Ginvin theAinvslot). Reproduces an independent dense MME and reproduces pedigree BLUP whenG = A; experimental, supplied-variance only, no variance-component estimation, no single-step, no external comparator.Added
fit_snp_blup()andcentered_markers()— SNP-BLUP / RR-BLUP marker effects via the existing Henderson MME (Z = Wcentered markers, identity prior,σ²_marker = σ²_g/k).gebv = W·âequals GBLUP to machine precision (the GBLUP↔SNP-BLUP equivalence, validated via the marginalV); experimental, supplied-variance only, unweighted VanRaden method-1.Added
test/fixtures/genomic_gblup_snpblup_target/— a Julia-native #49 comparator target for supplied-variance GBLUP/SNP-BLUP with a positive-definite supplied-frequency VanRadenG,Ginv, beta, GEBVs, marker effects, metadata, and a generator. This pins a future external comparator target; it is not external comparator evidence.Added
single_marker_scan()— a Phase-5 fixed-effect single-marker screening utility that residualizesyand centered marker dosages againstX, then reports marker effects, supplied-variance standard errors, Wald z-scores, and chi-square statistics plus approximate two-sided Gaussian/Wald p-values, Bonferroni-adjusted p-values, Benjamini-Hochberg q-values, and fixed-effect known-variance LOD-equivalent scores.marker_manhattan_data()prepares plot-ready marker IDs, chromosome/position metadata, cumulative plot positions, raw p-values, and-log10(p)values, including overloads for already-validatedHSMarkerMapSpec/HSDatamarker metadata.marker_region_data()prepares one-chromosome or coordinate-window data slices from the same row-aligned scan fields, preserving scan indices and optional marker-variance proportions for future regional plot/fine-mapping front ends without activating them.marker_significance_summary()reports nominal returned-marker-set raw, Bonferroni, and BH significance flags/counts plus top-marker provenance from the same scan fields; it is not a calibrated genome-wide threshold workflow.marker_effects()prepares deterministic sorted marker-effect summaries from the same scan fields, with optional metadata alignment.marker_scan_table()prepares row-aligned scan tables from the same direct scan fields in original scan order, with allele variances, marker-variance contributions, optional total-variance proportions, optional mixed/LOCO fields when present, and optional marker-map metadata alignment.gwas_table(),qtl_table(), andeqtl_table()are semantic wrappers over those already-computed direct scan tables; they add an analysis label plus optional trait or expression-feature metadata, but do not run GWAS, interval mapping, or expression-wide eQTL scans.marker_variance_explained()prepares deterministic marker-level variance-contribution summaries as2p(1-p) * effect^2, with optional total-variance proportions and metadata alignment.marker_qq_data()prepares sorted observed/expected QQ plot data from the same direct scan output.marker_genomic_inflation()computes a genomic-control-style lambda_GC diagnostic from returned chi-square statistics. This is direct Julia engine tooling only: no mixed-model GWAS/QTL, no interval-mapping or mixed-model LOD workflow, no marker-file parser, no plotting backend, noregional_plot()or fine-mapping activation, no p-value calibration or calibrated/correlated-marker genome-wide threshold workflow, no calibrated PVE/model R² claim, no Rmarker_scan()syntax, no bridge payload change, and no comparator parity.Added opt-in
sim/phase5_marker_scan_recovery.jlfor seeded marker-signal recovery of the direct fixed, supplied-variance mixed, and supplied LOCO marker-scan helpers outside CI. Default seed20260614passes all three cases on a half-sib simulated design. This is internal recovery smoke evidence, not broad multiple-testing calibration, QTL/eQTL validation, public R syntax, bridge payload change, or comparator parity.Added
mixed_model_marker_scan()— a dense, supplied-variance GLS marker-screening helper that formsV = sigma_a2 * Z * A * Z' + sigma_e2 * Ifrom supplied variance components and a supplied relationship precision, then runs marker-by-marker Wald tests conditional onX. Tests pin reduction tosingle_marker_scan()when the random-effect design contributes zero covariance and agreement with an independent GLS calculation. Direct Julia engine tooling only: no marker-scan variance-component estimation, no LOCO, no sparse production scan, no calibrated p-values, no calibrated PVE/model R² claim, no plotting backend, no Rmarker_scan()syntax, no bridge payload change, and no comparator parity. The sharedmarker_effects(),marker_variance_explained(), Manhattan, QQ, and inflation helpers work over the returned scan fields.Added
loco_relationship_precisions()andloco_mixed_model_marker_scan()— dense validation-scale leave-one-group-out marker-screening helpers. The construction helper drops each marker group, builds a VanRaden relationship matrix from the remaining markers, and applies the existing ridge-regularized inverse. The scan helper selects the matching precision before running the dense supplied-variance GLS scan. Tests pin explicit LOCO precision identities and marker-wise agreement with separatemixed_model_marker_scan()calls. Direct Julia engine tooling only: no public LOCO defaults, no marker-scan variance-component estimation, no sparse production scan, no calibrated p-values, no calibrated PVE/model R² claim, no plotting backend, no Rmarker_scan()syntax, no bridge payload change, and no comparator parity. The same effect-summary, marker-variance, and diagnostic helpers remain direct Julia-only.Internal: deduped the numerator-relationship recursion into
_numerator_relationship(pedigree[, rows])(one source shared byinbreeding_coefficientsand single-stepA₂₂construction); removed the test-only duplicate. No public API or behavior change.Documented and tested genomic reliability/PEV/accuracy: for a GBLUP fit the existing extractors use the regularized genomic self-relationship
diag(inv(Ginv)) = diag(G) + ridgeas the denominator (the ridge perturbs the reported reliability/accuracy), andmethod = :selinvPEV matches the dense diagonal. No logic change.Added an internal single-step H-inverse construction utility
_single_step_Hinv—H⁻¹ = A⁻¹ + scatter(τG⁻¹ − ωA₂₂⁻¹)on the genotyped rows (with the subtleA₂₂⁻¹ = inv(A[g,g]), not(A⁻¹)[g,g]). Property-checked (reduction, locality, symmetry, distinctness, scattered rows, singular-Gguard); unexported, not wired into fitting, blending/τ/ω/ridge defaults not comparator-validated.Validated genomic REML variance-component estimation: the existing REML optimizers (
fit_ai_reml,fit_sparse_reml) estimate σ²g/σ²e on a genomicGinvspec — AI and NelderMead reach the same optimum andfit_gblupat the estimate reproduces the REML breeding values. Experimental; no external comparator; no new code (reuses the Phase-1 optimizers).Added a "Genomic models" documentation page with runnable examples for
G/Ginv, GBLUP, genomic REML, SNP-BLUP, and the single-stepH⁻¹utility, and an explicit experimental / not-yet-R-wired / no-external-comparator boundary.Added experimental heritability uncertainty:
variance_component_covariance,variance_component_standard_errors,heritability_standard_error, andheritability_interval— a logit-transform delta interval (always in (0,1)) built on the REML AI matrix, with a self-contained standard-normal quantile. Asymptotic / REML-only; wide and unreliable at small n.Added
repeatability_mme()— the first Phase-3 slice: a supplied-variance Henderson solve of the two-random-effect repeatability / permanent-environment animal model (additivea+ permanent-environmentpe) via a block-diagonal relationship precision. Matches an independent marginal-GLS BLUP and reduces to the animal model asσ²pe → 0. Experimental, supplied-variance only — no REML estimation of the three components, no R-facing model-spec yet.Added
fit_repeatability_reml()— REML estimation of (σ²a, σ²pe, σ²e) for the repeatability / permanent-environment model (dense two-random-effect REML loglik + NelderMead), returning the repeatability coefficienttandh². Deterministic checks (loglik reduces to the animal-model REML at σ²pe=0; BLUPs matchrepeatability_mme; optimum beats a grid) + a one-off seeded recovery. Experimental, dense/validation-scale, REML-only; no committed recovery test (suite kept RNG-free), no intervals, no R-facing model-spec.Added
two_effect_mme()— the general supplied-variance kernel for two independent random effects (each with its own incidence, relationship inverse, and variance), covering common-environment and maternal-environment models;repeatability_mmeis now itsZ2=Z1, A2=Ispecial case. Validated against an independent marginal-GLS BLUP. Experimental; no correlated direct–maternal genetic, no estimation, no R-facing model-spec.Added
fit_two_effect_reml()— REML estimation of the two-effect-model variances (common-environmentc², maternal variance, etc.) via the dense two-effect REML loglik;fit_repeatability_remlis theZ2=Z1, A2=Ireduction. Experimental, dense/validation-scale, REML-only; no committed recovery test, no intervals, no R-facing model-spec.Added a "Standard QG models" documentation page with runnable examples for repeatability (MME + REML, repeatability coefficient), common-environment, and maternal-environment models, with the experimental / not-yet-R-wired boundary.
Validation status (v0.1 gate): corrected the
V1-AI-REMLevidence to cite the committed finite-difference REML Hessian check (V1-HERIT-CI) instead of an uncommitted "250-animal observed-information" claim, and recorded the R-lane external validation —V1-MRODE-FITandV1-COMPARATORSmove tocovered_external(the engine recovers the published gryphon REML estimate, Wilson 2010 h²=0.470, via suppliedA_gryphon, and agrees with sommer, within the maintainer-signed-off band). Status rows + tests only; no engine change.Reconciled
ROADMAP.mdwith merged reality: Phases 1-3 engine utilities (genomic prediction, single-stepH-inverse, AI-REML, the Takahashi selected inverse, repeatability / two-effect models) are now listed as landed experimental, validation-scale utilities instead of "not implemented".Added
multivariate_mme()— the first Phase-4 (multivariate Gaussian) slice: a supplied-covariance Henderson solve of the balanced multi-trait animal model (Kronecker MME with genetic precisionAinv⊗G0⁻¹and residualI⊗R0⁻¹), returning per-trait fixed effects and EBVs. Validated against an independent loop-built MME, an independent marginal-GLS BLUP, the univariate animal model att=1, andtindependent single-trait fits at diagonalG0/R0. Experimental, dense/validation-scale; balanced data with a shared design only; noG0/R0estimation, no missing records, no R-facing model-spec.Added
genetic_correlation()— converts a covariance matrix (e.g.G0,R0) to the corresponding correlation matrix; also extracts the genetic correlation from amultivariate_mmeresult.Extended
multivariate_mme()to unbalanced / missing-trait records: amissingorNaNentry inYmarks an unobserved trait, handled via the per-individual residual precisioninv(R0[Sᵢ, Sᵢ]); balanced data reduces to the previous fast path. Validated against an independent loop-built MME and a marginal-GLS BLUP with per-individual residual blocks.Added
fit_multivariate_reml()— estimates the multi-trait genetic/residual covariancesG0,R0by dense REML (log-Cholesky-parameterized Nelder–Mead; handles missing records), returning the estimates, correlations, per-trait heritabilities, and breeding values at the estimate. Validated by thet = 1reduction to the univariate REML (the loglik equalssparse_reml_loglikexactly), grid-beating, and EBV consistency withmultivariate_mme. Experimental, dense/validation-scale; the opt-in recovery harness accepts--seedor explicit--seedslists and prints summaries outside CI, but is not broadly multi-seed calibrated and has no external-comparator parity yet.Added copy-returning Julia-side multivariate result accessors:
variance_components,fixed_effects,breeding_values/EBV/BLUP, and REML-onlyheritability. These wrap existingmultivariate_mme/fit_multivariate_remlfields and do not changeresult_payload()or the R bridge contract.Added
test/fixtures/phase4_multitrait_parity/, a deterministic two-trait CSV fixture for R-lane sommer/ASReml/BLUPF90 parity work. It records a Julia REML target (G0,R0, beta, EBVs, h², and loglik) and CI checks fast self-consistency at the stored target covariances. Its README anddocs/dev-log/decisions/2026-06-14-multitrait-comparator-protocol.mddefine the comparator protocol; it is not external comparator evidence.Added opt-in
sim/phase4_multivariate_reml_recovery.jlfor seeded two-trait known-truth recovery of the unstructured multivariate REML estimator outside CI. Default seed20260616passed with relative errorsG = 0.174500andR = 0.131056against thresholds0.25and0.20. This is internal recovery evidence, not multi-seed calibration or external comparator parity.Updated the validation-status documentation table to include the current Phase 2, Phase 3, Phase 4, and Phase 4B validation rows and their claim boundaries.
Hardened the multivariate engine after a 7-lens adversarial review (each finding verified by running Julia):
multivariate_mme/fit_multivariate_remlnow reject non-finite observed phenotypes (Inf— onlymissing/NaNmark an unobserved trait), non-finiteX/Z/Ainv, and empty-trait columns with clear errors (previously a silent all-NaN result or an opaqueSingularException); andfit_multivariate_reml'sloglikis now the full REML log-likelihood including the(N−p')·log(2π)constant, on the same scale as the other loglik functions (safe for LRT/AIC). Regression-tested.Added Phase-4B structured genetic covariance support for the dense multivariate REML engine:
diagonal_covariance(),lowrank_covariance(), andfactor_analytic_covariance()builddiag(σ²),ΛΛ', andΛΛ' + Ψ, andfit_multivariate_reml(...; genetic_structure = :diagonal | :lowrank | :factor_analytic, rank = K)estimates constrained genetic covariance structures while keeping residualR0unstructured. Validated by deterministic constructor identities, structure metadata, copy-returninggenetic_structure()/genetic_loadings()/genetic_uniqueness()accessors, loglik equality to the existing evaluator, PSD/PD covariance checks, constrained loglik ≤ the unstructured fit, and deterministic sign-canonicalization of returned loading columns. The2026-06-14-loading-rotation-identifiabilitydecision note records this as a sign-only metadata convention and defers full loading rotation/interpretation. Experimental, dense/validation-scale; no R-facing covariance-structure syntax, bridge change, full loading rotation/interpretation convention, or external comparator yet.Added opt-in Phase-4B recovery harness
sim/phase4b_structured_covariance_recovery.jlfor seeded low-rank and factor-analytic covariance recovery outside CI. It now accepts explicit--seedslists and prints per-case summaries. This strengthens internal recovery tooling but does not add R-facing syntax, bridge payload fields, broad multi-seed calibration, or external comparator parity.Added
docs/dev-log/decisions/2026-06-14-multivariate-recovery-calibration-protocol.md, a shared protocol for the unstructured and structured multivariate recovery harnesses. It defines the minimum seed counts, run-plan fields, summaries, and claim gate required before any broad multi-seed calibration claim.Executed that recovery calibration protocol outside CI on the predeclared seed lists and recorded negative calibration evidence in
docs/dev-log/recovery-checkpoints/: unstructured passed 6/10, factor-analytic passed 8/10, and low-rank passed 9/10, with all fits converged. No broad multi-seed calibration claim is made.Added deterministic
sim/summarize_recovery_calibration.jltooling to parse committed recovery harness logs and regenerate the Markdown case summary, failure-mode table, and failed-seed list without rerunning stochastic simulations.Added
docs/dev-log/recovery-checkpoints/2026-06-14-multivariate-recovery-calibration-failure-modes.md, which records that unstructured failures were 3 G-only plus 1 G+R, factor-analytic failures were 1 G-only plus 1 G+R, and low-rank had 1 R-only failure.Documented a throttled local command form for opt-in recovery/calibration harnesses (
JULIA_NUM_THREADS=1,OPENBLAS_NUM_THREADS=1,OMP_NUM_THREADS=1,VECLIB_MAXIMUM_THREADS=1,nice -n 15) so long stochastic runs do not monopolize interactive workstations.Added
docs/dev-log/decisions/2026-06-14-calibration-failure-response.md, which records that the failed predeclared calibration run cannot be rescued by silently relaxing thresholds, dropping failed seeds, or rerunning until the pass count improves.Added a "Multivariate models" documentation page with a runnable balanced two-trait example and the experimental / not-yet-R-wired boundary.
Expanded planned backend marker/control vocabulary to include threaded CPU, AMDGPU, Metal, and oneAPI markers alongside CPU, CUDA, and auto metadata.
Added
backend_info()typed status diagnostics for planned backend rows with execution marked unavailable.Added planned genomic/QTL model-term vocabulary reservations:
genomic(),single_step(),markers(),marker_scan(), andqtl_scan().Added planned standard quantitative-genetic model-term vocabulary reservations:
permanent(),common_env(),maternal_genetic(),maternal_env(),paternal_genetic(),paternal_env(),cytoplasmic(),imprinting(),dominance(),epistasis(),relmat(), andHSquared.precision()in direct Julia code.Added a Documenter model-spec grammar page mirroring the R twin's status separation for parsed, reserved, and planned syntax.
Added
formula_status()grammar diagnostics and a Documenter status table mirroring the R twin's parsed/reserved/planned grammar rows.Added
validation_status()diagnostics for covered, external, partial, and planned validation rows.Added
max_dense_cellsguards for the temporary dense validation path.Added experimental dense variance-component optimization for validated low-level animal-model specs.
Added experimental low-level variance-component, fixed-effect, EBV/BLUP, fitted-value, and heritability extractors for the dense spec path.
Switched
breeding_values(fit)to the Henderson MME solve at the fit's variance components.Switched
fitted_values(fit)to the same Henderson MME solve at the fit's variance components.Added experimental dense prediction-error-variance and reliability extractors for the dense spec path.
Extended validation-scale prediction-error-variance and reliability extractors to supplied-variance
HendersonMMEResultobjects.Extended supplied-variance
variance_components()andheritability()extractors toHendersonMMEResultobjects.Added
EBV()andBLUP()aliases forbreeding_values(), plusaccuracy()as a checked square-root transformation of reliability.Added
fit_diagnostics()as metadata-only extraction for low-levelAnimalModelFitand supplied-varianceHendersonMMEResultobjects.Added experimental direct payload
fit_animal_model(y, X, Z, Ainv; ...)target for bridge-shaped inputs.Added explicit
fit_animal_model(...; target = :henderson_mme, variance_components = ...)dispatch for supplied-variance Henderson MME solving.Added
henderson_mme()for sparse Henderson mixed-model-equation solving at supplied variance components.Added a shared R/Julia Henderson mixed-model-equation validation fixture for the supplied-variance output path.
Added
result_payload()with field names aligned to the Rhsquared_fitextractor contract.Added
HSData,HSDataIDMap, andid_map()as an in-memory mirror of the Rhs_data()input-container contract.Added
HSDatamarker-map metadata validation and genotype-marker alignment checks.Added
data_status(::HSData)diagnostics mirroring the R twin'sdata_status()surface for component presence, ID-overlap counts, pedigree status, genotype status, marker-alignment status, expression status, annotation-feature status, and environment-key status. Diagnostic only; no bridge payload, raw-pedigree Ainv construction, genotype parsing, relationship construction, annotation or environment covariate joins, automatic expression-feature joins, environmental model terms, marker scan, genomic fitting, eQTL/omics fitting, or QTL/eQTL claim.Recorded the R twin's
hs_data()genotype-status diagnostics fromhsquaredheadf067cd9and mirrored them in JuliaHSDataas metadata diagnostics only.Recorded the R twin's
hs_data()expression-status diagnostics fromhsquaredhead06cdf59and mirrored them in JuliaHSDataas metadata diagnostics only.Recorded the R twin's
hs_data()annotation-feature diagnostics fromhsquaredhead87888d9and mirrored them in JuliaHSDataas metadata diagnostics only.Recorded the R twin's
hs_data()environment-key diagnostics fromhsquaredheade7fbb31and mirrored them in JuliaHSDataas metadata diagnostics only.Added
sparse_csc_matrix()for RMatrix::dgCMatrixslot marshalling.Recorded the R twin's PEV/reliability bridge extractor contract while keeping Julia
result_payload()fields unchanged.Recorded the R twin's tiny/local bridge enrichment of PEV/reliability from exported Julia extractors, still without widening base
result_payload().Recorded the R twin's supplied-variance
target = "henderson_mme"enrichment fromprediction_error_variance(mme)andreliability(mme), still without widening baseresult_payload().Recorded the R twin's opt-in supplied-variance Henderson MME bridge target, with explicit no-log-likelihood/no-variance-estimation boundary.
Recorded the R twin's sparse
Zbridge marshalling handoff.Recorded the R twin's optional
nadiv::Mrode9pedigree-Ainv comparator evidence.Recorded the R twin's
model_spec()preview surface for the v0.1 formula-to-bridge payload.Recorded the R twin's
hs_data()parser integration for the v0.1 formula-to-bridge payload without changing the Julia payload shape.Recorded the R twin's
animal(1 | id)shorthand fordata = hs_data(..., pedigree = ped)as R-side formula ergonomics only; the explicitanimal(1 | id, pedigree = ped)contract and Julia payload shape are unchanged.Added DocumenterVitepress documentation scaffold.
Added audience and comparator programme notes.
Added genomics/QTL/eQTL/GLLVM/GPU/HPC strategic roadmap.
Added a backend and algorithm roadmap page for CPU, threads, CUDA, AMDGPU, Metal, oneAPI, AI-REML, Takahashi selected inversion, Woodbury paths, APY, and claim gates.
Kept high-level fitting entry points as honest placeholders.