Data Containers

HSData is the first Julia-side mirror of the R hs_data() container.

It is an in-memory contract for matching phenotypes, pedigrees, genotypes, expression data, marker annotation, and environmental metadata. It does not read PLINK, VCF, Arrow, Parquet, HDF5, or Zarr files yet, and it does not build genomic relationship matrices or fit models.

Minimal Phenotype Data

using HSquared

phenotypes = (
    id = ["animal_1", "animal_1", "animal_2"],
    y = [1.0, 1.5, 2.0],
)

data = HSData(phenotypes; id = :id)
id_map(data).phenotype_ids

2-element Vector{Any}:
 "animal_1"
 "animal_2"

Repeated phenotype records are allowed. The ID map stores unique phenotype IDs in first-seen order.

Pedigree Matching

When a pedigree is supplied, every phenotyped ID must be present in the pedigree. Extra pedigree ancestors are allowed.

pedigree = normalize_pedigree(
    ["founder", "animal_1", "animal_2"],
    ["0", "founder", "founder"],
    ["0", "0", "0"],
)

data = HSData(phenotypes; pedigree = pedigree)
id_map(data).pedigree_ids

3-element Vector{Any}:
 "founder"
 "animal_1"
 "animal_2"

Raw table-like pedigrees can also be stored if they have an ID column. HSData does not normalize raw pedigree parents; use normalize_pedigree() for the engine pedigree representation.

Raw pedigree tables are allowed to carry warning conditions such as duplicate IDs, missing known parent IDs, self-parent rows, or same-known-parent rows so data_status() can report them. normalize_pedigree() rejects duplicate IDs, missing known parent IDs, and self-parent rows. It rejects a same-known-parent row by default, but allow_selfing = true permits self-fertilization; the already-indexed Pedigree constructor also permits the same earlier parent in both parent slots. For a raw table, data_status() recognizes sire/father and dam/mother column names independently, so a single recognized parent column is retained in its correct diagnostic count. If neither alias is present, the existing three-column positional convention is used.

R Parser Integration

On the R side, hs_data() can now feed the v0.1 parser. R head 36efbf3 allows model_spec() and hsquared() to accept an hs_data() object as data; model variables are read from data$phenotypes, and formula components such as pedigree = pedigree can be resolved from the bundle. R heads 74eef82 and 39ca990 also allow the R-side shorthand animal(1 | id) when the pedigree is already stored inside data = hs_data(..., pedigree = ped).

This does not change the Julia bridge payload shape. The bridge still targets y, X, sparse Z, normalized pedigree and ID metadata, method, family, and Julia target metadata. The explicit animal(1 | id, pedigree = ped) spelling remains the shared portable contract, and Julia HSData object marshalling remains planned.

Genotype And Expression IDs

Matrix-like genotype data needs explicit row IDs because Julia base matrices do not have row names.

genotypes = [
    0.0 1.0
    1.0 0.0
    2.0 2.0
]

expression = (
    id = ["animal_2", "animal_4"],
    gene1 = [4.0, 5.0],
    gene3 = [3.0, 6.0],
)

data = HSData(
    phenotypes;
    pedigree = pedigree,
    genotypes = genotypes,
    genotype_ids = ["animal_1", "animal_3", "founder"],
    genotype_marker_ids = ["m2", "m1"],
    markers = (
        marker = ["m1", "m2"],
        chr = ["1", "1"],
        pos = [10, 20],
    ),
    expression = expression,
)

id_map(data).phenotypes_without_genotypes

1-element Vector{Any}:
 "animal_2"

When marker metadata is supplied, HSData validates common marker-map aliases:

• marker ID: marker, marker_id, snp, snp_id, or id;

• chromosome: chromosome, chr, or chrom;

• position: position, pos, bp, or base_pair.

Marker IDs must be unique and non-missing. Chromosomes must be non-missing. Positions must be finite, non-negative numeric values. If both genotypes and markers are supplied, genotype marker names must match marker-map IDs exactly after marker names are normalized to strings. For matrix-like genotypes, use genotype_marker_ids because Julia base matrices do not carry column names.

data.marker_spec.marker_ids

2-element Vector{String}:
 "m1"
 "m2"

data.genotype_marker_spec.marker_map_index

2-element Vector{Int64}:
 2
 1

data_status() exposes the same diagnostics directly:

status = data_status(data)
[row.metric => row.count for row in status.id_overlap]

9-element Vector{Pair{String, Int64}}:
                 "phenotype_ids" => 2
                  "pedigree_ids" => 3
                  "genotype_ids" => 3
                "expression_ids" => 2
   "phenotypes_without_pedigree" => 0
  "phenotypes_without_genotypes" => 1
  "genotypes_without_phenotypes" => 2
 "phenotypes_without_expression" => 1
 "expression_without_phenotypes" => 1

[row.metric => row.count for row in status.pedigree_status]

12-element Vector{Pair{String, Int64}}:
               "pedigree_rows" => 3
                "pedigree_ids" => 3
 "phenotype_ids_with_pedigree" => 2
           "pedigree_only_ids" => 1
                    "founders" => 1
                 "nonfounders" => 2
            "known_sire_links" => 2
             "known_dam_links" => 0
    "missing_known_parent_ids" => 0
      "duplicate_pedigree_ids" => 0
            "self_parent_rows" => 0
      "same_known_parent_rows" => 0

Genotype Metadata

When genotypes are supplied, data_status() reports genotype rows, matched genotype IDs, marker-column count, named marker-column count, unnamed marker-column count, duplicate named marker-column count, missing genotype value count, and component type.

[row.metric => row.value for row in status.genotype_status]

8-element Vector{Pair{String, String}}:
                     "genotype_rows" => "3"
                      "genotype_ids" => "3"
           "genotype_marker_columns" => "2"
     "named_genotype_marker_columns" => "0"
   "unnamed_genotype_marker_columns" => "2"
 "duplicate_genotype_marker_columns" => "0"
           "missing_genotype_values" => "0"
                    "component_type" => "matrix"

Plain Julia matrices do not carry marker column names. Matrix-like genotypes therefore need explicit genotype_ids for ID matching and report marker columns as unnamed unless marker names are supplied separately through genotype_marker_ids for marker-map alignment.

matrix_genotype_data = HSData(
    phenotypes;
    genotypes = [0.0 1.0; 1.0 missing],
    genotype_ids = ["animal_1", "animal_2"],
)

[row.metric => row.value for row in data_status(matrix_genotype_data).genotype_status]

8-element Vector{Pair{String, String}}:
                     "genotype_rows" => "2"
                      "genotype_ids" => "2"
           "genotype_marker_columns" => "2"
     "named_genotype_marker_columns" => "0"
   "unnamed_genotype_marker_columns" => "2"
 "duplicate_genotype_marker_columns" => "0"
           "missing_genotype_values" => "1"
                    "component_type" => "matrix"

This is metadata hygiene only. HSData does not parse PLINK/VCF files, impute genotypes, construct genomic relationships, run marker scans, or fit genomic, QTL, GWAS, or eQTL models.

[row.metric => row.value for row in status.marker_status]

7-element Vector{Pair{String, String}}:
      "marker_map_markers" => "2"
 "genotype_marker_columns" => "2"
  "aligned_marker_columns" => "2"
             "chromosomes" => "1"
            "position_min" => "10.0"
            "position_max" => "20.0"
               "alignment" => "checked"

Expression Metadata

When expression data are supplied, data_status() reports expression rows, matched expression IDs, feature count, named feature count, unnamed feature count, duplicate named feature count, and component type.

[row.metric => row.value for row in status.expression_status]

7-element Vector{Pair{String, String}}:
               "expression_rows" => "2"
                "expression_ids" => "2"
           "expression_features" => "2"
     "named_expression_features" => "2"
   "unnamed_expression_features" => "0"
 "duplicate_expression_features" => "0"
                "component_type" => "table"

Plain Julia matrices do not carry row or column names. Matrix-like expression therefore needs explicit expression_ids for ID matching and reports feature columns as unnamed:

matrix_expression = [
    1.0 2.0 3.0
    4.0 5.0 6.0
]

matrix_expr_data = HSData(
    phenotypes;
    expression = matrix_expression,
    expression_ids = ["animal_1", "animal_2"],
)

[row.metric => row.value for row in data_status(matrix_expr_data).expression_status]

7-element Vector{Pair{String, String}}:
               "expression_rows" => "2"
                "expression_ids" => "2"
           "expression_features" => "3"
     "named_expression_features" => "0"
   "unnamed_expression_features" => "3"
 "duplicate_expression_features" => "0"
                "component_type" => "matrix"

This is metadata hygiene only. HSData does not join expression features into model matrices, fit eQTL or other omics models, or run GLLVM workflows.

Annotation Metadata

HSData can store feature annotation metadata. If annotation_id is supplied, the key column must be present in annotation; data_status() then reports overlap between expression feature columns and annotation feature keys.

expr_features = (
    id = ["animal_1", "animal_2"],
    gene1 = [4.0, 5.0],
    gene3 = [3.0, 6.0],
)

annotation = (
    gene_id = ["gene1", "gene2", "gene2"],
    chromosome = ["1", "1", "2"],
)

ann_data = HSData(
    phenotypes;
    expression = expr_features,
    annotation = annotation,
    annotation_id = :gene_id,
)

ann_data.annotation_spec.expression_without_annotation

1-element Vector{String}:
 "gene3"

ann_status = data_status(ann_data)
[row.metric => row.value for row in ann_status.annotation_status]

8-element Vector{Pair{String, String}}:
                        "annotation_rows" => "3"
                         "annotation_key" => "gene_id"
                    "annotation_features" => "2"
                    "expression_features" => "2"
    "expression_features_with_annotation" => "1"
               "annotation_only_features" => "1"
 "expression_features_without_annotation" => "1"
          "duplicate_annotation_features" => "1"

If an annotation table is supplied without annotation_id, it is stored but reported as unkeyed:

unkeyed_ann_data = HSData(phenotypes; annotation = (gene = ["gene1"], chr = ["1"]))
[row.metric => row.value for row in data_status(unkeyed_ann_data).annotation_status]

8-element Vector{Pair{String, String}}:
                        "annotation_rows" => "1"
                         "annotation_key" => "not_checked_no_annotation_id"
                    "annotation_features" => "not_available"
                    "expression_features" => "not_available"
    "expression_features_with_annotation" => "not_available"
               "annotation_only_features" => "not_available"
 "expression_features_without_annotation" => "not_available"
          "duplicate_annotation_features" => "not_available"

This is metadata hygiene only. HSData does not join annotation metadata into model matrices, fit eQTL or other omics models, or run GLLVM workflows. In this slice, keyed annotation diagnostics require table-like expression inputs with feature columns; plain Julia matrices do not carry feature column names.

Environment Metadata

HSData can also store an environment metadata table. If environment_id is supplied, the same key column must be present in phenotypes and environment; data_status() then reports overlap between phenotype environment keys and environment metadata keys.

environment = (
    env = ["E1", "E2", "E2"],
    temperature = [18.0, 20.0, 21.0],
)

pheno_env = (
    id = ["animal_1", "animal_1", "animal_2"],
    env = ["E1", "E1", "E3"],
    y = [1.0, 1.5, 2.0],
)

env_data = HSData(pheno_env; environment = environment, environment_id = :env)
env_data.environment_spec.phenotypes_without_environment

1-element Vector{String}:
 "E3"

env_status = data_status(env_data)
[row.metric => row.value for row in env_status.environment_status]

8-element Vector{Pair{String, String}}:
                           "environment_rows" => "3"
                            "environment_key" => "env"
                            "environment_ids" => "2"
                  "phenotype_environment_ids" => "2"
    "phenotype_environment_ids_with_metadata" => "1"
                       "environment_only_ids" => "1"
 "phenotype_environment_ids_without_metadata" => "1"
                  "duplicate_environment_ids" => "1"

If an environment table is supplied without environment_id, it is stored but reported as unkeyed:

unkeyed_env_data = HSData(phenotypes; environment = (site = ["S1"],))
[row.metric => row.value for row in data_status(unkeyed_env_data).environment_status]

8-element Vector{Pair{String, String}}:
                           "environment_rows" => "1"
                            "environment_key" => "not_checked_no_environment_id"
                            "environment_ids" => "not_available"
                  "phenotype_environment_ids" => "not_available"
    "phenotype_environment_ids_with_metadata" => "not_available"
                       "environment_only_ids" => "not_available"
 "phenotype_environment_ids_without_metadata" => "not_available"
                  "duplicate_environment_ids" => "not_available"

This is metadata hygiene only. HSData does not join environment covariates into model matrices, add environmental model terms, or fit multi-environment models.

id_map(data).genotypes_without_phenotypes

2-element Vector{Any}:
 "animal_3"
 "founder"

id_map(data).phenotypes_without_expression

1-element Vector{Any}:
 "animal_1"

id_map(data).expression_without_phenotypes

1-element Vector{Any}:
 "animal_4"

Current Boundary

HSData currently provides:

• exact-ID matching;

• repeated phenotype ID support;

• marker-map metadata validation and genotype-marker alignment checks;

• data_status() diagnostics for components, ID-overlap counts, pedigree status, genotype status, marker status, expression status, annotation-feature status, and environment-key status;

• optional pedigree, genotype, expression, marker, annotation, and environment storage;

• conservative mismatch fields for later bridge and genomic work.

Planned later:

• file-backed phenotype and genotype storage;

• PLINK, VCF/BCF, Arrow, Parquet, HDF5, and Zarr readers;

• genotype parsing and imputation;

• automatic genotype-to-relationship construction;

• QTL/eQTL scans and genomic relationship construction from genotype/marker data;

• automatic expression-feature joins;

• eQTL/omics workflows from expression/annotation metadata;

• direct Julia HSData to AnimalModelSpec construction;

• live Julia HSData object marshalling;

• production model fitting from data-container inputs.

IDs are not coerced. For example, 1 and "1" are different IDs until an explicit normalization rule is added.