Skip to content

Pipeline Overview

The ClinVar-GKS pipeline transforms ClinVar XML release data into GA4GH GKS format through a series of BigQuery stored procedures with an external VRS Python processing step.

Pipeline Steps

The pipeline executes in the following order. Each step is a BigQuery stored procedure unless otherwise noted.

┌──────────────────────────────┐
│ 1. Variation Identity        │  variation_identity_proc
│    Extract & normalize       │  → variation_identity table
│    variant data              │
└──────────────┬───────────────┘
               │
┌──────────────▼───────────────┐
│ 2. VRS Processing            │  External: vrs-python
│    Export → VRS Python →     │  → gks_vrs table
│    Import back to BigQuery   │
└──────────────┬───────────────┘
               │
┌──────────────▼───────────────┐
│ 3. Cat-VRS Generation        │  gks_catvar_proc
│    Canonical alleles &       │  → gks_catvar table
│    categorical variants      │
└──────────────┬───────────────┘
               │
┌──────────────▼───────────────┐
│ 4. Conditions & Traits       │  gks_scv_condition_proc
│    Map traits, build         │  → condition mapping &
│    conditions & condition    │    condition set tables
│    sets                      │
└──────────────┬───────────────┘
               │
┌──────────────▼───────────────┐
│ 5. SCV Statements            │  gks_scv_statement_proc
│    Build SCV records,        │  → gks_dict_scv table
│    propositions & statements │
└──────────────┬───────────────┘
               │
┌──────────────▼───────────────┐
│ 6. VCV Statements            │  gks_vcv_proc +
│    Aggregate SCVs into       │  gks_vcv_statement_proc
│    variant-level statements  │  → gks_dict_vcv table
└──────────────┬───────────────┘
               │
┌──────────────▼───────────────┐
│ 7. RCV Statements            │  gks_rcv_proc +
│    Aggregate SCVs into       │  gks_rcv_statement_proc
│    condition-level statements│  → gks_dict_rcv table
└──────────────┬───────────────┘
               │
┌──────────────▼───────────────┐
│ 8. JSON Output               │  gks_json_proc
│    Build dictionary tables   │  → gks_dict_* tables
│    for bundle assembly       │
└──────────────┬───────────────┘
               │
┌──────────────▼───────────────┐
│ 9. Export & Distribute       │  export-gks-dicts.sh
│    Export dicts to GCS,      │  assemble-gks-dicts.py
│    assemble JSON bundle +    │  upload-gks-to-r2.sh
│    Parquet, upload to R2     │  → R2 public bucket
└──────────────────────────────┘

Running the Pipeline

Step 1: Variation Identity

From the BigQuery console:

CALL `clinvar_ingest.variation_identity_proc`(CURRENT_DATE(), FALSE);

Step 2: VRS Processing

Export, process externally with vrs-python, and load back. See VRS Processing.

Step 3: Cat-VRS through JSON Output

From the BigQuery console:

CALL `clinvar_ingest.gks_catvar_proc`(CURRENT_DATE(), FALSE);
CALL `clinvar_ingest.gks_scv_condition_proc`(CURRENT_DATE(), FALSE);
CALL `clinvar_ingest.gks_scv_statement_proc`(CURRENT_DATE(), FALSE);
CALL `clinvar_ingest.gks_vcv_proc`(CURRENT_DATE(), FALSE);
CALL `clinvar_ingest.gks_vcv_statement_proc`(CURRENT_DATE(), FALSE);
CALL `clinvar_ingest.gks_rcv_proc`(CURRENT_DATE(), FALSE);
CALL `clinvar_ingest.gks_rcv_statement_proc`(CURRENT_DATE(), FALSE);
CALL `clinvar_ingest.gks_json_proc`(CURRENT_DATE(), 'all');

Step 4: Export & Distribute

# Export dictionary tables from BigQuery to GCS as NDJSON
./src/scripts/export-gks-dicts.sh clinvar_2026_06_14_v2_5_0 clinvar-gks gks-dicts

# Assemble NDJSON files into JSON bundle + Parquet
python3 ./src/scripts/assemble-gks-dicts.py \
  gs://clinvar-gks/gks-dicts/ \
  2026-06-14 \
  --parquet-dir /tmp/parquet-output

# Upload bundle to Cloudflare R2
./src/scripts/upload-gks-to-r2.sh 2026-06-14 v2_5_0 /tmp/clinvar-gks-2026-06-14.json.gz \
  --parquet-dir=/tmp/parquet-output

See Export for details on each step.

Documentation Tracks

The pipeline documentation serves two audiences:

  • Pipeline (this section) — documents how data flows through BigQuery stored procedures, including internal table schemas, transformation logic, and step-by-step workflows. Each step is tagged as Pipeline table, JSON artifact, or Internal to indicate its role
  • Output Reference — documents the JSON output files from a consumer perspective, covering record structure, field meanings, and usage guidance

Detailed Documentation

Each pipeline step has its own documentation page: