FHIR Validation in CI: Profiles, Terminology and IPS Checks That Fail the Build

A FHIR profile is a contract. The International Patient Summary says a patient has a birth date, an allergy names its patient, and a coded element draws its code from a stated value set. A contract that is checked by a person, in a connectathon week or during acceptance testing, is checked late and checked once. The examples in the repository, the fixtures the tests use and the resources a mapper emits drift in between, and the first system to notice is the one on the other side of the interface.

This article puts the check where the drift happens: in the build. Two validators, the HL7 validator and the Firely .NET SDK, check the same resources against the IPS profiles in a GitHub Actions workflow, and a non-conformant resource fails the build. The part that takes most of the design effort is not running a validator. It is knowing that the gate would refuse a broken resource if it saw one, and knowing which kinds of defect it cannot see at all.

What the gate has to prove

The requirement sounds like one sentence: fail the build when a resource does not conform. In practice it splits into several, and the non-functional ones decide whether the team keeps the gate switched on.

• Refuse non-conformant resources. Any error-level finding against the declared profile stops the build, with the file, the element path and the message in the log.
• Prove the refusal. The pipeline carries resources that are broken on purpose and fails if any of them passes. A gate that has only ever seen valid input has not been tested.
• Be repeatable. The same commit gives the same verdict next month. That means a pinned validator, pinned packages and no silent dependency on whatever a public server returns today.
• Fail closed. A validator that crashed, wrote no output or validated nothing is a failed build, not a passed one.
• Stay fast enough to run on every push. Package loading dominates the run time, so packages are cached and all files go through one validator start.
• Never touch patient data. Everything validated in CI is synthetic. Build logs and artefacts are not a place for clinical records.

Validation itself has three layers, and they differ in what they need. That difference is the main architectural fact of this article.

Pipeline design

The workflow has one job. It prepares the tools, runs the self-check twice with the HL7 validator (without and with a terminology server), runs the gate the way a product build would, repeats the self-check with the Firely tool, and publishes every output file before deciding the result.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
    A[Push to the repository] --> B[Set up Java, .NET and Python]
    B --> C[Restore FHIR package cache]
    C --> D[Download pinned validator and verify checksum]
    D --> E[Self-check: HL7 validator, no terminology server]
    E --> F[Self-check: HL7 validator with terminology server]
    F --> G[Gate: conformant folder must exit 0]
    G --> H[Gate: broken folder must exit 1]
    H --> I[Build and run the Firely tool, self-check its findings]
    I --> J[Publish all reports]
    J --> K{Every check as expected?}
    K -->|Yes| L[Build passes]
    K -->|No| M[Build fails]
    classDef compute fill:#FFF3E0,stroke:#FF9900,stroke-width:2px,color:#232F3E
    classDef store fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
    classDef danger fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B
    classDef net fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74
    class A,B,D,E,G,H,I,K compute
    class C,J,L store
    class F net
    class M danger

Each check step is allowed to fail without stopping the job, and a final step turns any failure into a failed build. The order matters: when a gate fails, the reports are what you need, so they are published first.

Test data that is wrong on purpose

The test data is eleven small resources and one file that says what the gate must conclude about each. Two conform. Seven are broken in exactly one way that a validator can see from the definitions alone. Two can only be judged by a terminology server.

The last file is the one worth looking at. It is well-formed, it meets the profile, the code exists, and it is clinically wrong:

{
  "resourceType": "AllergyIntolerance",
  "id": "ips-allergy-wrong-display",
  "meta": {
    "profile": [
      "http://hl7.org/fhir/uv/ips/StructureDefinition/AllergyIntolerance-uv-ips"
    ]
  },
  "clinicalStatus": {
    "coding": [
      {
        "system": "http://terminology.hl7.org/CodeSystem/allergyintolerance-clinical",
        "code": "active"
      }
    ]
  },
  "code": {
    "coding": [
      {
        "system": "http://snomed.info/sct",
        "code": "91936005",
        "display": "Fracture of femur"
      }
    ]
  },
  "patient": {
    "reference": "Patient/ips-patient-ok"
  }
}

The self-check reads the expectations, runs the gate and gives each case one of four labels: as expected, WRONG, blind spot (a terminology case that passed because no terminology server was in use) and caught anyway. Any WRONG fails the build. Blind spots are counted and printed, because a number in the log is harder to forget than a caveat in a wiki.

The HL7 validator behind a Python wrapper

The HL7 validator is the reference implementation: a Java command-line tool, validator_cli.jar, that loads the core definitions and any implementation guide packages, validates the files it is given and writes its findings as OperationOutcome resources. The wrapper adds the parts a build needs: one validator start for all files, a verdict per file, expectations, and exit codes with a fixed meaning.

"""A validation gate for FHIR resources in a build pipeline.

Two commands:

  gate       Validate every resource under a directory. Exit 1 if any has an error. This is the build step.
  selfcheck  Validate the deliberately broken resources in testdata/ and check that each one is rejected, and
             that the valid ones pass. Exit 1 if the gate lets something through. This tests the gate itself.

Two engines produce the findings: the HL7 validator (a Java command-line tool, run from here) and a small
Firely SDK tool (run separately; its JSON output is read from --firely-results).

    python fhir_gate.py selfcheck --engine hl7 --jar validator_cli.jar --ig hl7.fhir.uv.ips#2.0.0 --tx n/a
    python fhir_gate.py gate --dir resources --engine hl7 --jar validator_cli.jar --ig hl7.fhir.uv.ips#2.0.0
"""
from __future__ import annotations

import argparse
import json
import pathlib
import subprocess
import sys
import time

FILE_EXTENSION = "http://hl7.org/fhir/StructureDefinition/operationoutcome-file"
BLOCKING = {"fatal", "error"}


def run_hl7_validator(jar: str, files: list[pathlib.Path], ig: list[str], tx: str | None, work: pathlib.Path,
                      java: str = "java", timeout: int = 1800) -> dict[str, list[dict]]:
    """Run the HL7 validator once over all files and return {file name: [issue, ...]}."""
    work.mkdir(parents=True, exist_ok=True)
    output = work / "hl7-output.json"
    command = [java, "-jar", jar, *[str(f) for f in files], "-version", "4.0.1", "-output", str(output)]
    for package in ig:
        command += ["-ig", package]
    if tx:
        command += ["-tx", tx]
    started = time.monotonic()
    completed = subprocess.run(command, capture_output=True, text=True, timeout=timeout)
    (work / "hl7-console.txt").write_text(
        f"$ {' '.join(command)}\nexit code {completed.returncode}, {time.monotonic() - started:.0f} s\n\n"
        f"{completed.stdout}\n{completed.stderr}", encoding="utf-8")
    if not output.exists():
        raise RuntimeError(f"The validator wrote no output (exit code {completed.returncode}). See {work / 'hl7-console.txt'}")
    return outcomes_by_file(json.loads(output.read_text(encoding="utf-8")), files)


def outcomes_by_file(document: dict, files: list[pathlib.Path]) -> dict[str, list[dict]]:
    """The validator writes one OperationOutcome for a single file and a Bundle of them for several."""
    outcomes = [document] if document.get("resourceType") == "OperationOutcome" else \
        [entry["resource"] for entry in document.get("entry", [])]
    by_name = {f.name: f for f in files}
    result: dict[str, list[dict]] = {}
    for index, outcome in enumerate(outcomes):
        named = next((e.get("valueString") for e in outcome.get("extension", []) if e.get("url") == FILE_EXTENSION), None)
        name = pathlib.Path(named).name if named else (files[index].name if index < len(files) else f"unknown-{index}")
        if name not in by_name:
            raise RuntimeError(f"The validator reported on '{named}', which is not one of the input files")
        result[name] = [{
            "severity": issue.get("severity", "error"),
            "code": issue.get("code", ""),
            "message": issue.get("details", {}).get("text") or issue.get("diagnostics", ""),
            "location": "; ".join(issue.get("expression", []) or issue.get("location", [])),
        } for issue in outcome.get("issue", [])]
    missing = set(by_name) - set(result)
    if missing:
        raise RuntimeError(f"The validator returned nothing for: {', '.join(sorted(missing))}")
    return result


def load_firely_results(path: str) -> dict[str, list[dict]]:
    return {pathlib.Path(name).name: issues for name, issues in json.loads(pathlib.Path(path).read_text(encoding="utf-8")).items()}


def errors(issues: list[dict]) -> list[dict]:
    return [i for i in issues if i["severity"] in BLOCKING]


def command_gate(args: argparse.Namespace) -> int:
    files = sorted(pathlib.Path(args.dir).rglob("*.json"))
    if not files:
        print(f"No resources under {args.dir}: refusing to pass an empty check")
        return 1
    findings = findings_for(args, files)
    failed = 0
    for file in files:
        blocking = errors(findings[file.name])
        failed += bool(blocking)
        print(f"{'FAIL' if blocking else 'ok  '}  {file}")
        for issue in blocking:
            print(f"        {issue['location'] or '-'}: {issue['message']}")
    print(f"\n{len(files) - failed} of {len(files)} resources conform")
    return 1 if failed else 0


def command_selfcheck(args: argparse.Namespace) -> int:
    root = pathlib.Path(args.testdata)
    cases = json.loads((root / "expectations.json").read_text(encoding="utf-8"))["cases"]
    files = [root / case["file"] for case in cases]
    findings = findings_for(args, files)
    terminology_on = args.engine == "hl7" and bool(args.tx) and args.tx.lower() != "n/a" or \
        (args.engine == "firely" and args.firely_terminology)
    rows, wrong = [], 0
    for case, file in zip(cases, files):
        blocking = errors(findings[file.name])
        verdict = "fail" if blocking else "pass"
        needs_terminology = case.get("needs") == "terminology-server"
        if needs_terminology and not terminology_on:
            status = "blind spot" if verdict == "pass" else "caught anyway"
        else:
            status = "as expected" if verdict == case["expect"] else "WRONG"
            wrong += status == "WRONG"
        rows.append({"file": case["file"], "expected": case["expect"], "verdict": verdict, "status": status,
                     "because": case.get("because", ""), "errors": [f"{i['location']}: {i['message']}"[:300] for i in blocking]})
        print(f"{status:<13} {verdict:<5} {case['file']}")
        for issue in blocking[:3]:
            print(f"              {issue['message'][:160]}")
    summary = {"engine": args.engine, "terminology": "on" if terminology_on else "off", "cases": len(rows), "wrong": wrong,
               "blindSpots": sum(r["status"] == "blind spot" for r in rows), "rows": rows}
    if args.report:
        pathlib.Path(args.report).write_text(json.dumps(summary, indent=2), encoding="utf-8")
    print(f"\n{len(rows) - wrong} of {len(rows)} cases as expected, {summary['blindSpots']} blind spot(s) without a terminology server")
    return 1 if wrong else 0


def findings_for(args: argparse.Namespace, files: list[pathlib.Path]) -> dict[str, list[dict]]:
    if args.engine == "firely":
        return load_firely_results(args.firely_results)
    return run_hl7_validator(args.jar, files, args.ig, args.tx, pathlib.Path(args.work))


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    commands = parser.add_subparsers(dest="command", required=True)
    for name in ("gate", "selfcheck"):
        sub = commands.add_parser(name)
        sub.add_argument("--engine", choices=["hl7", "firely"], default="hl7")
        sub.add_argument("--jar", default="validator_cli.jar")
        sub.add_argument("--ig", action="append", default=[], help="implementation guide package, for example hl7.fhir.uv.ips#2.0.0")
        sub.add_argument("--tx", help="terminology server URL, or n/a to validate without one")
        sub.add_argument("--work", default="out")
        sub.add_argument("--firely-results", help="JSON written by the FhirGate tool")
        sub.add_argument("--firely-terminology", action="store_true", help="the Firely run used a terminology server")
        sub.add_argument("--report", help="write the self-check result here as JSON")
    commands.choices["gate"].add_argument("--dir", required=True)
    commands.choices["selfcheck"].add_argument("--testdata", default="testdata")
    args = parser.parse_args()
    try:
        return command_gate(args) if args.command == "gate" else command_selfcheck(args)
    except (RuntimeError, subprocess.TimeoutExpired, OSError, json.JSONDecodeError) as error:
        print(f"The check could not run: {error}")
        return 2


if __name__ == "__main__":
    sys.exit(main())

Four decisions in that file are worth defending.

• The verdict comes from the output file, not the process. The wrapper asks for -output, reads the Bundle of OperationOutcomes and counts error and fatal issues per file. It does not rely on the exit code of the Java process to say whether resources were valid.
• One start, many files. Loading the core definitions and the IPS package costs far more than validating a small resource. Starting the JVM once per file multiplies that cost by the number of files.
• Every input must be accounted for. The validator names the file each outcome belongs to in an extension. If an outcome names a file that was not submitted, or a submitted file has no outcome, the check stops with exit code 2.
• An empty folder is a failure. A glob that matches nothing after a directory rename would otherwise turn the gate into a step that always passes.

Warnings do not fail the build. That is a policy choice, and a team that wants best-practice warnings to block can add them to BLOCKING. Start with errors: a gate that fails on advice gets switched off.

The same check on the Firely SDK

A .NET team may not want a JVM in the build, and may want the same validation inside its own tests or services. The Firely SDK provides a validator as a library. This console tool on .NET 10 validates each file against the profile it declares in meta.profile and writes findings in the shape the Python self-check reads, so both engines are judged against the same expectations.

using System.Text.Json;
using Firely.Fhir.Packages;
using Firely.Fhir.Validation;
using Hl7.Fhir.Model;
using Hl7.Fhir.Serialization;
using Hl7.Fhir.Specification;
using Hl7.Fhir.Specification.Source;
using Hl7.Fhir.Specification.Terminology;

// Usage: FhirGate --package hl7.fhir.uv.ips@2.0.0 [--package <dependency>@<version>] --out firely.json <file.json> ...
// Validates each resource against the profile it declares in meta.profile, with the Firely .NET SDK, and
// writes {file: [{severity, code, message, location}]} in the shape fhir_gate.py reads.
// Exit code: 0 when the run completed (whatever it found), 2 when it could not run.

List<string> packages = [], files = [];
string output = "firely.json";
for (int i = 0; i < args.Length; i++)
{
    switch (args[i])
    {
        case "--package": packages.Add(args[++i]); break;
        case "--out": output = args[++i]; break;
        default: files.Add(args[i]); break;
    }
}

if (files.Count == 0)
{
    Console.Error.WriteLine("No files given.");
    return 2;
}

const string PackageServer = "https://packages.simplifier.net";
try
{
    // Core definitions plus the implementation guides, resolved from the package server and cached on disk.
    // The source loads exactly the packages it is given: an implementation guide's own dependencies have to
    // be listed too, or its profiles fail to load.
    FhirPackageSource core = FhirPackageSource.CreateCorePackageSource(ModelInfo.ModelInspector, FhirRelease.R4, PackageServer);
    FhirPackageSource guides = new(ModelInfo.ModelInspector, PackageServer, [.. packages]);
    CachedResolver resolver = new(new MultiResolver(guides, core));

    // Value sets that can be expanded from the packages are checked locally. Codes from large external code
    // systems such as SNOMED CT are not: that takes a terminology server.
    LocalTerminologyService terminology = new(resolver);
    Validator validator = new(resolver, terminology);
    FhirJsonParser parser = new();

    Dictionary<string, List<Finding>> results = [];
    foreach (string file in files)
    {
        List<Finding> findings = [];
        results[Path.GetFileName(file)] = findings;
        Resource resource;
        try
        {
            resource = parser.Parse<Resource>(await File.ReadAllTextAsync(file));
        }
        catch (Exception ex) when (ex is FormatException or DeserializationFailedException or InvalidOperationException)
        {
            // Content that is not valid FHIR at all never reaches profile validation.
            findings.Add(new Finding("error", "structure", ex.Message, ""));
            continue;
        }

        string? profile = resource.Meta?.Profile?.FirstOrDefault();
        OperationOutcome outcome = profile is null ? validator.Validate(resource) : validator.Validate(resource, profile);
        findings.AddRange(outcome.Issue.Select(issue => new Finding(
            issue.Severity?.ToString().ToLowerInvariant() ?? "error",
            issue.Code?.ToString() ?? "",
            issue.Details?.Text ?? issue.Diagnostics ?? "",
            string.Join("; ", issue.Expression ?? []))));
    }

    await File.WriteAllTextAsync(output, JsonSerializer.Serialize(results, new JsonSerializerOptions { WriteIndented = true, PropertyNamingPolicy = JsonNamingPolicy.CamelCase }));
    foreach ((string file, List<Finding> findings) in results)
    {
        int errors = findings.Count(f => f.Severity is "error" or "fatal");
        Console.WriteLine($"{(errors == 0 ? "ok  " : "FAIL")}  {file}  ({errors} error(s), {findings.Count - errors} other)");
    }

    return 0;
}
catch (Exception ex)
{
    Console.Error.WriteLine($"The validation run could not complete: {ex.GetType().Name}: {ex.Message}");
    return 2;
}

internal sealed record Finding(string Severity, string Code, string Message, string Location);

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Firely.Fhir.Validation.R4" Version="3.6.0" />
    <PackageReference Include="Firely.Fhir.Packages" Version="5.0.2" />
  </ItemGroup>
</Project>

Three things differ from the HL7 validator, and each one can catch you out.

• Package dependencies are yours to list. The package source loads the packages it is given. IPS profiles refer to extensions published in a separate package, hl7.fhir.uv.extensions.r4; with only the IPS package listed, the IPS Patient profile does not load. The workflow passes the dependency as a second --package. The HL7 validator resolves dependencies itself.
• Parsing comes first. A resource with an unknown element or a malformed date fails in the parser before profile validation starts. The tool reports that as a structure error so the file still gets a verdict.
• Terminology is local unless you say otherwise. LocalTerminologyService checks codes against value sets it can expand from the loaded packages. It has no SNOMED CT. The SDK can call an external terminology server as well; that configuration is not part of this sample.

The package library has to belong to the same SDK generation as the validator. Both references are pinned in the project file for that reason.

The workflow

name: FHIR validation gate

on:
  push:
    paths:
      - "fhir-validation-ci/**"
      - ".github/workflows/fhir-validation.yml"
  workflow_dispatch:

permissions:
  contents: write   # only to publish the result files to the ci-results branch

concurrency:
  group: fhir-validation-${{ github.ref }}
  cancel-in-progress: true

jobs:
  validate:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    defaults:
      run:
        working-directory: fhir-validation-ci
    env:
      # Pinned, with a checksum. An unpinned validator changes what "valid" means between two builds.
      VALIDATOR_VERSION: "7.0.5"
      VALIDATOR_SHA256: 9dfe9f305b0f9389eceade2a4a1c9a2740d7b0e614066fdf07a78b571c76d92f
      IPS_PACKAGE_HL7: hl7.fhir.uv.ips#2.0.0
      IPS_PACKAGE_FIRELY: hl7.fhir.uv.ips@2.0.0
      # The Firely package source loads what it is given: the packages IPS depends on are listed explicitly.
      IPS_DEPENDENCIES_FIRELY: hl7.fhir.uv.extensions.r4@5.3.0-ballot-tc1
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: "21"
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: "10.0.x"
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"

      # FHIR packages are downloaded once and reused. Without the cache every run depends on the registry.
      - uses: actions/cache@v4
        with:
          path: ~/.fhir/packages
          key: fhir-packages-${{ hashFiles('.github/workflows/fhir-validation.yml') }}

      - name: Download the HL7 validator
        run: |
          mkdir -p out
          curl -fsSL -o validator_cli.jar "https://github.com/hapifhir/org.hl7.fhir.core/releases/download/${VALIDATOR_VERSION}/validator_cli.jar"
          echo "${VALIDATOR_SHA256}  validator_cli.jar" | sha256sum --check | tee out/validator-jar.txt

      - name: Self-check, HL7 validator without a terminology server
        id: hl7_notx
        continue-on-error: true
        run: python gate/fhir_gate.py selfcheck --engine hl7 --ig "$IPS_PACKAGE_HL7" --tx n/a --work out/hl7-notx --report out/hl7-notx/report.json | tee out/hl7-notx.txt

      - name: Self-check, HL7 validator with tx.fhir.org
        id: hl7_tx
        continue-on-error: true
        run: python gate/fhir_gate.py selfcheck --engine hl7 --ig "$IPS_PACKAGE_HL7" --tx https://tx.fhir.org --work out/hl7-tx --report out/hl7-tx/report.json | tee out/hl7-tx.txt

      # The build step itself: conformant resources pass, and one non-conformant resource fails the step.
      - name: Gate, conformant resources pass
        id: gate_valid
        continue-on-error: true
        run: python gate/fhir_gate.py gate --dir testdata/valid --engine hl7 --ig "$IPS_PACKAGE_HL7" --tx n/a --work out/gate-valid | tee out/gate-valid.txt

      - name: Gate, non-conformant resources are refused
        id: gate_invalid
        continue-on-error: true
        run: |
          set +e
          python gate/fhir_gate.py gate --dir testdata/invalid --engine hl7 --ig "$IPS_PACKAGE_HL7" --tx n/a --work out/gate-invalid > out/gate-invalid.txt
          code=$?
          echo "exit code $code" | tee -a out/gate-invalid.txt
          test "$code" -eq 1

      - name: Build the Firely tool
        id: firely_build
        continue-on-error: true
        run: |
          set -o pipefail
          dotnet build firely/FhirGate/FhirGate.csproj -c Release -o out/firely-bin 2>&1 | tee out/firely-build.txt
          dotnet list firely/FhirGate/FhirGate.csproj package --include-transitive > out/firely-packages.txt

      - name: Self-check, Firely SDK
        id: firely
        continue-on-error: true
        run: |
          set -o pipefail
          dotnet out/firely-bin/FhirGate.dll --package "$IPS_PACKAGE_FIRELY" --package "$IPS_DEPENDENCIES_FIRELY" --out out/firely.json testdata/valid/*.json testdata/invalid/*.json testdata/terminology/*.json 2>&1 | tee out/firely-run.txt
          python gate/fhir_gate.py selfcheck --engine firely --firely-results out/firely.json --report out/firely-report.json | tee out/firely.txt

      - name: Publish the results
        if: always()
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          rm -rf out/firely-bin
          {
            echo "commit: $GITHUB_SHA"
            echo "run: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
            echo "date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
            echo "hl7_notx: ${{ steps.hl7_notx.outcome }}"
            echo "hl7_tx: ${{ steps.hl7_tx.outcome }}"
            echo "gate_valid: ${{ steps.gate_valid.outcome }}"
            echo "gate_invalid: ${{ steps.gate_invalid.outcome }}"
            echo "firely_build: ${{ steps.firely_build.outcome }}"
            echo "firely: ${{ steps.firely.outcome }}"
            java -version 2>&1 | head -1
            dotnet --version
          } > out/run.txt
          cat out/run.txt >> "$GITHUB_STEP_SUMMARY"
          publish="$RUNNER_TEMP/publish"
          rm -rf "$publish" && mkdir -p "$publish" && cp -r out/. "$publish/"
          cd "$publish"
          git init -q -b ci-results
          git config user.name "github-actions[bot]"
          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git add -A
          git commit -q -m "Validation results for $GITHUB_SHA"
          git push -f "https://x-access-token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" ci-results

      - name: Fail the build if the gate let something through
        if: steps.hl7_notx.outcome != 'success' || steps.hl7_tx.outcome != 'success' || steps.gate_valid.outcome != 'success' || steps.gate_invalid.outcome != 'success' || steps.firely_build.outcome != 'success' || steps.firely.outcome != 'success'
        run: |
          echo "One or more validation checks failed. See the ci-results branch."
          exit 1

The points that carry over to any CI system:

• Pin the validator and verify it. The validator is released often, and each release can change which findings are errors. Downloading the latest release on every run means the meaning of “valid” changes without a commit. The workflow names a version and checks the SHA-256 of the download. Upgrading becomes a pull request that either passes the self-check or shows what changed.
• Cache the FHIR packages. Both tools keep packages in ~/.fhir/packages. Caching that folder removes the package registry from the critical path of most builds.
• Collect, publish, then fail. Check steps use continue-on-error and a final step fails the job. The reports exist whether the build passed or not.
• Least privilege. This sample needs contents: write only because it pushes its reports to a results branch so they can be read without signing in. A product pipeline should upload them as build artefacts and keep contents: read.

Terminology: the layer that needs a server

Structure and profile checks are computed from definitions that fit in a package. Terminology is different in kind. SNOMED CT has several hundred thousand concepts, is released in editions, and is licensed. Neither validator carries it. When a resource holds a SNOMED CT code, the validator has two options: ask a server, or accept the code unchecked.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','signalColor':'#0073BB','signalTextColor':'#0F172A','actorBkg':'#FFF3E0','actorBorder':'#FF9900','actorTextColor':'#232F3E','noteBkgColor':'#FFF8E7','noteBorderColor':'#FF9900','noteTextColor':'#232F3E'}}}%%
sequenceDiagram
    participant P as Pipeline
    participant V as Validator
    participant T as Terminology server
    P->>V: Validate allergy with SNOMED CT code
    V->>V: Structure and profile checks from packages
    V->>T: Is this code valid, and is this its display?
    T-->>V: Unknown code, or wrong display
    V-->>P: OperationOutcome with error
    Note over P,V: Without the server the code is accepted unchecked

The recommendation that follows from this: run the offline check on every push and treat it as the fast gate. Run the terminology check against a server you control, on pull requests to the main branch or on a schedule, and pin the code system edition there. Use the public server to get started and for open, synthetic content.

What the pipeline reported

The gate step on the broken folder printed this, shortened here to one finding per file:

FAIL  testdata/invalid/allergy-bad-status.json
        AllergyIntolerance.clinicalStatus.coding[0].code: Unknown code 'ongoing' in the CodeSystem 'http://terminology.hl7.org/CodeSystem/allergyintolerance-clinical' version '1.0.1'
FAIL  testdata/invalid/allergy-no-patient.json
        AllergyIntolerance: AllergyIntolerance.patient: minimum required = 1, but only found 0 (from http://hl7.org/fhir/uv/ips/StructureDefinition/AllergyIntolerance-uv-ips|2.0.0)
FAIL  testdata/invalid/patient-bad-date.json
        Patient.birthDate: Not a valid date format: '12/03/1984'
FAIL  testdata/invalid/patient-empty-name.json
        Patient.name[0]: Constraint failed: ips-pat-1: 'Patient.name.given, Patient.name.family or Patient.name.text SHALL be present' (defined in http://hl7.org/fhir/uv/ips/StructureDefinition/Patient-uv-ips)
FAIL  testdata/invalid/patient-no-birthdate.json
        Patient: Patient.birthDate: minimum required = 1, but only found 0 (from http://hl7.org/fhir/uv/ips/StructureDefinition/Patient-uv-ips|2.0.0)
FAIL  testdata/invalid/patient-unknown-element.json
        Patient: Unrecognized property 'birthday'

0 of 7 resources conform
exit code 1

Five observations from the run are worth more than the pass counts.

• The validator process returned 0 every time. In all four invocations, including the one where seven of seven resources had errors, the Java process exited with code 0. A pipeline step that runs the jar and trusts the exit status would have passed that folder. The wrapper’s decision to read the OperationOutcome is what makes it a gate.
• Without a terminology server, wrong SNOMED CT codes pass in both engines. The code that does not exist and the display from another concept were accepted. The HL7 validator reported one narrative best-practice warning for those files and nothing about the code. The Firely tool reported no issue at all.
• With the terminology server, both were refused, with messages a developer can act on: Unknown code '999999999999' in the CodeSystem 'http://snomed.info/sct' and Wrong Display Name 'Fracture of femur' for http://snomed.info/sct#91936005, followed by the valid displays.
• The two engines agree on verdicts and differ in wording. For the missing birth date the HL7 validator says minimum required = 1, but only found 0 and names the profile; the Firely tool says Missing required member: 'birthDate'. Assert on verdicts and element paths in tests, not on message text.
• The first Firely runs were wrong in a way that looked right. Before the extensions package was listed, the IPS Patient profile failed to load, every broken patient was “refused”, and the only visible symptom was that the conformant patient failed too. This is the case the warning above describes.

Run time was dominated by loading. The two self-check runs, which started with an empty package cache, took 27 and 23 seconds for eleven files. The two gate steps that followed, with packages on disk, took 12 and 7 seconds. These are single measurements on a shared hosted runner, useful for order of magnitude only.

Choosing a validator

For a build gate the HL7 validator is the default choice, because it is what profile authors and other implementers will run when they dispute your resource. A .NET team gains from running the Firely validator as well, inside its own tests, provided it keeps the self-check. Running both in one pipeline, as this sample does, is a way to learn where they differ; it is not something every product build needs. $validate against the target server belongs later, in integration testing. It was not run for this article.

IPS, European and national profiles

Nothing in the gate is specific to IPS. The profile set is a package coordinate, and the resources declare which profile they claim in meta.profile. To gate on the HL7 Europe base and core profiles, or on a national core, change the -ig argument (and, for the Firely tool, the --package list including dependencies) and add test resources for that guide. Three cautions apply. Ballot versions change between releases, so pin the exact version you have agreed with your partners. National guides often bind to national SNOMED CT extensions, which the public terminology server may not hold. And a resource that claims several profiles has to pass all of them; validate against each one explicitly if your tooling only reads the first. Only IPS 2.0.0 was run here.

Security and operations

• Supply chain. The validator is an executable downloaded into the build. Pin the version, verify the checksum, and mirror it to an internal artefact store if your build network is restricted. The same goes for FHIR packages: a private package feed removes two public registries from the build.
• Secrets. The offline gate needs none. A private terminology server does; give the pipeline a read-only credential scoped to validation operations.
• Observability. Keep the machine-readable report per run. A rising count of warnings, or a change in the number of files validated, is a signal even when the build is green.
• Ownership. Someone has to own the upgrade of validator, packages and code system editions. Put it on a schedule; do not let it happen by accident.
• Scope. Gate what you emit: mapper outputs, examples in your own implementation guide, test fixtures. Validating inbound production traffic is a different design with different latency and privacy constraints.

Cost

The cost of the gate is runner minutes and the time developers wait. Both are driven by loading definitions, not by the number of resources, so the levers are the package cache, one validator start per run and keeping the terminology check off the per-push path. A hosted terminology server is the only component with a standing cost, and most programmes that exchange coded clinical data need one at run time anyway. The larger saving is not on the bill: a conformance defect found in the build costs a commit, and the same defect found by a partner costs a release.

Earlier in this series: migrating from Azure API for FHIR to Azure Health Data Services and FHIR subscriptions and events on Azure.

Key terms

• Profile— A set of constraints on a FHIR resource type, published as a StructureDefinition in an implementation guide.
• IPS— International Patient Summary, an HL7 implementation guide for a minimal, cross-border patient summary.
• Implementation guide package— A versioned archive of profiles, value sets and extensions, identified by a name and version such as hl7.fhir.uv.ips#2.0.0.
• OperationOutcome— The FHIR resource a validator returns: a list of issues, each with a severity, a code, a location and a message.
• Terminology server— A service that answers questions about code systems and value sets, such as whether a code exists and what its display is.
• Invariant— A rule in a profile written as a FHIRPath expression, for example ips-pat-1.

References & Further Reading

• → Using the FHIR Validator— HL7 documentation for validator_cli.jar and its parameters
• → org.hl7.fhir.core releases— Validator releases and release notes
• → Firely .NET SDK: validating data against profiles— The validator API used by the C# tool
• → Firely .NET SDK: using terminology services— Local and external terminology services
• → International Patient Summary Implementation Guide— The profiles validated in this article
• → HL7 Europe Base and Core FHIR IG— European base and core profiles
• → FHIR R4: Validating Resources— What the specification means by validation, and its layers
• → tx.fhir.org— The public HL7 terminology server
• → Sample code for this article— Test data, the Python gate, the Firely tool and the workflow