Technical methodology for clinical data licensing and model evaluation

This specification applies to all data units delivered under a Nurture AI Data Licence Agreement (DLA) and to all evaluation tasks delivered under an Evaluation Services Agreement (ESA). It does not apply to raw source material, which never leaves the controlling organisation’s jurisdiction (§2.2).

1.1Data unit. The atomic licensable object: one consultation transcript, one encounter record, one imaging series with report, or one evaluation task instance. Each unit carries a unique unit_id (UUIDv7) and a content digest.

1.2Controller. The healthcare organisation holding the source records. The controller remains data controller throughout; Nurture AI acts as processor for de-identification and as licensor of the derived, non-identifiable output.

1.3Quasi-identifier (QI). Any attribute that, alone or in combination, could plausibly link a unit to a natural person: dates, ages above 89, postal geography finer than three digits, rare diagnoses, free-text descriptors of occupation or family structure.

1.4Task instance. A tuple ⟨c, q, R, g⟩ of clinical context, prompt, rubric and gold reasoning trace, authored and adjudicated by licensed clinicians under §5.

1.5Failure mode. A labelled category from the NAI failure taxonomy (§5.3) describing how a model’s output is unsafe or incorrect, independent of whether the final answer matches gold.

2.1Extraction

Structured records are extracted as FHIR R4 bundles where the source system exposes a conformant API, otherwise via HL7v2 message parsing or direct CDM export. Audio is transcoded to 16 kHz mono FLAC; video is retained only where a licence explicitly covers it, otherwise discarded at extraction. Every extracted object is assigned a source_ref (an HMAC-SHA256 of the controller-side primary key under a controller-held key) so that units can be recalled or withdrawn on request without Nurture AI ever holding the original key.

2.2Jurisdictional constraint

Identifiable data MUST NOT transit outside the controller’s jurisdiction. Where the controller lacks compute capacity, Nurture AI deploys an enclave (container image with pinned digests, no outbound network except the ledger endpoint) inside the controller’s environment. The enclave’s egress is limited to de-identified units that have passed §3.4 risk assessment and to ledger writes.

2.3Provenance ledger

Each unit’s lifecycle is recorded as an append-only chain of signed events. The chain root for a release is published in the release manifest so a buyer can verify, for any unit, the ordered sequence of transformations it underwent and the identity of the enclave build that produced it.

event {
  unit_id        : uuidv7
  seq            : uint32
  ts             : rfc3339
  stage          : enum{extract, deid, risk, normalise, qa, release, withdraw}
  actor          : { kind: enum{enclave, clinician, system}, key_id }
  input_digest   : sha256
  output_digest  : sha256
  params_digest  : sha256          // hash of stage configuration
  prev           : sha256          // digest of previous event
  sig            : ed25519(prev ‖ output_digest ‖ params_digest)
}

De-identification is layered. Direct identifiers are removed deterministically; quasi-identifiers are generalised or perturbed until the release-level risk bound in §3.4 is met; free text and audio are scrubbed by a PHI recogniser whose recall is measured on a held-out, clinician-annotated audit sample for every corpus.

3.1Direct identifiers

All eighteen HIPAA Safe Harbor identifier classes are removed regardless of jurisdiction, as a floor. Names, MRNs, device serials, URLs, IPs and biometric identifiers are dropped. Dates are shifted per patient by a uniform offset δ ∈ [−365, +365] days, preserving intra-patient intervals; the offset is derived from the controller-held key and is not recoverable by Nurture AI.

3.2Quasi-identifier generalisation

QIs are generalised along fixed hierarchies (age → 5-year band; geography → NUTS-2 or equivalent; rare ICD-10 codes with fewer than 20 carriers in the corpus → parent category). Generalisation is applied greedily by information loss, measured as normalised certainty penalty, until the equivalence-class constraints in §3.4 hold.

3.3Free text and speech

Free text passes through an ensemble PHI recogniser (a fine-tuned token classifier per language, plus rule-based detectors for structured identifiers) with the decision threshold set for recall, not F1. Detected spans are replaced by typed surrogates ([NAME], [DATE:+3d], [LOC]) so that downstream models see well-formed text. Recogniser recall on each corpus’s audit sample MUST be ≥ 0.995 at the span level with a 95% Clopper–Pearson lower bound ≥ 0.99; corpora that do not meet this bound are re-run with a wider surrogate policy or withheld.

Speech is transcribed in-enclave, the transcript is scrubbed as above, and the corresponding audio segments are excised and cross-faded. Speaker embeddings are not released.

3.4Release-level risk bound

A release is publishable only if, over the released QI set, every equivalence class has size at least k and the sensitive attribute distribution within each class satisfies ℓ-diversity, with (k, ℓ) fixed by the DLA (defaults: k = 11, ℓ = 3). Residual re-identification risk is reported as the prosecutor-model bound:

Structured content is mapped to OMOP CDM v5.4 with SNOMED CT as the standard vocabulary for conditions and procedures, LOINC for observations and RxNorm (or the applicable national drug dictionary, cross-walked) for medications. Source codes are retained alongside standard concepts so that buyers can audit the mapping. Mapping coverage and unmapped-term rates are reported per domain in the release manifest.

Free-text clinical language is not translated. Units are delivered in the source language with language tags (BCP 47); an optional English gloss layer MAY be licensed separately and is always marked as derived.

Evaluation datasets are constructed to elicit failure, not to measure recall of textbook facts. Each task instance is derived from a real, de-identified encounter and is authored so that the clinically correct action depends on reasoning under incomplete or conflicting information.

5.1Task construction

1. Seed selection. Encounters are sampled with stratification over specialty, acuity, language and setting, with oversampling of cases where the documented diagnosis was revised, escalated or reversed within 72 hours.
2. Authoring. A licensed clinician in the relevant specialty writes the prompt, the gold reasoning trace, and a rubric of 6–14 weighted items. Each item is tagged with the failure mode it detects (§5.3).
3. Adversarial review. A second clinician attempts to produce a plausible but unsafe answer that scores well on the rubric. If one exists, the rubric is revised until it does not.
4. Adjudication. A third clinician resolves any disagreement on gold; unresolved items are dropped, not averaged.
5. Model probing. Instances are run against a panel of reference models; instances solved by all models with rubric score ≥ 0.9 are demoted to a calibration split so the main split retains discriminative power.

5.2Rubric structure

rubric_item {
  id            : string
  criterion     : text              // clinician-legible statement
  weight        : float ∈ (0, 1]   // normalised within task
  polarity      : enum{required, forbidden}
  failure_mode  : enum{ANC, PMC, HAL, OMS, ESC, DEF, CAL, TMP}
  severity      : enum{S1, S2, S3, S4}   // S4 = potential for serious harm
  evidence_span : [char_start, char_end] | null
}

5.3Failure taxonomy

Each model response is graded against the rubric by two independent clinician raters, with a third on disagreement (§7). The primary metric is the harm-weighted rubric score, which penalises high-severity failures super-linearly so that a single S4 omission cannot be offset by many minor correct items.

Secondary metrics reported per split, per specialty and per language:

• Unsafe rate — proportion of responses carrying the unsafe flag.
• Failure-mode incidence — per-code frequency, normalised by the number of items tagged with that code.
• Abstention quality — precision and recall of the model declining to answer, against clinician judgement of whether abstention was appropriate.
• Expected calibration error — computed over the model’s stated confidence where elicited, using 10 equal-mass bins:

Confidence intervals are bootstrap percentile intervals (2,000 resamples) at the task level, clustered by seed encounter so that instances derived from the same encounter are resampled together.

Inter-rater agreement is computed per rubric item as Krippendorff’s α (nominal), with Cohen’s κ reported for two-rater pairs. Items with α < 0.67 after adjudication are rewritten or removed; releases report the distribution of α across items rather than a single pooled figure, since pooled agreement masks systematically ambiguous criteria.

Rater drift is monitored with seeded control instances (5% of assignments) whose gold is fixed; a rater whose control agreement falls below 0.9 over a rolling window of 40 instances is paused and recalibrated.

Every release ships as a versioned bundle. Buyers receive the manifest before data so that scope, coverage and risk figures can be reviewed against the licence.

release/
├── MANIFEST.json          # schema_version, unit count, splits, ρ_max, ρ_avg,
│                          # PHI-recall bound, vocabulary coverage, ledger root
├── LICENCE.txt            # DLA reference, permitted uses, exclusivity flags
├── units/
│   ├── {unit_id}.json     # OMOP-aligned structured content + language tag
│   ├── {unit_id}.txt      # scrubbed free text with typed surrogates
│   └── {unit_id}.flac     # optional, scrubbed audio
├── tasks/                 # ESA deliveries only
│   ├── {task_id}.json     # context, prompt, rubric, gold trace, split
│   └── grading/           # rater judgements, adjudication log, α per item
├── ledger/
│   └── events.jsonl       # signed provenance chain (§2.3)
└── SHA256SUMS.sig

Withdrawal: if a controller withdraws consent for a source record, all derived units are added to a signed revocation list published with the next manifest, and buyers are contractually required to purge them within 30 days.

A release conforms to this specification if and only if:

1. all identifiable processing occurred within the controller’s jurisdiction (§2.2) and is evidenced by ledger events signed by an enclave key registered to that jurisdiction;
2. PHI-recogniser recall on the corpus audit sample meets the bound in §3.3;
3. the release-level risk bound in §3.4 is satisfied, or an expert-determination opinion is attached;
4. vocabulary coverage per domain is reported in the manifest (§4);
5. for evaluation deliveries, every task instance has passed adversarial review and adjudication (§5.1) and every rubric item reports α (§7);
6. the manifest’s ledger root matches the shipped event chain.

Non-conforming material is not delivered. Questions about this specification, or requests for the enclave build manifest and audit-sample protocol, can be sent to ali@nurturecap.co.