SAIL logo SAIL

We gave a model a question and accidentally also gave it the answer.

This is that story — and it's not about a bad model. It's about a widely-used clinical score that was never designed to be a machine-learning feature, and the open-source detector that catches it before it ships into one.

pip install sail-leakage

Also available as a VS Code extension (setup wizard, plain-language results, no Python typing required). It isn't on the Marketplace yet — build the .vsix from source: see the extension's README.

A two-minute annotated tour of SAIL on synthetic data.

Read the video as text
  • Two synthetic patient windows can have very different blood pressure — 78 and 52 mmHg — but the identical SOFA cardiovascular score, 3. Once a vasopressor dose is running, the dose sets the score, not blood pressure.
  • That's why validation can look excellent: the input already carries the dose, so the model reads it off and scores well. On a new patient, no dose has been chosen yet, the shortcut is gone, and performance drops below what was reported.
  • The extension's three steps: choose your data, run the checks — everything runs locally, on your own computer — and read the report.
  • SAIL runs five checks, each asking a plain question about how the data was built: construction, reconstruction, temporal overlap, timing violation, persistence dominance.
  • A flag means your validation numbers may overstate what the model will do on new patients, with a specific fix per card. When a check lacks what it needs, SAIL says which specification is missing — "not run" is an honest answer too, not a guess.

How it works

1

Pick your dataset

A CSV or Parquet file from your workspace, or try the example data first — no setup beyond having sail-leakage installed.

2

Confirm the columns

SAIL auto-detects the patient ID, time step, treatment and outcome fields from typical column names — you just confirm or correct them.

3

Read the report

Five checks, each explained in plain language: what it checks, what SAIL found, and what to do about it. Everything runs on your computer.

What it checks

Five categories, each a generalization of a mathematically proven mechanism from the underlying research — not a heuristic guess.

Construction leakage

Is a treatment dose built into one of the patient-state scores?

Reconstruction leakage

Can a hidden score be rebuilt from the scores the model can see?

Temporal overlap

Does the treatment period overlap the time window the state describes?

Timing violation

Is any state value measured after the treatment it should come before?

Persistence dominance

Would simply repeating the last treatment beat your model?

See the checks in depth →

SAIL is open source (MIT licence) and built in the open: code, results and decisions are reviewed in public pull requests.

About Detectors Papers Datasets & Models Privacy Understanding Leakage