HIP-1145: Research — The Experiment Record
Abstract
/v1/research is every experiment an org has ever run, kept and comparable:
versioned, append-only evidence under one discriminator — kind ∈ benchmark |
kernel-perf | training | ablation | policy-eval — with provenance (git sha,
branch, dirty, library versions) as queryable columns. It is implemented in
hanzoai/cloud at apps/research. A correction appends a new version under
the same stable id and the prior version is RETAINED, never mutated; faulted
and failed runs are retained too, because a negative result is evidence
(apps/research/research.go:17-22).
Motivation
A measurement that cannot be re-found, re-attributed to its commit, or compared against its own history is an anecdote. The record exists so the question "which library version regressed this benchmark" is a query over the longitudinal record rather than an archaeology project — and so dedup never reads as loss: RETAINED is the full history, CANONICAL is the deterministic deduped view over it.
Specification
The key words MUST, MUST NOT and SHOULD are to be interpreted as in RFC 2119.
Two planes, one owner
Each org's transactional SQLite — cloud.OrgStore, one physically separate
research.db per org (apps/research/research.go:191-206) — is the local
source of truth, riding whatever HA path the deployment configured. It rolls up
best-effort into hanzoai/datastore, the platform's column-oriented OLAP
plane, for the cross-project query surface: losing a roll-up MUST never fail an
ingest whose SQLite write already committed (apps/research/datastore.go:11-16).
The stronger "immutable · replicated · recovery-tested" claim is deliberately
NOT yet made — retry, reconciliation and backup/restore are still on the
critical path (apps/research/research.go:12-15).
Addresses
Seventeen operations under /v1/research, sixteen typed and one declared
(apps/research/research.go). The measurement plane: ingest
(POST /experiments, idempotent by content — re-running a backfill appends
nothing), the canonical listing (GET /experiments), GET /projects,
GET /totals, the grant (POST /grants), and the diary (POST /artifacts,
GET /artifacts, GET /artifacts/{sha256}). The last is the one route that
cannot be a typed op: it streams the artifact's raw bytes under the artifact's
own Content-Type, which a typed op — always JSON from a Go value — has no
vocabulary for; its prose is declared beside the route.
The experiment record adds four kinds in one shape — a POST that records a
batch, a GET that lists it narrowed by the filters that matter, with ?id=
selecting one so there is no second address for fetching one thing:
/benchmarks, /runs, /studies, /papers — plus GET /compare.
An artifact's identity is the SERVER-derived sha256 of its bytes, never a client-asserted hash, so the address space is genuinely content-addressed and un-poisonable; a re-POST of the same bytes is a no-op.
The execution, and whether it finished
Experiment is one measured number and Attempt is one scored item; neither is
the EXECUTION. Run is (apps/research/record.go): the benchmark and split it
measured against, the system under test and its version, the configuration that
moves a number (embedder, reader, k, temperature, max-tokens, the prompt file and
its digest), the digests that pin its inputs (dataset, retrieval store, fact
layer), the commit it was frozen at, its per-(category, metric) measures with the
intervals the harness reported, when it ran, by whom — and questions,
answered and ended.
completion is DERIVED from those three and MUST NOT be accepted from a write:
complete where the record says the run covered its set and says when it ended,
partial where it says it covered less, inconsistent where the two counts
cannot both be true, unknown where the record does not say. A harness that
checkpoints as it goes reports whatever has been answered so far, so a run read
at question 57 of 282 is otherwise indistinguishable from a finished one — which
is how locomo-all-context-k20-enso-flash reached hanzo.ai/benchmarks 9.4 F1
points and 12.5 EM points above where it landed. Absence MUST stay absent:
questions, answered, k, temperature and max_tokens are nullable, and a
null is the record declining to answer rather than a zero.
ResearchBenchmark is the TASK — dataset, that dataset's licence and origin, the
splits it declares, the metric its authors score it by, the citation — and it is
independent of anyone's run. Study is the question a set of runs was run to
answer; Paper cites runs, and a run names at most its study, never a paper. A
baseline is a run carrying baseline, not a kind of its own and not a category
inside somebody else's numbers.
GET /compare reads two runs together and states FIRST what stops them being
compared — a different benchmark, a different split, different dataset digests,
either run not established as complete, either run not naming its system — then
every configuration field they disagree about, then the change over what BOTH
measured, with whether the reported intervals overlap. A metric only one of them
reported is left out rather than compared against nothing.
All four ride the same per-org SQLite through the same orm records, the same
content-hash version identity, the same server-assigned append clock and the same
private-by-default grant: a correction to a run APPENDS a version and the
published mistake stays retained. Runs also roll up to the warehouse with
completion as a column; their per-category measures do not, because a second
copy of the numbers with no reconciliation between them is worse than a join.
Tenancy
The org is the validated principal's and selects the physical file
(storeFor → cloud.OrgStore); the project is the server's validated
sub-scope, stamped positionally into every row and every warehouse write — a
payload cannot forge either (apps/research/datastore.go:31-32). One org
cannot read another's evidence even in the presence of a query defect, because
the other org's rows are not in the file being queried.
Consent is a separate grant
An upload records visibility=private and grants NO training or
commons-publication rights. Board visibility, trainable and publishable are
each a SEPARATE authorized decision through POST /v1/research/grants — never
implied by uploading a run (apps/research/research.go:28-31). Ingest MUST
ignore any visibility or consent field a payload carries.
The BYO endpoint is stored, not dialed
An experiment may name the endpoint it was measured against. Ingest refuses the
obviously-hostile URL (https only, every resolved address publicly routable —
apps/research/ssrf.go), and that check is declared to be ingest hygiene, not
the dial-time control: nothing in this capability dials a recorded endpoint,
and code that ever does MUST carry a DialContext IP-pin first.
Money, events, telemetry
Free, said in those words: plugin/research/main.go declares cloud.Free. It
publishes nothing to the bus, so a customer's webhooks receive nothing from it.
Beyond the request span, its only extra emission is the warehouse roll-up rows
themselves — its own data plane, not telemetry a customer reads back under
/v1/o11y.
Stage
beta: the manifest row declares Stage: Beta (manifest/apps.go:410). The
durability contract is still rolling out — the roll-up is best-effort and
reconciliation is unbuilt, so the record is versioned-append-only today and no
more is claimed.
Loading a measurement tree
research.Load reads a tree's OWN output — the harness's generated
benchmarks.json for the numbers and each run directory's meta.json and
metrics.json for what the run was — and recomputes no metric: a second
normalization would be a second answer to the same question, free to disagree
with the one the published tables came from. Where a field is absent it is
recorded absent and counted, never defaulted. Measured over bench/brain as it
stands: 56 runs and 2936 measures, of which 9 are complete, 3 are inconsistent
(their own meta.json records more answers than questions), and 44 are unknown;
no run records who ran it or the version of what it measured; 44 carry no
completion counts, 28 none of the reader knobs, 4 no commit.
Upstream
Derives from none. The OLAP half reuses the platform's own
hanzoai/datastore connection beside the account-usage warehouse; nothing
external is forked, embedded or mirrored.
Rationale
Versioned-retained, rather than mutate-in-place with an audit log, is the load bearing choice: an evidence plane that can silently replace a number is a marketing plane (HIP-0512 makes the same argument for verdicts). Idempotency by content rather than by a client key follows — a backfill has no stable client key, and content identity makes replays free. Consent as a separate grant, rather than a flag on upload, exists because the uploader of a run is routinely a CI job with no authority to publish or license anything.
Security Considerations
The record carries two things worth stealing and one thing worth forging.
Stealing: another org's benchmark evidence (closed physically, per-org files)
and an artifact's raw bytes, which may carry licensed or personal material —
the artifact read is org-scoped before the hash is looked up, so a known hash
alone retrieves nothing across tenants. Forging: consent. The wrong
implementation lets an upload set trainable:true or a client assert an
artifact's hash — the first turns a CI credential into a licensing authority,
the second poisons the content address so a later honest upload resolves to
attacker bytes. Ingest forcing private/withheld and the server deriving every
hash are the closures. The stored BYO endpoint is the residual SSRF surface,
held closed by nothing dialing it.
References
- HIP-0026 — Identity and Access Management
- HIP-0106 — The Hanzo Plugin Contract
- HIP-0139 — Capability
- HIP-0512 — Experiment — The Evidence Plane
Copyright
Released under CC0 1.0 Universal Public Domain Dedication.