HIP-1333: Train — One Endpoint for Training

Abstract

/v1/train is the cloud's one training endpoint. It teaches a base model one new capability while holding what the model already does, and its first-class output is a capability artifact, not a new model. It is implemented in hanzoai/cloud at apps/train. Clients run on hanzoai/engine (HIP-0043); managed jobs run on the org's linked machines or on Hanzo's executor, Kai's (HIP-1332) through train serve in hanzoai/decision. A job Hanzo runs is billed by the device-second inside a window held ahead of it, so it never runs past what its payer can pay.

Two nouns and no others. A client is the loop the caller drives: the create → forward_backward → optim_step → sample → save_weights wire. A job is the loop the platform drives. Adaptation, protection, objective, evaluation and output are fields of those two objects; artifacts, bases and evaluations are what a job produces.

Motivation

Training has to teach a base one capability without regressing what it already does, and has to produce something that can be evaluated, merged or published apart from that base. It also has to sit behind the gateway's bearer and org. The surfaces this replaces (§14) do none of that: the engine's wire is an admin-only ingress carve, because the engine authenticates nobody; hanzoai/ai's managed jobs take only a console session cookie and submit training-job resources the cluster does not serve; Kai's stage training runs only from a shell.

Specification

The key words MUST, MUST NOT, SHOULD, SHOULD NOT and MAY are to be interpreted as in RFC 2119.

§1 The store

One per-org SQLite file, train.db, through cloud.OrgStore (HIP-0106): jobs, their tasks, their events, the org's clients, the org's artifacts and the objects holding their bytes. Artifact bytes are in Hanzo S3, bucket TRAIN_BUCKET (default hanzo-train), key <org>/<sha256>, each object encrypted under the org's own key (§6). Nothing else is stored and no other capability's store is read.

§2 The addresses

All under /v1/train. Every operation is typed.

The client wire is the engine's, byte for byte.

§3 The job

{
  "base_model": "kai", "revision": "a7",
  "dataset": {"uri": "stage:a11", "splits": {"train": "train", "validation": "validation"}},
  "objective": {"loss": "cross_entropy",
                "terms": [{"kind": "pairwise_margin", "weight": 1, "margin": 0.5},
                          {"kind": "consistency", "weight": 0.5}]},
  "adaptation": {"mode": "full"},
  "protect": {"suites": ["typed_decisions", "ag_news"],
              "projection": {"strength": 0.5}, "distillation": true,
              "budget": {"accuracy": 0.02, "ece": 0.03}},
  "resources": {"machines": ["gpu-1", "gpu-2"], "steps": 2000},
  "evaluation": {"suites": ["labels"]},
  "output": {"kind": "capability", "name": "open-labels"}
}

adaptation says how the trained parameters are represented; protect says what they must not damage. The two do not mix: every protection applies to whatever the adaptation trains.

• base_model names a trainable base (§4); revision pins it. The executor resolves the revision to the weights' SHA-256, which the job records as base.sha256.
• dataset.uri is hf://<repo>[@<rev>], s3://<bucket>/<key>, artifact:<sha256> or, for Kai, stage:<name>: that stage's data build as the executor holds it, pinned by its manifest hash. artifact:<sha256> names JSON Lines rows the org uploaded through POST /v1/train/artifacts (§6); a create reads the upload back before it accepts the job, and a sha256 the org did not upload is 400 invalid_request.
• objective.loss is cross_entropy. terms add pairwise_margin (a hinge between a group's attack and benign rows, with margin), hard_margin (a hinge between gold and the hardest negative) and consistency (symmetric KL across a group's views), each with a weight.
• adaptation.mode is full, readout, lora, qlora, basis or auto. readout trains the head alone over the frozen base, and its capability is of form head (§6). lora and qlora take rank, alpha and targets. basis trains coefficients in a subspace: basis names a basis artifact (its sha256, or basis://<base>/<name>) or sources names lora artifacts to build one from; rank is how many of its directions are used, and residual_rank the rank of an orthonormal residual learned beside them (0: the coefficients alone). auto chooses, and adaptation.chose says what and why.
• protect.suites names capabilities the base already has, by suite; capabilities names capability artifacts, by sha256. projection projects every update — a full update, an adapter or a residual — off the protected directions: the subspace P the suites' gradients span at the base, and each named capability's subspace. Its strength λ in [0, 1] (default 1) scales it: g' = g − λ·P g. At λ = 1 the update keeps only what lies outside P; at 0 it is untouched. Each step reports the share of the gradient removed, per layer. distillation adds KL to the base's answers on a preservation set drawn from the suites. budget bounds the regression the job may cause on each protected suite: accuracy down at most accuracy, calibration error up at most ece.
• resources.machines names linked machines (the first leads); absent, Hanzo's executor runs the job and its compute is charged to the creator's wallet (§10). devices names the accelerator kinds it may run on — cuda, rocm, metal, vulkan, cpu — and, absent, any its base runs. steps bounds optimizer steps. max_seconds bounds the device-seconds the job's tasks use together, and budget what a job Hanzo runs may cost, in US cents. A job Hanzo runs needs one (400 budget_required); a job on the org's machines costs nothing, so budget with machines is 400 invalid_request. Whichever bound is reached first stops the job (§10).
• evaluation.suites are the target suites (§8).
• output.kind is checkpoint, lora, capability, basis or merged, with a name. A job's result names one artifact of that kind; a job stopped before its result keeps what its executor stored before the stop (§5).
• from names an artifact the job starts from. A job with from and no dataset trains nothing: it evaluates (evaluation set), merges (output.kind merged) or extends a basis (output.kind basis). No executor runs from yet (§4).

A create is validated in this order, and the first failure is the answer: the shape (400 invalid_request), the base (400 unsupported_base, naming the bases), the inputs' agreement (400 incompatible_adapters when sources differ in base, revision or module set; 400 incompatible_basis when a basis belongs to another base), then each choice against §4 (400 unsupported_adaptation, unsupported_objective, unsupported_protect, unsupported_output, unsupported_device), each naming in supported what that base runs. A refusal is an RFC 9457 problem document carrying code. A job Hanzo runs is then weighed against its payer (§10): 402 insufficient_balance or 402 spend_cap_exceeded when the payer cannot hold its first window, 503 balance_unavailable when the balance cannot be read, 403 when the caller names no wallet to charge. Those three refusals are the money wire's body, {"error": {"code", "message"}}.

§4 What each base runs

GET /v1/train/models answers this table from the same code that validates a create; the two cannot disagree.

A client on kai and a job on an engine LLM are refused as unsupported_base for that noun. qlora, basis, protect.capabilities, the basis and merged outputs, from, and lora on Kai are specified and run nowhere; asking for them is a 400, naming the supported set where there is one. A base gains a row when its executor implements the row.

For Kai the job becomes a stage file (HIP-1332): the stage named by dataset.uri, its init replaced by revision, objective.terms its terms, protect.suites with projection and distillation its protect.project (λ its strength) and protect.distill, resources.steps its max_steps. readout holds every parameter outside the head fixed (the stage's frozen). An artifact: dataset is fetched through GET /v1/train/artifacts/{sha256} and built into the stage's data.

For Kai an artifact: dataset trains a readout, and each line of it is one decision:

{"state": "my card was charged twice",
 "questions": {"queue": {"type": "choice", "instructions": "which queue",
                         "criteria": ["billing", "bug", "other"]}},
 "target": {"queue": [1, 0, 0]}}

state and the question are as POST /v1/decisions takes them; lang may name the row's language. A row asks one question, every row the same one (its type and its labels, in order), and its target is a distribution over that question's options, in order: each at or above 0, summing to 1. Any other row fails the job, naming its line. The rows split by their state, about a fifth of the states held out, the same rows every time; the head trains on the rest and the job's result grades the holdout, its one suite. The capability answers that question alone (§6): a noul by one logit, a choice or a score by one output an option, at the temperature it was graded at, which its capability.json names beside the question. A route that names another question under a key, its labels in another order included, is refused when the decision service installs it.

§5 Lifecycle

queued → running → evaluating → one of succeeded, rejected, failed, cancelled, stopped. rejected means the run finished and broke its regression budget: its artifacts are kept and it is not publishable by default. stopped means the job's own bounds or its payer's money ended it (§10): its error is the reason, what its executor stored is kept, and it is not publishable.

A job is one lead task and one join task per further machine. A task is claimed, held under a lease, reported on and ended.

Claim. POST /v1/train/jobs/claim, held up to 25 seconds, is answered one task or 204:

{"machine": "gpu-0", "platform": true, "bases": ["kai"], "devices": ["cuda", "cuda"],
 "supports": {"adaptations": ["full", "readout"], "protect": ["projection", "distillation"],
              "terms": ["pairwise_margin", "hard_margin", "consistency"],
              "outputs": ["capability", "checkpoint"]}}

machine names the claiming host. devices has one entry per accelerator, 1 to 64 of cuda, rocm, metal, vulkan, cpu. supports is what the executor's build runs. An executor acting as a principal of an org (platform false) is offered that org's tasks placed on machine: a lead, or a join once its lead has reported the address it coordinates on. Hanzo's executor sets platform: true under the platform credential (§9) and is offered the oldest open lead of any org's job that names no machines. Either is offered only a job whose base_model is in bases, whose adaptation, output, protections and objective terms are in supports, and one of whose device kinds the claim has; the task runs on the first such kind in the claim's order, on as many of those devices as the claim lists, or on fewer when its payer's money holds fewer (§10). The answer:

{"org": "acme",
 "task": {"id": "tsk_0123456789abcdef", "job": "job_0123456789abcdef", "role": "lead",
          "lease": "trl_…", "coordinator": "", "device": "cuda", "devices": 2,
          "held": 600, "report": 20},
 "job": {"id": "job_0123456789abcdef", "status": "running", "…": "the job as read"}}

org is the job's org, which Hanzo's executor acts in for every later call on the task. lease is answered once and stored as its SHA-256. devices is how many devices the task runs on and held the device-seconds held for it (§10); the job is stopped before the task runs past them. report is the most seconds that may pass between its reports.

Report. POST /v1/train/jobs/{id}/events:

{"task": "tsk_0123456789abcdef", "lease": "trl_…",
 "events": [{"type": "metric", "name": "nll", "step": 100, "value": 0.66}]}

Each report meters the task by the clock (§10); the executor reports no time. A report's events are status (a lead's running, evaluating, failed with error, or stopped when it stopped on its own; a join's done, failed or stopped), coordinator (the lead's addr, host:port), metric (name, step, value), log (line, cut to 4096 bytes), artifact (sha256) and result, at most 500 a report. A report with no events is a heartbeat, due at least every report seconds. The answer:

{"cancel": false, "held": 900, "status": "running", "seq": 42}

held is the device-seconds held now. cancel: true says the job was cancelled, or stopped at what its bounds or its payer could hold (§10): the executor MUST kill its trainer and report nothing more. Once the job has succeeded, been rejected, cancelled or stopped, a report records nothing and answers its status. A report on a failed job, or under a lease that is not the task's current one, is 409 lease_lost, and the executor stops.

A task with no report or registration for TRAIN_LEASE_TTL (default 10 minutes) loses its lease. A lost lead fails its job with executor_lost: a Kai run resumes only where its checkpoint is, so it is not moved.

§6 Artifacts

An artifact is one object named by the SHA-256 of its bytes. A directory is packed as a tar of its files in name order, with zero times, owners and modes 0644, so the same files are the same artifact. The executor asks POST /v1/train/jobs/{id}/artifacts (task, lease, kind, name, sha256, size, meta) and is answered stored: true when the org already holds those bytes, or a grant:

{"sha256": "…", "stored": false,
 "upload": {"method": "PUT", "url": "https://…", "headers": {"x-amz-checksum-sha256": "…",
            "content-type": "application/x-tar", "content-length": "1048576",
            "x-amz-server-side-encryption": "aws:kms",
            "x-amz-server-side-encryption-aws-kms-key-id": "train-acme"},
            "expires": 1800007200}}

The PUT is accepted only with those headers, which the signature covers: that checksum, that length, and encryption under the org's own key. Over 64 MiB the grant is parts instead, each {number, offset, size, url, headers} a presigned PUT of one 64 MiB range of a multipart upload opened under the org's key. A grant lives two hours: every part's address is signed when the grant is, and the largest artifact, 64 GiB, takes about that long at 80 Mbit/s. The artifact is stored once an artifact event names it and the object reads back as those bytes, that size, encrypted under the org's key; an object that reads back otherwise is deleted.

The org's key is minted into KMS the first time the org stores anything, 32 random bytes under the key id train-<org>, and only S3 reads it; the cloud keeps no key material and answers none.

The org's own rows are uploaded the same way: POST /v1/train/artifacts (kind: dataset, name, sha256, size) answers the same grant for JSON Lines bytes (application/jsonl), and a job names them as dataset.uri artifact:<sha256> (§3). GET /v1/train/artifacts lists the org's objects, one per sha256, with what produced each, whether it is published and when retention deletes it; GET /v1/train/artifacts/{sha256} answers one with a download URL; DELETE /v1/train/artifacts/{sha256} deletes one, refused 409 published while a publish names it and 409 in_use while a job that has not ended reads it. A job reads an object when one of the fields that name an artifact to read — dataset.uri as artifact:<sha256>, adaptation.basis, from, an element of protect.capabilities or adaptation.sources — is its sha256; no other mention counts. An object stored more than TRAIN_RETENTION_DAYS (default 30; 0 keeps everything) ago that is not published and that no job which has not ended reads is deleted by the hourly sweep. The org may hold at most 128 GiB granted and not confirmed (409 pending_limit past it); bytes whose every grant lapsed unconfirmed are deleted by the same sweep, their multipart uploads aborted. Each deletion the sweep makes is on the platform's audit trail before it is made, and a deployment with no trail deletes nothing on its own. A deleted object's artifacts read deleted.

Bytes the org stored before objects were encrypted are not held as its own: a registration of them is answered a sealed grant, and their read-back seals them.

• checkpoint: the trained model whole (Kai: kai.json, model.safetensors, tokenizer/).
• capability: what the job taught, apart from its base: capability.json (the base and its SHA-256, the adaptation, the protection, the evaluation, and form) and its tensors — delta.safetensors, the changed tensors as W − W₀ (form delta, a full update); lora.safetensors, the adapter's factors per target (lora); or coefficients.safetensors and residual.safetensors over a basis the manifest names by SHA-256 (basis). Applied to that base it is the trained model.
• lora: the adapter alone, as the engine loads it.
• basis: a subspace, basis://<base>/<name>, versioned by the jobs that extend it: basis.json (layer names, rank, singular spectrum, explained variance by rank, the capabilities it was built from, their principal-angle matrix, functional reconstruction at r = 4, 8, 16 and 32, residual energy by capability, the protected directions, and its parent's SHA-256 and version) and basis.safetensors.
• merged: a base with a capability or adapter folded in.

Publishing. POST /v1/train/jobs/{id}/publish publishes a succeeded job's output (a rejected one's with force: true, which the record keeps) under name, the output's name when absent; a name the org already published moves to this job. A capability is published with its route, the shape of a release's routing.json (HIP-1332):

{"name": "triage",
 "route": {"keys": ["topic"],
           "options": {"topic": {"type": "choice", "labels": ["billing", "bug", "other"]}}}}

keys are the question keys it answers, 1 to 64, each 1 to 256 bytes with no control characters and none twice. options narrows a key to one question: one under that key routes here only when its type (choice, score or noul) and its option labels, in order, are exactly these. Options for a key the route does not answer, a question with no labels, and a label twice are 400. Two capabilities of one org may not claim one question — both on a key with no options, or both naming the same question under it — so a publish that would is 409 route_conflict. The record carries the base's SHA-256 from the job's result, and how the capability is served, its form: head for a readout, which applies over the frozen base and shares its encoder, full for every other adaptation. It is a different fact from capability.json's form, which says how the tensors are encoded. A route is refused for an output that is not a capability.

GET /v1/train/published answers every org's published capabilities, what the decision service installs, under the platform credential alone (§9):

{"revision": "4f1c…", "changed": true,
 "data": [{"org": "acme", "name": "triage", "sha256": "…", "base_sha256": "…",
           "form": "full", "route": {"keys": ["topic"]}, "published": 1800000000,
           "url": "https://…", "expires": 1800000600}]}

revision names the list: it moves when a capability is published, replaced or moved to another name, and at nothing else. A poll that sends ?revision= with the one it holds is answered changed: false and no data while it is current; one without it gets the whole list with download URLs that live 10 minutes. An org whose store cannot be read, or that the serving process holds and does not own, fails the answer (503) rather than reading as having published nothing.

§7 Clients

A client holds a model in the engine's memory, which every org's inference shares, so until the engine isolates orgs only a SuperAdmin (HIP-0118) may create one; anyone else is refused 403. A create is validated like a job's (§3, §4), then forwarded to the engine (ENGINE_UPSTREAM); the client's id is recorded under the org. Every other client operation looks the id up in the org's store first, so another org's client answers exactly as an unknown one does, and one the engine no longer holds leaves the org. An org holds at most TRAIN_CLIENTS (default 2) live clients and the engine at most TRAIN_ENGINE_CLIENTS (default 2) across every org, past which a create is 503 engine_full; an engine serving no training plane answers 503 training_unavailable. save_weights writes the adapter on the engine, as the engine's wire says; exporting it as an artifact is not specified.

§8 Evaluation and research

A job with protect or evaluation is evaluated by its executor before it ends: the base and the trained model on the target and protected suites of the dataset's validation split, per suite accuracy, log loss and 15-bin expected calibration error. The verdict is accepted when every protected suite stays within the budget, else rejected; the target suites' change is reported and does not decide. The executor files each suite as a research run (HIP-1145) under its credential's project, the base as the baseline, and reports the project and run ids, which the job carries in result.research.

HIP-1334 replaces protect.budget and evaluation with a stored gate the job names, moves metrics, artifacts and the result into the job's run, and makes publishing require a passed claim on that run.

§9 Tenancy

The org is the gateway's verdict, never a body, query or path value (HIP-0026). A job, task, client or artifact of another org answers 404 exactly as an unknown one does. An org's executor acts as a principal of the org whose jobs it claims; it can claim no other org's, and is offered only the tasks placed on its machine. Hanzo's executor holds the platform credential (a SuperAdmin, HIP-0118): it claims with platform: true a task of any org's job that names no machines, and makes every later call on that task acting in the assignment's org (X-Org-Id). A platform claim by anyone else, and a report or registration on a task Hanzo runs under any other credential, is 403. Each platform claim is put on the platform's audit trail — the org, job, task, machine, device class and count, the rate, the seconds held and the payer — before it is answered; one the trail cannot record, or a deployment with no trail, is answered 503 and the task is given back. A task's reports need its lease besides the credential. Artifact grants are scoped to one key under the org's prefix and are accepted for two hours, after which bytes put under them and never confirmed are deleted (§6); download URLs live 10 minutes. A platform claim, a payer's holds (§10) and GET /v1/train/published read the org stores held by the process that serves them.

§10 Money

The plugin declares Price: cloud.Metered. Every charge goes through the org's meter (HIP-1313) to a payer's wallet, and every rate is a row of the meter authority under product train, with a compiled floor when there is none:

Who pays. A job that names no machines is run by Hanzo's executor and its compute is charged to the wallet that created it. A job on the org's linked machines runs on the org's own hardware: its seconds are counted and its max_seconds holds, and it is charged nothing.

Hold a window, charge what ran. A task may use only the device-seconds held for it. A claim holds the first window, 300 seconds of wall time on each of the task's devices. The executor reports at least every 20 seconds (§5), and each report meters the task by the clock: its devices times the seconds since it was claimed, no more than is held, at the rate read when the task was claimed; the report is charged what that adds. When less than 180 seconds a device is left, the report asks for the next window: 300 seconds a device, clipped to what max_seconds and budget leave, then weighed against the payer's balance and spend caps — a project's cap as hard as at create, when the job's project was claim-bound — together with everything the payer has committed that the ledger does not know yet: its live holds beyond what they used, and its charges not yet sent, in every org the serving process holds. One payer's weighings and holds are taken one at a time, so two claims or reports cannot weigh the same balance. Granted, held grows. Refused, the task runs what is held and asks again at every report; once less than 40 seconds a device is left, the job ends stopped with the refusal's reason — max_seconds, budget, insufficient_balance, spend_cap, no_payer — and the report's answer says cancel. A balance or a hold that cannot be read is answered the same way, and a job it stops is stopped balance_unavailable. So a job is never charged past what its payer could pay when its last window was held, and an executor that reports on time and obeys cancel never runs past it.

A report's seconds and the charge they owe are recorded together, and the charge is sent to the ledger once the record is durable; a report that cannot be made durable is answered 503 and its charge is sent by the next report, the answer to its retry, or the hourly sweep. The ledger knows each charge by its name, so one sent twice is charged once.

A platform claim whose payer cannot hold a window on every device the claim offers is given fewer, halving down to one. A job whose first window its bounds or its payer refuse outright is ended stopped before it runs; one whose payer's money is held by the payer's other work, or whose balance cannot be read, waits for a later claim.

Ends. A task that goes silent is charged through its last report, and what it held beyond that is released when its lease lapses. A job that succeeds, is rejected, fails, is cancelled or is stopped releases what its tasks held; the seconds a cancelled task ran after its last report are not charged.

Create. A job Hanzo runs is refused at create when its payer cannot hold one device's first window on its own — 300 seconds clipped to max_seconds, costing at most budget — at the job's device class (cpu when that is all it allows, else gpu): 402 insufficient_balance, 402 spend_cap_exceeded, 503 balance_unavailable (§3). A job admitted while the payer's other work holds its money waits for a claim.

Storage. Stored objects are charged by the GB-month for as long as they are kept, accrued hourly: each payer of an org carries the second through which its storage was charged, and each hourly sweep charges the span since at the bytes it keeps now: every stored object, whatever produced it and however that job ended, from the first sweep after it is stored. A deleted object stops being charged at the next sweep. Bytes granted and not confirmed are not charged; they are bounded and deleted once their grants lapse (§6). Each debit is named by its span, so a span is charged once, and the span's watermark is made durable before the debit is sent. An upload or an executor's output is granted only when its payer holds a month of what it keeps in the org, what the org has pending, and the declared bytes (402/503 as a create).

Client compute is charged per engine-second of each forwarded operation that succeeds.

§11 Events and observability

It publishes nothing on the bus. A job's events and metrics are its own record, read back through /v1/train/jobs/{id}/events and /metrics. Beyond the request span every route gets, it emits structured log lines at each job transition.

§12 Stage

ga for every org, with no flag (HIP-0139 §8), and this HIP declares capability: train (§5). Money is the gate: a job Hanzo runs needs a budget and a wallet that holds its first window (§10), and every new stored object a wallet that pays a month of what it keeps. An org has at most 64 jobs that have not ended (409 job_limit), and a job keeps at most 100,000 log and metric events. Clients stay a SuperAdmin's (Security Considerations).

§13 Upstream

It forks no project. The client wire is the engine's, a wire we implement. Low-rank adaptation, shared subspace bases and gradient projection are published methods; no code of theirs is used.

§14 Migration

/v1/train replaces /v1/training/* (the engine serves its wire at /v1/train/clients) and /v1/ai/finetune/* (hanzoai/ai). Neither answers, and neither is an alias.

In hanzoai/ai, delete routers/finetune_router.go, controllers/finetune.go, controllers/zap_finetune_test.go, object/finetune_job.go, object/finetune_runtime.go, object/finetune_billing.go, object/finetune_billing_test.go, object/finetune_hf.go, object/finetune_serve.go and cluster/finetune.go, and edit:

Then in hanzoai/cloud, with the new hanzoai/ai pinned: make -f mk/fleet.mk describe/ai && make closure && make -f mk/fleet.mk openapi, which drops /v1/ai/finetune/* from private.yaml and openapi.yaml and so from the generated clients and the CLI.

Rationale

One endpoint puts every training call behind the gateway's bearer and org (§9). The client wire is the engine's, so a loop written against that wire runs against it as written.

Jobs run on the org's linked machines because that is where Kai trains and where a model's data already is. A scheduler that placed Kai on cluster pods would copy a data build per run and discard the resume a local checkpoint gives. An org with no machine of its own has its job run by Hanzo's executor, which claims through the same wire as the org's own and differs only in its credential and in being charged.

A job is billed by the second inside a window held ahead of it, rather than on an estimate held at create, because no estimate bounds a run that stops early or runs long: holding five minutes at a time keeps at most one window of a payer's money committed to a task, a payer whose money runs out mid-run is stopped before what is held runs out, and a run is charged what the clock metered, never what was guessed. The cloud meters by its own clock rather than an executor's count, so no executor's word sets a charge, and a stop is a cancel because a cancel is what every executor already obeys.

A window is held the way the prepaid gate holds an in-flight call — committed, then weighed with the payer's balance and caps — and not as a ledger hold, because a ledger hold is paid only by a payment, and a payment is not metered usage: it would count toward no spend cap.

Storage is charged for as long as an object is kept, at the bytes kept each hour, because a charge taken once at store time bills an object kept for a year the same as one deleted the next day. Unpublished objects are deleted after a retention period because nothing else would ever remove what an org stopped using.

The capability artifact is a delta over a named base rather than a new model because a delta can be evaluated against its base, merged into it, or refused, and a new model can only replace one.

Security Considerations

A wrong org scope hands one tenant another's weights or data: every read is keyed by the gateway's org, and absence and foreignness answer the same 404. A forged report could mark a regressing run accepted: reports need the task's lease, fenced per claim, and the verdict records the executor and machine that produced it, but an org's own executor is trusted with its org's verdicts. A client holds a model in the engine's memory, which also serves inference, so clients are a SuperAdmin's until per-org engine isolation exists. Upload grants are write-only, single-key, and bound to the declared size and checksum, so a grant cannot overwrite another object, store more bytes than it declared, or store bytes under a hash they do not have; an object that reads back wrong is deleted. Bytes put under a grant and never confirmed are uncharged, so an org may hold at most 128 GiB of them, and they are deleted once the grant lapses. Every object is encrypted under its org's own key, which only the object store reads from KMS; a read-back refuses one that is not, and bytes stored before encryption are uploaded again sealed before they count, so a store copied without KMS discloses no org's bytes.

The platform credential reads and charges every org, so it is a SuperAdmin's alone, every platform claim is on the audit trail before it is answered — a deployment with no trail answers none — and a task Hanzo runs accepts reports under that credential only, so no org's principal can keep a task running against another payer. Every task is metered by the cloud's clock and bounded by what is held, so no executor sets its own charge; an executor that ignores cancel runs on Hanzo's devices uncharged. Deletion by retention destroys an org's data on the platform's schedule, so each one is audited before it is made, and nothing published or in use is ever taken.

References

• HIP-0026 — Identity & Access Management Standard
• HIP-0043 — Hanzo Engine — LLM Inference Engine Standard
• HIP-0106 — The Hanzo Plugin Contract
• HIP-0118 — SuperAdmin & Tenant Isolation Model
• HIP-0139 — Capability
• HIP-1145 — Research — The Experiment Record
• HIP-1313 — Usage — The Metered Record
• HIP-1332 — Kai — The Decision Model and the Decision Plane
• HIP-1334 — Research — The Unified Research Runtime

Copyright

Released under CC0 1.0 Universal Public Domain Dedication.