HIP-0050: Hanzo Edge — Edge AI Runtime Standard

Abstract

This proposal defines Hanzo Edge, the on-device AI inference runtime for mobile, web, and embedded platforms. Edge is the lightweight counterpart to Hanzo Engine (HIP-0043): where Engine runs on cloud/datacenter GPUs, Edge runs on end-user hardware -- iPhones, Android phones, web browsers, and embedded ARM devices.

Edge is built on the same Rust ML framework (Candle, at ~/work/hanzo/ml) as Engine. Both share the same model format and quantization pipeline. A model quantized for Engine can be further compressed for Edge deployment. This shared foundation means a single model development workflow produces artifacts for both cloud and on-device inference.

Edge is optimized for small Zen models: zen3-nano (4B parameters) and zen4-mini (8B parameters) at 4-bit quantization. It provides streaming inference within fixed memory budgets, local MCP tool execution, and platform-native SDKs for Swift (iOS), Kotlin (Android), JavaScript/WASM (Web), and Rust (embedded).

Repository: github.com/hanzoai/edge (Rust, built on Engine + ML) ML Framework: github.com/hanzoai/ml (Candle, Rust) Engine: github.com/hanzoai/engine (cloud counterpart) Target Models: zen3-nano (4B), zen4-mini (8B), zen3-embedding, zen3-guard Binary: hanzo-edge

Specification

}

// Cloud fallback for complex tasks let response = try await edge.chatWithFallback( messages: messages, localModel: .zen3Nano, cloudModel: "zen4", fallbackThreshold: .complexity(0.8) )

#### Kotlin SDK (Android)

```kotlin
import ai.hanzo.edge.HanzoEdge

val edge = HanzoEdge.Builder()
    .model(Model.ZEN3_NANO)
    .quantization(Quantization.AFQ_4)
    .maxContextTokens(4096)
    .maxMemoryBytes(3L * 1024 * 1024 * 1024)
    .build()

// Streaming inference
edge.chat(
    messages = listOf(
        Message.system("You are a helpful assistant."),
        Message.user("Explain recursion.")
    ),
    onToken = { token -> print(token) },
    onComplete = { response -> /* handle completion */ }
)

JavaScript/WASM SDK (Web)

import { HanzoEdge } from '@hanzo/edge';

const edge = await HanzoEdge.init({
  model: 'zen3-nano',
  quantization: 'afq-4',
  maxContextTokens: 2048,
  wasmUrl: '/hanzo-edge.wasm',  // Self-hosted WASM binary
});

// Streaming inference
const stream = edge.chat([
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'Hello!' },
]);

for await (const token of stream) {
  document.getElementById('output').textContent += token;
}

Rust SDK (Embedded)

use hanzo_edge::{Edge, EdgeConfig, Model, Message};

let edge = Edge::new(EdgeConfig {
    model: Model::Zen3Nano,
    quantization: Quantization::Afq4,
    max_context_tokens: 4096,
    max_memory_bytes: 3 * 1024 * 1024 * 1024,
    backend: Backend::Cpu,  // or Backend::Metal, Backend::Vulkan
})?;

let mut stream = edge.chat(vec![
    Message::user("Summarize this document."),
])?;

while let Some(token) = stream.next().await {
    print!("{}", token?);
}

Local MCP Support

Edge implements a local MCP client (HIP-0010) that enables on-device tool use without network access:

Local MCP Tools:
  - file_read: Read files from the app sandbox
  - file_write: Write files to the app sandbox
  - clipboard: Read/write system clipboard
  - calendar: Query local calendar events (with permission)
  - contacts: Search local contacts (with permission)
  - location: Get current GPS coordinates (with permission)
  - camera: Capture photo/video (with permission)
  - sensor: Read device sensors (accelerometer, gyroscope, etc.)

Tools are permission-gated by the host application. The Edge runtime never accesses device capabilities without explicit SDK-level authorization.

Cloud Fallback

When a task exceeds the on-device model's capability, Edge transparently falls back to cloud inference:

User Request
  --> Local inference attempt
  --> If confidence < threshold OR context > local_max:
      --> Forward to LLM Gateway (HIP-0004) --> Engine (HIP-0043)
  --> Cache cloud response locally for future reference

Fallback triggers:

• Context length exceeds local max_context_tokens
• Model outputs low-confidence tokens (high entropy)
• User explicitly requests a cloud model
• Device is thermally throttled and latency would be unacceptable

API Specification

Edge exposes an OpenAI-compatible local API for applications that prefer HTTP over native SDK calls:

The local HTTP server binds to 127.0.0.1 only and is optional (disabled by default on mobile, enabled on embedded/desktop).

Prometheus Metrics

Metrics exported on the local health endpoint with namespace hanzo_edge:

Performance Targets

Benchmarks on target hardware with zen3-nano (4B, AFQ-4):

Benchmarks with zen4-mini (8B, AFQ-4):

Edge binaries are signed with Hanzo's code signing keys. Model weights are distributed with SHA-256 checksums verified on download. The update pipeline uses certificate pinning to prevent MITM attacks on model distribution.

Relationship to Other HIPs

References

1. Hanzo Edge Repository
2. Hanzo Engine (HIP-0043)
3. Hanzo ML Framework (Candle)
4. Metal Performance Shaders
5. Vulkan Compute
6. WebGPU Specification
7. WASM SIMD
8. AWQ: Activation-aware Weight Quantization
9. GPTQ: Accurate Post-Training Quantization
10. HIP-0039: Zen Model Architecture
11. HIP-0004: LLM Gateway - Unified AI Provider Interface
12. HIP-0010: MCP Integration Standards

Copyright

Copyright and related rights waived via CC0.