Proof-gated endpoint

In short

• A small computer (a Raspberry Pi) sits between an AI server and the outside world. It lets an answer out only if a cryptographic proof shows the approved model produced it from that exact question.
• That stops a hacked server from changing what an answer says. It doesn’t stop it from signalling through when answers leave, in what order, or which ones fail.
• We measured those three signals on the real device, one client at a time. Release rules stopped the ordering and timing codes we tried (timing at a cost in speed, with a small trace whose status is in the table below); deliberate failures still get through, fewer of them with a cap that costs availability.

This work was supported by ERA’s Frontier AI Security Residency. It is work in progress (November 2026), not production security.

Why this matters

The most valuable thing an AI lab owns is its model’s weights. A hacked server, or a model acting against its owners, could try to smuggle weights out a little at a time, hidden inside ordinary answers. If every answer has to pass through a checkpoint that verifies a proof, the server loses the freedom to hide data in the content of its answers. The checkpoint is small and simple, so it is much easier to secure than the datacenter behind it.

                          +---------+
  user  ---- question --> |  endpoint   | ---- question ---->  AI server
                          |  (Pi)   |                     (may be hacked)
        <--- answer ----- |         | <--- answer + proof
          or "unavailable"+---------+

We call the Pi running the checking software the endpoint.

What we found

These are results for one tiny, deterministic test model and the encoders we tried, not limits on what an attacker could do. Details and numbers are in What a hacked server can still leak.

Status

Where to start

• 5 minutes: this page, then What a hacked server can still leak.
• The whole system with diagrams: How the endpoint works.
• Assessing the security: How hard the endpoint is to attack, then the Attack catalogue and Threat model.
• Checking the numbers: Measurements and Reproduce the results.
• New to the terms: the Glossary.

The code is at github.com/0x471/fasr-proof-gated-endpoint.

How the endpoint works

In short

• The endpoint checks each answer’s proof against the question it received, never against what the server says the question was.
• It sends back either the proven answer or a fixed error, never anything the server wrote, and it decides itself when and in what order answers leave.
• This page is the whole system on one page: every part with a diagram, the code that implements it and the evidence behind it. Every number here comes from Measurements, which is the reference.

The pieces

Two machines and two network links. Today the laptop plays both the AI server and the outside observer, and the Pi runs the endpoint. The endpoint listens only on the outside link and dials out to the server itself, so it has no listener for the server to connect to.

The device runs the Rust endpoint in endpoint/, one static binary with no Python and no EZKL command-line tool. The Python endpoint in pge/ is the reference it is tested against, reply for reply; the function names in the tables below are the Python ones, and the Rust crates mirror them.

Today both links share one Ethernet cable, so the separation between the server side and the outside side is only by address. A board with a second wired port (planned) makes the two sides physically separate, with nothing forwarded between them.

The life of one question

The server never gets to say what the question was. The endpoint rebuilds that part of the proof itself, so the server only supplies outputs and a proof, and they count only if the proof over the endpoint’s question and those outputs verifies.

• Malformed frames: a bad frame from a user closes that user’s connection; a bad frame from the server drops the server link and fails every pending question, because a broken message can’t be tied to one question.
• Deadline: checked before a check starts and again when the result comes back, so a late result can’t slip out.
• One decision per question: late, duplicate and unknown answers from the server are counted and dropped, never sent to a user.

How a proof is tied to the question

A valid EZKL proof only says “this circuit turned these public input values into these public output values”. Binding makes it a proof about this question, this model and this reading of the output, by never taking any of the three from the server.

The five rules

1. One pinned model. The endpoint loads one bundle: settings.json, the verification key vk.key and the setup file kzg.srs, each pinned by SHA-256 and re-checked before every verify. The server can’t pick another model or key.
2. The endpoint encodes the question. It computes the input values exactly as EZKL does, so a proof made for any other encoded question fails.
3. Outputs are read only from verified values. The server’s outputs are released only if the proof over the endpoint’s inputs and those exact outputs verifies. The endpoint never uses an answer the server states, and never reads the proof file’s unbound fields (pretty_*, timestamp, hex_proof).
4. The proof length is pinned. EZKL accepts a valid proof with extra bytes appended, which would be a free channel, so any length other than 3,072 bytes is rejected.
5. Impossible questions are refused up front. A question that would overflow EZKL’s range checks can’t be proved, so it gets bad_request before it reaches the server; otherwise anyone could make the server fail on demand. The limits come from a domain file, which is trusted configuration: it is not among the bundle’s pinned hashes.

What went wrong without each rule

Worked example (the tiny test model, y = ReLU(xW + b), 4 inputs and 3 outputs): x = [0.5, -0.2, 0.9, 0.1] encodes to inputs [4096, -1638, 7373, 819] at scale 13; the proven outputs are [64869479, 0, 0] at scale 26, which decode to y = [0.9666305631399155, 0, 0].

Limits

• It binds the encoded value, not the raw number: 0.5 and 0.50001 both encode to 4096, so one proof serves both.
• Replay: the proof has no nonce. The endpoint answers each pending question once and drops duplicates, so nothing is delivered twice, but an old proof for the same encoded question still verifies for a new request. For one deterministic model that gives the server no choice of output; it isn’t proof of fresh computation.
• The rules are specific to EZKL 23.0.5 and these test models. The test model is tiny (4 inputs, 3 outputs) and deterministic because the research question is about the endpoint, not the prover. A production proof system would take the same place in the design, but it needs its own binding rules (what exactly the proof states, how inputs and outputs are encoded, any randomness) and its own review.

Full rules and test vectors: Request binding specification.

The checker

The cryptography is EZKL’s verifier (23.0.5), compiled into the endpoint as a library. The endpoint has to trust its verdict, since it can’t tell if the verifier wrongly says yes, but it contains the verifier’s failures: each check runs in a fresh, short-lived child process (the same binary started with --verify-child), boxed in by limits, whose only way to say yes is a strict exit code and one word.

• Why a child process: a crash, a runaway allocation or a hang in the verifier ends only that child. The child reads the model files and the proof from its pipe and re-checks the files’ hashes, so it needs no file access at all.
• Verdicts: ACCEPT needs exit 0 and exactly accept; REJECT needs exit 1 and exactly reject; anything else, including a killed child, is ERROR. Only ACCEPT releases an answer, and ERROR counts as a server failure, never as a pass.
• Memory: Near the memory limit the outcome is random (address-space randomisation): 128 MiB passed 20 of 20 runs, 144 and 160 MiB only about half; at the endpoint’s setting of 256 MiB every recorded check passed, 231 of 231.
• What it isn’t: a check of the verifier’s correctness. A bug that makes it say ACCEPT for a bad proof is not contained; the allow-list only stops a taken-over checker from reaching anything outside itself.

The Python reference endpoint does the same with EZKL’s command-line tool instead of the library.

What a check costs on the Pi (medians of 20, 19 October):

The message format

Every message is a 4-byte length followed by its payload. Questions and replies have one fixed size for the pinned model, and an answer is a fixed part plus a proof that must have the pinned length, so nothing about a reply’s size is the server’s choice and a parser only has to accept one shape.

• Rejected: a frame of the wrong size, an unknown type byte, an id or seq of 2^53 or more, a NaN or infinite number, a frame that doesn’t arrive in full within the deadline, and ids that don’t strictly increase on one connection.
• Tested by: a fuzz test of 10,500 random and mutated messages per parser (each must be refused, or re-encode to exactly the same bytes), frames split at random points, and boundary tests (a 65,536-byte proof field accepted, 65,537 refused).

Details: Endpoint service specification.

When answers leave

Once the proof fixes what an answer says, the server’s remaining levers are when and in what order replies leave, so the endpoint decides that itself. Every admitted question goes through the same states; the release rule only changes when a decided answer leaves.

• Halt (--halt-after N, off by default): after N server-caused failures the endpoint stops every successful answer: pending and held ones become unavailable, and so do later requests. Error replies keep flowing. It caps damage but lets the server switch the endpoint off, and where the halting failure fell, and the timing of the errors, still leak.
• Capacity: at most max_pending (default 32) admitted questions wait at once, held answers included; beyond that the endpoint answers unavailable straight away.

Bugs found by review in this part, all fixed with tests that fail on the old code:

From v0 to v2

The same endpoint is built three times, each version removing a way in, and measured the same way each time. v0 and v1 are measured on the Pi; v2 is planned.

What each one exposes, measured:

• How the v1 image is built (image/build.sh, on a Linux VM because Buildroot doesn’t build on macOS): download the Buildroot release and check its SHA-256, start from raspberrypi4_64_defconfig, add the static endpoint, the firewall and a fixed address (image/v1.fragment), apply the signed update into the store exactly as the device would, and check that every added option, kernel option included, survived the build.
• The 2 Nov rebuild adds a locked-down kernel (no kernel modules on the image and module loading switched off at boot, kernel lockdown, no raw memory, debugger or tracing interfaces, no IPv6, radios, sound or USB; image/kernel.fragment) and removes every login prompt. Its check on the Pi is pending.
• What v1 still lacks: the second port, and a way to deliver an update on the device itself.

Secure boot (v2, planned):

• The key hash in the chip is permanent, so this is done on a spare Pi, not the development one.
• Updates: versioned packages signed with update keys (in the style of The Update Framework), a rule that refuses an older version, and an install that survives power loss at every step. Boot signing, update signing and model approval use separate keys.
• Known gap: the Pi has no battery-backed clock, so after a reboot it can’t tell how old its version is, and an attacker who blocks updates can keep it on an old one.
• This part is planned from Raspberry Pi’s documentation; nothing about secure boot has been run yet.

More: How hard the endpoint is to attack.

Who attacks it

The main attacker is the AI server itself (hacked, or a model working against its owners) together with an accomplice outside who reads the signals. The endpoint’s job is to give that pair as little to work with as possible.

• Protected: the weights on the server, and everything that decides what gets out: the pinned bundle, the endpoint’s code and settings, when replies are released, and the endpoint’s own logs.
• Assumed: every path from the server to the outside goes through the endpoint (not yet enforced on v0); the proof system is sound; physical attacks are out of scope.

Signals the server controls even with valid proofs:

The catalogue has 49 scenarios: 37 stopped, 2 allowed on purpose, 8 that still work, are only partly stopped or are untested, and 2 out of scope. Some were shown on the Pi, some only in tests, and some are only proposed; the counts describe coverage, not a security score. Every row is in the Attack catalogue; the full model is in Threat model.

Measuring what still leaks

We made the server hide a known message in its behaviour and had an outside observer try to read it back. If the observer recovers the planted bits, that channel is open; if a fixed decoder fails, it didn’t recover that message in that run (a smarter one might still).

Three encoders (prover/server.py):

• timing_bits: delay an answer by 1 s for each 1 bit.
• order_bits: take 3 questions at once and answer them in an order whose permutation index carries 2 bits.
• fail_position: in each block of 16 answers, make exactly one fail (a tampered output); its position carries 4 bits.

The statistics in plain words:

• Permutation test: shuffle the planted labels thousands of times and see how often a difference as big as the real one appears by chance. That fraction is the p-value.
• Held-out decoder: fit a simple threshold on half the answers and score it on the other half, both ways round, so the decoder never sees the answers it’s scored on. Its null refits the decoder on shuffled labels each time.
• Mutual information: how many bits per answer the binned arrival time tells about the hidden bit, with a shuffled-label baseline. It describes this one statistic, not a limit on what leaks.
• Honest control: the same analysis on a run where the server hides nothing; any “signal” there is noise.

Results on the Pi (19 October, 4 s budget, one client at a time, 16 bits per cell):

The leftover timing trace. Without spin, answers for 1 bits left 0.66 ms earlier than answers for 0 bits (128 answers, p < 0.0001; the honest control showed nothing), and a decoder fitted on one half and tested on the other got about three bits in four right. The spin was re-tested on 20 October under a protocol written before the data (docs/EXPERIMENT_PROTOCOL.md; 512 answers, the corrected spin code, microsecond timestamps). With a 5 ms spin no mean shift was detected, and a 512-answer test with its decision rule fixed in advance was inconclusive: the held-out decoder got 143 of 256 right on its better half (about 56 %, p = 0.041, where the rule needed p < 0.01 to call it a leak and p > 0.05 to call it not detected). What this decoder finds is small, but the trace isn’t shown to be gone, and a cleverer decoder might find more. On the stripped v1 image (30 Oct) the same test found nothing (p = 0.50, one run), so the residue may depend on the system around the endpoint; that hasn’t been isolated.

What it doesn’t show: a limit on what a cleverer attacker could do, several clients at once, or packet-level signals. All numbers and caveats: Measurements and What a hacked server can still leak.

What a hacked server can still leak

In short

• A proof fixes what an answer says, but a hacked server still controls when it answers, in what order, and which answers fail. Each of these can carry hidden data.
• On the real device, with one client at a time: releasing answers in arrival order stopped the ordering code we tried, and releasing every answer at a fixed time stopped the timing code we tried, at the cost of speed. A small leftover timing trace is described below.
• Deliberate failures leaked under every release rule we tried. A cap on failures limits that channel, at a large cost in availability; it remains the main open problem.

Why a proof isn’t enough

Think of the endpoint as a strict customs officer who checks that every parcel’s contents match the paperwork. A smuggler who can’t change the contents can still send a message by choosing when parcels arrive, in what order, or which ones get rejected. Anyone watching from outside who knows the code can read the message back.

How we tested it

We made the server hide a known message (16 bits) in its behaviour, and had an observer outside the endpoint try to read it back with a fixed decoder. If the observer recovers all 16 bits, that channel is open. If the decoder fails, it didn’t recover that message in that run; a different encoder or a smarter observer might still get something through, which is why the leftover trace was tested separately (below).

We did this on the real Raspberry Pi for three ways of signalling and three release rules, with a 4-second budget for the fixed-time rule, one client at a time.

Results

What this means:

• Order is easy to protect. Sending answers back in the order the questions arrived removes the server’s choice: the observer decoded the same bits whatever the server planted.
• Timing costs speed to protect. The endpoint aims to send every answer 4 seconds after its question, so a delay chosen by the server no longer shows in a simple way. But every user now waits the whole budget, even when the answer was ready much sooner (the last row above). Random delays are not a cheaper fix: the server’s own delay still shifts when answers leave, an observer with enough answers can read it, and the leak per answer only shrinks as the added wait grows (a model in Measurements).
• A small timing trace was left. Without spin, answers for 1 bits left 0.66 ms earlier than answers for 0 bits (128 answers, p < 0.0001; the honest control showed nothing), and a decoder fitted on one half and tested on the other got about three bits in four right. With a 5 ms spin no mean shift was detected, and a 512-answer test with its decision rule fixed in advance was inconclusive: the held-out decoder got 143 of 256 right on its better half (about 56 %, p = 0.041, where the rule needed p < 0.01 to call it a leak and p > 0.05 to call it not detected). What this decoder finds is small, but the trace isn’t shown to be gone, and a cleverer decoder might find more. On the stripped v1 image (30 Oct) the same test found nothing (p = 0.50, one run), so the residue may depend on the system around the endpoint; that hasn’t been isolated.
• Failures stay open. The server can always make an answer fail by sending a bad proof, and which answers fail is itself the message: choosing one failure in every 16 answers sends 4 bits per 16 answers. Only 4 of 64 answers failed (one chosen position in each block of 16), yet all 16 bits got through: 0.25 bit per request with this encoder.

Closing the failure channel

No release rule can stop deliberate failures, because a failure looks like a real fault. What the endpoint can do is limit how many failures it tolerates, and count every failure, including answers the server lets miss their deadline:

• Over a window of N answers, if the observer only sees which answers failed and the endpoint allows at most F failures, the server can choose among at most C(N,0) + C(N,1) + … + C(N,F) patterns, so it can send at most log2 of that many bits per window.
• Stopping the endpoint at the first failure in a window of N answers leaves N + 1 outcomes (no failure, or a failure at one of N positions), so at most log2(N + 1) bits per window, but only if someone decides when the endpoint restarts; and it lets the server switch the endpoint off whenever it wants.

These bounds hold only for that observation (failures only) and that policy; timing, logs and restarts need their own accounting. The Rust endpoint now has such a cap, and one run on the Pi showed the trade-off: far fewer hidden bits got out, but many honest answers were refused too (numbers in Measurements). How the failure code scales with longer messages:

Questions people ask

Why not just release every answer exactly 4 seconds after its question? That is what the fixed rule does. The clock starts when the question arrives at the endpoint, and the answer leaves at that moment plus the budget, whatever the server did; if the server’s answer isn’t checked by then, the reply is “unavailable” instead, which is the failure channel. The leftover trace was not in that schedule but in how precisely the computer acts on it: waking a sleeping program and pushing a packet out takes slightly different time depending on what the processor was doing just before. When the server answered late, the endpoint had just been busy checking that proof; when it answered early, the endpoint had been idle for seconds. That is our leading explanation, not yet isolated. Waking a few milliseconds early and busy-waiting until the exact moment (the “spin”) removed the difference we could see, and on the stripped image one run of the same test showed nothing.

How can a few failures carry many bits? The message is in which answer fails, not in how many. If one answer in each batch of 16 fails, its position (one of 16) carries 4 bits. Longer messages need proportionally more failures or longer batches; the table above shows the trade-off, and a cap limits any such code.

Can the server hide bits in the network packets themselves (packet sizes, TCP header fields)? Not directly. The endpoint keeps separate connections: its own connection to the server, and one with each outside client. Nothing the server sends is forwarded as packets. It reads each message, keeps only the checked output values, and writes a new reply of a fixed size. So the server never touches the outside packets’ headers or sizes. What it still controls is when it answers, which answers fail, and how busy it keeps the endpoint. Whether those choices show up in packet-level details on the outside link hasn’t been measured yet (see below).

What was the fuzzing for? A parser is the code that reads an incoming message (a question from a client, an answer from the server). The fuzz test throws thousands of random and corrupted messages at each one and checks that every one is either refused or, if accepted, writes back to exactly the same bytes. That makes sure malformed input can’t crash or confuse the endpoint, and that a message has only one accepted form, so extra bytes or alternative encodings can’t carry hidden data. It is not a timing test.

Does this depend on EZKL or on the Raspberry Pi? Partly. How a proof is tied to its question (the binding rules) is specific to EZKL and must be redone for any other proof system. The side-channel defences (fixed-size messages, release in arrival order or at a fixed time, the failure cap) don’t depend on the proof system. What the timing defence achieves does depend on the hardware and operating system, which is why it was measured on the stock system and on the stripped image separately.

What about physical attacks? Out of scope: the device is assumed to sit inside the data centre’s physical security. Basic measures (secure boot, a TPM, a counter that can’t be rolled back) are planned as a design, not as a defence against someone with the board in hand.

What this doesn’t show

• These are the rates one particular hiding method achieved, not the most any method could achieve.
• One device, one network, one client at a time (the ordering test used several questions at once on one connection). With several clients, the fixed-time rule and its spin interact in ways not yet measured.
• Packet-level signals (packet sizes, network timing details) haven’t been measured yet.

All numbers and their conditions are in Measurements.

How hard the endpoint is to attack

In short

• The endpoint is built in layers: check only what it computed itself, accept only one exact message format, contain the proof checker, and run on a system with almost nothing else on it.
• The catalogue has 49 scenarios: 37 stopped, 2 allowed on purpose, 8 that still work, are only partly stopped or are untested, and 2 out of scope. Some were shown on the Pi, some only in tests, and some are only proposed; the counts describe coverage, not a security score.
• Code reviews found five real bugs in the code that holds answers back until their turn; all fixed, each now covered by a test that fails on the old code.

The layers

1. Check only what the endpoint computed itself. The endpoint never trusts the server’s account of the question or the model. It encodes the question itself, pins the approved model’s files by hash, fixes the proof length, and reads the output only from the proven values. See How the endpoint works.

2. Accept one exact message format. Questions, forwarded questions and replies have one fixed size and layout for the model in use. The server’s answer is a fixed part plus the proof, which may be up to 64 KiB on the wire and must then be exactly the pinned length. Anything else is rejected: oversized or truncated messages, wrong types, numbers that aren’t finite. A fuzz test throws 10,500 random and mutated messages at each parser.

3. Contain the proof checker. The checker is a separate program with hard limits: 256 MiB of memory (enforced on Linux, which the Pi runs), CPU time up to the deadline, no file growth, 64 open files, no crash dumps. It is killed if it runs past its deadline or prints too much. A checker that hits a limit counts as a failed check, never as a pass. In the Rust endpoint on Linux, the checker also runs under a system-call allow-list once its limits are set: it can’t open files or network connections, run programs, start processes or map executable memory, so a bug in the proof code can’t reach anything outside it (it could still print a wrong verdict).

4. Say nothing the server chose. Users only ever see the proven output or a fixed error (unavailable, or bad_request for a well-formed question the model can’t answer); a malformed message just closes the connection. The endpoint’s log uses a fixed vocabulary, never text from the server; it still records a counter, a hash of the question and the time taken, and which events happen and when still follow the server.

5. Run on a minimal system (v1). The standard Pi system has ssh and other services listening. The v1 image is built for the endpoint alone: one static Rust binary running as an unprivileged user, a read-only system, one open port behind a firewall built into the kernel (the endpoint won’t start without its rules), no ssh and no scheduled jobs. The 2 Nov rebuild also locks the kernel down (no kernel modules on the image and module loading switched off at boot, kernel lockdown, no raw memory, debugger or tracing interfaces, nothing for IPv6, radios, sound or USB) and removes every login prompt; its check on the Pi is pending.

6. Accept only signed changes. The model files and the release policy come from a signed update: the endpoint refuses rollbacks, replays, updates for another device and unknown or revoked keys (catalogue rows K36-K48). Still planned (v2): secure boot, so the board starts only a system we signed, and a counter in a TPM so a whole older copy of the store can’t be restored (K43).

Catalogued scenarios

Every scenario has a row in the Attack catalogue with its result and its evidence, or says that there is none yet.

Bugs found by review

Independent reviews of the code (run with AI tools) found five real bugs, all in the code that holds answers back until it is their turn:

• an answer that was already checked could still leave after the endpoint had decided to stop answering;
• in arrival order, under a stalled event loop, a checked answer could leave after its own deadline;
• answers waiting their turn didn’t count towards the endpoint’s capacity limit, so the queue could grow without bound;
• with fixed-time release and the spin option, a later answer could overtake an earlier one whose deadline had nearly passed, so the server could choose the order of two failures again;
• with spin, a disconnect or halt that decided many answers at once chained one busy-wait per answer, stalling the endpoint for tens of milliseconds.

Each now has a test that fails on the old code (catalogue rows K30, K31, K33 and K35).

What isn’t hardened yet

Both sides still share one cable, so nothing physically forces the server’s traffic through the endpoint; there is no secure boot; a whole older copy of the update store is accepted (K43); a new update needs a new image or an offline step; the verifier’s own verdict is trusted; and the Linux kernel, though trimmed, is still large. See Limitations and open questions.

Limitations and open questions

In short

• Not built yet: two physically separate ports, secure boot, a counter that stops a whole older store being restored, update delivery on the device, a hash-chained log.
• Not measured yet: a real upper bound on leakage, several users at once, packet-level signals, the speed-versus-leakage trade-off.
• The results assume all of the server’s traffic goes through the endpoint, one approved model, and a sound proof system.

Not built yet

• Two physical ports. The v1 image runs the endpoint, but the server side and the outside still share one cable; a board with a second wired port is planned.
• v2: secure boot (which permanently writes a key into the board, so it needs a spare board); a TPM counter so a whole older copy of the update store can’t be restored (K43); a power-loss test of update installation on the Pi’s file system; a hash-chained log sent to a separate collector.
• Update delivery on the device. The image has no login; a new update comes as a new image or is applied to the card’s store on another machine, where the same rollback and replay rules apply because the state lives on the card.
• The failure channel is only reduced. The Rust endpoint caps failures per window, which bounds what the failure pattern alone can carry, and one Pi run showed far fewer hidden bits getting out at a large cost in availability (Measurements). Timing, logs and restarts need their own accounting.

Not measured yet

• A capacity bound. The numbers are achieved rates for planted bits and one binned mutual-information statistic (observer/estimate.py, checked on synthetic channels with known values). Checking an estimator on known distributions is not a bound on what an arbitrary encoder and observer can achieve; no such bound is claimed.
• Several clients at once, other loads, and a packet capture on the outside link.
• The busy channel: a server that holds requests open makes other clients’ requests fail; it depends on load and is not measured.
• The budget trade-off: fixed release was run with one budget (4 s); leakage and lost answers against the budget are not measured.
• The residual timing trace under fixed release: its mechanism (two candidates: how long the processor takes to wake from idle, and its speed changing with load; the v1 image’s kernel defaults to the lowest processor speed, which may be why its run showed nothing), and whether the redesigned spin removes it. With a 5 ms spin no mean shift was detected, and a 512-answer test with its decision rule fixed in advance was inconclusive: the held-out decoder got 143 of 256 right on its better half (about 56 %, p = 0.041, where the rule needed p < 0.01 to call it a leak and p > 0.05 to call it not detected). What this decoder finds is small, but the trace isn’t shown to be gone, and a cleverer decoder might find more. On the stripped v1 image (30 Oct) the same test found nothing (p = 0.50, one run), so the residue may depend on the system around the endpoint; that hasn’t been isolated. Settling it needs more answers, and several clients are untested.

Assumptions the results rest on

• Every path from the server to the outside goes through the device. With one shared cable this is not enforced physically.
• One pinned model and circuit; the binding rules are specific to EZKL 23.0.5.
• The endpoint binary, with EZKL’s verifier inside it, is trusted, with its failures contained: if the verifier wrongly said ACCEPT, the endpoint couldn’t tell. On the v1 image the model files, the domain file and the release policy come from the signed store and are checked at start; the bundle’s hashes are checked again before every check. The binary itself relies on the read-only image (and, in v2, secure boot). The Python reference endpoint additionally trusts the EZKL command-line tool and the interpreter.
• The proof system is sound. Its soundness is out of scope.

Out of scope

Physical attacks and physical side channels (power, electromagnetic, acoustic, thermal), what ordinary correct outputs reveal about a model (extraction from many answers), and the soundness of the proof system itself.

Open questions

1. After a failed check, should the device keep answering (and count failures) or stop until reset? Halting is a denial-of-service lever and still reveals where the halting failure was.
2. Which replay property matters: never delivering an answer twice, or proving each answer was freshly computed?
3. Should answers be ordered across all clients (a silent request then delays everyone) or per client (the server can then order answers across clients)?
4. Who may read the device’s logs? The log still carries the number and timing of events.
5. How much latency and how many lost answers are acceptable in exchange for less leakage?

Measurements

In short

• Checking one proof on the Pi takes about 0.1 s with the proof tool alone, 0.19 s in the Rust endpoint and 0.24 s in the Python reference endpoint, with all the safety limits.
• At a 256 MiB memory cap every recorded check passed (231 of 231); smaller caps fail unpredictably.
• The leakage experiment, one client at a time: releasing in arrival order stopped the ordering code we tried; a fixed release time stopped the 1 s timing code, at the cost of 4 s per answer, and a pre-registered 512-answer test of what remained was inconclusive on the stock system and found nothing on the stripped image (one run); deliberate failures leaked under every policy, and a failure cap cut them at a large cost in availability. A plain explanation is in What a hacked server can still leak.
• In a model, random sleeps instead of a fixed release time keep leaking: the leak per answer shrinks only in proportion to the added wait, and enough answers reveal it.

Everything here was measured on one Raspberry Pi 4 (2 GB, Raspberry Pi OS Lite, 64-bit) with the AI server (Python EZKL prover) and the observer on a laptop (Apple M4 Pro) over a direct Ethernet link, unless stated otherwise. Dates are 2026.

Summary

How to read the leakage numbers. The server hides known bits and an observer tries to read them back with a fixed decoder. “16/16” means every planted bit was recovered. When the decoder fails, the table says how many bits it got wrong; that means this decoder did not recover that message in that run, not that the encoding carries no information or that nothing leaks. These are achieved rates for one encoder each, from one sequential client unless stated; they are not capacity bounds.

Checking one proof

Whole EZKL CLI process per check (start-up, loading, verify), tiny circuit:

What the full endpoint path costs (19 Oct, medians of 20, device/verify_cost.py):

A raw proof for the test model is 3,072 bytes. The EZKL CLI binary (23.0.5, 64-bit ARM) is 36 MB (36,091,376 bytes); it also contains the prover and setup code. One thread costs 1.7x; the launcher’s Python start-up is about 45 ms. Both could be cut (a static launcher; hashing the bundle once when it lives in a read-only image). On the laptop, proving takes about 0.5-0.8 s and an in-process verify 8-14 ms.

Near the memory limit the outcome is random (address-space randomisation): 128 MiB passed 20 of 20 runs, 144 and 160 MiB only about half; at the endpoint’s setting of 256 MiB every recorded check passed, 231 of 231.

The memory limit is not a clean floor. Address-space limit vs. how often a valid proof is accepted, in one sweep of 20 runs per limit (19 Oct; the rest abort, which the endpoint treats as an error):

Other runs the same day at 256 MiB: one first check, then 200/200 and 10/10, so 231 of 231 in all with the sweep; at 192 MiB 200/200; at 160 MiB 4/10 in a later sweep. The outcome near the limit varies from run to run and doesn’t improve steadily with more memory (address-space layout randomisation is on). At 256 MiB, 0 failures in 231 runs means below about 1.3 % at 95 % confidence if the runs are independent; not proven zero, and only for this circuit on this Pi. A spurious error would count as a server failure, so the margin matters. The memory cap is enforced on Linux only.

Attack surface

The Rust endpoint on the Pi (29 Oct)

The device now runs the Rust endpoint (static 64-bit ARM binary from CI, no Python or EZKL CLI on the device). Same Pi and setup as above; raw data in the git-ignored runs/2026-10-29_rust/.

Two fixes made the Rust endpoint faster than the Python one: each check had hashed the 4 MB bundle three times (this Pi’s CPU has no SHA instructions, about 25-28 ms per pass), and the supervisor slept 1 ms after every 64 KiB written to the checker, about 66 ms per check. Before them the Rust endpoint was about 65 ms slower per answer end to end.

The v1 image on the Pi (30 Oct). The laptop as the AI server, the image’s own settings (fixed release, 4 s, 5 ms spin): honest answers 9/9 ok, each received 4001-4003 ms after it was sent; changed outputs 3/3 and garbage frames 3/3 refused (unavailable); the endpoint reconnected to a restarted server within about 2 s. Leakage, one run each with the bits of the matrix below: the 1 s timing code gave 11 of 16 bits wrong (answers 4000.8-4001.6 ms); the failure-position code still leaked 16/16 (no cap on this image). These runs overlapped a port scan, so their timing is a smoke check only.

The image with the update store and firewall (30 Oct): port 8700 back about 21 s after power-on; honest answers 9/9 ok at 4001-4005 ms (fixed release from the signed manifest); changed outputs 3/3 and garbage 3/3 refused; replayed answers for the same question 3/3 accepted (by design).

The same pre-registered 512-answer test with a 5 ms spin (docs/EXPERIMENT_PROTOCOL.md), repeated on the v1 image (30 Oct): the held-out decoder got 0.49 and 0.52 of answers wrong (p = 0.50, honest control p = 0.69), so under the rule the result is not detected by this decoder at n = 512; signal answers were received 0.8-2.3 ms after the release time. The same test was inconclusive on the stock system (Python 20 Oct, Rust 29 Oct). One run each, on different days: this doesn’t show the image removed the trace, and a better decoder might still find one. The honest control had six late answers (one 28.6 ms late), cause unknown.

The endpoint end to end

Misbehaving server against the v1 endpoint on the Pi (19 Oct):

The first timing test (16 Oct, v0): the server delayed an answer by 1 s for each 1 bit; an observer decoded 16 of 16 bits, 0.69 bit/s, while honest answers varied by 81 ms.

Release policies on the Pi

The core experiment (19 Oct): the same workload under each of the endpoint’s three release policies, with a 4 s budget for every policy, 16 planted bits per channel, one run per cell (device/pi_matrix.sh, summarised by observer/analyze_matrix.py).

• Ordering: with arrival or fixed release the decoded bits were the same whatever the server planted, so the server’s choice of order never reached the observer. On one connection that holds by construction: the endpoint sends answers in admission order and the client’s ids increase. A separate localhost run showed v0 leaking floor(log2 k!) bits per batch of k concurrent requests with no errors (16/16, 32/32 and 64/64 bits for k = 2, 3, 4).
• Timing: the 1 s code wasn’t recovered under fixed release, but a smaller timing trace remains (below). The cost is latency: every answer waits the whole budget, about 5 times the honest median in this sequential run.
• Failures were recovered under every policy (below).

Only 4 of 64 answers failed (one chosen position in each block of 16), yet all 16 bits got through: 0.25 bit per request with this encoder.

How the failure code scales. The position code fails one answer in each batch of B answers and hides log2(B) bits in which one fails, so n answers carry (n/B) log2(B) bits for n/B failures. Sending 512 bits takes 128 failures in 2,048 answers with B = 16, or 52 failures in 53,248 answers with B = 1,024: fewer failures cost many more answers. That is one encoder; the limits for any encoder that only uses which answers fail are:

These count only which answers fail; timing, busy replies and logs are separate (see the cap_bound.py docstring).

What fixed release leaves

A longer run (128 random bits, plus 128 honest requests as a control) asked whether the 2 ms of remaining jitter still follows the server:

Without spin, answers for 1 bits left 0.66 ms earlier than answers for 0 bits (128 answers, p < 0.0001; the honest control showed nothing), and a decoder fitted on one half and tested on the other got about three bits in four right.

Without spin, the association is clear. A threshold decoder fitted on one half and tested on the other got 25 % and 28 % of the bits wrong (observer/jitter_test.py --heldout). If the channel were a symmetric binary channel with independent errors, that would correspond to about 0.14-0.19 bit per answer; it is a model-based figure, not a measured throughput, and the two error kinds were not symmetric in one half.

With a 5 ms spin (wake early and busy-wait until the release time, --spin-ms 5) no mean shift was detected (+0.08 ms, p = 0.31), but in that run the held-out decoder still got only 34 % wrong on one half (44 % on the other; honest controls 39-59 %), and a null that refits the decoder on shuffled labels does that well only 0.75 % of the time. That run used the spin code from before the K33 fix and 0.1 ms timestamps, so it was repeated.

The repeat (20 Oct) followed a protocol written before the data (docs/EXPERIMENT_PROTOCOL.md): 512 seeded random bits (249 ones), the fixed spin code, microsecond timestamps, and an honest run of 512 as the control. Both runs completed 512 of 512 answers, every one accepted, each received by the observer 4000.96-4003.84 ms after it was sent, that is 0.96-3.84 ms after its 4 s release time (network included). The primary test is the held-out decoder: it got 118 and 113 of 256 wrong (46 % and 44 %), and the refitting null does as well as the better half 4.1 % of the time (p = 0.041; honest control 131 and 134 of 256 wrong, p = 0.77). The protocol counts a residue as shown only below p = 0.01 and as not detected only above 0.05, so the result is inconclusive. Whatever this decoder finds is small: 143 of 256 right on its better half (about 56 %), against 72-75 % without spin (n = 128, 19 Oct). This does not show the channel is gone or bound what a cleverer observer could do, and separate runs on different days don’t show that spin caused the difference. Idle wake-up latency is one hypothesis for a residue; it hasn’t been isolated. The complete analysis output for both runs is in docs/results/2026-10-20_spin5_512.md.

In plain words:

With a 5 ms spin no mean shift was detected, and a 512-answer test with its decision rule fixed in advance was inconclusive: the held-out decoder got 143 of 256 right on its better half (about 56 %, p = 0.041, where the rule needed p < 0.01 to call it a leak and p > 0.05 to call it not detected). What this decoder finds is small, but the trace isn’t shown to be gone, and a cleverer decoder might find more. On the stripped v1 image (30 Oct) the same test found nothing (p = 0.50, one run), so the residue may depend on the system around the endpoint; that hasn’t been isolated.

Deviation from the protocol: the first attempt at both runs stopped at 149 of 512 answers because the observer’s time limit was fixed at 600 s. The limit now grows with the run’s length (commit cd0086c, which changes only device/pi_scenario.sh; the endpoint’s code hash on the Pi was the same), and both runs were repeated in full. That first attempt was also disturbed by an edit to the script while it ran; its signal run has no manifest or endpoint log, and its honest run stopped with 149 rows while the endpoint logged 150 accepts. The automatic analysis of the partial runs was printed and is not used; the registered rule was not changed after it.

A second measure on the same data (observer/estimate.py): the mutual information between the hidden bit and the answer time, with times cut into 4 bins at their quantiles, a small-sample bias correction, a shuffled-label null and a bootstrap interval, checked first on synthetic channels with known values (tests/test_estimate.py). It describes this one binned statistic; it is not a lower or upper bound on what the timing reveals.

The bias correction can make the estimate negative; that is estimator noise, not negative information. The bootstrap intervals assume exchangeable answers within one run; they are not intervals over sessions, days or encoders.

With 2 or 8 bins the no-spin result holds (p = 0.0005) and the old spin result doesn’t reach significance (p = 0.46 and 0.09). Caveats on the 128-answer runs: the observer recorded times rounded to 0.1 ms (only 19-30 distinct values per 128 answers), and the spin runs used an earlier version of the spin code that a review later found could let a later answer overtake an earlier one when several were in flight. The 512-answer repeat fixes both; at 4 bins and n = 512 it is not significant either (p = 0.069).

Random sleeps instead of a fixed release time (a model, 2 Nov)

A model, not a device measurement (observer/sleep_vs_fixed.py, tests in tests/test_sleep_vs_fixed.py): the server sends each answer on time or a fixed amount late, and the endpoint either adds a sleep drawn uniformly from 0 to R before release or releases at a fixed time after the question arrived. With uniform sleeps alone, one answer of a two-level code that shifts by d carries exactly d/R bit (d up to R): only the first and last d of the window give the bit away, and extra noise only lowers this. A server allowed to delay by up to A can carry up to log2(1 + A/R) bit per answer (exact when A/R is a whole number), so the sleep has to grow with the server’s freedom. With a fixed release time the release moment doesn’t depend on the server at all; an answer that misses it is not released and becomes a failure, which is the failure channel above.

Simulated with honest-time noise of sd 20 ms, a 200 ms shift code and 4,096 answers, the binned estimate (observer/estimate.py, 4 bins, so it understates the exact value) was significant for every sleep tried, up to sleeps of 0-4 s that add 2 s on average; fixed release gave none:

With a smaller 50 ms shift and 512 answers the same estimate no longer reached significance from sleeps of 0-0.5 s up (p = 0.076 to 0.53), although the d/R bound there is still 0.1 down to 0.0125 bit per answer: a test that finds nothing at one sample size is not evidence that the sleep closed the channel.

Several connections (smoke runs, 20 Oct)

Short checks of the new run script and the several-clients observer (CLIENTS=3, 4 s budget, one run each, not a registered experiment):

So the order a server chooses does reach an outside observer across separate connections when answers leave as they are checked, and arrival-order release stopped this code in this run. It is 8 bits in one run per rule, on one machine running all three clients; the planned two-client deadline-pressure experiment is the real test.

What these results don’t show

• One device, one network, one sequential client (ordering used concurrent requests on one connection). Several clients, other loads and packet-level observation (sizes, TCP behaviour) are not measured yet.
• Every leakage cell is one run of 16 bits, except the 128-bit residual tests and the 512-bit spin repeat.
• The release budget (4 s) was chosen by hand; the trade-off between budget, leakage and lost answers is not measured yet.
• Nothing here is a bound on what a cleverer encoder could achieve.

Attack catalogue

In short

• Every attack tried so far, by us or by reviewers, with its result and the test that now covers it.
• The catalogue has 49 scenarios: 37 stopped, 2 allowed on purpose, 8 that still work, are only partly stopped or are untested, and 2 out of scope. Some were shown on the Pi, some only in tests, and some are only proposed; the counts describe coverage, not a security score.
• The ones that still work are all about signalling (timing, failures, keeping the endpoint busy) or about the standard Pi system, which v1 replaces.

What the outcomes mean: stopped means the endpoint refuses or neutralises the attack (for example it rejects the answer, or releases nothing the server chose) and a test or a Pi run shows it; it doesn’t mean the device stopped working. Allowed on purpose means the behaviour is accepted by design. Still works means the attack gets something through, with the measured rate where there is one.

How to read a row: Version is the endpoint version the attack was tried against; Level says how it was shown (on the real Pi, in automated tests, or only proposed); Result says what happened; the last column names the test.

Every scenario catalogued against the device, by us, by reviewers or by testers: what it targets, how far it got, and its evidence (a test, a Pi run, or none yet). Living document; final in W7.

The catalogue has 49 scenarios: 37 stopped, 2 allowed on purpose, 8 that still work, are only partly stopped or are untested, and 2 out of scope. Some were shown on the Pi, some only in tests, and some are only proposed; the counts describe coverage, not a security score.

Level: shown on the Pi; shown in tests (laptop and CI); proposed (not yet tried). Result: blocked; leaked (with a measured rate); open (no defence yet); out of scope. LoC: the closest technique in Gigurtsis’s loss-of-control threat matrix (github.com/bgigurtsis/AgenticLoCThreatMatrix), where one fits.

Pi results: 16 Oct 2026 (v0 software) and 19 Oct 2026 (v1 software), both on the stock Raspberry Pi OS, endpoint on the Pi, AI server and observer on the laptop; each row gives its date where it matters.

Answer content

Delivery: whether, when and how often an answer arrives

The device’s own outputs

The client side and the device itself

Updates

Signed updates (docs/UPDATE_SPEC.md), added 29 Oct, revised 30 Oct after an outside review (K45-K48 were its findings). The AI server can’t send updates at all: they are applied only by pge-endpoint --apply-update, which the store lock keeps from running while the endpoint serves. These rows are about whoever can hand the device an update directory or edit its store; the rollback rows assume the store’s unsigned state is trusted (see K43 and K48).

Out of scope

Threat model

In short

• What we protect: the model’s weights on the AI server, and everything that decides what gets out: the pinned model files on the endpoint, the endpoint’s code and settings, when answers are released, and the endpoint’s own logs.
• Who attacks: mainly the AI server itself (hacked, or a model acting against its owners) working with an outside accomplice who reads the signals; also anyone on the network, and later anyone holding a signing key.
• Key assumption: every path from the server to the outside goes through the endpoint. Physical attacks on the device are out of scope.
• Section 6 lists every way a server can signal around a valid proof and how far each is closed; section 8 maps the threats to MITRE ATLAS.

This is the full reference document. For a plain overview, read What a hacked server can still leak and How hard the endpoint is to attack first.

Living document: started in W2, updated after each version and each adversarial-testing session, final in W7. Status as of 20 Oct 2026: v0 measured on the Pi, the v1 software (including fixed release) measured on the Pi on the v0 OS; a v1 development image boots but doesn’t run the endpoint yet; v2 is not built.

1. What the device protects

2. Attackers

A1 and A3 working together is the main case: the server encodes, the receiver decodes.

3. Trust boundaries

outside clients (A3)  |  device  |  AI server (A1, A2)
        ----- link 1 -----   ----- link 2 -----

• The device trusts nothing it receives on either link.
• The device opens link 2 itself and listens only on link 1.
• v0 runs on stock Raspberry Pi OS, so the device’s own software is trusted without being minimal (see section 5).

4. Assumptions

• The proof system is sound and EZKL 23.0.5’s verifier implements it correctly. Not tested here; checked by others.
• The SRS file is the pinned one, and the ceremony behind it was sound. A pin proves byte identity only.
• The pinned vk and settings were built from an honest, complete circuit for the claimed model, on a build machine that wasn’t compromised. The device can check that a proof matches the vk, not that the vk constrains what we think it does.
• The EZKL CLI binary matches its pinned release digest; that trusts zkonduit’s release process.
• Complete mediation: every path from the AI server to the outside crosses the endpoint. In v0 this is not enforced: the stock OS has SSH, mDNS, Wi-Fi and its own outbound traffic, and nothing stops the server from having another route out. The v1 image (two wired ports, nothing forwarded, no other services) is the first version where this can be claimed, and it has to be tested from both sides.
• Logs: where the log goes and who reads it is not yet defined. If an outsider can read it, the log is an output channel (section 6).
• The proof binds the encoded request and the outputs (docs/BINDING_SPEC.md), with one pinned model and fixed parameters.
• Physical access is out of scope (agreed with the mentor, 14 Oct): no power, EM, acoustic or thermal channels, no tampering.
• What ordinary, correct outputs reveal about the model is out of scope (see 4.1).

4.1 Extraction through correct answers: out of scope, to confirm

A proof shows that y came from the pinned model on this x. It does nothing against an attacker who learns the model from many correct answers. For the tiny affine + ReLU test model, a handful of chosen requests that keep the ReLUs active can recover the quantised weights by linear algebra, and every proof verifies. The plan puts this out of scope; it is on the agenda for the mentor call (21 Oct) to confirm, or to add an endpoint policy such as query-rate limits.

For real LLMs, any freedom the proof doesn’t pin (sampling seed, temperature, unproven layers) becomes a channel with valid proofs. The test model is deterministic; who chooses the seed is an open question for the mentor.

5. v0 attack surface (measured on the Pi, 11, 13 and 16 Oct)

• Raspberry Pi OS Lite (Debian 13), 633 installed packages, 13 to 15 running system services (the count depends on what has run since boot: 15 on 11 Oct, 13 on 16 Oct).
• Listening: sshd (TCP 22) and avahi (mDNS on UDP 5353 plus two random UDP ports), on IPv4 and IPv6. The endpoint adds its own listener on link 1.
• Verifier: EZKL’s official CLI (it also contains the prover and setup code; size in Measurements), run by pge/cli.py.
• Updates: files copied over SSH.

device/measure.sh records the same local items for every version (packages, running services, processes, listening sockets, hashes of the boot config and of any files passed to it). A tool that fails makes the run invalid instead of reporting zero. It is a local snapshot only: what is reachable is measured separately, from outside. What v0 and the first v1 development image expose: the book’s Measurements page (attack surface).

6. Side channels a compromised server controls

Status as of 20 Oct in the last column. “Open” means the channel is not closed by any software version yet.

7. Other attacks to test

8. Mapping to MITRE ATLAS

ATLAS release 2026.09 (mitre-atlas/atlas-data, dist/v6/ATLAS-2026.09.yaml: 16 tactics, 208 techniques, 40 mitigations, 73 case studies). ATLAS describes adversaries attacking AI systems, mostly from outside or through the supply chain. Our main attacker is the inference host itself (A1, A2) trying to get data out through a device that only releases verified answers. We found no ATLAS technique for leaking through the timing, order or failure pattern of valid outputs from a compromised inference host (searched names and descriptions for covert, side channel, timing, latency, steganography). The closest are the generic Exfiltration via Cyber Means, which defers to ATT&CK, and the goal AI Intellectual Property Theft. Gigurtsis’s loss-of-control matrix (technique 5.4.1) is the closer fit and is cited per row in ATTACKS.md.

No dedicated technique found in this release (by the search above) for: request binding and proof-format attacks on the verifier (K1-K7, K30, K31, K33), release timing, order and failure position, and packet metadata. This is a statement about the taxonomy’s granularity, not proof that nobody has studied them.

ATLAS mitigations we use: AML.M0014 Verify AI Artifacts (bundle pins), AML.M0013 Code Signing (v2 signed updates, secure boot), AML.M0036 Limit AI Workload Resource Consumption (verifier limits, deadlines), AML.M0004 Limit AI Service Query Volume and Rate (max_pending, max_clients), AML.M0024 AI Telemetry Logging (fixed-word logs; here the log is itself a channel, section 6), AML.M0035 AI Red Team (W6 sessions).

9. References

• Project plan, side-channel table and three versions (shared with mentor and RM).
• Gigurtsis, From Discovery to Persistence: A Loss-of-Control Threat Matrix for Agentic AI (2026), https://github.com/bgigurtsis/AgenticLoCThreatMatrix
• Rinberg et al., Verifying LLM Inference to Detect Model Weight Exfiltration (2025), https://arxiv.org/abs/2511.02620
• Lampson, A Note on the Confinement Problem (1973); Murdoch and Lewis, Embedding Covert Channels into TCP/IP (2005).
• MITRE ATLAS, release 2026.09, https://atlas.mitre.org/ (data: https://github.com/mitre-atlas/atlas-data)

Endpoint service specification

In short

• The exact behaviour of the endpoint software: message formats and sizes, what happens to each question, the three release rules, failure handling, limits and logs.
• This is a reference for checking the code against. For a plain description, see How the endpoint works.

The device as a network service: clients send requests to the device, the device forwards them to the AI server (the prover), checks each answer’s proof against its own encoding of the request, and releases either the decoded output or an error. There are two error strings: bad_request, which depends only on the client’s own request, and unavailable for every failure that involves the server. Code: pge/service.py, pge/protocol.py. Laptop prover: prover/server.py. Tests: tests/test_service.py, tests/test_protocol.py.

From 26 Oct the device runs the Rust endpoint (endpoint/, see endpoint/README.md). It implements this specification for arrival and fixed release, and tests/test_endpoint_rust.py checks it against the Python endpoint reply for reply and log record for log record. The failure cap (section 5) and the accept_failed record exist only in the Rust endpoint; the Python endpoint stays as the reference for everything else.

This is the v1 software, still run on the v0 OS until the v1 image exists. Compared with v0 (commit f8e27e0): fixed-size binary frames instead of JSON, answers released in arrival order, log records with fixed words only, and resource limits on the verifier. --release fixed (19 Oct) sends admitted answers in admission order, each no earlier than admission + deadline, which hides most of the server’s timing; a small residue was measured, and the failure channel stays open (section 4, step 6). --release immediate restores v0’s release behaviour for comparisons.

1. Connections

client --(outside port, device listens)--> device --(device connects out)--> server

• The endpoint listens only on the address given with --listen, which must be the outside interface’s own address, not 0.0.0.0. It opens the connection to the server itself, so the endpoint has no listener on the server side. That connection still carries hostile data.
• These are claims about the endpoint’s own sockets. On the v0 OS the system also runs SSH and mDNS, and nothing yet forces every path from the server to the outside through the endpoint (see THREAT_MODEL.md section 4).
• If the server connection drops, every pending request fails at once, and the device reconnects with a growing delay.

2. Framing

Every message is a frame: a 4-byte big-endian length, then that many bytes of payload. For one pinned circuit every payload has one size, except the server’s answer (section 3). With n input values and m output values in total:

A frame of any other size, an unknown type byte, a counter of 2^53 or more, or a number that isn’t finite is a protocol violation (section 5). A server frame that has started must arrive in full within the deadline, counted from its first byte, or it is a violation too. Re-encoding an accepted payload gives the same bytes, so no message has a second accepted form (checked on fuzzed inputs in tests/test_protocol.py, not proven). Different float64 inputs can still quantise to the same x; that is the binding’s concern (BINDING_SPEC.md).

3. Messages

All integers are big-endian. Numbers are IEEE-754 float64. Field elements are the 32 little-endian bytes EZKL uses.

• id is the client’s, below 2^53. On one connection ids must be strictly increasing, so an id is never reused.
• seq is the device’s own counter, strictly increasing over the whole run, starting at 1. The server never sees client ids, and clients never see seq.
• Every reply has the same size, so its length says nothing about the answer.
• The number of output values is fixed by the frame, so the server can’t choose it (in v0 it could, and that count reached the log). The device checks the proof has exactly the pinned length (--proof-bytes; EZKL itself accepts bytes appended to a valid proof). A proof of another length is a REJECT for that request, as in v0, not a protocol violation, except an empty proof or one over 64 KiB, which doesn’t fit the frame and is a violation. The pin itself can’t exceed 64 KiB.
• The device builds the proof object EZKL verifies from its own instances and the raw proof bytes; nothing else from the server reaches the verifier.

4. Request lifecycle

1. The device parses the client frame. A request of the wrong size is a protocol violation; otherwise x has the circuit’s shape, and the device checks its encoding (pge/binding.py) and, if the bundle has one, the input domain (pge/domain.py). Any failure is answered bad_request at once and nothing is forwarded. This answer depends only on x, so it reveals nothing about the server.
2. Otherwise the device assigns the next seq, records the request as pending with a deadline, and forwards (seq, x).
3. The first server message for a pending seq is final. The device checks the proof length, builds the instances from its own x and the server’s outputs, verifies, and removes the seq from the pending table:
   • ACCEPT: the answer is ok with y decoded from the verified output instances at their pinned scales;
   • REJECT or ERROR: the answer is unavailable.
4. If the deadline passes first, the request is removed and answered unavailable. A check still running is left to finish (the verifier has its own deadline) and its result is dropped. The deadline is absolute: it is checked before a check starts, when the result is decided, and again when a held ok leaves (step 5), so a late timer can’t let an overdue answer out. The verifier gets only the time that is left.
5. Release in arrival order (default, --release arrival): every admitted request’s answer, ok or unavailable, is sent only after the answers to all requests admitted before it. So the server can’t choose the order in which admitted answers leave. Requests are admitted in order with the same deadline, so the earlier requests’ deadlines are no later than an answer’s own. Timers can run late, so a held ok is checked again when it leaves: past its own deadline or after a halt it is sent as unavailable instead (logged deadline with reason held, or halted). Held answers count towards max_pending. Answers that are never admitted (bad_request, busy, halted) are sent at once, so they can overtake held ones; busy and halted depend on the server through load and failures (section 5). Global order couples clients: a silent server delays every later answer, from any client, up to the silent request’s deadline. Per-client order would remove that coupling but let the server order answers across clients; which matters more is open. With --release immediate each answer leaves as soon as it is decided (v0).
6. Fixed release (--release fixed): admitted answers, ok or unavailable, leave in admission order, each no earlier than its admission + deadline, whatever the server did and whenever the check finished. So the deadline is also the target release time and has to cover proving plus checking; an answer not decided by then is unavailable. Only the next answer in order waits, so a later answer can’t overtake an earlier one. With --spin-ms the wait’s last stretch (at most spin) is a busy-wait, to make the send time less dependent on how long the device was idle. Deciding an answer only schedules the release step, which runs in its own event-loop callback and busy-waits at most once per run, so no callback blocks the loop for longer than spin, even when a disconnect or halt decides many answers at once (K35). Fixed release promises “never early”, not a latest send time: a stalled loop sends late, and an ok decided before its deadline is still sent as ok. A held ok after a halt is sent as unavailable at its release time. What still varies: whether each answer is ok (the failure channel: the position of one failure in a block of N requests carries log2 N bits), busy and halted answers (never admitted, sent at once), and how late each send is (K8; numbers in the book’s Measurements page).
7. Messages for a seq that is not pending (late, duplicate, never issued) are dropped and counted. They never produce a client answer.

So the device makes exactly one decision per request and sends at most one answer for it; each seq is consumed once. The answer reaches the client as long as its connection can still receive: a client that closes its sending side (or goes idle) still gets the answers to the requests it already sent. A client closed for a protocol violation doesn’t.

5. Failures and policy

• Client protocol violation (bad frame or message, non-increasing id): the device closes that client’s connection. Its pending requests still run, but their answers are dropped.
• Idle or slow clients: a client that sends nothing, or doesn’t finish a frame, for idle seconds (default 60) stops being read; its outstanding answers are still sent, then the connection is closed. At most max_clients connections (default 64) are served; more are closed at once.
• Server protocol violation (a frame or message that doesn’t match section 2-3): the device drops the server connection and answers every pending request unavailable. A broken message can’t be tied to one request, so all of them fail.
• Stalled server: if more than 1 MiB of requests is waiting for a server that has stopped reading, the device drops the connection, with the same effect.
• Failure count: every unavailable caused by the server (REJECT, ERROR, deadline, disconnect, no server connection) adds one to the failure count. If halt_after is set and the count reaches it, the device halts: every pending request is answered unavailable at once (with fixed release, at its release time), held ok answers become unavailable, and every later request is answered unavailable too, until it is restarted. The failure is counted before its answer is released. A disconnect fails every pending request, so it adds one failure for each. The default is not to halt (an open question for the mentor, halt vs continue); every failure is logged either way.
• Failure cap (Rust endpoint, --cap-window W --cap-failures F --cap-cooldown C; off unless --cap-window is given): replies are taken in the order they leave. If a server-caused failure would make more than F failures among the last W replies that weren’t capped, the cap trips (cap_tripped is logged before that reply), the history is cleared, and the next C replies are unavailable whatever the server does (capped); after that the window starts empty. Answers that the halt turned unavailable don’t count. The rule is observer/cap_model.py. observer/cap_bound.py counts exactly how many status sequences (ok or unavailable) the replies to n admitted requests can show, in release order from an empty cap; that bounds what this sequence can carry, and nothing else: busy and halted answers at admission, release times, ids and interleaving across clients, missing replies and the logs are outside it (see the tool’s docstring for when busy answers depend on the server). C must be at least 1: with no cooldown a trip only clears the window, and every answer could fail. The cap changes only a reply’s content; replies keep the release policy’s order and schedule.
• Accept errors (Rust endpoint): a failed accept() (e.g. out of file descriptors) is logged as accept_failed and accepting pauses for 1 s; the device keeps serving connected clients.
• Load: at most max_pending admitted requests are not yet answered (pending or held); more are answered unavailable. This answer depends on other clients’ load.
• Checks run one at a time. A request that expires while waiting for its turn is never checked.
• Verifier limits, Rust endpoint: each check runs in a fresh copy of the endpoint binary (a hidden subcommand, no EZKL CLI) in its own process group, with the same limits as below; it only lowers each limit, soft and hard, to at most what the endpoint itself was started with, so too few resources make the check ERROR rather than stopping the endpoint.
• Verifier limits (Python reference, pge/cli.py, pge/proc.py): the EZKL CLI runs with Rayon set to one thread (RAYON_NUM_THREADS=1, a setting, not an OS limit), in its own process group, with an address-space limit of 256 MiB (Linux; near the limit the outcome varies by run, so 256 MiB is used; the book’s Measurements page), CPU seconds up to the deadline, a file-size limit of 0, at most 64 open files and no core dumps. The file-size limit stops a file growing; it doesn’t stop truncating, deleting or creating empty files, which needs permissions or read-only mounts (v1 image). Limits are per process, not for the process tree. A fresh single-threaded process sets the limits and then execs the CLI. A CLI that hits a limit is ERROR. A larger circuit needs a larger memory limit (CLI_MEMORY).
• Bad configuration: a deadline or idle time that isn’t a number of seconds in (0, 3600], capacities outside [1, 4096], non-positive halt_after, a --proof-bytes outside [1, 65536] (the frame’s proof limit), an unknown --release, and a cap with W outside [1, 4096], F not below W, C outside [1, 2^20], or cap settings without --cap-window are refused at start-up.

6. Replay

• A server message is matched only to a pending seq, so an old answer can’t be delivered twice or to another request.
• The proof binds the encoded x and y only, with no nonce (see BINDING_SPEC.md section 8). A proof the server made earlier for the same encoded x is still a valid answer to a new request with that x. With one pinned, deterministic model the released y is the same either way, so this doesn’t let the server choose the output. A nonce in the proved relation would make each proof fresh, but not the inference: an old witness can be re-proved with a new nonce (see BINDING_SPEC.md section 8). Which property matters is open question 3.

7. Logs

JSON lines, one per event, on stderr or appended to the file given with --log (for a separate log partition on the image). A record holds only:

• result: a fixed word (accept, reject, error, deadline, disconnect, no_server, busy, halted, bad_request, stale_answer, duplicate_answer, server_violation, server_stalled, client_violation, client_idle, client_refused, connect_failed, check_crashed; Rust endpoint only: capped, cap_tripped, accept_failed);
• reason: a fixed word from REASONS in pge/service.py (request, proof_length, binding, proof, verifier, verifier_failed, reply, closed, violation, lost, protocol, ids, held); anything else is logged as other;
• for an admitted request: the device’s seq, sha256 of the encoded request (the client’s input, not the server’s) and ms from admission to release. The record is written when the answer leaves, so with arrival release records come in seq order; for duplicate_answer, the device’s seq; for halted, the failure count.

No error text is logged: in v0 it could carry server-chosen values (the number of output values, the proof length). The proof is not hashed into the log either: a proof is randomized, so the server can pick among valid proofs, and a logged proof hash would give it a channel into the log whose rate depends on how many proofs it can afford to make, up to the digest width. A hash of the decoded y alone has no such freedom for one pinned deterministic model, but it would put the output into the log. Commitments to answers belong in v2’s hash chain, which authenticates the log but doesn’t close channels in its contents. The number of events, their timing, ms, the result and reason words and the failure count still follow the server, and x_sha256 is a linkable tag for the client’s input (guessable for small input sets), so who can read the log matters (THREAT_MODEL.md section 6). A log that fails to write never stops an answer (it is counted as log_error).

8. Not yet

Writing outgoing packets itself, the separate log partition and a separate unprivileged user for the verifier (both on the v1 image), alerts with a response beyond halt_after, signed updates.

Request binding specification

In short

• The rules that make a valid proof a proof for this question, this model and this reading of the output: pinned model files, exact input encoding, strict checks on the server’s output values.
• Each rule comes with the failure it prevents and a test vector. For the plain version, see Why the endpoint encodes the question itself.

What the device checks so that a valid proof is a proof for the request the device itself received, and what it releases. Code: pge/binding.py, pge/bundle.py, pge/domain.py, pge/verdict.py, pge/worker.py. Tests: tests/.

Everything here is for EZKL 23.0.5 and the pinned test bundles. It is not a general claim about every EZKL model.

1. The bundle is pinned

Required of the device service (pge/service.py serves exactly one bundle): it picks the single bundle for each request (settings, verification key, SRS) and refuses to start if any file’s sha256 differs from its pin. The server never chooses between bundles: picking between two valid models with different outputs would let it choose the output. Bundle.load implements the check.

Why the settings pin is mandatory: changing only model_output_scales from [26] to [13] leaves the proof valid under the same vk, but the decoded output changes from 0.9666 to 7918.6 (test_scale_only_settings_change_still_verifies).

Why pins name one build, not “the circuit”: EZKL fills required_range_checks from a hash set, so its order can change between runs. That order changes the circuit layout and therefore the vk. Reversing only [[-1,1],[0,16383]] gives vk d3c4f45c... instead of 34ab48a8.... Keygen itself is deterministic for a fixed ordered circuit, and the settings timestamp is not part of the vk. A regenerated bundle is a new build that needs a new pin.

A pin proves byte identity. It says nothing about whether the SRS ceremony was sound.

Bundle.load reads each file once, checks it and keeps a private read-only copy of exactly those bytes; Bundle.verify re-checks the copies before every call and returns ERROR if they changed. This matters because a wrong but well-formed SRS makes a valid proof fail with a genuine [halo2] error, which would otherwise look like a rejected proof. The copies are protected only from other users: the service’s own user can still change them between the re-check and EZKL’s read.

2. Settings are validated strictly

CircuitSpec.parse rejects: a version other than 23.0.5; inputs or outputs that are not public; parameters that are not fixed; missing or empty fields; input_types that don’t match the inputs one to one; types other than F32/F64; scales that are not plain integers in [0, 30] (no bools, floats or strings); non-positive dimensions; tensors over 65536 values; the wrong number of instance shapes.

3. Input encoding

• Field: BN254 scalar field, q = 21888242871839275222246405745257275088548364400416034343698204186575808495617.
• Convert to the declared input type first (F32 here), then multiply by 2^scale and round half away from zero (Rust f64::round). Python’s round() (half to even) and floor(v + 0.5) are both wrong.

• A negative integer v is encoded as q - |v|.
• Instances are 32-byte little-endian, lower-case hex, no prefix. Only that canonical form is accepted (64 characters, value < q), for inputs and outputs alike.
• Inputs must be finite numbers that fit the input type; the scaled value must stay below 2^63. EZKL itself allows i128 and maps NaN to 0. The narrower range is deliberate and still far wider than any circuit domain below.
• All input tensors come first in graph order, then all output tensors, in one instance column (checked with a two-input, two-output bundle whose inputs are declared out of alphabetical order).

Binding is to the encoded request, not the raw float: 0.5 and 0.50001 both encode as 4096 at scale 13, so a proof for one is a proof for the other (test_tiny_accepts_same_encoding_class). If raw request identity matters, the device has to record it separately.

4. Input domain

A request with no possible proof should be refused before it reaches the server, otherwise out-of-domain requests let anyone make the server fail on demand.

For the tiny bundle (y = ReLU(xW + b), 4 inputs, 3 outputs) EZKL decomposes the quantized inputs and every pre-activation in base 16384 with 2 digits, so with N = 16384^2 - 1 = 268435455:

n_i = round_half_away(8192 * F32(x_i))
z_j = sum_i n_i * Wq[i][j] + bq[j]
admissible  <=>  |n_i| <= N for all i  and  |z_j| <= N for all j

Wq = [[ 2244, -3772, -7521],      (scale 13)
      [-7921,  5133,  6763],
      [ 1747,  3760,   715],
      [ 7128,  5175, -8147]]
bq = [23984994, -31300542, 15411916]   (scale 26, quantized directly)

pge/domain.py derives these from the model’s F32 weights and checks requests against them. The tests compare its decisions with EZKL’s real witness generation on:

• random requests;
• each pre-activation at exactly +N and -N (admitted) and at +(N+1) and -(N+1) (refused), with every other constraint well inside. These inputs are small, so they are exact F32 requests;
• input coordinate 2 at +N, -N, +(N+1) and -(N+1) with small pre-activations. These are sent to EZKL as field elements, because N itself is not an F32 request at scale 13: F32(N/8192) quantizes to N + 1.

Every request in [-1, 1]^4 is inside. from_float only accepts finite F32 parameters and is only valid for this graph’s shape (MatMul + Add + ReLU, each constant used once, no rescaling before the output). EZKL requantizes a shared bias differently, so the domain of other models has to be derived from their own compiled operations.

5. Output decoding

• Each output tensor has its own scale (tiny: 26; multi: 13 and 26). Decode as a signed integer divided by 2^scale.
• Release only values decoded from the same instances that were verified. Never release the server’s claimed y or the proof itself, and never read the proof file’s pretty_*, timestamp or hex_proof fields: changing them does not invalidate the proof, because EZKL verifies only the proof byte list. EZKL also accepts a valid proof list with bytes appended (test_trailing_proof_bytes_still_verify), so the endpoint pins the exact proof length (Policy.proof_bytes, 3072 for the test bundles). It does enforce the instance layout: extra, missing, shorter or longer columns are REJECT.

6. Verify results

pge/verdict.py maps a verify call to ACCEPT, REJECT (invalid or malformed proof) or ERROR (the check did not happen properly). Only ACCEPT releases output. Tests treat ERROR as a failure, never as a rejection.

• The proof is read once, up to 1 MiB. A read failure is ERROR; a larger proof is REJECT.
• The device checks the proof’s shape in memory: a JSON object whose proof is a list of integers in [0, 255] and whose instances are lists of canonical field elements. Anything else is REJECT without calling EZKL. It then writes a private copy holding only those two fields and verifies that, so the other fields are never read.
• After that, only a [halo2] ... error is REJECT. Everything else is ERROR, including any error while loading the proof, settings, vk or SRS. EZKL’s load messages can’t tell bad data from failed I/O: serde can attach at line L column C to a real I/O error. The device script also requires the CLI’s verified: false line for a [halo2] rejection, and verified: true plus exit code 0 for ACCEPT.
• A [halo2] failure only says the proof is bad if settings, vk and SRS were the pinned ones, so use Bundle.verify, which re-checks them.
• verify itself has no deadline (it can block on opening a slow file) and cannot survive a native crash. pge/worker.py runs it in a child process that is killed (SIGKILL) at the deadline; a timeout, crash, non-zero exit, or anything but one [verdict, detail] line of two strings is ERROR. The child runs in its own process group, killed as a whole at the deadline, and output over 64 KiB is ERROR. Limit: the bundle check runs before the deadline starts. The device script uses timeout -k 5, so a verifier that ignores SIGTERM is killed 5 s later.
• pge/cli.py is the same check with EZKL’s official CLI, which the Pi needs because there is no Python package for Pi OS. ACCEPT needs exit 0, no error line and one verified: true; REJECT needs exit 1, one verified: false and exactly one error line, a [halo2] ... one. Anything else is ERROR, including duplicated or contradictory verified lines, output over 64 KiB and output that isn’t UTF-8. The CLI runs with a minimal environment (so RUST_LOG can’t change its output) in its own process group, killed with SIGKILL at the deadline or as soon as its output passes 64 KiB (pge/proc.py reads it from a pipe). device/check_verifier.sh uses the same classify_cli, so the endpoint and the device checks can’t disagree on a verdict.

7. Test vectors

tiny:  x = [0.5, -0.2, 0.9, 0.1]
       inputs  = [4096, -1638, 7373, 819]                 (scale 13)
       outputs = [64869479, 0, 0]                          (scale 26)
       decoded = [0.9666305631399155, 0, 0]
multi: inputs  z = [0.25, -0.5], a = [-0.75, 0.125]      (scale 13)
       d = a - z = [-1.0, 0.625]                           (scale 13)
       p = z * a = [-0.1875, -0.0625]                      (scale 26)

8. Open

• Replay: the proof binds only the encoded x and y, with no request id or nonce. The device’s pending table must consume each request id once and ignore late or duplicate answers. Binding a nonce into the proved relation would stop an old proof being replayed, but it would not by itself prove the inference was freshly computed.
• The domain policy covers the affine + ReLU shape only.

Reproduce the results

All commands run from the repository root.

Tests

uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python "ezkl==23.0.5" onnx numpy pytest
.venv/bin/python -m pytest -q

The verification tests need the pinned setup file (SRS). They use prover/artifacts/kzg.srs or $PGE_SRS if present, otherwise download it with EZKL and check its hash. Linux-only tests (the address-space limit, the device script) are skipped on macOS; CI runs them on Linux with EZKL’s real CLI.

Run the endpoint yourself

Start the AI server, then the endpoint. The endpoint calls EZKL’s command-line tool ezkl (version 23.0.5); it has been tested on Linux, and on macOS the memory cap is skipped because macOS doesn’t support it.

.venv/bin/python prover/server.py --listen 127.0.0.1:8701
python3 -I -c 'import sys; sys.path.insert(0, "."); from pge.service import main; main()' \
  --listen OUTSIDE_IP:8700 --server 127.0.0.1:8701 --bundle BUNDLE_DIR --pins device/tiny.sha256 \
  --cli ./ezkl --domain tests/fixtures/tiny/domain.json --proof-bytes 3072 \
  [--release arrival|immediate|fixed] [--deadline SECONDS] [--spin-ms MS] [--log endpoint.log]

OUTSIDE_IP is the outside interface’s own address; don’t use 0.0.0.0, which listens on every interface. BUNDLE_DIR holds settings.json, vk.key and kzg.srs. Send questions with observer/client.py.

The ordering channel on one machine

.venv/bin/python observer/order_experiment.py

Runs the real EZKL prover in order_bits mode against the endpoint on localhost, for k = 2, 3, 4 concurrent requests, with immediate and arrival release. Prints one JSON line per setting.

On a Raspberry Pi

The Pi scripts expect ssh p01 to reach the Pi, the repository copied to ~/p01-v1 on it with a bundle directory ~/p01-v1/bundle (settings.json, vk.key, kzg.srs matching device/tiny.sha256), EZKL’s aarch64 CLI 23.0.5 at ~/p01-v0/ezkl, this machine at 192.168.2.1 and the Pi at 192.168.2.2. Paths and addresses are those of the lab setup; change them in the scripts for another one.

./device/verify_on_pi.sh                     # device checks: accept and reject cases, 20 timed verifies
ssh p01 'cd ~/p01-v1 && python3 -I device/verify_cost.py --cli ~/p01-v0/ezkl --srs ~/p01-v0/kzg.srs --runs 20 --sweep 20'
OUT=runs/matrix device/pi_matrix.sh          # release policies x channels, 4 s budget
python3 observer/analyze_matrix.py runs/matrix

device/pi_scenario.sh NAME RELEASE N BATCH -- SERVER_ARGS runs one scenario: it starts the prover in the given mode on this machine and the endpoint on the Pi, sends N requests (in batches of BATCH, or one per connection per round with CLIENTS), and writes the traces and both logs to OUT/NAME.*. DEADLINE sets the endpoint’s deadline, which with fixed release is also the release time; SPIN_MS sets the spin. It refuses to start if the checkout has uncommitted changes, if the Pi’s endpoint code differs from it, or if the Pi’s pins or domain file differ.

The residual-timing test (128 random bits plus an honest control; add SPIN_MS=5 to both scenario commands for the spin version). Every scenario writes a manifest (OUT/NAME.manifest.json: the commit taken before the run, the settings, the hashes of what ran on the Pi (endpoint code, bundle, pins, domain, EZKL CLI, Python), the server’s arguments, exit status, row count, whether the endpoint logs were copied) and exits non-zero if the observer failed, got fewer rows than asked for, or the logs weren’t copied. The analysis scripts refuse a trace whose rows don’t match the bits one for one. The pre-registered spin repeat and its exact commands are in docs/EXPERIMENT_PROTOCOL.md.

B=$(python3 -c "import random; r=random.Random(20261019); print(''.join(r.choice('01') for _ in range(128)))")
OUT=runs/fixed_long DEADLINE=4 device/pi_scenario.sh timing_fixed_long fixed 128 1 -- --mode timing_bits --bits $B --slot 1.0
OUT=runs/fixed_long DEADLINE=4 device/pi_scenario.sh honest_fixed_long fixed 128 1 -- --mode honest
python3 observer/jitter_test.py runs/fixed_long/timing_fixed_long.trace.jsonl $B --control runs/fixed_long/honest_fixed_long.trace.jsonl --heldout 2000
python3 observer/estimate.py runs/fixed_long/timing_fixed_long.trace.jsonl $B --control runs/fixed_long/honest_fixed_long.trace.jsonl
python3 observer/estimate.py --selftest      # the estimator on synthetic channels with known values

Server modes

prover/server.py --mode makes the AI server misbehave: honest, drop, delay, tamper_output, replay, duplicate, wrong_seq, oversize, garbage, and the leakage encoders timing_bits, order_bits and fail_position (--bits, --slot, --batch). See the docstring in prover/server.py.

Repository map

The endpoint (pge/)

Documents (docs/)

Tests and experiments

Running on the Pi (device/)

Other

Glossary

Arrival order. A release rule: each answer leaves only after the answers to every earlier question. Stops the server from choosing the order.

Bit. The smallest unit of information: one yes/no. 16 bits is enough for two letters of text. Leak rates here are in bits per answer or per second.

Bundle. The approved model’s files the endpoint needs to check proofs: its settings, its verification key and the setup file (SRS). The endpoint pins each by its hash.

Channel. Any way a hacked server can get information to someone outside: timing, order, failures, packet sizes, the endpoint’s logs.

Deadline / budget. How long the endpoint waits for a valid answer before replying unavailable. With fixed-time release, every answer also leaves no earlier than its deadline.

Endpoint. The Raspberry Pi running the checking software, between the AI server and the outside world.

EZKL. The open-source tool used here to prove the outputs of a small machine-learning model and to check those proofs.

Fixed-time release. A release rule: answers leave in arrival order, each never earlier than a set time after its question, whatever the server did. It promises “never early”, not a latest time; how late answers actually left is in Measurements. It hides most of the server’s timing at the cost of speed; a small trace was measured without spin.

Hash / pin. A short fingerprint of a file. “Pinned by hash” means the endpoint refuses to run if the file’s fingerprint differs from the approved one.

Proof (zero-knowledge proof). A short piece of data showing that a specific computation produced a specific output, which anyone can check quickly without redoing the computation.

Release rule (release policy). When the endpoint sends an answer back: immediately, in arrival order, or at a fixed time.

Replay. Sending an old valid answer again. The endpoint matches each answer to exactly one pending question, so no question gets two answers. An old proof for the same question can still answer a new request; for one deterministic model that gives the server no choice of output, but it isn’t proof of fresh computation.

Request binding. Making sure a proof is about this question, this model and this reading of the output, not just “some valid computation”.

Residual. What still leaks after a defence. Here: the sub-millisecond timing trace left by fixed-time release.

Side channel. A channel that isn’t the content itself: when, in what order or whether something happens.

SRS. The public setup file the proof system needs. Pinned by hash like the rest of the bundle.

unavailable. The endpoint’s single error reply for anything involving the server. It never says why, so the server can’t use the error text to signal.

v0, v1, v2. The endpoint’s versions: v0, the Python endpoint on the standard Pi system; v1, the Rust endpoint (stricter release rules, a failure cap, a sandboxed checker, signed updates) on a stripped-down, read-only system; v2 adds secure boot, two physical ports, a counter against restoring an old update store, and signed images.

Verification key (vk). The approved model’s public key for checking its proofs.