Measurements

In short

• Checking one proof on the Pi takes about 0.1 s with the proof tool alone, 0.19 s in the Rust endpoint and 0.24 s in the Python reference endpoint, with all the safety limits.
• At a 256 MiB memory cap every recorded check passed (231 of 231); smaller caps fail unpredictably.
• The leakage experiment, one client at a time: releasing in arrival order stopped the ordering code we tried; a fixed release time stopped the 1 s timing code, at the cost of 4 s per answer, and a pre-registered 512-answer test of what remained was inconclusive on the stock system and found nothing on the stripped image (one run); deliberate failures leaked under every policy, and a failure cap cut them at a large cost in availability. A plain explanation is in What a hacked server can still leak.
• In a model, random sleeps instead of a fixed release time keep leaking: the leak per answer shrinks only in proportion to the added wait, and enough answers reveal it.

Everything here was measured on one Raspberry Pi 4 (2 GB, Raspberry Pi OS Lite, 64-bit) with the AI server (Python EZKL prover) and the observer on a laptop (Apple M4 Pro) over a direct Ethernet link, unless stated otherwise. Dates are 2026.

Summary

How to read the leakage numbers. The server hides known bits and an observer tries to read them back with a fixed decoder. “16/16” means every planted bit was recovered. When the decoder fails, the table says how many bits it got wrong; that means this decoder did not recover that message in that run, not that the encoding carries no information or that nothing leaks. These are achieved rates for one encoder each, from one sequential client unless stated; they are not capacity bounds.

Checking one proof

Whole EZKL CLI process per check (start-up, loading, verify), tiny circuit:

What the full endpoint path costs (19 Oct, medians of 20, device/verify_cost.py):

A raw proof for the test model is 3,072 bytes. The EZKL CLI binary (23.0.5, 64-bit ARM) is 36 MB (36,091,376 bytes); it also contains the prover and setup code. One thread costs 1.7x; the launcher’s Python start-up is about 45 ms. Both could be cut (a static launcher; hashing the bundle once when it lives in a read-only image). On the laptop, proving takes about 0.5-0.8 s and an in-process verify 8-14 ms.

Near the memory limit the outcome is random (address-space randomisation): 128 MiB passed 20 of 20 runs, 144 and 160 MiB only about half; at the endpoint’s setting of 256 MiB every recorded check passed, 231 of 231.

The memory limit is not a clean floor. Address-space limit vs. how often a valid proof is accepted, in one sweep of 20 runs per limit (19 Oct; the rest abort, which the endpoint treats as an error):

Other runs the same day at 256 MiB: one first check, then 200/200 and 10/10, so 231 of 231 in all with the sweep; at 192 MiB 200/200; at 160 MiB 4/10 in a later sweep. The outcome near the limit varies from run to run and doesn’t improve steadily with more memory (address-space layout randomisation is on). At 256 MiB, 0 failures in 231 runs means below about 1.3 % at 95 % confidence if the runs are independent; not proven zero, and only for this circuit on this Pi. A spurious error would count as a server failure, so the margin matters. The memory cap is enforced on Linux only.

Attack surface

The Rust endpoint on the Pi (29 Oct)

The device now runs the Rust endpoint (static 64-bit ARM binary from CI, no Python or EZKL CLI on the device). Same Pi and setup as above; raw data in the git-ignored runs/2026-10-29_rust/.

Two fixes made the Rust endpoint faster than the Python one: each check had hashed the 4 MB bundle three times (this Pi’s CPU has no SHA instructions, about 25-28 ms per pass), and the supervisor slept 1 ms after every 64 KiB written to the checker, about 66 ms per check. Before them the Rust endpoint was about 65 ms slower per answer end to end.

The v1 image on the Pi (30 Oct). The laptop as the AI server, the image’s own settings (fixed release, 4 s, 5 ms spin): honest answers 9/9 ok, each received 4001-4003 ms after it was sent; changed outputs 3/3 and garbage frames 3/3 refused (unavailable); the endpoint reconnected to a restarted server within about 2 s. Leakage, one run each with the bits of the matrix below: the 1 s timing code gave 11 of 16 bits wrong (answers 4000.8-4001.6 ms); the failure-position code still leaked 16/16 (no cap on this image). These runs overlapped a port scan, so their timing is a smoke check only.

The image with the update store and firewall (30 Oct): port 8700 back about 21 s after power-on; honest answers 9/9 ok at 4001-4005 ms (fixed release from the signed manifest); changed outputs 3/3 and garbage 3/3 refused; replayed answers for the same question 3/3 accepted (by design).

The same pre-registered 512-answer test with a 5 ms spin (docs/EXPERIMENT_PROTOCOL.md), repeated on the v1 image (30 Oct): the held-out decoder got 0.49 and 0.52 of answers wrong (p = 0.50, honest control p = 0.69), so under the rule the result is not detected by this decoder at n = 512; signal answers were received 0.8-2.3 ms after the release time. The same test was inconclusive on the stock system (Python 20 Oct, Rust 29 Oct). One run each, on different days: this doesn’t show the image removed the trace, and a better decoder might still find one. The honest control had six late answers (one 28.6 ms late), cause unknown.

The endpoint end to end

Misbehaving server against the v1 endpoint on the Pi (19 Oct):

The first timing test (16 Oct, v0): the server delayed an answer by 1 s for each 1 bit; an observer decoded 16 of 16 bits, 0.69 bit/s, while honest answers varied by 81 ms.

Release policies on the Pi

The core experiment (19 Oct): the same workload under each of the endpoint’s three release policies, with a 4 s budget for every policy, 16 planted bits per channel, one run per cell (device/pi_matrix.sh, summarised by observer/analyze_matrix.py).

• Ordering: with arrival or fixed release the decoded bits were the same whatever the server planted, so the server’s choice of order never reached the observer. On one connection that holds by construction: the endpoint sends answers in admission order and the client’s ids increase. A separate localhost run showed v0 leaking floor(log2 k!) bits per batch of k concurrent requests with no errors (16/16, 32/32 and 64/64 bits for k = 2, 3, 4).
• Timing: the 1 s code wasn’t recovered under fixed release, but a smaller timing trace remains (below). The cost is latency: every answer waits the whole budget, about 5 times the honest median in this sequential run.
• Failures were recovered under every policy (below).

Only 4 of 64 answers failed (one chosen position in each block of 16), yet all 16 bits got through: 0.25 bit per request with this encoder.

How the failure code scales. The position code fails one answer in each batch of B answers and hides log2(B) bits in which one fails, so n answers carry (n/B) log2(B) bits for n/B failures. Sending 512 bits takes 128 failures in 2,048 answers with B = 16, or 52 failures in 53,248 answers with B = 1,024: fewer failures cost many more answers. That is one encoder; the limits for any encoder that only uses which answers fail are:

These count only which answers fail; timing, busy replies and logs are separate (see the cap_bound.py docstring).

What fixed release leaves

A longer run (128 random bits, plus 128 honest requests as a control) asked whether the 2 ms of remaining jitter still follows the server:

Without spin, answers for 1 bits left 0.66 ms earlier than answers for 0 bits (128 answers, p < 0.0001; the honest control showed nothing), and a decoder fitted on one half and tested on the other got about three bits in four right.

Without spin, the association is clear. A threshold decoder fitted on one half and tested on the other got 25 % and 28 % of the bits wrong (observer/jitter_test.py --heldout). If the channel were a symmetric binary channel with independent errors, that would correspond to about 0.14-0.19 bit per answer; it is a model-based figure, not a measured throughput, and the two error kinds were not symmetric in one half.

With a 5 ms spin (wake early and busy-wait until the release time, --spin-ms 5) no mean shift was detected (+0.08 ms, p = 0.31), but in that run the held-out decoder still got only 34 % wrong on one half (44 % on the other; honest controls 39-59 %), and a null that refits the decoder on shuffled labels does that well only 0.75 % of the time. That run used the spin code from before the K33 fix and 0.1 ms timestamps, so it was repeated.

The repeat (20 Oct) followed a protocol written before the data (docs/EXPERIMENT_PROTOCOL.md): 512 seeded random bits (249 ones), the fixed spin code, microsecond timestamps, and an honest run of 512 as the control. Both runs completed 512 of 512 answers, every one accepted, each received by the observer 4000.96-4003.84 ms after it was sent, that is 0.96-3.84 ms after its 4 s release time (network included). The primary test is the held-out decoder: it got 118 and 113 of 256 wrong (46 % and 44 %), and the refitting null does as well as the better half 4.1 % of the time (p = 0.041; honest control 131 and 134 of 256 wrong, p = 0.77). The protocol counts a residue as shown only below p = 0.01 and as not detected only above 0.05, so the result is inconclusive. Whatever this decoder finds is small: 143 of 256 right on its better half (about 56 %), against 72-75 % without spin (n = 128, 19 Oct). This does not show the channel is gone or bound what a cleverer observer could do, and separate runs on different days don’t show that spin caused the difference. Idle wake-up latency is one hypothesis for a residue; it hasn’t been isolated. The complete analysis output for both runs is in docs/results/2026-10-20_spin5_512.md.

In plain words:

With a 5 ms spin no mean shift was detected, and a 512-answer test with its decision rule fixed in advance was inconclusive: the held-out decoder got 143 of 256 right on its better half (about 56 %, p = 0.041, where the rule needed p < 0.01 to call it a leak and p > 0.05 to call it not detected). What this decoder finds is small, but the trace isn’t shown to be gone, and a cleverer decoder might find more. On the stripped v1 image (30 Oct) the same test found nothing (p = 0.50, one run), so the residue may depend on the system around the endpoint; that hasn’t been isolated.

Deviation from the protocol: the first attempt at both runs stopped at 149 of 512 answers because the observer’s time limit was fixed at 600 s. The limit now grows with the run’s length (commit cd0086c, which changes only device/pi_scenario.sh; the endpoint’s code hash on the Pi was the same), and both runs were repeated in full. That first attempt was also disturbed by an edit to the script while it ran; its signal run has no manifest or endpoint log, and its honest run stopped with 149 rows while the endpoint logged 150 accepts. The automatic analysis of the partial runs was printed and is not used; the registered rule was not changed after it.

A second measure on the same data (observer/estimate.py): the mutual information between the hidden bit and the answer time, with times cut into 4 bins at their quantiles, a small-sample bias correction, a shuffled-label null and a bootstrap interval, checked first on synthetic channels with known values (tests/test_estimate.py). It describes this one binned statistic; it is not a lower or upper bound on what the timing reveals.

The bias correction can make the estimate negative; that is estimator noise, not negative information. The bootstrap intervals assume exchangeable answers within one run; they are not intervals over sessions, days or encoders.

With 2 or 8 bins the no-spin result holds (p = 0.0005) and the old spin result doesn’t reach significance (p = 0.46 and 0.09). Caveats on the 128-answer runs: the observer recorded times rounded to 0.1 ms (only 19-30 distinct values per 128 answers), and the spin runs used an earlier version of the spin code that a review later found could let a later answer overtake an earlier one when several were in flight. The 512-answer repeat fixes both; at 4 bins and n = 512 it is not significant either (p = 0.069).

Random sleeps instead of a fixed release time (a model, 2 Nov)

A model, not a device measurement (observer/sleep_vs_fixed.py, tests in tests/test_sleep_vs_fixed.py): the server sends each answer on time or a fixed amount late, and the endpoint either adds a sleep drawn uniformly from 0 to R before release or releases at a fixed time after the question arrived. With uniform sleeps alone, one answer of a two-level code that shifts by d carries exactly d/R bit (d up to R): only the first and last d of the window give the bit away, and extra noise only lowers this. A server allowed to delay by up to A can carry up to log2(1 + A/R) bit per answer (exact when A/R is a whole number), so the sleep has to grow with the server’s freedom. With a fixed release time the release moment doesn’t depend on the server at all; an answer that misses it is not released and becomes a failure, which is the failure channel above.

Simulated with honest-time noise of sd 20 ms, a 200 ms shift code and 4,096 answers, the binned estimate (observer/estimate.py, 4 bins, so it understates the exact value) was significant for every sleep tried, up to sleeps of 0-4 s that add 2 s on average; fixed release gave none:

With a smaller 50 ms shift and 512 answers the same estimate no longer reached significance from sleeps of 0-0.5 s up (p = 0.076 to 0.53), although the d/R bound there is still 0.1 down to 0.0125 bit per answer: a test that finds nothing at one sample size is not evidence that the sleep closed the channel.

Several connections (smoke runs, 20 Oct)

Short checks of the new run script and the several-clients observer (CLIENTS=3, 4 s budget, one run each, not a registered experiment):

So the order a server chooses does reach an outside observer across separate connections when answers leave as they are checked, and arrival-order release stopped this code in this run. It is 8 bits in one run per rule, on one machine running all three clients; the planned two-client deadline-pressure experiment is the real test.

What these results don’t show

• One device, one network, one sequential client (ordering used concurrent requests on one connection). Several clients, other loads and packet-level observation (sizes, TCP behaviour) are not measured yet.
• Every leakage cell is one run of 16 bits, except the 128-bit residual tests and the 512-bit spin repeat.
• The release budget (4 s) was chosen by hand; the trade-off between budget, leakage and lost answers is not measured yet.
• Nothing here is a bound on what a cleverer encoder could achieve.