atproto utils for zig zat.dev
atproto sdk zig

feat: configurable CAR size limits + devlog 005

CAR parser now accepts max_size and max_blocks options for large repo
verification. export jwt module for verify tool. devlog 005 covers the
three-way trust chain comparison (zig vs go vs rust).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

+75 -3
+67
devlog/005-three-way-verify.md
··· 1 + # three-way trust chain verification 2 + 3 + the previous devlogs covered decode throughput and signature verification as isolated benchmarks. this one puts it all together: given a handle, resolve identity, fetch the full repo, and cryptographically verify everything — zig vs Go vs Rust. 4 + 5 + ## what we're measuring 6 + 7 + the full AT Protocol trust chain: 8 + 9 + ``` 10 + handle → DID → DID document → signing key 11 + 12 + repo CAR → commit → signature ← verified against key 13 + 14 + MST root CID → walk nodes → rebuild tree → CID match 15 + ``` 16 + 17 + all three implementations do the same work: resolve the handle, resolve the DID, extract the signing key, fetch the repo CAR, parse every block with SHA-256 CID verification, verify the commit signature, walk the MST to count records, and (where possible) rebuild the MST to verify the root CID. 18 + 19 + ## the implementations 20 + 21 + **zig (zat)** — uses zat's own primitives end to end: `HandleResolver`, `DidResolver`, `car.read()` with CID verification, `jwt.verifySecp256k1`, `mst.Mst` for walk + rebuild. 22 + 23 + **go (indigo)** — uses bluesky's official Go SDK: `identity.BaseDirectory` for handle/DID resolution, `repo.LoadRepoFromCAR` for parsing, `commit.VerifySignature` for sig verify, `MST.Walk()` + `MST.RootCID()` for MST. 24 + 25 + **rust (RustCrypto)** — manual implementation since no indigo-equivalent exists in Rust. HTTP + DNS TXT handle resolution, plc.directory DID resolution, hand-rolled CAR parser with SHA-256, k256/p256 for ECDSA, recursive CBOR MST traversal. skips MST rebuild (no crate for it). 26 + 27 + ## the O(n) bug 28 + 29 + first run against pfrazee.com (192k records, 243k blocks): zig's MST walk took **79 seconds**. go finished in 6ms. 30 + 31 + the cause: `findBlock()` was doing a linear scan through 243k blocks on every lookup. MST walk calls `findBlock()` once per node (~50k nodes). that's ~12 billion comparisons. 32 + 33 + Go's `TinyBlockstore` uses a `map[string]blocks.Block` — O(1) by CID key. replaced the flat block slice with `std.StringHashMapUnmanaged([]const u8)` in zig and `HashMap<Vec<u8>, Vec<u8>>` in rust. 34 + 35 + result: 79s → 48ms (zig), 14s → 125ms (rust). 36 + 37 + ## results 38 + 39 + _pfrazee.com — 192,144 records, 243,470 blocks, 70.6 MB CAR, macOS arm64 (M3 Max)_ 40 + 41 + | SDK | CAR parse | sig verify | MST walk | MST rebuild | compute total | 42 + |-----|----------:|----------:|---------:|------------:|-------------:| 43 + | zig (zat) | 81.6ms | 0.6ms | 45.5ms | 172.6ms | **300.4ms** | 44 + | go (indigo) | 403.8ms | 0.4ms | 5.8ms | 0.0ms | **410.0ms** | 45 + | rust (RustCrypto) | 301.0ms | 0.2ms | 120.9ms | N/A | **422.1ms** | 46 + 47 + network time (handle + DID resolution + repo fetch) dominates total wall clock — 8-20 seconds depending on PDS response time. compute is under 500ms for all three. 48 + 49 + the story is different from the decode benchmarks. there, zig was 19x faster than Go. here, the gap is ~1.4x. the reason: signature verification is a single ECDSA verify (sub-millisecond for everyone), and CAR parsing on a 70 MB file is less dominated by per-block overhead than the firehose's thousands of small CARs. the MST rebuild (zig-only) is the biggest single cost — serializing 192k entries into a fresh tree and hashing. 50 + 51 + go's MST walk is fastest (5.8ms vs zig's 45.5ms) because indigo's `LoadRepoFromCAR` builds the MST in memory during CAR parse. walking it is just pointer chasing. zig and rust decode MST nodes from raw CBOR on each visit. 52 + 53 + ## the chart tool 54 + 55 + added a script (`scripts/verify_chart.py`) that parses `just verify` output and generates SVG charts — stacked horizontal bars with a dark theme. two variants: compute-only (the interesting comparison) and total (dominated by network). see the [atproto-bench README](https://tangled.sh/@zzstoatzz.io/atproto-bench) for the charts. 56 + 57 + ```sh 58 + just chart pfrazee.com # run verify + generate SVGs 59 + ``` 60 + 61 + ## what's in this release 62 + 63 + - **go-verify**: full trust chain verification using indigo 64 + - **rust-verify**: full trust chain verification using RustCrypto + hand-rolled CAR/MST 65 + - **zig verify**: O(1) block lookup fix (HashMap instead of linear scan) 66 + - **scripts/verify_chart.py**: SVG chart generation from verify output 67 + - **justfile**: `just verify`, `just chart` recipes
+6 -2
src/internal/repo/car.zig
··· 45 45 /// this is the correct behavior for untrusted data (e.g. from the network). 46 46 /// set to false only for trusted local data where you want raw decode speed. 47 47 verify_block_hashes: bool = true, 48 + /// max total CAR size in bytes. null = use default (2 MB). 49 + max_size: ?usize = null, 50 + /// max number of blocks. null = use default (10,000). 51 + max_blocks: ?usize = null, 48 52 }; 49 53 50 54 /// parse a CAR v1 file from raw bytes ··· 54 58 55 59 /// parse a CAR v1 file from raw bytes with options 56 60 pub fn readWithOptions(allocator: Allocator, data: []const u8, options: ReadOptions) CarError!Car { 57 - if (data.len > max_blocks_size) return error.BlocksTooLarge; 61 + if (data.len > (options.max_size orelse max_blocks_size)) return error.BlocksTooLarge; 58 62 59 63 var pos: usize = 0; 60 64 ··· 105 109 try verifyBlockHash(cid_bytes, content); 106 110 } 107 111 108 - if (blocks.items.len >= max_block_count) return error.TooManyBlocks; 112 + if (blocks.items.len >= (options.max_blocks orelse max_block_count)) return error.TooManyBlocks; 109 113 110 114 try blocks.append(allocator, .{ 111 115 .cid_raw = cid_bytes,
+2 -1
src/root.zig
··· 23 23 pub const json = @import("internal/xrpc/json.zig"); 24 24 25 25 // crypto 26 - pub const Jwt = @import("internal/crypto/jwt.zig").Jwt; 26 + pub const jwt = @import("internal/crypto/jwt.zig"); 27 + pub const Jwt = jwt.Jwt; 27 28 pub const multibase = @import("internal/crypto/multibase.zig"); 28 29 pub const multicodec = @import("internal/crypto/multicodec.zig"); 29 30