this string has no description
ngerakines.me
pds_js_readonly.txt
1Read-Only PDS Implementation Plan
2
3 Overview
4
5 Implement a read-only PDS that loads repositories from CAR files on startup and serves AT Protocol read endpoints while rejecting all write operations
6 with authentication errors.
7
8 Architecture Approach
9
10 Create a new @pds/readonly package with a CLI tool that:
11 1. Parses CAR files to extract repository data
12 2. Populates SQLite storage for efficient querying
13 3. Serves read-only XRPC endpoints
14 4. Returns AuthenticationRequired for all write endpoints
15
16 Files to Create
17
18 1. CAR Parser - /packages/core/src/car.js
19
20 The codebase has buildCarFile() but no parser. Implement:
21
22 export function readVarint(bytes, offset) // Decode varint, return [value, newOffset]
23 export function parseCarFile(carBytes) // Returns { roots: string[], blocks: Map<cid, data> }
24 export async function* iterateCarBlocks(bytes) // Memory-efficient streaming
25
26 Uses existing: cborDecode, cidToString from repo.js
27
28 2. Repository Loader - /packages/core/src/loader.js
29
30 Extract and index repository data from parsed CAR:
31
32 export async function loadRepositoryFromCar(carBytes, actorStorage)
33 // 1. Parse CAR, get root CID (commit)
34 // 2. Decode commit: { did, version, rev, prev, data (MST root), sig }
35 // 3. Walk MST using walkMst() from mst.js
36 // 4. Populate: blocks, records, commits, metadata tables
37
38 3. Read-Only Package - /packages/readonly/
39
40 packages/readonly/
41 package.json
42 src/
43 index.js # createReadOnlyServer(options)
44 cli.js # CLI entry point
45
46 CLI Usage:
47 pds-readonly \
48 --car ./repos/did:plc:abc123.car \
49 --car ./repos/did:plc:xyz789.car \
50 --blobs ./blobs \
51 --port 3000
52
53 4. Multi-Repository Support
54
55 Use per-DID SQLite databases with a routing layer:
56
57 class MultiRepoManager {
58 repos = new Map() // did -> { db, actorStorage }
59
60 async loadCar(carPath) {
61 // Create DB, load CAR, register DID
62 }
63
64 getStorage(did) {
65 return this.repos.get(did)?.actorStorage
66 }
67 }
68
69 Files to Modify
70
71 /packages/core/src/pds.js
72
73 Add read-only mode with guards on write handlers:
74
75 // In constructor:
76 this.readOnly = options.readOnly ?? false;
77
78 // Guard for write endpoints:
79 if (this.readOnly) {
80 return Response.json(
81 { error: 'AuthenticationRequired', message: 'This PDS is read-only' },
82 { status: 401 }
83 );
84 }
85
86 Write endpoints to guard:
87 - handleInit
88 - handleCreateSession, handleRefreshSession
89 - handleCreateRecord, handlePutRecord, handleDeleteRecord, handleApplyWrites
90 - handleUploadBlob
91 - handlePutPreferences
92 - OAuth: handleOAuthPar, handleOAuthToken, handleOAuthRevoke
93
94 /packages/core/src/index.js
95
96 Export new modules:
97 export * from './car.js';
98 export * from './loader.js';
99
100 /package.json (root)
101
102 Add to workspaces:
103 "workspaces": ["packages/*", "examples/*"]
104 // readonly package auto-included via packages/*
105
106 Read-Only Endpoint Behavior
107 ┌────────────────────┬───────────┬─────────────────────────────────┐
108 │ Endpoint │ Status │ Notes │
109 ├────────────────────┼───────────┼─────────────────────────────────┤
110 │ describeServer │ Supported │ Indicates read-only in response │
111 ├────────────────────┼───────────┼─────────────────────────────────┤
112 │ describeRepo │ Supported │ Returns repo metadata │
113 ├────────────────────┼───────────┼─────────────────────────────────┤
114 │ getRecord │ Supported │ Retrieve single record │
115 ├────────────────────┼───────────┼─────────────────────────────────┤
116 │ listRecords │ Supported │ Paginated record listing │
117 ├────────────────────┼───────────┼─────────────────────────────────┤
118 │ listRepos │ Supported │ Lists all loaded DIDs │
119 ├────────────────────┼───────────┼─────────────────────────────────┤
120 │ getRepoStatus │ Supported │ Status for specific DID │
121 ├────────────────────┼───────────┼─────────────────────────────────┤
122 │ getRepo │ Supported │ Full CAR export │
123 ├────────────────────┼───────────┼─────────────────────────────────┤
124 │ sync.getRecord │ Supported │ Record with MST proof │
125 ├────────────────────┼───────────┼─────────────────────────────────┤
126 │ getLatestCommit │ Supported │ Latest commit info │
127 ├────────────────────┼───────────┼─────────────────────────────────┤
128 │ getBlob │ Supported │ If blobs directory provided │
129 ├────────────────────┼───────────┼─────────────────────────────────┤
130 │ listBlobs │ Supported │ List blob CIDs │
131 ├────────────────────┼───────────┼─────────────────────────────────┤
132 │ subscribeRepos │ Partial │ Historical events only │
133 ├────────────────────┼───────────┼─────────────────────────────────┤
134 │ resolveHandle │ Supported │ Handle-to-DID resolution │
135 ├────────────────────┼───────────┼─────────────────────────────────┤
136 │ All POST endpoints │ 401 │ AuthenticationRequired error │
137 └────────────────────┴───────────┴─────────────────────────────────┘
138 Blob Handling
139
140 CAR files don't contain blobs (blobs are stored separately). Options:
141
142 1. Filesystem directory (recommended): Point to existing blob storage
143 --blobs ./blobs/
144 # Structure: blobs/{did}/{shard}/{cid}
145 2. No blobs: getBlob returns 404, listBlobs returns empty
146
147 Deployment
148
149 Docker
150
151 FROM node:20-slim
152 WORKDIR /app
153 COPY . .
154 RUN npm install && npm run build
155
156 ENTRYPOINT ["node", "packages/readonly/src/cli.js"]
157 CMD ["--port", "3000"]
158
159 Docker Compose
160
161 services:
162 pds-readonly:
163 build: .
164 ports:
165 - "3000:3000"
166 volumes:
167 - ./data/repos:/repos:ro
168 - ./data/blobs:/blobs:ro
169 command:
170 - --car
171 - /repos/*.car
172 - --blobs
173 - /blobs
174 - --port
175 - "3000"
176
177 Environment Variables
178
179 PDS_PORT=3000
180 PDS_CAR_DIR=/repos # Directory containing .car files
181 PDS_BLOBS_DIR=/blobs # Optional blob storage
182 PDS_HOSTNAME=pds.example.com # Public hostname
183
184 Verification Steps
185
186 1. Unit tests: CAR parser roundtrip with buildCarFile
187 2. Loader tests: Load test CAR, verify storage populated
188 3. Integration tests:
189 - Start server, call listRepos - returns loaded DIDs
190 - Call getRecord - returns record data
191 - Call createRecord - returns 401
192 4. E2E test: Export from real PDS via getRepo, import to read-only, verify data matches
193
194 Implementation Sequence
195
196 1. Phase 1: CAR parser (car.js) + tests
197 2. Phase 2: Repository loader (loader.js) + tests
198 3. Phase 3: Read-only guards in pds.js
199 4. Phase 4: @pds/readonly package with CLI
200 5. Phase 5: Multi-repo routing layer
201 6. Phase 6: Documentation and deployment configs
202
203 Design Decisions
204
205 - Multi-repo: Support multiple CAR files from the start, each representing a different DID
206 - Blobs: Filesystem directory support (--blobs ./blobs/ with structure blobs/{did}/{shard}/{cid})
207 - WebSocket: subscribeRepos serves historical events only (from cursor), no live updates since data is static
208
209 Obtaining CAR Files
210
211 CAR files can be obtained via:
212
213 1. Export from existing PDS: GET /xrpc/com.atproto.sync.getRepo?did=<did>
214 2. Relay/BGS: Many relays provide repo export endpoints
215 3. Direct backup: If you have database access to a PDS
216
217 Example: Export a repo
218
219 curl "https://pds.example.com/xrpc/com.atproto.sync.getRepo?did=did:plc:abc123" \
220 -o did_plc_abc123.car
221
222 Blob Export
223
224 Blobs must be copied separately. From a PDS with filesystem blobs:
225 cp -r /path/to/pds/blobs/<did>/ ./blobs/<did>/
226
227 Complete Deployment Example
228
229 # 1. Create data directories
230 mkdir -p data/repos data/blobs
231
232 # 2. Export repositories
233 curl "https://bsky.social/xrpc/com.atproto.sync.getRepo?did=did:plc:abc123" \
234 -o data/repos/abc123.car
235
236 # 3. Copy blobs (if available)
237 # Blobs would need to come from the original PDS's blob storage
238
239 # 4. Start read-only PDS
240 docker compose up -d
241
242 # 5. Verify
243 curl http://localhost:3000/xrpc/com.atproto.sync.listRepos
244 # Returns: { "repos": [{ "did": "did:plc:abc123", ... }] }
245
246 curl http://localhost:3000/xrpc/com.atproto.repo.listRecords?repo=did:plc:abc123&collection=app.bsky.feed.post
247 # Returns posts from the loaded repository