this string has no description
ngerakines.me
lexicon_garden_autocomplete_plan.md
1# NSID Autocomplete Implementation Plan
2
3Redis-based autocomplete for lexicon NSIDs using `FT.SUGADD`/`FT.SUGGET` commands.
4
5## Overview
6
7- **Index command**: `FT.SUGADD autocomplete {nsid} 1.0 PAYLOAD "{did} {nsid}"`
8- **Query command**: `FT.SUGGET autocomplete "{query}" FUZZY WITHPAYLOADS MAX 10`
9- **API endpoint**: `GET /api/autocomplete-nsid?q={query}`
10- **UI**: Search input in navbar with dropdown autocomplete
11
12## Prerequisites
13
14- **Redis Stack** or **RediSearch module** required (standard Redis does not support `FT.SUG*` commands)
15- Existing `REDIS_CACHE_ENABLED` and `REDIS_URL` configuration will be reused
16
17---
18
19## Implementation Steps
20
21### 1. Extend Redis Cache Module
22
23**File**: `src/cache/redis.rs`
24
25Add autocomplete methods to `RedisCache`:
26
27```rust
28// Constants
29const AUTOCOMPLETE_KEY: &str = "lexicon:autocomplete";
30
31// New struct for results
32#[derive(Debug, Clone, Serialize, Deserialize)]
33pub struct AutocompleteSuggestion {
34 pub nsid: String,
35 pub did: String,
36}
37
38impl RedisCache {
39 /// FT.SUGADD autocomplete {nsid} 1.0 PAYLOAD "{did} {nsid}"
40 pub async fn add_nsid_suggestion(&self, nsid: &str, did: &str) { ... }
41
42 /// FT.SUGDEL autocomplete {nsid}
43 pub async fn remove_nsid_suggestion(&self, nsid: &str) { ... }
44
45 /// FT.SUGGET autocomplete "{query}" FUZZY WITHPAYLOADS MAX {limit}
46 pub async fn get_nsid_suggestions(&self, query: &str, limit: Option<usize>) -> Vec<AutocompleteSuggestion> { ... }
47
48 /// FT.SUGLEN autocomplete
49 pub async fn get_suggestion_count(&self) -> Option<i64> { ... }
50}
51```
52
53Use `deadpool_redis::redis::cmd()` for raw commands since `FT.SUG*` aren't in `AsyncCommands`.
54
55**Update**: `src/cache/mod.rs` - export `AutocompleteSuggestion`
56
57---
58
59### 2. TAP Consumer Integration
60
61**File**: `src/tap_consumer.rs`
62
63Add autocomplete indexing inside the `if is_authoritative` block (after graph edge insertion, around line ~260):
64
65```rust
66// Inside the `if is_authoritative { ... }` block, after graph edge processing:
67
68// Add NSID to autocomplete index (best-effort, authoritative only)
69if let Some(ref cache) = self.redis_cache {
70 cache.add_nsid_suggestion(nsid, did).await;
71}
72```
73
74This ensures only authoritative schemas are indexed for autocomplete, matching the behavior of `handle_admin_autocomplete_rebuild`.
75
76Note: No need to remove on delete since the rebuild can clean up stale entries.
77
78---
79
80### 3. Admin Backfill Endpoint (Authoritative Only)
81
82**File**: `src/http/handle_admin.rs`
83
84Add handler following the same pattern as `handle_admin_graph_rebuild` - only indexes authoritative, non-deleted schemas:
85
86```rust
87/// POST /admin/autocomplete/rebuild
88/// Only indexes authoritative schemas (where the NSID authority matches the record DID)
89pub async fn handle_admin_autocomplete_rebuild(
90 State(web_context): State<WebContext>,
91) -> Result<impl IntoResponse, WebError> {
92 let redis_cache = web_context.redis_cache.as_ref()
93 .ok_or_else(|| WebError::Internal(anyhow::anyhow!("Redis not configured")))?;
94
95 // Get all distinct NSIDs (reuse existing function)
96 let nsids = schema_list_distinct_nsids(&web_context.pool).await?;
97 tracing::info!(nsid_count = nsids.len(), "Rebuilding autocomplete index from authoritative schemas");
98
99 let mut indexed = 0usize;
100 let mut authority_failures = 0usize;
101
102 // For each NSID, resolve authority and index only if authoritative schema exists
103 for nsid in &nsids {
104 // Resolve the authority DID for this NSID
105 let authority_did = match resolve_schema_authority(
106 &web_context.pool,
107 web_context.dns_resolver.as_ref(),
108 nsid,
109 web_context.redis_cache.as_ref(),
110 )
111 .await
112 {
113 Ok(did) => did,
114 Err(e) => {
115 tracing::debug!(nsid = %nsid, error = %e, "Failed to resolve authority during autocomplete rebuild");
116 authority_failures += 1;
117 continue;
118 }
119 };
120
121 // Construct the AT-URI for the authoritative schema
122 let aturi = format!("at://{}/com.atproto.lexicon.schema/{}", authority_did, nsid);
123
124 // Get the latest schema from the authoritative source
125 let schema = match schema_get_latest(&web_context.pool, &aturi).await? {
126 Some(s) if s.deleted_at.is_none() => s,
127 _ => {
128 tracing::debug!(
129 nsid = %nsid,
130 authority_did = %authority_did,
131 "No authoritative schema found during autocomplete rebuild"
132 );
133 continue;
134 }
135 };
136
137 // Add to autocomplete index
138 redis_cache.add_nsid_suggestion(&schema.nsid, &schema.did).await;
139 indexed += 1;
140 }
141
142 tracing::info!(
143 indexed = indexed,
144 authority_failures = authority_failures,
145 "Autocomplete index rebuild complete"
146 );
147
148 Ok(Redirect::to("/admin"))
149}
150```
151
152**File**: `src/http/server.rs` - Add routes under admin nest:
153```rust
154.route("/autocomplete", get(handle_admin_autocomplete))
155.route("/autocomplete/rebuild", post(handle_admin_autocomplete_rebuild))
156```
157
158**File**: `templates/admin/autocomplete.html` - Admin page template showing:
159- Indexed NSIDs count
160- Redis connection status
161- Rebuild Index button
162
163**Note**: No new storage function needed - reuses existing `schema_list_distinct_nsids` and `schema_get_latest`.
164
165---
166
167### 4. Autocomplete API Endpoint
168
169**File**: `src/http/handle_autocomplete.rs` (new)
170
171```rust
172#[derive(Debug, Deserialize)]
173pub struct AutocompleteQuery {
174 pub q: String,
175 pub limit: Option<usize>,
176}
177
178#[derive(Debug, Serialize)]
179pub struct AutocompleteResponse {
180 pub suggestions: Vec<AutocompleteSuggestionItem>,
181}
182
183#[derive(Debug, Serialize)]
184pub struct AutocompleteSuggestionItem {
185 pub nsid: String,
186 pub did: String,
187 pub url: String, // e.g., "/lexicon/{did}/{nsid}"
188}
189
190/// GET /api/autocomplete-nsid?q={query}&limit={limit}
191pub async fn handle_autocomplete_nsid(
192 State(web_context): State<WebContext>,
193 Query(query): Query<AutocompleteQuery>,
194) -> impl IntoResponse {
195 // Validate: return empty if q < 2 chars
196 // Cap limit at 20
197 // Call redis_cache.get_nsid_suggestions() if available
198 // Return JSON response
199}
200```
201
202**File**: `src/http/mod.rs` - Add `pub mod handle_autocomplete;`
203
204**File**: `src/http/server.rs` - Add route:
205```rust
206.route("/api/autocomplete-nsid", get(handle_autocomplete_nsid))
207```
208
209---
210
211### 5. Navbar Search UI
212
213**File**: `templates/base.html`
214
215Replace nav section (lines 100-108):
216
217```html
218<header class="container">
219 <nav>
220 <ul>
221 <li><strong><a href="/">Lexicon Garden</a></strong></li>
222 </ul>
223 <ul>
224 <li style="position: relative;">
225 <input type="search" id="nsid-search" placeholder="Search NSIDs..."
226 autocomplete="off" style="margin-bottom: 0; min-width: 200px;">
227 <ul id="autocomplete-dropdown" class="autocomplete-dropdown"></ul>
228 </li>
229 <li><a href="/lexicon">Graph</a></li>
230 </ul>
231 </nav>
232</header>
233```
234
235Add CSS for dropdown (in `<style>` block):
236
237```css
238.autocomplete-dropdown {
239 position: absolute;
240 top: 100%;
241 left: 0;
242 right: 0;
243 z-index: 1000;
244 background: var(--pico-background-color);
245 border: 1px solid var(--pico-muted-border-color);
246 border-radius: 0 0 var(--pico-border-radius) var(--pico-border-radius);
247 list-style: none;
248 margin: 0;
249 padding: 0;
250 max-height: 300px;
251 overflow-y: auto;
252 display: none;
253}
254.autocomplete-dropdown li { padding: 0.5rem 1rem; cursor: pointer; }
255.autocomplete-dropdown li:hover, .autocomplete-dropdown li.selected {
256 background: var(--pico-primary-background);
257 color: var(--pico-primary-inverse);
258}
259.autocomplete-dropdown .nsid { font-weight: 500; }
260.autocomplete-dropdown .did { font-size: 0.75rem; color: var(--pico-muted-color); }
261```
262
263Add JavaScript before `{% block scripts %}`:
264
265```html
266<script>
267(function() {
268 const input = document.getElementById('nsid-search');
269 const dropdown = document.getElementById('autocomplete-dropdown');
270 if (!input || !dropdown) return;
271
272 let timer, selected = -1, items = [];
273
274 async function search(q) {
275 if (q.length < 2) { dropdown.style.display = 'none'; return; }
276 const res = await fetch(`/api/autocomplete-nsid?q=${encodeURIComponent(q)}`);
277 const data = await res.json();
278 items = data.suggestions || [];
279 render();
280 }
281
282 function render() {
283 if (!items.length) { dropdown.innerHTML = '<li>No results</li>'; }
284 else {
285 dropdown.innerHTML = items.map((s, i) =>
286 `<li data-i="${i}" data-url="${s.url}">
287 <div class="nsid">${s.nsid}</div>
288 <div class="did">${s.did}</div>
289 </li>`).join('');
290 }
291 dropdown.style.display = 'block';
292 selected = -1;
293 }
294
295 input.addEventListener('input', e => {
296 clearTimeout(timer);
297 timer = setTimeout(() => search(e.target.value.trim()), 200);
298 });
299
300 input.addEventListener('keydown', e => {
301 if (dropdown.style.display === 'none') return;
302 if (e.key === 'ArrowDown') { selected = Math.min(selected + 1, items.length - 1); update(); e.preventDefault(); }
303 if (e.key === 'ArrowUp') { selected = Math.max(selected - 1, 0); update(); e.preventDefault(); }
304 if (e.key === 'Enter' && selected >= 0) { window.location.href = items[selected].url; e.preventDefault(); }
305 if (e.key === 'Escape') { dropdown.style.display = 'none'; }
306 });
307
308 function update() {
309 dropdown.querySelectorAll('li').forEach((li, i) => li.classList.toggle('selected', i === selected));
310 }
311
312 dropdown.addEventListener('click', e => {
313 const li = e.target.closest('li[data-url]');
314 if (li) window.location.href = li.dataset.url;
315 });
316
317 document.addEventListener('click', e => {
318 if (!input.contains(e.target) && !dropdown.contains(e.target)) dropdown.style.display = 'none';
319 });
320})();
321</script>
322```
323
324---
325
326## Files to Modify
327
328| File | Changes |
329|------|---------|
330| `src/cache/redis.rs` | Add `AutocompleteSuggestion`, `add_nsid_suggestion`, `remove_nsid_suggestion`, `get_nsid_suggestions`, `get_suggestion_count` |
331| `src/cache/mod.rs` | Export `AutocompleteSuggestion` |
332| `src/tap_consumer.rs` | Call `add_nsid_suggestion` after schema insert (authoritative only) |
333| `src/http/handle_autocomplete.rs` | New file - autocomplete API endpoint |
334| `src/http/handle_admin.rs` | Add `handle_admin_autocomplete` and `handle_admin_autocomplete_rebuild` |
335| `src/http/mod.rs` | Add `pub mod handle_autocomplete` |
336| `src/http/server.rs` | Add routes for `/api/autocomplete-nsid`, `/admin/autocomplete`, and `/admin/autocomplete/rebuild` |
337| `templates/base.html` | Add search input, CSS, and JavaScript |
338| `templates/admin/autocomplete.html` | New file - admin page for autocomplete management |
339
340**Note**: No new storage functions needed - reuses existing `schema_list_distinct_nsids`, `schema_get_latest`, and `resolve_schema_authority`.
341
342---
343
344## Graceful Degradation
345
346- If Redis is not configured (`redis_cache: None`), API returns empty results
347- If RediSearch commands fail (no module), errors are logged but application continues
348- Search input remains visible but returns no suggestions
349
350---
351
352## Testing
353
3541. Start Redis Stack locally: `docker run -p 6379:6379 redis/redis-stack-server`
3552. Run app with `REDIS_CACHE_ENABLED=true REDIS_URL=redis://localhost:6379`
3563. Trigger admin backfill: `POST /admin/autocomplete/rebuild`
3574. Test API: `curl "http://localhost:4800/api/autocomplete-nsid?q=app.bsky"`
3585. Test UI: Type in navbar search input