NSID Autocomplete Implementation Plan#

Redis-based autocomplete for lexicon NSIDs using FT.SUGADD/FT.SUGGET commands.

Overview#

• Index command: FT.SUGADD autocomplete {nsid} 1.0 PAYLOAD "{did} {nsid}"
• Query command: FT.SUGGET autocomplete "{query}" FUZZY WITHPAYLOADS MAX 10
• API endpoint: GET /api/autocomplete-nsid?q={query}
• UI: Search input in navbar with dropdown autocomplete

Prerequisites#

• Redis Stack or RediSearch module required (standard Redis does not support FT.SUG* commands)
• Existing REDIS_CACHE_ENABLED and REDIS_URL configuration will be reused

Implementation Steps#

1. Extend Redis Cache Module#

File: src/cache/redis.rs

Add autocomplete methods to RedisCache:

// Constants
const AUTOCOMPLETE_KEY: &str = "lexicon:autocomplete";

// New struct for results
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AutocompleteSuggestion {
    pub nsid: String,
    pub did: String,
}

impl RedisCache {
    /// FT.SUGADD autocomplete {nsid} 1.0 PAYLOAD "{did} {nsid}"
    pub async fn add_nsid_suggestion(&self, nsid: &str, did: &str) { ... }

    /// FT.SUGDEL autocomplete {nsid}
    pub async fn remove_nsid_suggestion(&self, nsid: &str) { ... }

    /// FT.SUGGET autocomplete "{query}" FUZZY WITHPAYLOADS MAX {limit}
    pub async fn get_nsid_suggestions(&self, query: &str, limit: Option<usize>) -> Vec<AutocompleteSuggestion> { ... }

    /// FT.SUGLEN autocomplete
    pub async fn get_suggestion_count(&self) -> Option<i64> { ... }
}

Use deadpool_redis::redis::cmd() for raw commands since FT.SUG* aren't in AsyncCommands.

Update: src/cache/mod.rs - export AutocompleteSuggestion

2. TAP Consumer Integration#

File: src/tap_consumer.rs

Add autocomplete indexing inside the if is_authoritative block (after graph edge insertion, around line ~260):

// Inside the `if is_authoritative { ... }` block, after graph edge processing:

// Add NSID to autocomplete index (best-effort, authoritative only)
if let Some(ref cache) = self.redis_cache {
    cache.add_nsid_suggestion(nsid, did).await;
}

This ensures only authoritative schemas are indexed for autocomplete, matching the behavior of handle_admin_autocomplete_rebuild.

Note: No need to remove on delete since the rebuild can clean up stale entries.

3. Admin Backfill Endpoint (Authoritative Only)#

File: src/http/handle_admin.rs

Add handler following the same pattern as handle_admin_graph_rebuild - only indexes authoritative, non-deleted schemas:

/// POST /admin/autocomplete/rebuild
/// Only indexes authoritative schemas (where the NSID authority matches the record DID)
pub async fn handle_admin_autocomplete_rebuild(
    State(web_context): State<WebContext>,
) -> Result<impl IntoResponse, WebError> {
    let redis_cache = web_context.redis_cache.as_ref()
        .ok_or_else(|| WebError::Internal(anyhow::anyhow!("Redis not configured")))?;

    // Get all distinct NSIDs (reuse existing function)
    let nsids = schema_list_distinct_nsids(&web_context.pool).await?;
    tracing::info!(nsid_count = nsids.len(), "Rebuilding autocomplete index from authoritative schemas");

    let mut indexed = 0usize;
    let mut authority_failures = 0usize;

    // For each NSID, resolve authority and index only if authoritative schema exists
    for nsid in &nsids {
        // Resolve the authority DID for this NSID
        let authority_did = match resolve_schema_authority(
            &web_context.pool,
            web_context.dns_resolver.as_ref(),
            nsid,
            web_context.redis_cache.as_ref(),
        )
        .await
        {
            Ok(did) => did,
            Err(e) => {
                tracing::debug!(nsid = %nsid, error = %e, "Failed to resolve authority during autocomplete rebuild");
                authority_failures += 1;
                continue;
            }
        };

        // Construct the AT-URI for the authoritative schema
        let aturi = format!("at://{}/com.atproto.lexicon.schema/{}", authority_did, nsid);

        // Get the latest schema from the authoritative source
        let schema = match schema_get_latest(&web_context.pool, &aturi).await? {
            Some(s) if s.deleted_at.is_none() => s,
            _ => {
                tracing::debug!(
                    nsid = %nsid,
                    authority_did = %authority_did,
                    "No authoritative schema found during autocomplete rebuild"
                );
                continue;
            }
        };

        // Add to autocomplete index
        redis_cache.add_nsid_suggestion(&schema.nsid, &schema.did).await;
        indexed += 1;
    }

    tracing::info!(
        indexed = indexed,
        authority_failures = authority_failures,
        "Autocomplete index rebuild complete"
    );

    Ok(Redirect::to("/admin"))
}

File: src/http/server.rs - Add routes under admin nest:

.route("/autocomplete", get(handle_admin_autocomplete))
.route("/autocomplete/rebuild", post(handle_admin_autocomplete_rebuild))

File: templates/admin/autocomplete.html - Admin page template showing:

• Indexed NSIDs count
• Redis connection status
• Rebuild Index button

Note: No new storage function needed - reuses existing schema_list_distinct_nsids and schema_get_latest.

4. Autocomplete API Endpoint#

File: src/http/handle_autocomplete.rs (new)

#[derive(Debug, Deserialize)]
pub struct AutocompleteQuery {
    pub q: String,
    pub limit: Option<usize>,
}

#[derive(Debug, Serialize)]
pub struct AutocompleteResponse {
    pub suggestions: Vec<AutocompleteSuggestionItem>,
}

#[derive(Debug, Serialize)]
pub struct AutocompleteSuggestionItem {
    pub nsid: String,
    pub did: String,
    pub url: String,  // e.g., "/lexicon/{did}/{nsid}"
}

/// GET /api/autocomplete-nsid?q={query}&limit={limit}
pub async fn handle_autocomplete_nsid(
    State(web_context): State<WebContext>,
    Query(query): Query<AutocompleteQuery>,
) -> impl IntoResponse {
    // Validate: return empty if q < 2 chars
    // Cap limit at 20
    // Call redis_cache.get_nsid_suggestions() if available
    // Return JSON response
}

File: src/http/mod.rs - Add pub mod handle_autocomplete;

File: src/http/server.rs - Add route:

.route("/api/autocomplete-nsid", get(handle_autocomplete_nsid))

5. Navbar Search UI#

File: templates/base.html

Replace nav section (lines 100-108):

<header class="container">
    <nav>
        <ul>
            <li><strong><a href="/">Lexicon Garden</a></strong></li>
        </ul>
        <ul>
            <li style="position: relative;">
                <input type="search" id="nsid-search" placeholder="Search NSIDs..."
                       autocomplete="off" style="margin-bottom: 0; min-width: 200px;">
                <ul id="autocomplete-dropdown" class="autocomplete-dropdown"></ul>
            </li>
            <li><a href="/lexicon">Graph</a></li>
        </ul>
    </nav>
</header>

Add CSS for dropdown (in <style> block):

.autocomplete-dropdown {
    position: absolute;
    top: 100%;
    left: 0;
    right: 0;
    z-index: 1000;
    background: var(--pico-background-color);
    border: 1px solid var(--pico-muted-border-color);
    border-radius: 0 0 var(--pico-border-radius) var(--pico-border-radius);
    list-style: none;
    margin: 0;
    padding: 0;
    max-height: 300px;
    overflow-y: auto;
    display: none;
}
.autocomplete-dropdown li { padding: 0.5rem 1rem; cursor: pointer; }
.autocomplete-dropdown li:hover, .autocomplete-dropdown li.selected {
    background: var(--pico-primary-background);
    color: var(--pico-primary-inverse);
}
.autocomplete-dropdown .nsid { font-weight: 500; }
.autocomplete-dropdown .did { font-size: 0.75rem; color: var(--pico-muted-color); }

Add JavaScript before {% block scripts %}:

<script>
(function() {
    const input = document.getElementById('nsid-search');
    const dropdown = document.getElementById('autocomplete-dropdown');
    if (!input || !dropdown) return;

    let timer, selected = -1, items = [];

    async function search(q) {
        if (q.length < 2) { dropdown.style.display = 'none'; return; }
        const res = await fetch(`/api/autocomplete-nsid?q=${encodeURIComponent(q)}`);
        const data = await res.json();
        items = data.suggestions || [];
        render();
    }

    function render() {
        if (!items.length) { dropdown.innerHTML = '<li>No results</li>'; }
        else {
            dropdown.innerHTML = items.map((s, i) =>
                `<li data-i="${i}" data-url="${s.url}">
                    <div class="nsid">${s.nsid}</div>
                    <div class="did">${s.did}</div>
                </li>`).join('');
        }
        dropdown.style.display = 'block';
        selected = -1;
    }

    input.addEventListener('input', e => {
        clearTimeout(timer);
        timer = setTimeout(() => search(e.target.value.trim()), 200);
    });

    input.addEventListener('keydown', e => {
        if (dropdown.style.display === 'none') return;
        if (e.key === 'ArrowDown') { selected = Math.min(selected + 1, items.length - 1); update(); e.preventDefault(); }
        if (e.key === 'ArrowUp') { selected = Math.max(selected - 1, 0); update(); e.preventDefault(); }
        if (e.key === 'Enter' && selected >= 0) { window.location.href = items[selected].url; e.preventDefault(); }
        if (e.key === 'Escape') { dropdown.style.display = 'none'; }
    });

    function update() {
        dropdown.querySelectorAll('li').forEach((li, i) => li.classList.toggle('selected', i === selected));
    }

    dropdown.addEventListener('click', e => {
        const li = e.target.closest('li[data-url]');
        if (li) window.location.href = li.dataset.url;
    });

    document.addEventListener('click', e => {
        if (!input.contains(e.target) && !dropdown.contains(e.target)) dropdown.style.display = 'none';
    });
})();
</script>

Files to Modify#

Note: No new storage functions needed - reuses existing schema_list_distinct_nsids, schema_get_latest, and resolve_schema_authority.

Graceful Degradation#

• If Redis is not configured (redis_cache: None), API returns empty results
• If RediSearch commands fail (no module), errors are logged but application continues
• Search input remains visible but returns no suggestions

Testing#

1. Start Redis Stack locally: docker run -p 6379:6379 redis/redis-stack-server
2. Run app with REDIS_CACHE_ENABLED=true REDIS_URL=redis://localhost:6379
3. Trigger admin backfill: POST /admin/autocomplete/rebuild
4. Test API: curl "http://localhost:4800/api/autocomplete-nsid?q=app.bsky"
5. Test UI: Type in navbar search input