···1717 - Dev mode -- show did, copy did in profiles (remove "logged in as <did>" from home page)
1818 - Toggle for table view vs future post-style view
1919 - Toggle for "for" and "at" in pours view
2020+ - Pull bsky account management stuff in? (i.e. email verification, change password, enable two factor?)
2121+2222+- "Close bag" of coffee
2323+ - Remove it from the beans dropdown when adding a new brew
2424+ - Add a "closed"/"archived" field to the lexicon
2525+ - Maybe allow adding a rating?
2626+ - Question: Should it show up in the profile screen? (maybe change header to current beans? --
2727+ have a different list at bottom of previous beans -- show created date, maybe closed date?)
2828+ - should be below the brewers table
20292130## Far Future Considerations
22312323-- Consider fully separating API backend from frontend service
2424- - Currently using HTMX header checks to prevent direct browser access to internal API endpoints
2525- - If adding mobile apps, third-party API consumers, or microservices architecture, revisit this
2626- - For now, monolithic approach is appropriate for HTMX-based web app with decentralized storage
3232+- Pivot to full svelte-kit?
27332834- Maybe swap from boltdb to sqlite
2935 - Use the non-cgo library
30363137## Fixes
32383333-- Homepage still shows cached feed items on homepage when not authed. should show a cached version of firehose (last 5 entries, cache last 20) from the server.
3434- This fetch should not try to backfill anything
3939+- Migrate terms and about page text. Add links to about at top of non-authed home page
4040+4141+- Backfill on startup should be cache invalidated if time since last backfill exceeds some amount (set in code/env var maybe?)
35423636-- Feed database in prod seems to be showing outdated data -- not sure why, local dev seems to show most recent.
4343+- Fix always using celcius for units, use settings (future state) or infer from number (maybe also future state)
37443838-- View button for somebody else's brew leads to an invalid page. need to show the same view brew page but w/o the edit and delete buttons.
3939-- Back button in view should take user back to their previous page (not sure how to handle this exactly though)
4545+- Make rating color nicer, but on white background for selector on new/edit brew page
40464141-- Header should probably always be attached to the top of the screen?
4747+- Refactor: remove the `SECURE_COOKIES` env var, it should be unecessary
4848+ - For dev, we should know its running in dev mode by checking the root url env var I think?
4949+ - This just adds noise and feels like an antipattern
42504343-- Feed item "view details" button should go away, the "new brew" in "addded a new brew" should take to view page instead (underline this text)
5151+- Fix styling of manage records page to use rounded tables like everything else
5252+ - Should also use tab selectors the same way as the profile uses
Caddyfile
deploy/Caddyfile
Dockerfile
deploy/Dockerfile
-215
MIGRATION.md
···11-# Alpine.js → Svelte Migration Complete! 🎉
22-33-## What Changed
44-55-The entire frontend has been migrated from Alpine.js + HTMX + Go templates to a **Svelte SPA**.
66-77-### Before
88-- **Frontend**: Go HTML templates + Alpine.js + HTMX
99-- **State**: Alpine global components + DOM manipulation
1010-- **Routing**: Server-side (Go mux)
1111-- **Data**: Mixed (HTMX partials + JSON API)
1212-1313-### After
1414-- **Frontend**: Svelte SPA (single-page application)
1515-- **State**: Svelte stores (reactive)
1616-- **Routing**: Client-side (navaid)
1717-- **Data**: JSON API only
1818-1919-## Architecture
2020-2121-```
2222-/
2323-├── cmd/arabica-server/main.go # Go backend entry point
2424-├── internal/ # Go backend (unchanged)
2525-│ ├── handlers/
2626-│ │ ├── handlers.go # Added /api/me and /api/feed-json
2727-│ │ └── ...
2828-│ └── routing/
2929-│ └── routing.go # Added SPA fallback route
3030-├── frontend/ # NEW: Svelte app
3131-│ ├── src/
3232-│ │ ├── App.svelte # Root component with router
3333-│ │ ├── main.js # Entry point
3434-│ │ ├── routes/ # Page components
3535-│ │ │ ├── Home.svelte
3636-│ │ │ ├── Login.svelte
3737-│ │ │ ├── Brews.svelte
3838-│ │ │ ├── BrewView.svelte
3939-│ │ │ ├── BrewForm.svelte
4040-│ │ │ ├── Manage.svelte
4141-│ │ │ ├── Profile.svelte
4242-│ │ │ ├── About.svelte
4343-│ │ │ ├── Terms.svelte
4444-│ │ │ └── NotFound.svelte
4545-│ │ ├── components/ # Reusable components
4646-│ │ │ ├── Header.svelte
4747-│ │ │ ├── Footer.svelte
4848-│ │ │ ├── FeedCard.svelte
4949-│ │ │ └── Modal.svelte
5050-│ │ ├── stores/ # Svelte stores
5151-│ │ │ ├── auth.js # Authentication state
5252-│ │ │ ├── cache.js # Data cache (replaces data-cache.js)
5353-│ │ │ └── ui.js # UI state (notifications, etc.)
5454-│ │ └── lib/
5555-│ │ ├── api.js # Fetch wrapper
5656-│ │ └── router.js # Client-side routing
5757-│ ├── index.html
5858-│ ├── vite.config.js
5959-│ └── package.json
6060-└── static/app/ # Built Svelte output (served by Go)
6161-```
6262-6363-## Development
6464-6565-### Run Frontend Dev Server (with hot reload)
6666-6767-```bash
6868-cd frontend
6969-npm install
7070-npm run dev
7171-```
7272-7373-Frontend runs on http://localhost:5173 with Vite proxy to Go backend
7474-7575-### Run Go Backend
7676-7777-```bash
7878-go run cmd/arabica-server/main.go
7979-```
8080-8181-Backend runs on http://localhost:18910
8282-8383-### Build for Production
8484-8585-```bash
8686-cd frontend
8787-npm run build
8888-```
8989-9090-This builds the Svelte app into `static/app/`
9191-9292-Then run the Go server normally:
9393-9494-```bash
9595-go run cmd/arabica-server/main.go
9696-```
9797-9898-The Go server will serve the built Svelte SPA from `static/app/`
9999-100100-## Key Features Implemented
101101-102102-### ✅ Authentication
103103-- Login with AT Protocol handle
104104-- Handle autocomplete
105105-- User profile dropdown
106106-- Persistent sessions
107107-108108-### ✅ Brews
109109-- List all brews
110110-- View brew details
111111-- Create new brew
112112-- Edit brew
113113-- Delete brew
114114-- Dynamic pours list
115115-- Rating slider
116116-117117-### ✅ Equipment Management
118118-- Tabs for beans, roasters, grinders, brewers
119119-- CRUD operations for all entity types
120120-- Inline entity creation from brew form
121121-- Tab state persisted to localStorage
122122-123123-### ✅ Social Feed
124124-- Community feed on homepage
125125-- Feed items with author info
126126-- Real-time updates (via API polling)
127127-128128-### ✅ Data Caching
129129-- Stale-while-revalidate pattern
130130-- localStorage persistence
131131-- Automatic invalidation on writes
132132-133133-## API Changes
134134-135135-### New Endpoints
136136-137137-- `GET /api/me` - Current user info
138138-- `GET /api/feed-json` - Feed items as JSON
139139-140140-### Existing Endpoints (unchanged)
141141-142142-- `GET /api/data` - All user data
143143-- `POST /api/beans`, `PUT /api/beans/{id}`, `DELETE /api/beans/{id}`
144144-- `POST /api/roasters`, `PUT /api/roasters/{id}`, `DELETE /api/roasters/{id}`
145145-- `POST /api/grinders`, `PUT /api/grinders/{id}`, `DELETE /api/grinders/{id}`
146146-- `POST /api/brewers`, `PUT /api/brewers/{id}`, `DELETE /api/brewers/{id}`
147147-- `POST /brews`, `PUT /brews/{id}`, `DELETE /brews/{id}`
148148-149149-### Deprecated Endpoints (HTML partials, no longer needed)
150150-151151-- `GET /api/feed` (HTML)
152152-- `GET /api/brews` (HTML)
153153-- `GET /api/manage` (HTML)
154154-- `GET /api/profile/{actor}` (HTML)
155155-156156-## Files to Delete (Future Cleanup)
157157-158158-These can be removed once you're confident the migration is complete:
159159-160160-```bash
161161-# Old Alpine.js JavaScript
162162-static/js/alpine.min.js
163163-static/js/manage-page.js
164164-static/js/brew-form.js
165165-static/js/data-cache.js
166166-static/js/handle-autocomplete.js
167167-168168-# Go templates (entire directory)
169169-templates/
170170-171171-# Template rendering helpers
172172-internal/bff/
173173-```
174174-175175-## Testing Checklist
176176-177177-- [ ] Login with AT Protocol handle
178178-- [ ] View homepage with feed
179179-- [ ] Create new brew with dynamic pours
180180-- [ ] Edit existing brew
181181-- [ ] Delete brew
182182-- [ ] Manage beans/roasters/grinders/brewers
183183-- [ ] Tab navigation with localStorage persistence
184184-- [ ] Inline entity creation from brew form
185185-- [ ] Navigate between pages (client-side routing)
186186-- [ ] Logout
187187-188188-## Browser Support
189189-190190-- Chrome/Edge (latest)
191191-- Firefox (latest)
192192-- Safari (latest)
193193-194194-## Performance
195195-196196-The Svelte bundle is **~136KB** (before gzip, ~35KB gzipped), which is excellent for a full-featured SPA.
197197-198198-Compared to Alpine.js (+ individual page scripts):
199199-- **Before**: ~50KB Alpine + ~20KB per page = 70-90KB
200200-- **After**: ~35KB gzipped for entire app
201201-202202-## Next Steps
203203-204204-1. Test thoroughly in development
205205-2. Deploy to production
206206-3. Monitor for any issues
207207-4. Delete old template files once confident
208208-5. Update documentation
209209-210210-## Notes
211211-212212-- OAuth flow still handled by Go backend
213213-- Sessions stored in BoltDB (unchanged)
214214-- User data stored in PDS via AT Protocol (unchanged)
215215-- All existing Go handlers remain functional
+3-6
README.md
···39394040## Docker
41414242+Build and run with Docker Compose:
4343+4244```bash
4343-# Build and run with Docker Compose
4444-docker compose up -d
4545-4646-# Or build and run manually
4747-docker build -t arabica .
4848-docker run -p 18910:18910 -v arabica-data:/data arabica
4545+docker compose -f deploy/compose.yml up -d
4946```
50475148For production deployments, configure environment variables in `docker-compose.yml`:
compose.yml
deploy/compose.yml
-641
docs/firehose-plan.md
···11-# Firehose Integration Plan for Arabica
22-33-## Executive Summary
44-55-This document proposes refactoring Arabica's home page feed to consume events from the AT Protocol firehose via Jetstream, replacing the current polling-based approach. This will provide real-time updates, dramatically reduce API calls, and improve scalability.
66-77-**Recommendation:** Implement Jetstream consumer with local BoltDB index as Phase 1, with optional Slingshot/Constellation integration in Phase 2.
88-99----
1010-1111-## Problem Statement
1212-1313-### Current Architecture
1414-1515-The feed service (`internal/feed/service.go`) polls each registered user's PDS directly:
1616-1717-```
1818-For N registered users:
1919-- N profile fetches
2020-- N × 5 collection fetches (brew, bean, roaster, grinder, brewer)
2121-- N × 4 reference resolution fetches
2222-- Total: ~10N API calls per refresh
2323-```
2424-2525-### Issues
2626-2727-| Problem | Impact |
2828-| ------------------------ | ----------------------------------- |
2929-| High API call volume | Risk of rate limiting as users grow |
3030-| 5-minute cache staleness | Users don't see recent activity |
3131-| N+1 query pattern | Linear scaling, O(N) per refresh |
3232-| PDS dependency | Feed fails if any PDS is slow/down |
3333-| No real-time updates | Requires manual refresh |
3434-3535----
3636-3737-## Proposed Solution: Jetstream Consumer
3838-3939-### Architecture Overview
4040-4141-```
4242-┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
4343-│ AT Protocol │ │ Jetstream │ │ Arabica │
4444-│ Firehose │────▶│ (Public/Self) │────▶│ Consumer │
4545-│ (all records) │ │ JSON over WS │ │ (background) │
4646-└─────────────────┘ └──────────────────┘ └────────┬────────┘
4747- │
4848- ▼
4949- ┌─────────────────┐
5050- │ Feed Index │
5151- │ (BoltDB) │
5252- └────────┬────────┘
5353- │
5454- ▼
5555- ┌─────────────────┐
5656- │ HTTP Handler │
5757- │ (instant) │
5858- └─────────────────┘
5959-```
6060-6161-### How It Works
6262-6363-1. **Background Consumer** connects to Jetstream WebSocket
6464-2. **Filters** for `social.arabica.alpha.*` collections
6565-3. **Indexes** incoming events into local BoltDB
6666-4. **Serves** feed requests instantly from local index
6767-5. **Fallback** to direct polling if consumer disconnects
6868-6969-### Benefits
7070-7171-| Metric | Current | With Jetstream |
7272-| --------------------- | ---------------- | ----------------- |
7373-| API calls per refresh | ~10N | 0 |
7474-| Feed latency | 5 min cache | Real-time (<1s) |
7575-| PDS dependency | High | None (after sync) |
7676-| User discovery | Manual registry | Automatic |
7777-| Scalability | O(N) per request | O(1) per request |
7878-7979----
8080-8181-## Technical Design
8282-8383-### 1. Jetstream Client Configuration
8484-8585-```go
8686-// internal/firehose/config.go
8787-8888-type JetstreamConfig struct {
8989- // Public endpoints (fallback rotation)
9090- Endpoints []string
9191-9292- // Filter to Arabica collections only
9393- WantedCollections []string
9494-9595- // Optional: filter to registered DIDs only
9696- // Leave empty to discover all Arabica users
9797- WantedDids []string
9898-9999- // Enable zstd compression (~56% bandwidth reduction)
100100- Compress bool
101101-102102- // Cursor file path for restart recovery
103103- CursorFile string
104104-}
105105-106106-func DefaultConfig() *JetstreamConfig {
107107- return &JetstreamConfig{
108108- Endpoints: []string{
109109- "wss://jetstream1.us-east.bsky.network/subscribe",
110110- "wss://jetstream2.us-east.bsky.network/subscribe",
111111- "wss://jetstream1.us-west.bsky.network/subscribe",
112112- "wss://jetstream2.us-west.bsky.network/subscribe",
113113- },
114114- WantedCollections: []string{
115115- "social.arabica.alpha.brew",
116116- "social.arabica.alpha.bean",
117117- "social.arabica.alpha.roaster",
118118- "social.arabica.alpha.grinder",
119119- "social.arabica.alpha.brewer",
120120- },
121121- Compress: true,
122122- CursorFile: "jetstream-cursor.txt",
123123- }
124124-}
125125-```
126126-127127-### 2. Event Processing
128128-129129-```go
130130-// internal/firehose/consumer.go
131131-132132-type Consumer struct {
133133- config *JetstreamConfig
134134- index *FeedIndex
135135- client *jetstream.Client
136136- cursor atomic.Int64
137137- connected atomic.Bool
138138-}
139139-140140-func (c *Consumer) handleEvent(ctx context.Context, event *models.Event) error {
141141- if event.Kind != "commit" || event.Commit == nil {
142142- return nil
143143- }
144144-145145- commit := event.Commit
146146-147147- // Only process Arabica collections
148148- if !strings.HasPrefix(commit.Collection, "social.arabica.alpha.") {
149149- return nil
150150- }
151151-152152- switch commit.Operation {
153153- case "create", "update":
154154- return c.index.UpsertRecord(ctx, event.Did, commit)
155155- case "delete":
156156- return c.index.DeleteRecord(ctx, event.Did, commit.Collection, commit.RKey)
157157- }
158158-159159- // Update cursor for recovery
160160- c.cursor.Store(event.TimeUS)
161161-162162- return nil
163163-}
164164-```
165165-166166-### 3. Feed Index Schema (BoltDB)
167167-168168-```go
169169-// internal/firehose/index.go
170170-171171-// BoltDB Buckets:
172172-// - "records" : {at-uri} -> {record JSON + metadata}
173173-// - "by_time" : {timestamp:at-uri} -> {} (for chronological queries)
174174-// - "by_did" : {did:at-uri} -> {} (for user-specific queries)
175175-// - "by_type" : {collection:timestamp:at-uri} -> {} (for type filtering)
176176-// - "profiles" : {did} -> {profile JSON} (cached profiles)
177177-// - "cursor" : "jetstream" -> {cursor value}
178178-179179-type FeedIndex struct {
180180- db *bbolt.DB
181181-}
182182-183183-type IndexedRecord struct {
184184- URI string `json:"uri"`
185185- DID string `json:"did"`
186186- Collection string `json:"collection"`
187187- RKey string `json:"rkey"`
188188- Record json.RawMessage `json:"record"`
189189- CID string `json:"cid"`
190190- IndexedAt time.Time `json:"indexed_at"`
191191-}
192192-193193-func (idx *FeedIndex) GetRecentFeed(ctx context.Context, limit int) ([]*FeedItem, error) {
194194- // Query by_time bucket in reverse order
195195- // Hydrate with profile data from profiles bucket
196196- // Return feed items instantly from local data
197197-}
198198-```
199199-200200-### 4. Profile Resolution
201201-202202-Profiles are not part of Arabica's lexicons, so we need a strategy:
203203-204204-**Option A: Lazy Loading (Recommended for Phase 1)**
205205-206206-```go
207207-func (idx *FeedIndex) resolveProfile(ctx context.Context, did string) (*Profile, error) {
208208- // Check local cache first
209209- if profile := idx.getCachedProfile(did); profile != nil {
210210- return profile, nil
211211- }
212212-213213- // Fetch from public API and cache
214214- profile, err := publicClient.GetProfile(ctx, did)
215215- if err != nil {
216216- return nil, err
217217- }
218218-219219- idx.cacheProfile(did, profile, 1*time.Hour)
220220- return profile, nil
221221-}
222222-```
223223-224224-**Option B: Slingshot Integration (Phase 2)**
225225-226226-```go
227227-// Use Slingshot's resolveMiniDoc for faster profile resolution
228228-func (idx *FeedIndex) resolveProfileViaSlingshot(ctx context.Context, did string) (*Profile, error) {
229229- url := fmt.Sprintf("https://slingshot.microcosm.blue/xrpc/com.bad-example.identity.resolveMiniDoc?identifier=%s", did)
230230- // Returns {did, handle, pds} in one call
231231-}
232232-```
233233-234234-### 5. Reference Resolution
235235-236236-Brews reference beans, grinders, and brewers. The index already has these records:
237237-238238-```go
239239-func (idx *FeedIndex) resolveBrew(ctx context.Context, brew *IndexedRecord) (*FeedItem, error) {
240240- var record map[string]interface{}
241241- json.Unmarshal(brew.Record, &record)
242242-243243- item := &FeedItem{RecordType: "brew"}
244244-245245- // Resolve bean reference from local index
246246- if beanRef, ok := record["beanRef"].(string); ok {
247247- if bean := idx.getRecord(beanRef); bean != nil {
248248- item.Bean = recordToBean(bean)
249249- }
250250- }
251251-252252- // Similar for grinder, brewer references
253253- // All from local index - no API calls
254254-255255- return item, nil
256256-}
257257-```
258258-259259-### 6. Fallback and Resilience
260260-261261-```go
262262-// internal/firehose/consumer.go
263263-264264-func (c *Consumer) Run(ctx context.Context) error {
265265- for {
266266- select {
267267- case <-ctx.Done():
268268- return ctx.Err()
269269- default:
270270- if err := c.connectAndConsume(ctx); err != nil {
271271- log.Warn().Err(err).Msg("jetstream connection lost, reconnecting...")
272272-273273- // Exponential backoff
274274- time.Sleep(c.backoff.NextBackOff())
275275-276276- // Rotate to next endpoint
277277- c.rotateEndpoint()
278278- continue
279279- }
280280- }
281281- }
282282-}
283283-284284-func (c *Consumer) connectAndConsume(ctx context.Context) error {
285285- cursor := c.loadCursor()
286286-287287- // Rewind cursor slightly to handle duplicates safely
288288- if cursor > 0 {
289289- cursor -= 5 * time.Second.Microseconds()
290290- }
291291-292292- return c.client.ConnectAndRead(ctx, &cursor)
293293-}
294294-```
295295-296296-### 7. Feed Service Integration
297297-298298-```go
299299-// internal/feed/service.go (modified)
300300-301301-type Service struct {
302302- registry *Registry
303303- publicClient *atproto.PublicClient
304304- cache *publicFeedCache
305305-306306- // New: firehose index
307307- firehoseIndex *firehose.FeedIndex
308308- useFirehose bool
309309-}
310310-311311-func (s *Service) GetRecentRecords(ctx context.Context, limit int) ([]*FeedItem, error) {
312312- // Prefer firehose index if available and populated
313313- if s.useFirehose && s.firehoseIndex.IsReady() {
314314- return s.firehoseIndex.GetRecentFeed(ctx, limit)
315315- }
316316-317317- // Fallback to polling (existing code)
318318- return s.getRecentRecordsViaPolling(ctx, limit)
319319-}
320320-```
321321-322322----
323323-324324-## Implementation Phases
325325-326326-### Phase 1: Core Jetstream Consumer (2 weeks)
327327-328328-**Goal:** Replace polling with firehose consumption for the feed.
329329-330330-**Tasks:**
331331-332332-1. Create `internal/firehose/` package
333333- - `config.go` - Jetstream configuration
334334- - `consumer.go` - WebSocket consumer with reconnection
335335- - `index.go` - BoltDB-backed feed index
336336- - `scheduler.go` - Event processing scheduler
337337-338338-2. Integrate with existing feed service
339339- - Add feature flag: `ARABICA_USE_FIREHOSE=true` (just use a cli flag)
340340- - Keep polling as fallback
341341-342342-3. Handle profile resolution
343343- - Cache profiles locally with 1-hour TTL
344344- - Lazy fetch on first access
345345- - Background refresh for active users
346346-347347-4. Cursor management
348348- - Persist cursor to survive restarts
349349- - Rewind on reconnection for safety
350350-351351-**Deliverables:**
352352-353353-- Real-time feed updates
354354-- Reduced API calls to near-zero
355355-- Automatic user discovery (anyone using Arabica lexicons)
356356-357357-### Phase 2: Slingshot Optimization (1 week)
358358-359359-**Goal:** Faster profile and record hydration.
360360-361361-**Tasks:**
362362-363363-1. Add Slingshot client (`internal/atproto/slingshot.go`)
364364-2. Use `resolveMiniDoc` for profile resolution
365365-3. Use Slingshot as fallback for missing records
366366-367367-**Deliverables:**
368368-369369-- Faster profile loading
370370-- Resilience to slow PDS endpoints
371371-372372-### Phase 3: Constellation for Social (1 week)
373373-374374-**Goal:** Enable like/comment counts when social features are added.
375375-376376-**Tasks:**
377377-378378-1. Add Constellation client (`internal/atproto/constellation.go`)
379379-2. Query backlinks for interaction counts
380380-3. Display counts on feed items
381381-382382-**Deliverables:**
383383-384384-- Like count on brews
385385-- Comment count on brews
386386-- Foundation for social features
387387-388388-### Phase 4: Spacedust for Real-time Notifications (Future)
389389-390390-**Goal:** Push notifications for interactions.
391391-392392-**Tasks:**
393393-394394-1. Subscribe to Spacedust for user's content interactions
395395-2. Build notification storage and API
396396-3. WebSocket to frontend for live updates
397397-398398----
399399-400400-## Data Flow Comparison
401401-402402-### Before (Polling)
403403-404404-```
405405-User Request → Check Cache → [Cache Miss] → Poll N PDSes → Build Feed → Return
406406- ↓
407407- ~10N API calls
408408- 5-10 second latency
409409-```
410410-411411-### After (Jetstream)
412412-413413-```
414414-Jetstream → Consumer → Index (BoltDB)
415415- ↓
416416-User Request → Query Index → Return
417417- ↓
418418- 0 API calls
419419- <10ms latency
420420-```
421421-422422----
423423-424424-## Automatic User Discovery
425425-426426-A major benefit of firehose consumption is automatic user discovery:
427427-428428-**Current:** Users must explicitly register via `/api/feed/register`
429429-430430-**With Jetstream:** Any user who creates an Arabica record is automatically indexed
431431-432432-```go
433433-// When we see a new DID creating Arabica records
434434-func (c *Consumer) handleNewUser(did string) {
435435- // Auto-register for feed
436436- c.registry.Register(did)
437437-438438- // Fetch and cache their profile
439439- go c.index.fetchAndCacheProfile(did)
440440-441441- // Backfill their existing records
442442- go c.backfillUser(did)
443443-}
444444-```
445445-446446-This could replace the manual registry entirely, or supplement it for "featured" users.
447447-448448----
449449-450450-## Backfill Strategy
451451-452452-When starting fresh or discovering a new user, we need historical data:
453453-454454-**Option A: Direct PDS Fetch (Simple)**
455455-456456-```go
457457-func (c *Consumer) backfillUser(ctx context.Context, did string) error {
458458- for _, collection := range arabicaCollections {
459459- records, _ := publicClient.ListRecords(ctx, did, collection, 100)
460460- for _, record := range records {
461461- c.index.UpsertFromPDS(record)
462462- }
463463- }
464464- return nil
465465-}
466466-```
467467-468468-**Option B: Slingshot Fetch (Faster)**
469469-470470-```go
471471-func (c *Consumer) backfillUserViaSlingshot(ctx context.Context, did string) error {
472472- // Single endpoint, pre-cached records
473473- // Same API as PDS but faster
474474-}
475475-```
476476-477477-**Option C: Jetstream Cursor Rewind (Events Only)**
478478-479479-- Rewind cursor to desired point in time
480480-- Replay events (no records available before cursor)
481481-- Limited to ~24h of history typically
482482-483483-**Recommendation:** Use Option A for Phase 1, add Option B in Phase 2.
484484-485485----
486486-487487-## Configuration
488488-489489-```bash
490490-# Environment variables
491491-492492-# Enable firehose-based feed (default: false during rollout)
493493-ARABICA_USE_FIREHOSE=true
494494-495495-# Jetstream endpoint (default: public Bluesky instances)
496496-JETSTREAM_URL=wss://jetstream1.us-east.bsky.network/subscribe
497497-498498-# Optional: self-hosted Jetstream
499499-# JETSTREAM_URL=ws://localhost:6008/subscribe
500500-501501-# Feed index database path
502502-ARABICA_FEED_INDEX_PATH=~/.local/share/arabica/feed-index.db
503503-504504-# Profile cache TTL (default: 1h)
505505-ARABICA_PROFILE_CACHE_TTL=1h
506506-507507-# Optional: Slingshot endpoint for Phase 2
508508-# SLINGSHOT_URL=https://slingshot.microcosm.blue
509509-510510-# Optional: Constellation endpoint for Phase 3
511511-# CONSTELLATION_URL=https://constellation.microcosm.blue
512512-```
513513-514514----
515515-516516-## Monitoring and Metrics
517517-518518-```go
519519-// Prometheus metrics to track firehose health
520520-521521-var (
522522- eventsReceived = prometheus.NewCounterVec(
523523- prometheus.CounterOpts{
524524- Name: "arabica_firehose_events_total",
525525- Help: "Total events received from Jetstream",
526526- },
527527- []string{"collection", "operation"},
528528- )
529529-530530- indexSize = prometheus.NewGauge(
531531- prometheus.GaugeOpts{
532532- Name: "arabica_feed_index_records",
533533- Help: "Number of records in feed index",
534534- },
535535- )
536536-537537- consumerLag = prometheus.NewGauge(
538538- prometheus.GaugeOpts{
539539- Name: "arabica_firehose_lag_seconds",
540540- Help: "Lag between event time and processing time",
541541- },
542542- )
543543-544544- connectionState = prometheus.NewGauge(
545545- prometheus.GaugeOpts{
546546- Name: "arabica_firehose_connected",
547547- Help: "1 if connected to Jetstream, 0 otherwise",
548548- },
549549- )
550550-)
551551-```
552552-553553----
554554-555555-## Risk Assessment
556556-557557-| Risk | Mitigation |
558558-| ----------------------- | --------------------------------------------- |
559559-| Jetstream unavailable | Fallback to polling, rotate endpoints |
560560-| Index corruption | Rebuild from backfill, periodic snapshots |
561561-| Duplicate events | Idempotent upserts using AT-URI as key |
562562-| Missing historical data | Backfill on startup and new user discovery |
563563-| High event volume | Filter to Arabica collections only (~0 noise) |
564564-| Profile resolution lag | Local cache with background refresh |
565565-566566----
567567-568568-## Open Questions
569569-570570-1. **Should we remove the registry entirely?**
571571- - Pro: Simpler, automatic discovery
572572- - Con: Lose ability to curate "featured" users
573573- - Recommendation: Keep registry for admin features, but don't require it for feed inclusion
574574-575575-2. **Self-host Jetstream or use public?**
576576- - Public is free and reliable
577577- - Self-host gives control and removes dependency
578578- - Recommendation: Start with public, evaluate self-hosting if issues arise
579579-580580-3. **How long to keep historical data?**
581581- - Option: Rolling 30-day window
582582- - Option: Keep everything (disk is cheap)
583583- - Recommendation: Keep 90 days, prune older records
584584-585585-4. **Real-time feed updates to frontend?**
586586- - Could push new items via WebSocket/SSE
587587- - Or just reduce cache TTL to ~30 seconds
588588- - Recommendation: Phase 1 just reduces staleness; real-time push is future enhancement
589589-590590----
591591-592592-## Alternatives Considered
593593-594594-### 1. Tap (Bluesky's Full Sync Tool)
595595-596596-**Pros:** Full verification, automatic backfill, collection signal mode
597597-**Cons:** Heavy operational overhead, overkill for current scale
598598-**Verdict:** Revisit when user base exceeds 500+
599599-600600-### 2. Direct Firehose Consumption
601601-602602-**Pros:** No Jetstream dependency
603603-**Cons:** Complex CBOR/CAR parsing, high bandwidth
604604-**Verdict:** Jetstream provides the simplicity we need
605605-606606-### 3. Slingshot as Primary Data Source
607607-608608-**Pros:** Pre-cached records, single endpoint
609609-**Cons:** Still polling-based, no real-time
610610-**Verdict:** Use as optimization layer, not primary
611611-612612-### 4. Spacedust Instead of Jetstream
613613-614614-**Pros:** Link-focused, lightweight
615615-**Cons:** Only links, no full records
616616-**Verdict:** Use for notifications, not feed content
617617-618618----
619619-620620-## Success Criteria
621621-622622-| Metric | Target |
623623-| -------------------------- | ----------------------- |
624624-| Feed latency | <100ms (from >5s) |
625625-| API calls per feed request | 0 (from ~10N) |
626626-| Time to see new content | <5s (from 5min) |
627627-| Feed availability | 99.9% (with fallback) |
628628-| New user discovery | Automatic (from manual) |
629629-630630----
631631-632632-## References
633633-634634-- [Jetstream GitHub](https://github.com/bluesky-social/jetstream)
635635-- [Jetstream Blog Post](https://docs.bsky.app/blog/jetstream)
636636-- [Jetstream Go Client](https://pkg.go.dev/github.com/bluesky-social/jetstream/pkg/client)
637637-- [Microcosm.blue Services](https://microcosm.blue/)
638638-- [Constellation API](https://constellation.microcosm.blue/)
639639-- [Slingshot API](https://slingshot.microcosm.blue/)
640640-- [Existing Evaluation: Jetstream/Tap](./jetstream-tap-evaluation.md)
641641-- [Existing Evaluation: Microcosm Tools](./microcosm-tools-evaluation.md)
-34
docs/indigo-research.md
···11-# AT Protocol Integration
22-33-## Overview
44-55-Arabica uses the Bluesky indigo SDK for AT Protocol integration.
66-77-**Package:** `github.com/bluesky-social/indigo`
88-99-## Key Components
1010-1111-### OAuth Authentication
1212-1313-- Public OAuth client with PKCE
1414-- DPOP-bound access tokens
1515-- Scopes: `atproto`, `transition:generic`
1616-- Session persistence via BoltDB
1717-1818-### Record Operations
1919-2020-Standard AT Protocol record CRUD operations:
2121-- `com.atproto.repo.createRecord`
2222-- `com.atproto.repo.getRecord`
2323-- `com.atproto.repo.listRecords`
2424-- `com.atproto.repo.putRecord`
2525-- `com.atproto.repo.deleteRecord`
2626-2727-### Client Implementation
2828-2929-See `internal/atproto/client.go` for the XRPC client wrapper.
3030-3131-## References
3232-3333-- indigo SDK: https://github.com/bluesky-social/indigo
3434-- AT Protocol docs: https://atproto.com
-314
docs/jetstream-tap-evaluation.md
···11-# Jetstream and Tap Evaluation for Arabica
22-33-## Executive Summary
44-55-This document evaluates two AT Protocol synchronization tools - **Jetstream** and **Tap** - for potential integration with Arabica. These tools could help reduce API requests for the community feed feature and simplify real-time data synchronization.
66-77-**Recommendation:** Consider Jetstream for community feed improvements in the near term; Tap is overkill for Arabica's current scale but valuable for future growth.
88-99----
1010-1111-## Background: Current Arabica Architecture
1212-1313-Arabica currently interacts with AT Protocol in two ways:
1414-1515-1. **Authenticated User Operations** (`internal/atproto/store.go`)
1616- - Direct XRPC calls to user's PDS for CRUD operations
1717- - Per-session in-memory cache (5-minute TTL)
1818- - Each user's data stored in their own PDS
1919-2020-2. **Community Feed** (`internal/feed/service.go`)
2121- - Polls registered users' PDSes to aggregate recent activity
2222- - Fetches profiles, brews, beans, roasters, grinders, brewers from each user
2323- - Public feed cached for 5 minutes
2424- - **Problem:** N+1 query pattern - each registered user requires multiple API calls
2525-2626-### Current Feed Inefficiency
2727-2828-For N registered users, the feed service makes approximately:
2929-- N profile fetches
3030-- N x 5 collection fetches (brew, bean, roaster, grinder, brewer) for recent items
3131-- N x 4 collection fetches for reference resolution
3232-- **Total: ~10N API calls per feed refresh**
3333-3434----
3535-3636-## Tool 1: Jetstream
3737-3838-### What It Is
3939-4040-Jetstream is a streaming service that consumes the AT Protocol firehose (`com.atproto.sync.subscribeRepos`) and converts it into lightweight JSON events. It's operated by Bluesky at public endpoints.
4141-4242-**Public Instances:**
4343-- `jetstream1.us-east.bsky.network`
4444-- `jetstream2.us-east.bsky.network`
4545-- `jetstream1.us-west.bsky.network`
4646-- `jetstream2.us-west.bsky.network`
4747-4848-### Key Features
4949-5050-| Feature | Description |
5151-|---------|-------------|
5252-| JSON Output | Simple JSON instead of CBOR/CAR binary encoding |
5353-| Filtering | Filter by collection (NSID) or repo (DID) |
5454-| Compression | ~56% smaller messages with zstd compression |
5555-| Low Latency | Real-time event delivery |
5656-| Easy to Use | Standard WebSocket connection |
5757-5858-### Jetstream Event Example
5959-6060-```json
6161-{
6262- "did": "did:plc:eygmaihciaxprqvxpfvl6flk",
6363- "time_us": 1725911162329308,
6464- "kind": "commit",
6565- "commit": {
6666- "rev": "3l3qo2vutsw2b",
6767- "operation": "create",
6868- "collection": "social.arabica.alpha.brew",
6969- "rkey": "3l3qo2vuowo2b",
7070- "record": {
7171- "$type": "social.arabica.alpha.brew",
7272- "method": "pourover",
7373- "rating": 4,
7474- "createdAt": "2024-09-09T19:46:02.102Z"
7575- },
7676- "cid": "bafyreidwaivazkwu67xztlmuobx35hs2lnfh3kolmgfmucldvhd3sgzcqi"
7777- }
7878-}
7979-```
8080-8181-### How Arabica Could Use Jetstream
8282-8383-**Use Case: Real-time Community Feed**
8484-8585-Instead of polling each user's PDS every 5 minutes, Arabica could:
8686-8787-1. Subscribe to Jetstream filtered by:
8888- - `wantedCollections`: `social.arabica.alpha.*`
8989- - `wantedDids`: List of registered feed users
9090-9191-2. Maintain a local feed index updated in real-time
9292-9393-3. Serve feed directly from local index (instant response, no API calls)
9494-9595-**Implementation Sketch:**
9696-9797-```go
9898-// Subscribe to Jetstream for Arabica collections
9999-ws, _ := websocket.Dial("wss://jetstream1.us-east.bsky.network/subscribe?" +
100100- "wantedCollections=social.arabica.alpha.brew&" +
101101- "wantedCollections=social.arabica.alpha.bean&" +
102102- "wantedDids=" + strings.Join(registeredDids, "&wantedDids="))
103103-104104-// Process events in background goroutine
105105-for {
106106- var event JetstreamEvent
107107- ws.ReadJSON(&event)
108108-109109- switch event.Commit.Collection {
110110- case "social.arabica.alpha.brew":
111111- feedIndex.AddBrew(event.DID, event.Commit.Record)
112112- case "social.arabica.alpha.bean":
113113- feedIndex.AddBean(event.DID, event.Commit.Record)
114114- }
115115-}
116116-```
117117-118118-### Jetstream Tradeoffs
119119-120120-| Pros | Cons |
121121-|------|------|
122122-| Dramatically reduces API calls | No cryptographic verification of data |
123123-| Real-time updates (sub-second latency) | Requires persistent WebSocket connection |
124124-| Simple JSON format | Trust relationship with Jetstream operator |
125125-| Can filter by collection/DID | Not part of formal AT Protocol spec |
126126-| Free public instances available | No built-in backfill mechanism |
127127-128128-### Jetstream Verdict for Arabica
129129-130130-**Recommended for:** Community feed real-time updates
131131-132132-**Not suitable for:** Authenticated user operations (those need direct PDS calls)
133133-134134-**Effort estimate:** Medium (1-2 weeks)
135135-- Add WebSocket client for Jetstream
136136-- Build local feed index (could use BoltDB or in-memory)
137137-- Handle reconnection/cursor management
138138-- Still need initial backfill via direct API
139139-140140----
141141-142142-## Tool 2: Tap
143143-144144-### What It Is
145145-146146-Tap is a synchronization tool for AT Protocol that handles the complexity of repo synchronization. It subscribes to a Relay and outputs filtered, verified events. Tap is more comprehensive than Jetstream but requires running your own instance.
147147-148148-**Repository:** `github.com/bluesky-social/indigo/cmd/tap`
149149-150150-### Key Features
151151-152152-| Feature | Description |
153153-|---------|-------------|
154154-| Automatic Backfill | Fetches complete history when tracking new repos |
155155-| Verification | MST integrity checks, signature validation |
156156-| Recovery | Auto-resyncs if repo becomes desynchronized |
157157-| Flexible Delivery | WebSocket, fire-and-forget, or webhooks |
158158-| Filtered Output | DID and collection filtering |
159159-160160-### Tap Operating Modes
161161-162162-1. **Dynamic (default):** Add DIDs via API as needed
163163-2. **Collection Signal:** Auto-track repos with records in specified collection
164164-3. **Full Network:** Mirror entire AT Protocol network (resource-intensive)
165165-166166-### How Arabica Could Use Tap
167167-168168-**Use Case: Complete Feed Infrastructure**
169169-170170-Tap could replace the entire feed polling mechanism:
171171-172172-1. Run Tap instance with `TAP_SIGNAL_COLLECTION=social.arabica.alpha.brew`
173173-2. Tap automatically discovers and tracks users who create brew records
174174-3. Feed service consumes events from local Tap instance
175175-4. No manual user registration needed - Tap discovers users automatically
176176-177177-**Collection Signal Mode:**
178178-179179-```bash
180180-# Start Tap to auto-track repos with Arabica records
181181-TAP_SIGNAL_COLLECTION=social.arabica.alpha.brew \
182182- go run ./cmd/tap --disable-acks=true
183183-```
184184-185185-**Webhook Delivery (Serverless-friendly):**
186186-187187-Tap can POST events to an HTTP endpoint, making it compatible with serverless architectures:
188188-189189-```bash
190190-# Tap sends events to Arabica webhook
191191-TAP_WEBHOOK_URL=https://arabica.example/api/feed-webhook \
192192- go run ./cmd/tap
193193-```
194194-195195-### Tap Tradeoffs
196196-197197-| Pros | Cons |
198198-|------|------|
199199-| Automatic backfill when adding repos | Requires running your own service |
200200-| Full cryptographic verification | More operational complexity |
201201-| Handles cursor management | Resource requirements (DB, network) |
202202-| Auto-discovers users via collection signal | Overkill for small user bases |
203203-| Webhook support for serverless | Still in beta |
204204-205205-### Tap Verdict for Arabica
206206-207207-**Recommended for:** Future growth when feed has many users
208208-209209-**Not suitable for:** Current scale (< 100 registered users)
210210-211211-**Effort estimate:** High (2-4 weeks)
212212-- Deploy and operate Tap service
213213-- Integrate webhook or WebSocket consumer
214214-- Migrate feed service to consume from Tap
215215-- Handle Tap service reliability/monitoring
216216-217217----
218218-219219-## Comparison Matrix
220220-221221-| Aspect | Current Polling | Jetstream | Tap |
222222-|--------|----------------|-----------|-----|
223223-| API Calls per Refresh | ~10N | 0 (after connection) | 0 (after backfill) |
224224-| Latency | 5 min cache | Real-time | Real-time |
225225-| Backfill | Full fetch each time | Manual | Automatic |
226226-| Verification | Trusts PDS | Trusts Jetstream | Full verification |
227227-| Operational Cost | None | None (public) | Run own service |
228228-| Complexity | Low | Medium | High |
229229-| User Discovery | Manual registry | Manual | Auto via collection |
230230-| Recommended Scale | < 50 users | 50-1000 users | 1000+ users |
231231-232232----
233233-234234-## Recommendation
235235-236236-### Short Term (Now - 6 months)
237237-238238-**Stick with current polling + caching approach**
239239-240240-Rationale:
241241-- Current implementation works
242242-- User base is small
243243-- Polling N users with caching is acceptable
244244-245245-**Consider adding Jetstream for feed** if:
246246-- Feed latency becomes user-visible issue
247247-- Registered users exceed ~50
248248-- API rate limiting becomes a problem
249249-250250-### Medium Term (6-12 months)
251251-252252-**Implement Jetstream integration**
253253-254254-1. Add background Jetstream consumer
255255-2. Build local feed index (BoltDB or SQLite)
256256-3. Serve feed from local index
257257-4. Keep polling as fallback for backfill
258258-259259-### Long Term (12+ months)
260260-261261-**Evaluate Tap when:**
262262-- User base exceeds 500+ registered users
263263-- Want automatic user discovery
264264-- Need cryptographic verification for social features (likes, comments)
265265-- Building moderation/anti-abuse features
266266-267267----
268268-269269-## Implementation Notes
270270-271271-### Jetstream Client Library
272272-273273-Bluesky provides a Go client library:
274274-275275-```go
276276-import "github.com/bluesky-social/jetstream/pkg/client"
277277-```
278278-279279-### Tap TypeScript Library
280280-281281-For frontend integration:
282282-283283-```typescript
284284-import { TapClient } from '@atproto/tap';
285285-```
286286-287287-### Connection Resilience
288288-289289-Both tools require handling:
290290-- WebSocket reconnection
291291-- Cursor persistence across restarts
292292-- Backpressure when events arrive faster than processing
293293-294294-### Caching Integration
295295-296296-Can coexist with current `SessionCache`:
297297-- Jetstream/Tap updates the local index
298298-- Local index serves feed requests
299299-- SessionCache continues for authenticated user operations
300300-301301----
302302-303303-## Related Documentation
304304-305305-- Jetstream GitHub: https://github.com/bluesky-social/jetstream
306306-- Tap README: https://github.com/bluesky-social/indigo/blob/main/cmd/tap/README.md
307307-- Jetstream Blog Post: https://docs.bsky.app/blog/jetstream
308308-- Tap Blog Post: https://docs.bsky.app/blog/introducing-tap
309309-310310----
311311-312312-## Note on "Constellation" and "Slingshot"
313313-314314-These terms don't appear to correspond to official AT Protocol tools as of this evaluation. If these refer to specific community projects or internal codenames, please provide additional context for evaluation.
-505
docs/microcosm-tools-evaluation.md
···11-# Microcosm Tools Evaluation for Arabica
22-33-## Executive Summary
44-55-This document evaluates three community-built AT Protocol infrastructure tools from [microcosm.blue](https://microcosm.blue/) - **Constellation**, **Spacedust**, and **Slingshot** - for potential integration with Arabica's community feed feature.
66-77-**Recommendation:** Adopt Constellation immediately for future social features (likes/comments). Consider Slingshot as an optional optimization for feed performance. Spacedust is ideal for real-time notifications when social features are implemented.
88-99----
1010-1111-## Background: Current Arabica Architecture
1212-1313-### The Problem
1414-1515-Arabica's community feed (`internal/feed/service.go`) currently polls each registered user's PDS directly. For N registered users:
1616-1717-| API Call Type | Count per Refresh |
1818-|---------------|-------------------|
1919-| Profile fetches | N |
2020-| Brew collections | N |
2121-| Bean collections | N |
2222-| Roaster collections | N |
2323-| Grinder collections | N |
2424-| Brewer collections | N |
2525-| Reference resolution | ~4N |
2626-| **Total** | **~10N API calls** |
2727-2828-This approach has several issues:
2929-- **Latency**: Feed refresh is slow with many users
3030-- **Rate limits**: Risk of PDS rate limiting
3131-- **Reliability**: Feed fails if any PDS is slow/down
3232-- **Scalability**: Linear growth in API calls per user
3333-3434-### Future Social Features
3535-3636-Arabica plans to add likes, comments, and follows (see `AGENTS.md`). These interactions require **backlink queries** - given a brew, find all likes pointing at it. This is impossible with current polling approach.
3737-3838----
3939-4040-## Tool 1: Constellation (Backlink Index)
4141-4242-### What It Is
4343-4444-Constellation is a **global backlink index** that crawls every record in the AT Protocol firehose and indexes all links (AT-URIs, DIDs, URLs). It answers "who/what points at this target?" queries.
4545-4646-**Public Instance:** `https://constellation.microcosm.blue`
4747-4848-### Key Capabilities
4949-5050-| Feature | Description |
5151-|---------|-------------|
5252-| Backlink queries | Find all records linking to a target |
5353-| Like/follow counts | Get interaction counts instantly |
5454-| Any lexicon support | Works with `social.arabica.alpha.*` |
5555-| DID filtering | Filter links by specific users |
5656-| Distinct DID counts | Count unique users, not just records |
5757-5858-### API Examples
5959-6060-**Get like count for a brew:**
6161-```bash
6262-curl "https://constellation.microcosm.blue/links/count/distinct-dids" \
6363- -G --data-urlencode "target=at://did:plc:xxx/social.arabica.alpha.brew/abc123" \
6464- --data-urlencode "collection=social.arabica.alpha.like" \
6565- --data-urlencode "path=.subject.uri"
6666-```
6767-6868-**Get all users who liked a brew:**
6969-```bash
7070-curl "https://constellation.microcosm.blue/links/distinct-dids" \
7171- -G --data-urlencode "target=at://did:plc:xxx/social.arabica.alpha.brew/abc123" \
7272- --data-urlencode "collection=social.arabica.alpha.like" \
7373- --data-urlencode "path=.subject.uri"
7474-```
7575-7676-**Get all comments on a brew:**
7777-```bash
7878-curl "https://constellation.microcosm.blue/links" \
7979- -G --data-urlencode "target=at://did:plc:xxx/social.arabica.alpha.brew/abc123" \
8080- --data-urlencode "collection=social.arabica.alpha.comment" \
8181- --data-urlencode "path=.subject.uri"
8282-```
8383-8484-### How Arabica Could Use Constellation
8585-8686-**Use Case 1: Social Interaction Counts**
8787-8888-When displaying a brew in the feed, fetch interaction counts:
8989-9090-```go
9191-// Get like count for a brew
9292-func (c *ConstellationClient) GetLikeCount(ctx context.Context, brewURI string) (int, error) {
9393- url := fmt.Sprintf("%s/links/count/distinct-dids?target=%s&collection=%s&path=%s",
9494- c.baseURL,
9595- url.QueryEscape(brewURI),
9696- "social.arabica.alpha.like",
9797- url.QueryEscape(".subject.uri"))
9898-9999- // Returns {"total": 42}
100100- var result struct { Total int `json:"total"` }
101101- // ... fetch and decode
102102- return result.Total, nil
103103-}
104104-```
105105-106106-**Use Case 2: Comment Threads**
107107-108108-Fetch all comments for a brew detail page:
109109-110110-```go
111111-func (c *ConstellationClient) GetComments(ctx context.Context, brewURI string) ([]Comment, error) {
112112- // Constellation returns the AT-URIs of comment records
113113- // Then fetch each comment from Slingshot or user's PDS
114114-}
115115-```
116116-117117-**Use Case 3: "Who liked this" List**
118118-119119-```go
120120-func (c *ConstellationClient) GetLikers(ctx context.Context, brewURI string) ([]string, error) {
121121- // Returns list of DIDs who liked this brew
122122- // Can hydrate with profile info from Slingshot
123123-}
124124-```
125125-126126-### Constellation Tradeoffs
127127-128128-| Pros | Cons |
129129-|------|------|
130130-| Instant interaction counts (no polling) | Third-party dependency |
131131-| Works with any lexicon including Arabica's | Not self-hosted (yet) |
132132-| Handles likes from any PDS globally | Slight index delay (~seconds) |
133133-| 11B+ links indexed, production-ready | Trusts Constellation operator |
134134-| Free public instance | Query limits may apply |
135135-136136-### Constellation Verdict
137137-138138-**Essential for:** Social features (likes, comments, follows)
139139-140140-**Not needed for:** Current feed polling (Constellation indexes interactions, not record listings)
141141-142142-**Effort estimate:** Low (1 week)
143143-- Add HTTP client for Constellation API
144144-- Integrate counts into brew display
145145-- Cache counts locally (5-minute TTL)
146146-147147----
148148-149149-## Tool 2: Spacedust (Interactions Firehose)
150150-151151-### What It Is
152152-153153-Spacedust extracts **links** from every record in the AT Protocol firehose and re-emits them over WebSocket. Unlike Jetstream (which emits full records), Spacedust emits just the link relationships.
154154-155155-**Public Instance:** `wss://spacedust.microcosm.blue`
156156-157157-### Key Capabilities
158158-159159-| Feature | Description |
160160-|---------|-------------|
161161-| Real-time link events | Instantly know when someone likes/follows |
162162-| Filter by source/target | Subscribe to specific collections or targets |
163163-| Any lexicon support | Works with `social.arabica.alpha.*` |
164164-| Lightweight | Just links, not full records |
165165-166166-### Example: Subscribe to Likes on Your Brews
167167-168168-```javascript
169169-// WebSocket connection to Spacedust
170170-const ws = new WebSocket(
171171- "wss://spacedust.microcosm.blue/subscribe" +
172172- "?wantedSources=social.arabica.alpha.like:subject.uri" +
173173- "&wantedSubjects=did:plc:your-did"
174174-);
175175-176176-ws.onmessage = (event) => {
177177- const link = JSON.parse(event.data);
178178- // { source: "at://...", target: "at://...", ... }
179179- console.log("Someone liked your brew!");
180180-};
181181-```
182182-183183-### How Arabica Could Use Spacedust
184184-185185-**Use Case: Real-time Notifications**
186186-187187-When social features are added, Spacedust enables instant notifications:
188188-189189-```go
190190-// Background goroutine subscribes to Spacedust
191191-func (s *NotificationService) subscribeToInteractions(userDID string) {
192192- ws := dial("wss://spacedust.microcosm.blue/subscribe" +
193193- "?wantedSources=social.arabica.alpha.like:subject.uri" +
194194- "&wantedSubjects=" + userDID)
195195-196196- for {
197197- link := readLink(ws)
198198- // Someone liked a brew by userDID
199199- s.notify(userDID, "Someone liked your brew!")
200200- }
201201-}
202202-```
203203-204204-**Use Case: Live Feed Updates**
205205-206206-Push new brews to connected clients without polling:
207207-208208-```go
209209-// Subscribe to all Arabica brew creations
210210-ws := dial("wss://spacedust.microcosm.blue/subscribe" +
211211- "?wantedSources=social.arabica.alpha.brew:beanRef")
212212-213213-// When a link event arrives, a new brew was created
214214-// Fetch full record from Slingshot and push to feed
215215-```
216216-217217-### Spacedust Tradeoffs
218218-219219-| Pros | Cons |
220220-|------|------|
221221-| Real-time, sub-second latency | Requires persistent WebSocket |
222222-| Lightweight link-only events | Still in v0 (missing some features) |
223223-| Filter by collection/target | No cursor replay yet |
224224-| Perfect for notifications | Need to hydrate records separately |
225225-226226-### Spacedust Verdict
227227-228228-**Ideal for:** Real-time notifications, live feed updates
229229-230230-**Not suitable for:** Current feed needs (need full records, not just links)
231231-232232-**Effort estimate:** Medium (2-3 weeks)
233233-- WebSocket client with reconnection
234234-- Notification service for social interactions
235235-- Integration with frontend for live updates
236236-- Depends on social features being implemented first
237237-238238----
239239-240240-## Tool 3: Slingshot (Records & Identities Cache)
241241-242242-### What It Is
243243-244244-Slingshot is an **edge cache** for AT Protocol records and identities. It pre-caches records from the firehose and provides fast, authenticated access. Also resolves handles to DIDs with bi-directional verification.
245245-246246-**Public Instance:** `https://slingshot.microcosm.blue`
247247-248248-### Key Capabilities
249249-250250-| Feature | Description |
251251-|---------|-------------|
252252-| Fast record fetching | Pre-cached from firehose |
253253-| Identity resolution | `resolveMiniDoc` for handle/DID |
254254-| Bi-directional verification | Only returns verified handles |
255255-| Works with slow PDS | Cache serves even if PDS is down |
256256-| Standard XRPC API | Drop-in replacement for PDS calls |
257257-258258-### API Examples
259259-260260-**Resolve identity:**
261261-```bash
262262-curl "https://slingshot.microcosm.blue/xrpc/com.bad-example.identity.resolveMiniDoc?identifier=bad-example.com"
263263-# Returns: { "did": "did:plc:...", "handle": "bad-example.com", "pds": "https://..." }
264264-```
265265-266266-**Get record (standard XRPC):**
267267-```bash
268268-curl "https://slingshot.microcosm.blue/xrpc/com.atproto.repo.getRecord?repo=did:plc:xxx&collection=social.arabica.alpha.brew&rkey=abc123"
269269-```
270270-271271-**List records:**
272272-```bash
273273-curl "https://slingshot.microcosm.blue/xrpc/com.atproto.repo.listRecords?repo=did:plc:xxx&collection=social.arabica.alpha.brew&limit=10"
274274-```
275275-276276-### How Arabica Could Use Slingshot
277277-278278-**Use Case 1: Faster Feed Fetching**
279279-280280-Replace direct PDS calls with Slingshot for public data:
281281-282282-```go
283283-// Before: Each user's PDS
284284-pdsEndpoint, _ := c.GetPDSEndpoint(ctx, did)
285285-url := fmt.Sprintf("%s/xrpc/com.atproto.repo.listRecords...", pdsEndpoint)
286286-287287-// After: Single Slingshot endpoint
288288-url := fmt.Sprintf("https://slingshot.microcosm.blue/xrpc/com.atproto.repo.listRecords...")
289289-```
290290-291291-**Benefits:**
292292-- Eliminates N DNS lookups for N user PDS endpoints
293293-- Single, fast endpoint for all public record fetches
294294-- Continues working even if individual PDS is slow/down
295295-- Pre-cached records = faster response times
296296-297297-**Use Case 2: Identity Resolution**
298298-299299-Replace multiple API calls with single `resolveMiniDoc`:
300300-301301-```go
302302-// Before: Two calls
303303-handle := resolveHandle(did) // Call 1
304304-pds := resolvePDSEndpoint(did) // Call 2
305305-306306-// After: One call
307307-mini := resolveMiniDoc(did)
308308-// { handle: "user.bsky.social", pds: "https://...", did: "did:plc:..." }
309309-```
310310-311311-**Use Case 3: Hydrate Records from Constellation**
312312-313313-When Constellation returns AT-URIs (e.g., comments on a brew), fetch the actual records from Slingshot:
314314-315315-```go
316316-// Constellation returns: ["at://did:plc:a/social.arabica.alpha.comment/123", ...]
317317-commentURIs := constellation.GetComments(ctx, brewURI)
318318-319319-// Fetch each comment record from Slingshot
320320-for _, uri := range commentURIs {
321321- record := slingshot.GetRecord(ctx, uri)
322322- // ...
323323-}
324324-```
325325-326326-### Implementation: Slingshot-Backed PublicClient
327327-328328-```go
329329-// internal/atproto/slingshot_client.go
330330-331331-const SlingshotBaseURL = "https://slingshot.microcosm.blue"
332332-333333-type SlingshotClient struct {
334334- baseURL string
335335- httpClient *http.Client
336336-}
337337-338338-func NewSlingshotClient() *SlingshotClient {
339339- return &SlingshotClient{
340340- baseURL: SlingshotBaseURL,
341341- httpClient: &http.Client{Timeout: 10 * time.Second},
342342- }
343343-}
344344-345345-// ListRecords uses Slingshot instead of user's PDS
346346-func (c *SlingshotClient) ListRecords(ctx context.Context, did, collection string, limit int) (*PublicListRecordsOutput, error) {
347347- // Same XRPC API, different endpoint
348348- url := fmt.Sprintf("%s/xrpc/com.atproto.repo.listRecords?repo=%s&collection=%s&limit=%d&reverse=true",
349349- c.baseURL, url.QueryEscape(did), url.QueryEscape(collection), limit)
350350- // ... standard HTTP request
351351-}
352352-353353-// ResolveMiniDoc gets handle + PDS in one call
354354-func (c *SlingshotClient) ResolveMiniDoc(ctx context.Context, identifier string) (*MiniDoc, error) {
355355- url := fmt.Sprintf("%s/xrpc/com.bad-example.identity.resolveMiniDoc?identifier=%s",
356356- c.baseURL, url.QueryEscape(identifier))
357357- // ... returns { did, handle, pds }
358358-}
359359-```
360360-361361-### Slingshot Tradeoffs
362362-363363-| Pros | Cons |
364364-|------|------|
365365-| Faster than direct PDS calls | Third-party dependency |
366366-| Single endpoint for all users | May not have custom lexicons cached |
367367-| Identity verification built-in | Not all XRPC APIs implemented |
368368-| Resilient to slow/down PDS | Trusts Slingshot operator |
369369-| Pre-cached from firehose | Still in v0, some features missing |
370370-371371-### Slingshot Verdict
372372-373373-**Recommended for:** Feed performance optimization, identity resolution
374374-375375-**Not suitable for:** Authenticated user operations (still need direct PDS)
376376-377377-**Effort estimate:** Low (3-5 days)
378378-- Add SlingshotClient as optional PublicClient backend
379379-- Feature flag to toggle between direct PDS and Slingshot
380380-- Test with Arabica collections to ensure they're indexed
381381-382382----
383383-384384-## Comparison: Current vs. Microcosm Tools
385385-386386-| Aspect | Current Polling | + Slingshot | + Constellation | + Spacedust |
387387-|--------|-----------------|-------------|-----------------|-------------|
388388-| Feed refresh latency | Slow (N PDS calls) | Fast (1 endpoint) | N/A | Real-time |
389389-| Like/comment counts | Impossible | Impossible | Instant | N/A |
390390-| Rate limit risk | High | Low | Low | None |
391391-| PDS failure resilience | Poor | Good | N/A | N/A |
392392-| Real-time updates | No (5min cache) | No | No | Yes |
393393-| Effort to integrate | N/A | Low | Low | Medium |
394394-395395----
396396-397397-## Recommendation
398398-399399-### Immediate (Social Features Prerequisite)
400400-401401-**1. Integrate Constellation when adding likes/comments**
402402-403403-Constellation is essential for social features. When a brew is displayed, use Constellation to:
404404-- Show like count
405405-- Show comment count
406406-- Power "who liked this" lists
407407-- Power comment threads
408408-409409-**Implementation priority:** Do this alongside `social.arabica.alpha.like` and `social.arabica.alpha.comment` lexicon implementation.
410410-411411-### Short Term (Performance Optimization)
412412-413413-**2. Evaluate Slingshot for feed performance**
414414-415415-If feed latency becomes an issue:
416416-- Add SlingshotClient as alternative to direct PDS calls
417417-- A/B test performance improvement
418418-- Use for public record fetches only (keep direct PDS for authenticated writes)
419419-420420-**Trigger:** When registered users exceed ~20-30, or feed refresh exceeds 5 seconds
421421-422422-### Medium Term (Real-time Features)
423423-424424-**3. Add Spacedust for notifications**
425425-426426-When social features are live and users want notifications:
427427-- Subscribe to Spacedust for likes/comments on user's content
428428-- Push notifications via WebSocket to connected clients
429429-- Optional: background job for email notifications
430430-431431-**Trigger:** After social features launch, when users request notifications
432432-433433----
434434-435435-## Comparison with Official Tools (Jetstream/Tap)
436436-437437-See `jetstream-tap-evaluation.md` for official Bluesky tools. Key differences:
438438-439439-| Aspect | Microcosm Tools | Official Tools |
440440-|--------|-----------------|----------------|
441441-| Focus | Links/interactions | Full records |
442442-| Backlink queries | Constellation (yes) | Not available |
443443-| Record caching | Slingshot | Not available |
444444-| Real-time | Spacedust (links) | Jetstream (records) |
445445-| Self-hosting | Not yet documented | Available |
446446-| Community | Community-supported | Bluesky-supported |
447447-448448-**Recommendation:** Use Microcosm tools for social features (likes/comments/follows) where backlink queries are essential. Consider Jetstream for full feed real-time if needed later.
449449-450450----
451451-452452-## Implementation Plan
453453-454454-### Phase 1: Constellation Integration (with social features)
455455-456456-```
457457-1. Create internal/atproto/constellation.go
458458- - ConstellationClient with HTTP client
459459- - GetBacklinks(), GetLinkCount(), GetDistinctDIDs()
460460-461461-2. Create internal/social/interactions.go
462462- - GetBrewLikeCount(brewURI)
463463- - GetBrewComments(brewURI)
464464- - GetBrewLikers(brewURI)
465465-466466-3. Update templates to show interaction counts
467467- - Modify feed item display
468468- - Add like button (when like lexicon ready)
469469-```
470470-471471-### Phase 2: Slingshot Optimization (optional)
472472-473473-```
474474-1. Create internal/atproto/slingshot.go
475475- - SlingshotClient implementing same interface as PublicClient
476476-477477-2. Add feature flag: ARABICA_USE_SLINGSHOT=true
478478-479479-3. Modify feed/service.go to use SlingshotClient
480480- - Keep PublicClient as fallback
481481-```
482482-483483-### Phase 3: Spacedust Notifications (future)
484484-485485-```
486486-1. Create internal/notifications/spacedust.go
487487- - WebSocket client with reconnection
488488- - Subscribe to user's content interactions
489489-490490-2. Create notification storage (BoltDB)
491491-492492-3. Add /api/notifications endpoint for frontend polling
493493-494494-4. Optional: WebSocket to frontend for real-time
495495-```
496496-497497----
498498-499499-## Related Documentation
500500-501501-- Microcosm Main: https://microcosm.blue/
502502-- Constellation API: https://constellation.microcosm.blue/
503503-- Source Code: https://github.com/at-microcosm/microcosm-rs
504504-- Discord: https://discord.gg/tcDfe4PGVB
505505-- See also: `jetstream-tap-evaluation.md` for official Bluesky tools
···11-import navaid from 'navaid';
11+import navaid from "navaid";
2233/**
44 * Simple client-side router using navaid
55 * Handles browser history and navigation
66 */
77-const router = navaid('/');
77+const router = navaid("/");
8899/**
1010 * Navigate to a route programmatically
+24-14
frontend/src/routes/About.svelte
···11<script>
22- import { navigate } from '../lib/router.js';
22+ import { navigate } from "../lib/router.js";
33</script>
4455<div class="max-w-4xl mx-auto">
66- <div class="bg-gradient-to-br from-amber-50 to-brown-100 rounded-xl p-8 border-2 border-brown-300 shadow-lg">
66+ <div
77+ class="bg-gradient-to-br from-amber-50 to-brown-100 rounded-xl p-8 border-2 border-brown-300 shadow-lg"
88+ >
79 <h1 class="text-3xl font-bold text-brown-900 mb-6">About Arabica</h1>
88-1010+911 <div class="prose prose-brown max-w-none">
1012 <p class="text-lg text-brown-800 mb-4">
1111- Arabica is a coffee brew tracking application that leverages the AT Protocol for decentralized data storage.
1313+ Arabica is a coffee brew tracking application that leverages the AT
1414+ Protocol for decentralized data storage.
1215 </p>
1313-1616+1417 <h2 class="text-2xl font-bold text-brown-900 mt-8 mb-4">Features</h2>
1518 <ul class="space-y-2 text-brown-800">
1619 <li class="flex items-start">
1720 <span class="mr-2">🔒</span>
1818- <span><strong>Decentralized:</strong> Your data lives in your Personal Data Server (PDS)</span>
2121+ <span
2222+ ><strong>Decentralized:</strong> Your data lives in your Personal Data
2323+ Server (PDS)</span
2424+ >
1925 </li>
2026 <li class="flex items-start">
2127 <span class="mr-2">🚀</span>
2222- <span><strong>Portable:</strong> Own your coffee brewing history</span>
2828+ <span><strong>Portable:</strong> Own your coffee brewing history</span
2929+ >
2330 </li>
2431 <li class="flex items-start">
2532 <span class="mr-2">📊</span>
2626- <span>Track brewing variables like temperature, time, and grind size</span>
3333+ <span
3434+ >Track brewing variables like temperature, time, and grind size</span
3535+ >
2736 </li>
2837 <li class="flex items-start">
2938 <span class="mr-2">🌍</span>
···3443 <span>Add tasting notes and ratings to each brew</span>
3544 </li>
3645 </ul>
3737-4646+3847 <h2 class="text-2xl font-bold text-brown-900 mt-8 mb-4">AT Protocol</h2>
3948 <p class="text-brown-800 mb-4">
4040- The Authenticated Transfer Protocol (AT Protocol) is a decentralized social networking protocol
4141- that gives you full ownership of your data. Your brewing records are stored in your own PDS,
4242- not in Arabica's servers.
4949+ The Authenticated Transfer Protocol (AT Protocol) is a decentralized
5050+ social networking protocol that gives you full ownership of your data.
5151+ Your brewing records are stored in your own PDS, not in Arabica's
5252+ servers.
4353 </p>
4444-5454+4555 <div class="mt-8">
4656 <button
4747- on:click={() => navigate('/')}
5757+ on:click={() => navigate("/")}
4858 class="bg-gradient-to-r from-brown-700 to-brown-800 text-white px-6 py-3 rounded-lg hover:from-brown-800 hover:to-brown-900 transition-all font-semibold shadow-lg"
4959 >
5060 Get Started