+137

mlf-lexicon-fetcher/README.md

··· 1 1 + # mlf-lexicon-fetcher 2 2 + 3 3 + ATProto Lexicon fetcher with DNS resolution and HTTP client. 4 4 + 5 5 + Resolves lexicon NSIDs to DIDs via DNS TXT records and fetches lexicon JSON from ATProto repositories. 6 6 + 7 7 + ## Features 8 8 + 9 9 + - **DNS Resolution**: Resolves NSIDs to DIDs using DNS TXT records (RFC spec) 10 10 + - **HTTP Fetching**: Fetches lexicons from ATProto repositories via XRPC 11 11 + - **Pattern Matching**: Supports wildcard patterns (`.*` and `._`) 12 12 + - **Network Optimization**: Groups similar NSIDs to reduce HTTP requests 13 13 + - **Lockfile Support**: Fetches from known DIDs, bypassing DNS resolution 14 14 + - **Testable**: Mock DNS and HTTP clients for testing 15 15 + 16 16 + ## Usage 17 17 + 18 18 + ### Basic Fetching 19 19 + 20 20 + ```rust 21 21 + use mlf_lexicon_fetcher::ProductionLexiconFetcher; 22 22 + 23 23 + #[tokio::main] 24 24 + async fn main() -> Result<(), Box<dyn std::error::Error>> { 25 25 + // Create a production fetcher (real DNS + HTTP) 26 26 + let fetcher = ProductionLexiconFetcher::production().await?; 27 27 + 28 28 + // Fetch a single lexicon with metadata 29 29 + let result = fetcher.fetch_with_metadata("app.bsky.feed.post").await?; 30 30 + 31 31 + for lexicon in result.lexicons { 32 32 + println!("NSID: {}", lexicon.nsid); 33 33 + println!("DID: {}", lexicon.did); 34 34 + println!("Lexicon: {}", serde_json::to_string_pretty(&lexicon.lexicon)?); 35 35 + } 36 36 + 37 37 + Ok(()) 38 38 + } 39 39 + ``` 40 40 + 41 41 + ### Pattern Matching 42 42 + 43 43 + ```rust 44 44 + // Fetch all lexicons under a namespace 45 45 + let result = fetcher.fetch_with_metadata("app.bsky.feed.*").await?; 46 46 + println!("Fetched {} lexicons", result.lexicons.len()); 47 47 + 48 48 + // Fetch direct children only 49 49 + let result = fetcher.fetch_with_metadata("app.bsky._").await?; 50 50 + ``` 51 51 + 52 52 + ### Optimized Batch Fetching 53 53 + 54 54 + ```rust 55 55 + use mlf_lexicon_fetcher::ProductionLexiconFetcher; 56 56 + 57 57 + // Fetch many NSIDs with automatic optimization 58 58 + let nsids = vec![ 59 59 + "app.bsky.actor.profile".to_string(), 60 60 + "app.bsky.actor.defs".to_string(), 61 61 + "app.bsky.feed.post".to_string(), 62 62 + "app.bsky.feed.like".to_string(), 63 63 + ]; 64 64 + 65 65 + // Automatically groups into patterns like "app.bsky.actor.*" and "app.bsky.feed.*" 66 66 + let results = fetcher.fetch_many_optimized(&nsids).await?; 67 67 + ``` 68 68 + 69 69 + ### Lockfile Mode (Skip DNS) 70 70 + 71 71 + ```rust 72 72 + // When you already have the DID (e.g., from a lockfile) 73 73 + let result = fetcher.fetch_from_did_with_metadata( 74 74 + "did:plc:abc123", 75 75 + "app.bsky.feed.post" 76 76 + ).await?; 77 77 + ``` 78 78 + 79 79 + ### Advanced: Pattern Optimization 80 80 + 81 81 + ```rust 82 82 + use std::collections::HashSet; 83 83 + use mlf_lexicon_fetcher::optimize_fetch_patterns; 84 84 + 85 85 + let nsids: HashSet<String> = vec![ 86 86 + "app.bsky.actor.foo".to_string(), 87 87 + "app.bsky.actor.bar".to_string(), 88 88 + "app.bsky.feed.post".to_string(), 89 89 + ].into_iter().collect(); 90 90 + 91 91 + // Returns: ["app.bsky.actor.*", "app.bsky.feed.post"] 92 92 + let optimized = optimize_fetch_patterns(&nsids); 93 93 + ``` 94 94 + 95 95 + ## API 96 96 + 97 97 + ### Main Types 98 98 + 99 99 + - `ProductionLexiconFetcher` - Ready-to-use fetcher with real DNS and HTTP 100 100 + - `LexiconFetcher<D, H>` - Generic fetcher (for custom DNS/HTTP implementations) 101 101 + - `FetchResult` - Contains fetched lexicons with metadata (NSID, DID, JSON) 102 102 + 103 103 + ### Key Methods 104 104 + 105 105 + - `fetch_with_metadata(nsid)` - Fetch single/pattern, returns metadata 106 106 + - `fetch_many(nsids)` - Fetch multiple NSIDs sequentially 107 107 + - `fetch_many_optimized(nsids)` - Fetch multiple NSIDs with pattern optimization 108 108 + - `fetch_from_did_with_metadata(did, nsid)` - Skip DNS, fetch from known DID 109 109 + 110 110 + ### Utilities 111 111 + 112 112 + - `optimize_fetch_patterns(nsids)` - Group NSIDs into minimal patterns 113 113 + - `parse_nsid(nsid)` - Parse NSID into authority and name segments 114 114 + - `construct_dns_name(authority, name)` - Build DNS TXT record name 115 115 + 116 116 + ## Testing 117 117 + 118 118 + Use mock implementations for testing: 119 119 + 120 120 + ```rust 121 121 + use mlf_lexicon_fetcher::{LexiconFetcher, MockDnsResolver, MockHttpClient}; 122 122 + 123 123 + let mut dns = MockDnsResolver::new(); 124 124 + dns.add_record("app.bsky", "feed", "did:plc:test".to_string()); 125 125 + 126 126 + let mut http = MockHttpClient::new(); 127 127 + http.add_lexicon( 128 128 + "app.bsky.feed.post".to_string(), 129 129 + serde_json::json!({"lexicon": 1, "id": "app.bsky.feed.post"}) 130 130 + ); 131 131 + 132 132 + let fetcher = LexiconFetcher::new(dns, http); 133 133 + ``` 134 134 + 135 135 + ## License 136 136 + 137 137 + MIT

+75 -13

mlf-lexicon-fetcher/examples/usage.rs

··· 5 5 6 6 #[tokio::main] 7 7 async fn main() -> Result<(), Box<dyn std::error::Error>> { 8 8 - // Example 1: Fetch a single lexicon 9 9 - println!("=== Example 1: Fetch Single Lexicon ==="); 8 8 + // Example 1: Fetch with metadata (NEW!) 9 9 + println!("=== Example 1: Fetch with Metadata ==="); 10 10 11 11 let mut dns_resolver = MockDnsResolver::new(); 12 12 dns_resolver.add_record("place.stream", "chat.profile", "did:plc:test123".to_string()); ··· 28 28 29 29 let fetcher = LexiconFetcher::new(dns_resolver, http_client); 30 30 31 31 - match fetcher.fetch("place.stream.chat.profile").await { 32 32 - Ok(lexicon) => { 33 33 - println!("Successfully fetched lexicon:"); 34 34 - println!("{}", serde_json::to_string_pretty(&lexicon)?); 31 31 + // New API returns metadata (DID, NSID, lexicon) 32 32 + match fetcher.fetch_with_metadata("place.stream.chat.profile").await { 33 33 + Ok(result) => { 34 34 + println!("Successfully fetched {} lexicon(s):", result.lexicons.len()); 35 35 + for fetched in result.lexicons { 36 36 + println!(" NSID: {}", fetched.nsid); 37 37 + println!(" DID: {}", fetched.did); 38 38 + println!(" Lexicon: {}", serde_json::to_string_pretty(&fetched.lexicon)?); 39 39 + } 35 40 } 36 41 Err(e) => eprintln!("Error: {}", e), 37 42 } 38 43 39 39 - // Example 2: Fetch multiple lexicons with a pattern 40 40 - println!("\n=== Example 2: Fetch Multiple Lexicons with Pattern ==="); 44 44 + // Example 2: Fetch multiple with pattern 45 45 + println!("\n=== Example 2: Fetch Multiple with Pattern ==="); 41 46 42 47 let mut dns_resolver2 = MockDnsResolver::new(); 43 48 dns_resolver2.add_record("app.bsky", "feed", "did:plc:bsky123".to_string()); ··· 58 63 59 64 let fetcher2 = LexiconFetcher::new(dns_resolver2, http_client2); 60 65 61 61 - match fetcher2.fetch_pattern("app.bsky.feed.*").await { 62 62 - Ok(lexicons) => { 63 63 - println!("Successfully fetched {} lexicons:", lexicons.len()); 64 64 - for (nsid, _lexicon) in lexicons { 65 65 - println!(" - {}", nsid); 66 66 + match fetcher2.fetch_with_metadata("app.bsky.feed.*").await { 67 67 + Ok(result) => { 68 68 + println!("Successfully fetched {} lexicons:", result.lexicons.len()); 69 69 + for fetched in result.lexicons { 70 70 + println!(" - {} (from {})", fetched.nsid, fetched.did); 71 71 + } 72 72 + } 73 73 + Err(e) => eprintln!("Error: {}", e), 74 74 + } 75 75 + 76 76 + // Example 3: Optimized batch fetching (NEW!) 77 77 + println!("\n=== Example 3: Optimized Batch Fetching ==="); 78 78 + 79 79 + let mut dns_resolver3 = MockDnsResolver::new(); 80 80 + dns_resolver3.add_record("app.bsky", "actor", "did:plc:bsky123".to_string()); 81 81 + 82 82 + let mut http_client3 = MockHttpClient::new(); 83 83 + http_client3.add_lexicon( 84 84 + "app.bsky.actor.profile".to_string(), 85 85 + json!({"lexicon": 1, "id": "app.bsky.actor.profile"}), 86 86 + ); 87 87 + http_client3.add_lexicon( 88 88 + "app.bsky.actor.defs".to_string(), 89 89 + json!({"lexicon": 1, "id": "app.bsky.actor.defs"}), 90 90 + ); 91 91 + 92 92 + let fetcher3 = LexiconFetcher::new(dns_resolver3, http_client3); 93 93 + 94 94 + let nsids = vec![ 95 95 + "app.bsky.actor.profile".to_string(), 96 96 + "app.bsky.actor.defs".to_string(), 97 97 + ]; 98 98 + 99 99 + // Automatically optimizes to "app.bsky.actor.*" pattern 100 100 + match fetcher3.fetch_many_optimized(&nsids).await { 101 101 + Ok(results) => { 102 102 + println!("Optimized fetch completed:"); 103 103 + for result in results { 104 104 + println!(" Batch of {} lexicons", result.lexicons.len()); 105 105 + } 106 106 + } 107 107 + Err(e) => eprintln!("Error: {}", e), 108 108 + } 109 109 + 110 110 + // Example 4: Fetch from known DID (NEW!) 111 111 + println!("\n=== Example 4: Fetch from Known DID (Lockfile Mode) ==="); 112 112 + 113 113 + let dns_resolver4 = MockDnsResolver::new(); 114 114 + let mut http_client4 = MockHttpClient::new(); 115 115 + http_client4.add_lexicon( 116 116 + "app.bsky.feed.post".to_string(), 117 117 + json!({"lexicon": 1, "id": "app.bsky.feed.post"}), 118 118 + ); 119 119 + 120 120 + let fetcher4 = LexiconFetcher::new(dns_resolver4, http_client4); 121 121 + 122 122 + // Skip DNS resolution when DID is already known (e.g., from lockfile) 123 123 + match fetcher4.fetch_from_did_with_metadata("did:plc:bsky123", "app.bsky.feed.post").await { 124 124 + Ok(result) => { 125 125 + println!("Fetched from known DID:"); 126 126 + for fetched in result.lexicons { 127 127 + println!(" - {} (bypassed DNS)", fetched.nsid); 66 128 } 67 129 } 68 130 Err(e) => eprintln!("Error: {}", e),

+22 -3

website/content/docs/cli/07-fetch.md

··· 31 31 **Arguments:** 32 32 - `[NSID]` - Optional NSID or pattern to fetch: 33 33 - Specific lexicon: `com.example.forum.post` 34 34 - - Wildcard pattern: `com.example.forum.*` 34 34 + - All descendants: `com.example.forum.*` (matches `post`, `comment`, `comment.reply`, etc.) 35 35 + - Direct children only: `com.example.forum._` (matches `post`, `comment`, but NOT `comment.reply`) 35 36 - Real-world example: `app.bsky.feed.*` 36 37 37 38 **Options:** ··· 183 184 184 185 This downloads only the `com.example.forum.post` lexicon. 185 186 186 186 - ### Fetch with Wildcard 187 187 + ### Fetch with Wildcards 187 188 189 189 + **All descendants (`.*`):** 188 190 ```bash 189 191 mlf fetch com.example.forum.* 190 192 ``` 193 193 + Downloads all lexicons under the `com.example.forum` namespace (recursive). 191 194 192 192 - This downloads all lexicons under the `com.example.forum` namespace. 195 195 + **Example:** If the namespace contains: 196 196 + - `com.example.forum.post` 197 197 + - `com.example.forum.comment` 198 198 + - `com.example.forum.comment.reply` 199 199 + 200 200 + The `.*` pattern fetches **all three**. 201 201 + 202 202 + **Direct children only (`._`):** 203 203 + ```bash 204 204 + mlf fetch com.example.forum._ 205 205 + ``` 206 206 + Downloads only direct children of `com.example.forum` (non-recursive). 207 207 + 208 208 + Using the same example namespace, the `._` pattern fetches: 209 209 + - `com.example.forum.post` ✓ 210 210 + - `com.example.forum.comment` ✓ 211 211 + - `com.example.forum.comment.reply` ✗ (skipped, not a direct child) 193 212 194 213 ### Fetch and Save 195 214