seoul.systems
targeting
main
from
seoul.systems/microcosm-rs:
order_query
The following adds support for reverse-chronological ordering when fetching back-links.
We add a non-required "reverse" query parameter that allows reversing the default order of returned back-links.
Supports both current storage backends (mem_store and rocks_store).
None yet.
- Add "reverse" boolean parameter to three endpoints:
1. getManyToManyCounts (grouped counts)
2. getBacklinks (links to subject)
3. links (target links)
- Implement reverse iteration logic in both storage backends (mem_store
and rocks_store) for __reverse-chronological__ ordering
- Update HTML templates with (reverse) checkboxes
Before all our testing was done manually. This fixes the remaining failing automatic tests.
Adds test coverage for the reverse parameter in get_links and
get_many_to_many_counts methods. Tests verify that reverse: true
returns results in oldest-first/descending order while reverse:
false returns newest-first/ascending order.
The initial implementation of the reverse ordering feature had a flaw
in how it handled pagination. When `reverse=true` was set, the code
would reverse the entire collection before applying pagination
slicing, which broke the cursor logic. Cursor positions were
calculated relative to the original ordering but applied to a reversed
set of items, causing duplicate results or skipped items across page
boundaries.
The fix restructures pagination to account for reverse ordering:
- In reverse mode, pagination starts from the beginning (oldest items)
and moves forward, with begin/end indices calculated accordingly
- In normal mode, pagination continues working backward from the end
(newest items) as before
- The result slice is reversed __only after__ pagination boundaries
are established, ensuring cursor values remain consistent
Applied to both storage backends (mem_store and rocks_store) for
consistent behavior across implementations.
Refactor link query APIs to use an explicit Order enum
(OldestToNewest, NewestToOldest) instead of a boolean
reverse flag for better clarity and maintainability.
As Lexicons do not offer support for tuples as primitive field types I had to convert the return to use a new RecordsBySubject struct instead, which closely matches what you already did for the m2m counts endpoint.
The comments obviously don't belong here... please refer to the comments made for the PR #7
alright! reading back my own code gave me such a headache with this!
i'm reverting a few parts of this:
deprecated links endpoint: going to hard-code it to the existing order (deprecated endpoints don't get new features)
reverse ordering on many-to-many counts.
the many-to-many counts endpoint is a weird one to think about because it's not obvious what an "order" on it means. especially since the counts are aggregated, which row should be first in the response?
the current behaviour is deterministic but arbitrary: items are sorted by the other subject's target id (ie., the first time we saw the joined thingy), which is potentially unrelated to the actual many-to-many records themselves. if someone later creates a many-to-many record linking an earlier other subject, it'll become first in the list.
so: the determinism of that other-subject-target-id sorting is what enables paging to work at this endpoint, but it doesn't impose a client-meaningful order of items in the response. reversing an arbitrary order just gives you a different arbitrary order.
hopefully that makes sense ha
(additionally: there is a bit of optimization in the main join loop in the rocksdb implementation that assumes target-id-order to save work from unused pages, and i could be wrong, but i think it wouldn't stay valid under reverse paging)
(oh but i do think we can add a reverse option to the distinct-dids endpoint! probably under a new xrpc version.)
despite the above, this is merged!
i did a couple edits locally:
Just so that I understand you correctly here: We decided to not proceed with the implementation of a reverse order parameter for many-to-many related things as the current order is deterministic but arbitrary in nature, and thus reversing is essentially not providing any more information that is meaningful in the end.
If it's not too much trouble, do you mind explaining why you used Iterator::skip and ::take for the slicing? Is this simply more idiomatic or more performant even? :)
Really liked the way you resolved empty results with the empty impl by the way. Definitely less verbose and even clearer than what I opted for haha
many-to-many: yes exactly. it's not clear what expectation we would be trying to meet for a client requesting the many-to-many counts in reverse.
i'm open to being wrong here! when thinking about ordering, i was focused on the order of the many-to-many join-records themselves. which is where things felt meaningless/arbitrary because a) they're aggregated together (at this endpoint) and b) their order of creation is independent of the order this endpoint returns its results in.
but maybe having results ordered by the other-target-id does have client-relevant meaning. joined subjects are sorted by their age in a global sense?
thinking through it with concrete examples, i'm more inclined to bring it back:
hmm.
the thing that still feels off to me for these is the fact that this endpoint is an aggregate of many-to-many counts. that implies to me that the count is the thing of interest, and the order of responses in relation to the count is arbitrary again.
trying to keep the non-aggregated version of this endpoint in view, does ordering by the linked subject age make sense then? (ordering by the link actually feels more natural to me in that case) (i think i just talked myself out of and then back into bringing back your ordering here!!)
skip and take: yeah, it's a little more functional-style which works well for my brain. i like a big pipeline of chained transformations operating on sequences so i took the chance to do a little refactor with it. especially since it's immediately setting up an iterator right after slicing, it feels more cohesive to me.
it can be technically the other things too:
indexing ranges like thing[start..end] can panic if either index is out of bounds. in some cases the compiler can prove we already did the bounds checks and remove the panic hooks, but i suspect it can't here. even though we can prove that the indices are in range by reasoning about the code, it's nice not to have things-that-can-panic in the code.
i think it eliminated a Vec::reverse() in favour of an Iterator::rev(), which is potentially a very tiny efficiency improvement
m2m: I think this just boils down to how we communicate this fact to users of our API imo. As your internal back and forth shows there can be valid arguments in both ways. I think most users might naively - as I did at first - simply expect the links/records to be returned in the order they were created in. But, clearly documenting this in the endpoint description seems like a good idea here to make sure we don't leave any kind of ambiguity on the table.
skip and take: That's actually a great argument and I tend to agree with you on both fronts. Having worked professionally more with Typescript than, say, C, I always liked that you could chain transformations to a specific object together (map, filter and so on). The additional safety we get here from using iterators to slice data is new for me but definitely worth it.
m2m (again): Depending on how you see this I might reopen the PR again and introduce the parameter again. I let you make the final judgement call ofc :)
thanks for following up! i really appreciate the discussion!
100% agree the most important thing is communicating and setting accurate expectations in the api docs regardless of how this lands.
I really had convinced myself that the order was "meaningless" for clients, but then every example i was thinking of was like, welllll it kind of is actually meaningful. After a few days I'm still feeling that way, so I'm interested in bringing this back.
(the other thing that got me was switching to the other M2M PR, and remembering that this endpoint is useful as a "distinct" version of that query. it's not only useful for its counts)
i do think there's some extra logic needed in one of the loops that currently tries to avoid some work based on cursor assumptions.
aaaaaand i think there might be a bug in the existing logic, so this will be extra fun sorry! (i will see if i can finally get that to a proper repro or disprove it, so we can hopefully fix that first)
Sounds great! I think no matter how you look at it, the order is not entirely arbitrary as we collect results on the order they're stored.
Do you mind elaborating a bit further on your last two remarks before I open the PR again?! Might be easier for me to know what to look for then!
- Add "reverse" boolean parameter to three endpoints:
1. getManyToManyCounts (grouped counts)
2. getBacklinks (links to subject)
3. links (target links)
- Implement reverse iteration logic in both storage backends (mem_store
and rocks_store) for __reverse-chronological__ ordering
- Update HTML templates with (reverse) checkboxes
Before all our testing was done manually. This fixes the remaining failing automatic tests.
Adds test coverage for the reverse parameter in get_links and
get_many_to_many_counts methods. Tests verify that reverse: true
returns results in oldest-first/descending order while reverse:
false returns newest-first/ascending order.
The initial implementation of the reverse ordering feature had a flaw
in how it handled pagination. When `reverse=true` was set, the code
would reverse the entire collection before applying pagination
slicing, which broke the cursor logic. Cursor positions were
calculated relative to the original ordering but applied to a reversed
set of items, causing duplicate results or skipped items across page
boundaries.
The fix restructures pagination to account for reverse ordering:
- In reverse mode, pagination starts from the beginning (oldest items)
and moves forward, with begin/end indices calculated accordingly
- In normal mode, pagination continues working backward from the end
(newest items) as before
- The result slice is reversed __only after__ pagination boundaries
are established, ensuring cursor values remain consistent
Applied to both storage backends (mem_store and rocks_store) for
consistent behavior across implementations.
Looks like we might need a rebase, seems like this picked up (and is trying to undo) the changes from your other two PRs
✅ naming the query param reverse sounds good to me since it's consistent with com.atproto.repo.listRecords (and i assume/hopefully most other atproto lexicons)
i think we double-reverse for reverse-order backlinks:
could probably work it so we only call reverse once (when "forward". not confusing at all.)
Looks like we might need a rebase, seems like this picked up (and is trying to undo) the changes from your other two PRs
I think merging upstream into the fork might work as well and makes things easier for you as the reviewer. I don't mind rebasing ofc.
i think we double-reverse for reverse-order backlinks:
Yeah the existing .rev() calls were a bit confusing at first. Maybe changing the default order to return the oldest links per default might make things a bit clearer? Though I think this is mostly helping us as the maintainers as users might expect to get served the newest ones most of the time.
Another option could be introducing an "order" query parameter with two possible values asc and desc (or something similar) to make things a bit more clear what the default order is, but then we would lose the consistency between our endpoint and com.atproto.repo.listRecords.
We could reduce the number of times we reverse using the following I think:
microcosm.blue
let mut did_id_rkeys = linkers.0[begin..end].iter().collect::<Vec<_>>();
if !reverse {
did_id_rkeys.reverse();
}
Some comments might be justified though. That conditional looks super confusing lol
Upon thinking about this a bit more I think we should keep the reverse query parameter as is and don't touch the default order as this would be an unnecessary API break. Instead we could just introduce an enum and convert the query parameter to an enum value immediately after receiving. Other operations would be clearer then. We should still add some more context somewhere in the comments though. I had something like this in mind. What do you think?
// As backlinks are stored chronologically (oldest → newest) we need to reverse for newest-first queries
pub enum Order {
NewestToOldest, // default (reverse=false)
OldestToNewest, // reverse=true
}
impl From<bool> for Order {
fn from(reverse: bool) -> Self {
if reverse {
Order::OldestToNewest
} else {
Order::NewestToOldest
}
}
}
// In server/mod.rs, where we receive the query parameter
fn get_links(
accept: ExtractAccept,
query: axum_extra::extract::Query<GetLinkItemsQuery>,
store: impl LinkReader,
) -> Result<impl IntoResponse, http::StatusCode> {
// Convert boolean to enum right at the boundary
let order = Order::from(query.reverse);
// Now pass the enum to storage layer
let paged = store.get_links(
&query.target,
&query.collection,
&query.path,
order, // ← Clean enum instead of mysterious boolean
limit,
until,
&filter_dids,
)?;
// ...
}
// In rocks_store.rs get_links() - this replaces the confusing double-reverse!
let did_id_rkeys = match order {
Order::OldestToNewest => {
begin = until.map(|u| (u - 1) as usize).unwrap_or(0);
end = std::cmp::min(begin + limit as usize, total as usize);
next = if end < total as usize {
Some(end as u64 + 1)
} else {
None
};
// No reversal - storage is already chronological
linkers.0[begin..end].iter().collect::<Vec<_>>()
}
Order::NewestToOldest => {
end = until.map(|u| std::cmp::min(u, total)).unwrap_or(total) as usize;
begin = end.saturating_sub(limit as usize);
next = if begin == 0 { None } else { Some(begin as u64) };
// Reverse chronological storage to get newest-to-oldest
linkers.0[begin..end].iter().rev().collect::<Vec<_>>()
}
};
Thanks for the follow-up!
For the API, I agree with where you landed: keep it newest-first by default, optional reverse param being true makes it chronological. Small nit would be to skip the From<bool> impl since a bool itself doesn't inherently carry directional meaning; the endpoint gives it that meaning, so we can put the let order = if query.reverse { Order::OldestToNewest } ... directly in the endpoint.
i think you're right to just put the whole construction of the vec and cursor into branches on Order. part of me wants to try and encapsulate it a bit and switch the indexing to use linkers.0.iter().skip(begin).take(end) or whatever but the .rev() in there will still require a branch and/or make type things annoying and probably all of it less clear. i like your code.
comments about order are excellent thank you :)
Rebased the PR to match the current state of main and introduced the Order enum to make the sorting of items a bit clearer throughout the handlers that use the reverse parameter already (followed your suggestion regarding the From<bool> and put the definition of the order parameter directly in the endpoint itself before passing it as a parameter to the underlying handler).
Damn, I just realized I have not yet added a lexicon for the new endpoint. Will do that now. At least the endpoint logic itself should not be affected by that and thus be ready for review, given its similarity to the existing
getManyToManyCountsendpoint.