Feeds — Pagination & dedup

Every page of a feed is a snapshot, and snapshots move. The pagination block controls how pages compose into a stable scroll: cursor shape, dedup rules, and how many recent ids the cursor remembers.

The config surface

pagination: {
  pageSize?:    number            // default 20
  cursor?:      'rank' | 'rank+id'  // default 'rank+id'
  dedup?:       FeedDedupConfig     // default { by: 'sourceRowId' }
  seenWindow?:  number              // default 200
  excludeIds?:  string              // context path to ids to exclude
}

pageSize

How many items to return per call. Tier-clamped at the service layer, so an unreasonable value is capped silently — the response's meta.pageSize reports the effective cap.

cursor

The cursor carries the current scroll position:

• 'rank+id' (default) — encodes the last item's score and id. Stable under ties and under late-arriving rows with identical scores.
• 'rank' — encodes just the last score. Smaller cursor, but prone to duplication when scores tie.

Most feeds want 'rank+id'. Use 'rank' only when score is guaranteed unique (e.g. pure recency with a timestamp column).

dedup

Dedup is a discriminated union with three shapes:

dedup: false                                 // no dedup (rare)

dedup: { by: 'sourceRowId' }                 // dedup on row id, session-scoped

dedup: {
  by: 'author_id',                           // dedup on any mapped field
  within: 'session',                         // 'session' | 'page' | { window: N }
}

seenWindow

How many recent ids the cursor is allowed to remember. Default 200. Larger windows dedup better at the cost of bigger cursors. The signed cursor is still small (~a few hundred bytes at seenWindow: 200), but if you're scrolling into the thousands, drop it to 50 or use { within: { window: 50 } }.

excludeIds

A context path that resolves to ids to exclude from the result (e.g. the seed record in a "more like this" feed). Filtered after ranking.

pagination: {
  excludeIds: 'context.seedRecordId',   // single id
  // or
  excludeIds: 'context.already_shown',  // array of ids
}

Paginating from the client

foryou: {
  candidates: { from: 'embeddings', limit: 300 },
  rank: { similarity: { weight: 0.7, against: 'topics' },
          recency:    { weight: 0.3, halfLife: '7d' } },
  pagination: {
    pageSize:   20,
    cursor:     'rank+id',
    dedup:      { by: 'author_id', within: { window: 20 } },  // no author repeats in last 20
    seenWindow: 200,
  },
}

The evolved flag

FeedPage.evolved goes true when the supplied context's hash changed since the last page — it tells the UI "your context is different from the one this cursor was minted against, you probably want a fresh fetch."

The React hook watches context for you:

const { items, refetch } = useFeed(feed, {
  context:   { topics: ['security'] },
  onEvolve:  (tick) => refetch(),     // or 'topChanged' via evolveOn
})

Sweet spots by feed type

Gotchas

• Changing the feed config invalidates outstanding cursors. A cursor minted against seenWindow: 200 deserialized against seenWindow: 50 is rejected with 400 invalid_cursor. Your UI should fall back to a clean page fetch.
• Cursors are signed. Modifying them on the client is detected and rejected. Don't try to decode them; treat them as opaque.
• { within: 'page' } + small pageSize is essentially "no dedup." If you want cross-page dedup, stay on 'session'.