#62 — Distributed: solve replicas forth and back affect on caches
Repo: Twill-AI/facade State: open | Status: open Assignee: Unassigned
Created: 2024-08-02 · Updated: 2024-08-02
Description
Created during #56 implementation.
Approach
First search LLM history in cache, next try in db. At the end persist history with overwriting all.
may cause “missed some context” issues for replicas which appear for short time, serve some requests and go down.
In details: User started on replica A, chatted with it, next replica B appeared for a short period of time, user was switched to it by Gateway. In new replica B cache is empty so it took all history from db and user chatted with it without issues. But next replica B was disconnected/failed and user was switched back to replica A. Replica A here has old history in cache and:
- Will respond with outdated context.
- Will overwrite good context with broken one in db.
AC:
- Change LLM history caching behavior to resist against jittering replicas. See websocket_router.ConnectionManager.get_history comment.
Notes
Add implementation notes, blockers, and context here
Related
Add wikilinks to related people, meetings, or other tickets