#246 — Modify vector database to include a metadata JSON field
Repo: Twill-AI/twill-llm-engine State: closed | Status: done Assignee: meliascosta
Created: 2025-01-29 · Updated: 2025-03-27
Description
See general RAG context here: https://twillpayments.atlassian.net/wiki/spaces/TD/pages/250216449/RAG
AC:
- New metadata column exists in table with JSONB type and indices set
- Endpoints are modified to expose the new filtering strategy on the JSONB column
Implementation details
Filter Queries will support a limited query language like used by pinecone and Langchain (see references)
References:
- Pinecone metadata filters https://docs.pinecone.io/guides/data/understanding-metadata#metadata-query-language
- Langchains PGVector implementation of filters: https://github.com/langchain-ai/langchain-postgres/blob/36ea595f0a39da4ce8e6957de6cd67b5888baa85/langchain_postgres/vectorstores.py#L1074
- Piccolo JSONB data type: https://piccolo-orm.readthedocs.io/en/latest/piccolo/schema/column_types.html#json
Notes
Add implementation notes, blockers, and context here
Related
Add wikilinks to related people, meetings, or other tickets