#252 — NER improvements scoping

Repo: Twill-AI/twill-llm-engine State: closed | Status: done Assignee: meliascosta

Created: 2025-01-29 · Updated: 2025-02-25

Description

Based on https://twillpayments.atlassian.net/wiki/spaces/TD/pages/227770370/Beta+tester+s+error+analysis

We detected a number of errors in the responses generated by the LLM that are caused by the Named Entity Recognition step. This is a crucial first step to identify relevant tenant specific entities like vendors, customers, and accounts. We will need to review and modify the current fuzzy matching to address some of the errors found on the report.

For example, see the following query where the number 6 was matched with vendor ‘76 FUEL’

Original query	NER detected query
Create a chart showing sales, gross margin, operating profit, and net income for the last 6 months. Use monthly data for the x-axis and financial figures for the y-axis.	Create a chart showing sales, gross margin, operating profit, and net income for the last vendor ‘76 FUEL’ months. Use monthly data for the x-axis and financial figures for the y-axis.

AC:

• Caused of the wrong matches has been identified
• Mitigation strategy has been proposed
• Strategy has been implemented in LLM engine

Notes

Add implementation notes, blockers, and context here

Related

Add wikilinks to related people, meetings, or other tickets