When we joined the project, AquiLLM had a basic 1-5 rating system for responses, but it was broken in two important ways: ratings and the AI responses they were tied to weren't being stored properly in the database, and users had no way to explain specifically how a response failed. Without that data, AquiLLM's developers had no real way to evaluate how the system was performing, and no strong direction for where to focus development.

Our work for Capstone 2026 solved these issues. We finished the rating system, added a written feedback option, and built tools that allowed developers to easily make sense of the feedback that was now being properly stored. We also created a feature that allows researchers to directly improve AquiLLM's responses in real time through their own feedback.

What We Delivered

• Ratings and written feedback with context: Ratings, text feedback, and the conversation context they connect back to are now reliably stored together in the database. Every piece of feedback can be traced back to the exact message, the model that generated it, and the conversation it came from.
• A feedback dashboard for developers: We implemented a new admin-only page that lets the team explore that feedback data easily. They can filter by rating, model, date, content, etc, and build queries to access feedback information without tedious terminal commands. They can also group results into charts, drill into individual conversations for context, and share queries with other admins via URLs. It's backed by a small custom query language that is largely modeled after KQL, with a few small differences to make it more suitable for this specific project.
• Plain-English access to the dashboard: To make things even simpler for the team, we implemented a Natural Language Processing skill that allows developers to simply type their query in plain English into the chat, i.e., "show me last week's 1-star ratings broken down by model", and the assistant translates that into a dashboard query and responds with a clickable link.
• A way for researchers to actively improve AquiLLM's responses: Researchers can now attach notes to individual collections, which allows them to give added context to the AI such as formatting preferences, citation conventions, domain caveats, and corrections to common mistakes using an in-browser editor. Those notes are automatically pulled in at runtime alongside the context from the collection, so domain experts can fix systematic issues themselves without waiting on a developer to deploy fixes. They can also directly translate their feedback into notes with a press of a button, by asking the AI to suggest a notes edit based on their feedback. This brings the project full circle by allowing users to not only give feedback properly, but also use that feedback to directly improve AquiLLM's responses for their entire research team.

Tools and Technologies

We worked within AquiLLM's existing toolchain, so we mostly extended what was already there instead of building an entirely new stack.

• Backend: Python and Django run the application, with PostgreSQL as the database. The chat interface stays in sync with the server in real time through Django Channels.
• Frontend: Built with React and TypeScript and packaged with Vite. The feedback dashboard renders its charts with Chart.js. The contextual notes editor feature is powered by Monaco, the same editor that's inside Microsoft's VS Code.
• Development environment: Everything runs locally inside Docker Compose, which packages the database, cache, web server, and supporting services together so any developer can spin up the full system with one command.

Possible Ideas For Future Work

• Visual figure search across collections: Today AquiLLM can search through text, so the next step may be searching for figures. For instance, a future feature might allow researchers to type something like "show me the IMF slope vs stellar mass plot, I think it was in one of the Smith papers" and have AquiLLM surface the actual figures with their captions and surrounding context. Researchers spend a lot of time hunting figures across PDFs they vaguely remember reading, so being able to ask AquiLLM for them directly would be a big improvement.