A simple scoring system for testing LLM chatbots

One of the key challenges in LLM-based solutions like chatbots and AI assistants, especially those using Retrieval-Augmented Generation (RAG), is the probabilistic nature of LLMs. This often leads to different outputs for the same input, making testing more complex. While analyzing anonymized chat logs in production helps improve these systems, there's a need for a more deterministic testing method to compare different solution versions effectively.

You cannot treat LLM chatbot testing like unit tests on deterministic code. Probabilistic outputs need scenario scores, tolerances, and regression checks that compare quality over time.

To address this, we use a simple scoring system based on common scenarios. For example, we validate whether responses contain specific elements such as expected phone numbers, addresses, keywords, or price ranges. In more complex cases, we might use ML classifiers or even another LLM to measure similarity between actual and expected responses. Based on the percentage of correct responses, we calculate an overall solution score. This approach allows us to easily compare versions and determine if new features improve or degrade answer quality.

This process can be automated within development pipelines, enabling faster iterations and reducing dependence on live data. It's a cost-effective strategy for pre-deployment checks and regression testing. By leveraging open-source tools like Promptfoo to generate score reports after each code check-in, we ensure that changes are production-ready and maintain solution quality.