Is the Eva repo appropriate for testing an agent built on pipecat in another repo?

[spilliton]

The team I am on has built a pipecat voice agent and we are exploring the options out there to test it under various scenarios to gain confidence that is working and compare metrics when we make changes to the agent prompts/code/models.

Does using Eva for a use-case like this make sense? I'm trying to get a feel for the spirit of this project and it feels a bit more like it is for evaluating the LLMs against some consistent agent types (CSM/HRSD/ITSM) than it is for writing scenarios/tests against agents built for production use. Seems like a tool like https://www.coval.ai/ is more appropriate for what we are trying to accomplish?

[fanny-riols]

Hi, thanks for reaching out.

1. EVA-Bench isn't only evaluating LLMs, but really meant to evaluate all types of models: STT, LLMs & TTS models in cascade architectures, or LALMs + TTS in hybrid, or speech-to-speech models. We also have a few metrics specifically evaluating STT & TTS models, such as Speech Fidelity for TTS, and debugging metrics Transcription Accuracy (Key Entities) and WER for STT. EVA-Bench is meant to evaluate voice agents end-to-end.
2. EVA-Bench and Coval solve different problems. Coval is more of a logging/monitoring platform for deployed agents; it's about observability in production. EVA-Bench is about systematic pre-production testing and validation. Coval could also be used that way, but might be less flexible and it might take more time to iterate on the prompts/code/models and get results. EVA-Bench is better for debugging, identifying failure modes and understanding what is going wrong in your agent. Coval deals with the agent as a black box, so it’s more difficult to get the type of feedback needed to improve the agent. I would say that EVA-Bench & Coval are complementary, not alternatives.
3. We currently have 3 datasets, but the framework is flexible and anyone can add their own dataset with scenarios specific to their use case and domain. Here is an article explaining how we build dataset: https://huggingface.co/blog/ServiceNow-AI/eva-bench-data . You can also test different user personas, background noise or user accents.