org.llm4s.rag.benchmark.BenchmarkRunner
See theBenchmarkRunner companion object
class BenchmarkRunner(llmClient: LLMClient, embeddingClient: EmbeddingClient, resolveEmbeddingProvider: String => Result[EmbeddingProviderConfig], datasetManager: DatasetManager, val options: BenchmarkRunnerOptions)
Main execution engine for RAG benchmarks.
Orchestrates the full benchmark workflow:
- Load dataset
- For each experiment configuration: a. Create RAG pipeline with config b. Index documents c. Run queries and generate answers d. Evaluate with RAGAS metrics e. Collect timing and results
- Aggregate results and generate reports
Value parameters
- datasetManager
-
Dataset loading manager
- embeddingClient
-
Default embedding client
- llmClient
-
LLM client for answer generation and evaluation
- options
-
Runner configuration options
Attributes
- Example
-
val runner = BenchmarkRunner(llmClient, embeddingClient, resolveEmbeddingProvider) val suite = BenchmarkSuite.chunkingSuite("data/datasets/ragbench/test.jsonl") val results = runner.runSuite(suite) println(BenchmarkReport.console(results)) - Companion
- object
- Graph
-
- Supertypes
-
class Objecttrait Matchableclass Any
Members list
In this article