TestDataset

org.llm4s.rag.evaluation.TestDataset
See theTestDataset companion object
final case class TestDataset(name: String, samples: Seq[EvalSample], metadata: Map[String, String])

Test dataset for RAG evaluation.

Supports loading from JSON files and generating synthetic test cases from documents using LLM.

Value parameters

metadata

Additional metadata for tracking/filtering

name

Name identifier for this dataset

samples

The evaluation samples

Attributes

Example
{
// Load from file
val dataset = TestDataset.fromJsonFile("test_cases.json")
// Generate synthetic test cases
val generated = TestDataset.generateFromDocuments(
 documents = Seq("Paris is the capital of France...", "Tokyo is the capital of Japan..."),
 llmClient = client,
 samplesPerDoc = 3
)
// Save to file
TestDataset.save(dataset, "output.json")

}

Companion
object
Graph
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Members list

Value members

Concrete methods

def filter(predicate: EvalSample => Boolean): TestDataset

Filter samples based on a predicate.

Filter samples based on a predicate.

Attributes

def sample(n: Int, seed: Long): TestDataset

Get a random subset of samples.

Get a random subset of samples.

Attributes

def take(n: Int): TestDataset

Take first n samples.

Take first n samples.

Attributes

Get only samples that have ground truth.

Get only samples that have ground truth.

Attributes

def withMetadata(key: String, value: String): TestDataset

Add metadata to the dataset.

Add metadata to the dataset.

Attributes

Get samples without ground truth.

Get samples without ground truth.

Attributes

Inherited methods

def productElementNames: Iterator[String]

Attributes

Inherited from:
Product
def productIterator: Iterator[Any]

Attributes

Inherited from:
Product