core
API
Generated with
core/org.llm4s/org.llm4s.rag/org.llm4s.rag.evaluation/TestDataset

TestDataset

org.llm4s.rag.evaluation.TestDataset
See theTestDataset companion class
object TestDataset

Attributes

Companion
class
Graph
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type

Members list

Type members

Inherited and Abstract types

type MirroredElemLabels <: Tuple

The names of the product elements

The names of the product elements

Attributes

Inherited from:
Mirror
type MirroredLabel <: String

The name of the type

The name of the type

Attributes

Inherited from:
Mirror

Value members

Concrete methods

def create(name: String, samples: Seq[EvalSample]): TestDataset

Create a dataset from a sequence of samples.

Create a dataset from a sequence of samples.

Attributes

def empty(name: String): TestDataset

Create an empty dataset.

Create an empty dataset.

Attributes

def fromJson(json: String): Result[TestDataset]

Parse dataset from JSON string.

Parse dataset from JSON string.

Value parameters

json

The JSON string

Attributes

Returns

The parsed dataset or an error

def fromJsonFile(path: String): Result[TestDataset]

Load dataset from a JSON file.

Load dataset from a JSON file.

Expected JSON format:

{
{
 "name": "my_dataset",
 "metadata": {"source": "manual"},
 "samples": [
   {
     "question": "What is the capital of France?",
     "answer": "Paris is the capital of France.",
     "contexts": ["Paris is the capital and largest city of France."],
     "ground_truth": "The capital of France is Paris.",
     "metadata": {"category": "geography"}
   }
 ]
}

}

Value parameters

path

Path to the JSON file

Attributes

Returns

The loaded dataset or an error

def generateFromDocuments(documents: Seq[String], llmClient: LLMClient, samplesPerDoc: Int, datasetName: String): Result[TestDataset]

Generate synthetic test cases from documents using LLM.

Generate synthetic test cases from documents using LLM.

For each document, generates question-answer pairs with the document serving as both context and source of ground truth.

Value parameters

datasetName

Name for the generated dataset

documents

The source documents

llmClient

LLM client for generation

samplesPerDoc

Number of QA pairs to generate per document

Attributes

Returns

Generated dataset or an error

def save(dataset: TestDataset, path: String): Result[Unit]

Save dataset to a JSON file.

Save dataset to a JSON file.

Value parameters

dataset

The dataset to save

path

Path for the output file

Attributes

Returns

Success or an error

def single(question: String, answer: String, contexts: Seq[String], groundTruth: Option[String]): TestDataset

Create a simple dataset with a single sample (useful for quick testing).

Create a simple dataset with a single sample (useful for quick testing).

Attributes

def toJson(dataset: TestDataset, pretty: Boolean): String

Convert dataset to JSON string.

Convert dataset to JSON string.

Value parameters

dataset

The dataset to convert

pretty

Whether to format with indentation

Attributes

Returns

JSON string representation

Generated with