llm4s-docs
API
Generated with
llm4s-docs/org.llm4s/org.llm4s.rag/org.llm4s.rag.embed

org.llm4s.rag.embed

package org.llm4s.rag.embed

Members list

Type members

Classlikes

object FileEmbedder

FileEmbedder handles extracting content from various file types and passing it to the appropriate embedding models.

FileEmbedder handles extracting content from various file types and passing it to the appropriate embedding models.

Encodes files of arbitrary MIME types into embedding vector sequences.

MIME type is detected automatically via Apache Tika. Dispatch then depends on the media type:

  • Text-like files (plain text, HTML, PDF, source code, …): text is extracted by org.llm4s.extract.TikaDocumentExtractor, optionally chunked, then embedded via the supplied EmbeddingClient. Real embeddings are always produced.

  • Image / Audio / Video: behaviour depends on config.experimentalStubs. When false, the file bytes are read (bounded by config.maxMediaFileSize) and forwarded to client.embedMultimodal() to obtain real provider embeddings. When true, a deterministic L2-normalised stub vector is returned instead; the vector is seeded from the file name, size, and last-modified time, so the same file always produces the same stub vector.

== Stub dimensions ==

Stub vectors are capped at MAX_STUB_DIMENSION (8 192) regardless of the configured model dimension, to prevent OOM errors during testing.

== Modality disambiguation ==

Each modality (image, audio, video) uses a different XOR seed constant when generating stub vectors, so stubs for the same file differ across modalities.

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
final case class FileEmbeddingConfig(textModel: EmbeddingModelConfig, localModels: LocalEmbeddingModels, chunking: TextChunkingConfig, experimentalStubs: Boolean, maxMediaFileSize: Long)

Everything FileEmbedder needs beyond the file itself and the client that embeds it.

Everything FileEmbedder needs beyond the file itself and the client that embeds it.

Grouping these means a caller configures the embedder once rather than repeating five arguments - including a stubs toggle that is a deployment decision, not a call-site one - at every call.

Value parameters

chunking

Text chunking settings; if enabled, extracted text is split before embedding.

experimentalStubs

When false, image/audio/video files are read and forwarded to client.embedMultimodal() for real provider embeddings. When true, deterministic stub vectors are returned instead.

localModels

Model configurations for the image, audio and video modalities.

maxMediaFileSize

Maximum allowed media file size in bytes. Files exceeding this limit are rejected with an error. Defaults to 50 MB.

textModel

Model configuration (name + dimensions) used for text-like files.

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
final case class TextChunkingConfig(enabled: Boolean, size: Int, overlap: Int)

Controls how extracted text is split before embedding.

Controls how extracted text is split before embedding.

Value parameters

enabled

When false, the full extracted text is embedded as a single unit.

overlap

Number of characters shared between adjacent chunks, to preserve context at chunk boundaries.

size

Target chunk size in characters.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object TextChunkingConfig

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
Generated with