llm4s-core
API
Generated with
llm4s-core/org.llm4s/org.llm4s.rag/org.llm4s.rag.loader/org.llm4s.rag.loader.s3

org.llm4s.rag.loader.s3

package org.llm4s.rag.loader.s3

Members list

Type members

Classlikes

final case class S3DocumentSource(bucket: String, prefix: String, region: String, extensions: Set[String], credentials: Option[AwsCredentialsProvider], metadata: Map[String, String], endpointOverride: Option[String]) extends SyncableSource

Document source for AWS S3.

Document source for AWS S3.

Reads documents from an S3 bucket with support for:

  • Prefix filtering (e.g., "docs/", "reports/2024/")
  • File extension filtering
  • Automatic change detection via ETags
  • Pagination for large buckets

Authentication uses the AWS credential chain by default:

  1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
  2. System properties
  3. AWS credentials file (~/.aws/credentials)
  4. IAM role (for EC2/Lambda)

Usage:

val source = S3DocumentSource("my-bucket", prefix = "docs/")
val loader = SourceBackedLoader(source)
rag.sync(loader)

Value parameters

bucket

S3 bucket name

credentials

Optional credentials provider (default: AWS credential chain)

endpointOverride

Optional endpoint override (for LocalStack, MinIO, etc.)

extensions

File extensions to include (empty = all files)

metadata

Additional metadata to attach to all documents

prefix

Key prefix to filter objects (e.g., "docs/", "reports/")

region

AWS region (default: us-east-1)

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object S3DocumentSource

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
object S3Loader

Convenience factory for loading documents from AWS S3.

Convenience factory for loading documents from AWS S3.

S3Loader combines S3DocumentSource with SourceBackedLoader to provide a simple API for S3 document ingestion.

Usage:

// Basic usage with default credentials
val loader = S3Loader("my-bucket", prefix = "docs/")
rag.sync(loader)

// With explicit credentials
val loader = S3Loader.withCredentials(
 bucket = "my-bucket",
 accessKeyId = "...",
 secretAccessKey = "..."
)

// For LocalStack testing
val loader = S3Loader.forLocalStack("test-bucket")

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
S3Loader.type
Generated with