core
API
Generated with
core/org.llm4s/org.llm4s.rag/org.llm4s.rag.loader/org.llm4s.rag.loader.internal/RobotsTxtParser/RobotsTxt

RobotsTxt

org.llm4s.rag.loader.internal.RobotsTxtParser.RobotsTxt
See theRobotsTxt companion object
final case class RobotsTxt(allowRules: Seq[String], disallowRules: Seq[String], crawlDelay: Option[Int])

Parsed robots.txt rules for a domain.

Value parameters

allowRules

Paths that are explicitly allowed

crawlDelay

Suggested delay between requests in seconds

disallowRules

Paths that are disallowed

Attributes

Companion
object
Graph
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Members list

Value members

Concrete methods

def isAllowed(path: String): Boolean

Check if a path is allowed according to these rules.

Check if a path is allowed according to these rules.

Uses order of specificity: longer matches take precedence. Allow rules take precedence over Disallow rules of equal length.

Attributes

Inherited methods

def productElementNames: Iterator[String]

Attributes

Inherited from:
Product
def productIterator: Iterator[Any]

Attributes

Inherited from:
Product
Generated with