Ollama Quick Start Guide

Run LLM4S with Ollama for free, local LLM inference - no API keys required!

Table of contents

  1. Why Ollama?
  2. Prerequisites
  3. Step 1: Install Ollama
    1. macOS / Linux
    2. Windows
    3. Verify Installation
  4. Step 2: Start Ollama Server
    1. Start the Server
    2. Verify Server is Running
  5. Step 3: Pull a Model
    1. Available Models
    2. List Downloaded Models
  6. Step 4: Configure LLM4S
    1. Add the dependencies
    2. Add an ollama-local section
  7. Step 5: Write Your First LLM4S + Ollama App
    1. Run It!
    2. Example Output
  8. Step 6: Try Different Models
  9. Step 6a: Write Your First LLM4S + Ollama/Llama3.2 App
  10. Step 6b: Write Your First LLM4S + Ollama/Phi3 App
  11. Step 6c: Write Your First LLM4S + Ollama/CodeLlama App
    1. Configure for CodeLlama
    2. Run It!
    3. Example Output
  12. Streaming Responses
  13. Tool Calling with Ollama
  14. Model Comparison
  15. Configuration Options
    1. Temperature Control
    2. Context Length
  16. Troubleshooting
    1. “Connection refused” error
    2. “Model not found” error
    3. Slow inference
    4. Model comparison
  17. Running the Examples
  18. Production Deployment
    1. Docker Compose Setup
    2. Pre-pull Models
  19. Ollama vs Cloud Providers
  20. Next Steps
  21. Resources

Why Ollama?

Ollama is the easiest way to run large language models locally on your machine:

  • ✅ 100% Free - No API costs or rate limits
  • ✅ Private - Your data never leaves your machine
  • ✅ Fast - Low latency for local inference
  • ✅ Offline - Works without internet connection
  • ✅ Multiple Models - Easy model switching (llama2, mistral, phi, etc.)

Perfect for development, testing, and production workloads where privacy matters.

Prerequisites

  • JDK 21
  • Scala 3.7.x (llm4s 0.5.1 is built with 3.7.1)
  • sbt 1.10.6+
  • 4-8GB RAM (depending on model size)

No API keys needed! 🎉

Ollama support ships in its own artifact, alongside llm4s-core:

1
2
3
4
5
6
7

val llm4sVersion = "0.5.1"

libraryDependencies ++= Seq(
  "org.llm4s" %% "llm4s-core"   % llm4sVersion,
  "org.llm4s" %% "llm4s-ollama" % llm4sVersion,
  "org.llm4s" %% "llm4s-agent"  % llm4sVersion  // only for the agent under "Tool Calling"
)

Adding llm4s-ollama registers the provider; there is nothing else to wire up. Before 0.5.0, Ollama was part of llm4s-core. See the installation guide.

Step 1: Install Ollama

macOS / Linux 

1

curl -fsSL https://ollama.com/install.sh | sh

Windows

Download the installer from ollama.com/download

Or use PowerShell:

1
2

# Download and install Ollama
winget install Ollama.Ollama

Verify Installation 

1
2

ollama --version
# Should output: ollama version is 0.x.x

Step 2: Start Ollama Server

Start the Server 

1

ollama serve

The server will start on http://localhost:11434 by default.

On Windows/macOS, Ollama may start automatically as a background service. Check your system tray/menu bar.

Verify Server is Running 

1
2

curl http://localhost:11434
# Should output: Ollama is running

Step 3: Pull a Model

Ollama models are pulled on-demand. Let’s start with Mistral 7B (fast and capable):

1

ollama pull mistral

Available Models

Model Size RAM Required Best For Pull Command
mistral 4.1GB 8GB General purpose, fast ollama pull mistral
llama2 3.8GB 8GB Good balance ollama pull llama2
phi 1.6GB 4GB Lightweight, fast ollama pull phi
neural-chat 4.1GB 8GB Conversational ollama pull neural-chat
codellama 3.8GB 8GB Code generation ollama pull codellama
llama3.2 2.0GB 8GB Latest Llama ollama pull llama3.2
gemma2 5.4GB 8GB Google’s model ollama pull gemma2

Recommendation: Start with mistral for the best balance of speed and quality.

List Downloaded Models 

1

ollama list

Step 4: Configure LLM4S

LLM4S reads providers from named sections in your application.conf, not from environment variables such as LLM_MODEL - nothing in the library reads those. See the Configuration Guide for the full picture.

Add the dependencies 

1
2
3
4
5

// build.sbt
libraryDependencies ++= Seq(
  "org.llm4s" %% "llm4s-core"   % llm4sVersion,
  "org.llm4s" %% "llm4s-ollama" % llm4sVersion  // the Ollama provider
)

Before 0.5.0 the Ollama provider shipped inside llm4s-core, so llm4s-core alone was enough.

Add an ollama-local section

Create src/main/resources/application.conf:

1
2
3
4
5
6
7
8
9
10
11

llm4s {
  providers {
    provider = "ollama-local"          # the default: the name of a section below

    ollama-local {
      provider = "ollama"
      model    = "mistral"
      # baseUrl defaults to http://localhost:11434, where `ollama serve` listens
    }
  }
}

No API key is needed. If you want to take the model or server from the environment, bind the variables yourself - the names are yours to choose:

1
2
3
4
5
6

    ollama-local {
      provider = "ollama"
      model    = "mistral"
      model    = ${?OLLAMA_MODEL}            # optional override from the environment
      baseUrl  = ${?OLLAMA_BASE_URL}         # unset: the default, http://localhost:11434
    }

Step 5: Write Your First LLM4S + Ollama App

Create HelloOllama.scala:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model._
import org.llm4s.model.ModelRegistryService

object HelloOllama extends App {
  // Create a conversation with system and user messages
  val conversation = Conversation(Seq(
    SystemMessage("You are a helpful AI assistant."),
    UserMessage("Explain what Scala is in one sentence.")
  ))

  // Load the default provider section and make the request
  val result = for {
    providerConfig <- Llm4sConfig.defaultProvider()
    registry       <- Llm4sConfig.modelRegistryService()
    given ModelRegistryService = registry
    client     <- LLMConnect.getClient(providerConfig)
    completion <- client.complete(conversation)
  } yield completion

  result match {
    case Right(completion) =>
      println(s"Response from ${completion.model}:")
      println(completion.message.content)
    case Left(error) =>
      Console.err.println(s"Error: ${error.formatted}")
  }
}

Run It! 

1

sbt run

Example Output 

1
2
3
4

✓ Response from mistral:
Scala is a statically-typed programming language that combines
object-oriented and functional programming paradigms, running on
the Java Virtual Machine (JVM).

Step 6: Try Different Models

You can easily switch models by changing model in the ollama-local section (after ollama pull <model>):

1
2
3
4
5

    ollama-local {
      provider = "ollama"
      model    = "llama3.2"   # or "phi3", "codellama", ...
      baseUrl  = "http://localhost:11434"
    }

If you added the model = ${?OLLAMA_MODEL} binding from Step 4, you can switch from the shell instead - export OLLAMA_MODEL=llama3.2 - because your own application.conf reads it.

Then run your program again without code changes!

Step 6a: Write Your First LLM4S + Ollama/Llama3.2 App

Llama 3.2 is Meta’s latest with an impressive 128K context window. Perfect for processing large documents and long conversations.

For the code, use the same Scala example from Step 5 - simply change the model in the ollama-local section of application.conf (run ollama pull llama3.2 first):

1
2
3
4
5

    ollama-local {
      provider = "ollama"
      model    = "llama3.2"
      baseUrl  = "http://localhost:11434"
    }

Then run:

1

sbt run

Why Llama 3.2? Latest Llama model with 128K context window. Excellent for RAG applications, long-form content generation, and processing large documents. Available in 1B, 3B, 8B, 70B, and 405B sizes.

Step 6b: Write Your First LLM4S + Ollama/Phi3 App

Phi3 is Microsoft’s efficient model, even smaller than Phi. Ideal for ultra-low-latency applications and edge deployment.

For the code, use the same Scala example from Step 5 - simply change the model in the ollama-local section of application.conf (run ollama pull phi3 first):

1
2
3
4
5

    ollama-local {
      provider = "ollama"
      model    = "phi3"
      baseUrl  = "http://localhost:11434"
    }

Then run:

1

sbt run

Why Phi3? Microsoft’s compact model optimized for efficiency. Smaller than Phi (1.4GB) with competitive quality. Perfect for resource-constrained environments and real-time applications requiring minimal latency.

Step 6c: Write Your First LLM4S + Ollama/CodeLlama App

CodeLlama is purpose-built for code generation and understanding. Create HelloCodeLlama.scala:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model._
import org.llm4s.model.ModelRegistryService

object HelloCodeLlama extends App {
  // Create a conversation asking for code
  val conversation = Conversation(Seq(
    SystemMessage("You are an expert Scala developer. Write clean, idiomatic code."),
    UserMessage("Write a simple Scala function that reverses a list.")
  ))

  // Load the default provider section and make the request
  val result = for {
    providerConfig <- Llm4sConfig.defaultProvider()
    registry       <- Llm4sConfig.modelRegistryService()
    given ModelRegistryService = registry
    client     <- LLMConnect.getClient(providerConfig)
    completion <- client.complete(conversation)
  } yield completion

  result match {
    case Right(completion) =>
      println(s"Code suggestion from ${completion.model}:")
      println(completion.message.content)
    case Left(error) =>
      println(s"Error: ${error.formatted}")
  }
}

Configure for CodeLlama

Set the model in the ollama-local section of application.conf (run ollama pull codellama first):

1
2
3
4
5

    ollama-local {
      provider = "ollama"
      model    = "codellama"
      baseUrl  = "http://localhost:11434"
    }

Run It! 

1

sbt run

Example Output 

1
2
3
4
5
6
7
8
9
10
11
12
13

✓ Code suggestion from codellama:
def reverseList[T](list: List[T]): List[T] = {
  list.reverse
}

// Or for manual reversal:
def reverseList[T](list: List[T]): List[T] = {
  def helper(acc: List[T], remaining: List[T]): List[T] = {
    if (remaining.isEmpty) acc
    else helper(remaining.head :: acc, remaining.tail)
  }
  helper(Nil, list)
}

Why CodeLlama? CodeLlama is specialized for code-related tasks. Use it for code generation, refactoring suggestions, and explaining code. 16K context window perfect for larger code files.

Streaming Responses

Get real-time token streaming (like ChatGPT):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model._
import org.llm4s.model.ModelRegistryService

object StreamingOllama extends App {
  val conversation = Conversation(Seq(
    SystemMessage("You are a concise assistant."),
    UserMessage("Write a haiku about Scala programming.")
  ))

  val result = for {
    providerConfig <- Llm4sConfig.defaultProvider()
    registry       <- Llm4sConfig.modelRegistryService()
    given ModelRegistryService = registry
    client     <- LLMConnect.getClient(providerConfig)
    completion <- client.streamComplete(
      conversation,
      CompletionOptions(),
      chunk => chunk.content.foreach(print)  // Print tokens as they arrive
    )
  } yield completion

  result match {
    case Right(completion) =>
      println("\n--- Streaming complete! ---")
      println(s"Total content: ${completion.message.content}")
    case Left(error) =>
      Console.err.println(s"Error: ${error.formatted}")
  }
}

Tool Calling with Ollama

Ollama supports tool calling (function calling) with compatible models, and llm4s-ollama’s native client sends the tools and reads the calls (streamed or not) itself - no /v1 OpenAI-compatible workaround is needed.

Only models whose Ollama page lists the tools capability can do this (llama3.1 can; the original llama3 does not). Sending tools to any other model fails, and the client reports it instead of dropping the tools:

1

Invalid tools: Ollama model 'llama3:latest' does not support tool calling (the server said: registry.ollama.ai/library/llama3:latest does not support tools). Use a model whose Ollama page lists the 'tools' capability, or send no tools.

Pull a tool-capable model (ollama pull llama3.1) and set OLLAMA_MODEL to it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

import org.llm4s.agent.Agent
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.model.ModelRegistryService
import org.llm4s.toolapi._
import upickle.default._

object OllamaTools extends App {
  // Define result type
  case class WeatherResult(forecast: String)
  implicit val weatherResultRW: ReadWriter[WeatherResult] = macroRW

  // Define a weather tool with proper schema
  val weatherSchema = Schema
    .`object`[Map[String, Any]]("Weather parameters")
    .withProperty(
      Schema.property("location", Schema.string("City or location name"))
    )

  val getWeather = ToolBuilder[Map[String, Any], WeatherResult](
    "get_weather",
    "Get the current weather in a location",
    weatherSchema
  ).withHandler { extractor =>
    extractor.getString("location").map { location =>
      // Mock implementation
      WeatherResult(s"Weather in $location: Sunny, 72F")
    }
  }.buildSafe()

  val result = for {
    weatherTool    <- getWeather
    tools          = new ToolRegistry(Seq(weatherTool))
    providerConfig <- Llm4sConfig.defaultProvider()
    registry       <- Llm4sConfig.modelRegistryService()
    given ModelRegistryService = registry
    client <- LLMConnect.getClient(providerConfig)
    agent <- Agent.builder("weather-agent", client).withTools(tools).build()
    result <- agent.run("What's the weather in San Francisco?")
  } yield result

  result match {
    case Right(r) =>
      println("Final response:")
      println(r.answer.getOrElse(s"Run ended: ${r.status}"))
    case Left(error) =>
      Console.err.println(s"Error: ${error.formatted}")
  }
}

Model Comparison

Performance comparison running on Apple M1 Mac:

Model Speed (tokens/sec) Quality Memory Best Use Case
mistral ~40 ⭐⭐⭐⭐ 8GB General purpose, great balance
llama2 ~35 ⭐⭐⭐⭐ 8GB Conversational, creative
phi ~80 ⭐⭐⭐ 4GB Quick tests, development
codellama ~35 ⭐⭐⭐⭐ 8GB Code generation
neural-chat ~40 ⭐⭐⭐⭐ 8GB Dialogue, chat apps

Performance varies by hardware. These are approximate values on M1 MacBook Pro 16GB RAM.

Configuration Options

Temperature Control 

1
2
3
4
5

CompletionOptions(
  temperature = 0.7,  // Higher = more creative (0.0-2.0)
  maxTokens = Some(1000),
  topP = 0.9
)

Context Length

Ollama models have different context windows:

  • mistral: 8k tokens
  • llama2: 4k tokens
  • llama3.2: 128k tokens
  • codellama: 16k tokens
  • gemma2: 8k tokens

Troubleshooting

“Connection refused” error

Problem: Ollama server not running

Solution:

1

ollama serve

“Model not found” error

Problem: Model not pulled

Solution:

1

ollama pull mistral

Slow inference

Problem: Not enough RAM or CPU

Solutions:

  • Use a smaller model: ollama pull phi
  • Close other applications
  • Check if running on GPU (M-series Mac, CUDA GPU)

Model comparison 

1
2
3
4
5

# List all models with sizes
ollama list

# Delete a model to free space
ollama rm llama2

Running the Examples

Try the built-in LLM4S Ollama samples. The samples’ own modules/samples/src/main/resources/application.conf already defaults to an ollama-local section (model llama3:latest, server http://localhost:11434), and binds OLLAMA_MODEL and OLLAMA_BASE_URL to override it - those two variables are that file’s bindings, not something the library reads. See Running the samples.

The default model has to be pulled first; to use a model you already have instead, name it with OLLAMA_MODEL (ollama list shows what is installed):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# Either pull the samples' default model once...
ollama pull llama3

# ...or override the samples' ollama-local section
export OLLAMA_MODEL=mistral
export OLLAMA_BASE_URL=http://localhost:11434

# Run basic Ollama example
sbt "samples/runMain org.llm4s.samples.basic.OllamaExample"

# Run Ollama streaming example
sbt "samples/runMain org.llm4s.samples.basic.OllamaStreamingExample"

# Run Ollama streaming example with raw provider exchange logging
sbt "samples/runMain org.llm4s.samples.basic.OllamaStreamingExample /tmp/my-provider-exchanges"

# Run tool calling example (works with any provider)
sbt "samples/runMain org.llm4s.samples.toolapi.BuiltinToolsExample"

The logging-enabled streaming example writes one timestamped JSONL file per run to the directory you provide. Each entry captures the raw request and response exchange, including accumulated streaming payloads for streaming-capable providers.

Production Deployment

Docker Compose Setup 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# docker-compose.yml
version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    mem_limit: 8g

  llm4s-app:
    build: .
    environment:
      # Read by the app's own application.conf binding: baseUrl = ${?OLLAMA_BASE_URL}
      - OLLAMA_BASE_URL=http://ollama:11434
    depends_on:
      - ollama

volumes:
  ollama-data:

This assumes the app’s ollama-local section binds the server address, as in Step 4: baseUrl = ${?OLLAMA_BASE_URL}. Without the variable the section uses http://localhost:11434.

Pre-pull Models 

1
2
3

# Pull models into Docker volume
docker-compose exec ollama ollama pull mistral
docker-compose exec ollama ollama pull llama2

Ollama vs Cloud Providers

Feature Ollama OpenAI Anthropic
Cost Free $0.01-0.06/1K tokens $0.003-0.015/1K tokens
Privacy 100% local Cloud-based Cloud-based
Speed Depends on hardware Fast (API) Fast (API)
Offline ✅ Yes ❌ No ❌ No
Model quality Good (7B-70B) Excellent (GPT-4) Excellent (Claude)
Setup 5 minutes Instant Instant

Use Ollama for:

  • Development and testing
  • Privacy-sensitive applications
  • Cost-conscious deployments
  • Offline environments

Use cloud providers for:

  • Highest quality responses
  • Scale without hardware limits
  • Latest model capabilities

Next Steps

Resources

💡 Pro Tip

Use Ollama for development and testing, then switch to a cloud provider for production by pointing llm4s.providers.provider at another section - add the provider’s module (e.g. llm4s-openai) and a section such as:

1
2
3
4

openai-main {
  provider = "openai"
  model    = "gpt-4o"
}

then set provider = "openai-main" (or -Dllm4s.providers.provider=openai-main). Only the selected section is validated, so OPENAI_API_KEY is needed only where openai-main is used. See Switching providers.

Your code stays exactly the same! 🎉