ollama-local

---
name: ollama-local
description: Use when the user wants to run an LLM locally without cloud API costs or data leaving their machine — installing Ollama, pulling/managing models, calling the REST API (generate, chat, embeddings, streaming), picking a model for their RAM budget, or wiring Ollama into Hermes as a provider.
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [ollama, local-llm, self-hosted, embeddings, offline-ai]
    related_skills: [http-api-tester]
---

# ollama-local

## Overview

Set up and use Ollama for running large language models locally. Ollama runs models on your machine — no API keys, no cloud, no per-token costs. The agent installs Ollama, pulls models, and shows you how to use them via the REST API or command line.

## When to Use

- The user wants to run an LLM locally without paying for API access.
- The user wants privacy — no data leaves their machine.
- The user wants to use a local model with their agent or application.
- The user says "set up Ollama", "run a local LLM", or "I want offline AI".

## Installation

### Linux

```bash
curl -fsSL https://ollama.com/install.sh | sh
```

### macOS

```bash
# Via Homebrew
brew install ollama

# Or download from https://ollama.com/download
```

### Windows

Download from https://ollama.com/download and run the installer. Ollama runs as a background service on Windows.

### Verify installation

```bash
ollama --version
# ollama version is 0.x.x
```

## Model Management

### Pull a model

```bash
# Small, fast model (good for testing)
ollama pull llama3.2:3b

# Medium model (good balance of speed and quality)
ollama pull llama3.1:8b

# Large model (best quality, needs 16GB+ RAM)
ollama pull llama3.1:70b

# Coding-focused model
ollama pull qwen2.5-coder:7b

# Embedding model
ollama pull nomic-embed-text
```

### List installed models

```bash
ollama list
```

### Run a model (interactive chat)

```bash
ollama run llama3.1:8b
>>> Tell me about quantum computing
```

### Remove a model

```bash
ollama rm llama3.2:3b
```

## API Usage

Ollama exposes a REST API at `http://localhost:11434`:

### Generate a response

```bash
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain recursion in one sentence.",
  "stream": false
}'
```

### Chat (multi-turn)

```bash
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [
    {"role": "user", "content": "What is 2+2?"},
    {"role": "assistant", "content": "4"},
    {"role": "user", "content": "What about 3+5?"}
  ],
  "stream": false
}'
```

### Generate embeddings

```bash
curl http://localhost:11434/api/embeddings -d '{
  "model": "nomic-embed-text",
  "prompt": "The quick brown fox jumps over the lazy dog."
}'
```

### Python client

```python
import requests

response = requests.post('http://localhost:11434/api/generate', json={
    'model': 'llama3.1:8b',
    'prompt': 'Write a haiku about the ocean.',
    'stream': False
})
print(response.json()['response'])
```

### Streaming responses

```python
import requests

response = requests.post('http://localhost:11434/api/generate', json={
    'model': 'llama3.1:8b',
    'prompt': 'Tell me a story.',
    'stream': True
}, stream=True)

for line in response.iter_lines():
    if line:
        import json
        chunk = json.loads(line)
        print(chunk.get('response', ''), end='', flush=True)
```

## Integration with Hermes

Configure Hermes to use the local Ollama instance as a provider:

```bash
# Set Ollama as a custom provider
hermes config set model.provider custom
hermes config set model.base_url http://localhost:11434/v1
hermes config set model.api_key ollama  # Ollama doesn't require a real key
hermes config set model.default llama3.1:8b
```

Or use Ollama for specific tasks (like auxiliary/compression) while keeping a cloud model for main reasoning:

```bash
hermes config set auxiliary.compression.provider custom
hermes config set auxiliary.compression.base_url http://localhost:11434/v1
hermes config set auxiliary.compression.model llama3.2:3b
```

## Model Selection Guide

| Model | Size | RAM needed | Best for |
|---|---|---|---|
| `llama3.2:3b` | 2 GB | 4 GB | Fast responses, simple tasks |
| `llama3.1:8b` | 5 GB | 8 GB | General purpose, good balance |
| `qwen2.5-coder:7b` | 5 GB | 8 GB | Code generation, debugging |
| `llama3.1:70b` | 40 GB | 64 GB | High quality, complex reasoning |
| `nomic-embed-text` | 0.3 GB | 1 GB | Embeddings for RAG/search |

## Performance Tips

- **Use GPU if available** — Ollama auto-detects NVIDIA/AMD GPUs and Apple Silicon. GPU inference is substantially faster than CPU; measure on your own hardware, since the gap depends on the model, quantisation, and VRAM.
- **Match model size to your RAM** — A model that doesn't fit in RAM will spill to disk and become extremely slow. Check `ollama ps` to see if the model is fully in memory.
- **Use smaller models for simple tasks** — Don't use a 70B model for a one-sentence answer. Use 3B or 8B for quick tasks.
- **Keep models loaded** — Ollama keeps models in memory for 5 minutes after last use by default. Increase this with `OLLAMA_KEEP_ALIVE` env var if you're making frequent requests.
- **Quantization** — Ollama uses 4-bit quantization by default, which reduces memory usage by ~70% with minimal quality loss. No configuration needed.

## Common Pitfalls

1. **Model larger than available RAM.** Ollama will fall back to disk swap and performance becomes unusable rather than failing outright — check `ollama ps` to confirm the model is fully resident in memory, and use a smaller model or add RAM if not.
2. **First pull looks "stuck".** The first `ollama pull` downloads the full model file (multiple GB); this can take minutes on a slow connection. Subsequent runs use the cached model and start instantly — don't kill the process assuming it's hung.
3. **Port conflict on 11434.** If another service already binds Ollama's default port, the server fails to start silently in some setups. Set `OLLAMA_HOST=0.0.0.0:11435` before starting and update client URLs to match.
4. **GPU not detected.** On Linux, missing NVIDIA drivers/CUDA toolkit means Ollama silently falls back to CPU (much slower) instead of erroring. Verify with `nvidia-smi` before assuming GPU is in use.
5. **Prompts exceed local context comfortably.** Local models advertise large context windows (e.g., 128k for Llama 3.1) but running at full context requires far more RAM than the base model size suggests. Keep prompts under ~8k tokens for 8B-class models in practice.
6. **Requests queue instead of running in parallel.** Ollama processes requests sequentially by default, so concurrent callers block each other. Set `OLLAMA_NUM_PARALLEL` if concurrency is needed.

## Verification Checklist

- [ ] `ollama --version` succeeds and `ollama list` shows the pulled model
- [ ] `ollama ps` confirms the model is loaded and shows a reasonable memory footprint (not swapping)
- [ ] A test `curl http://localhost:11434/api/generate` call returns a non-empty `response` field
- [ ] If GPU acceleration was expected, `ollama ps` or system GPU monitor (`nvidia-smi`) confirms it's actually being used
- [ ] If wired into Hermes as a provider, `hermes config get model.base_url` reflects the correct local URL and a real Hermes call round-trips successfully

# ollama-local



Run large language models locally with Ollama — no API keys, no cloud, no per-token costs.



## What it does



The agent installs Ollama, pulls a model that fits your hardware, and shows you how to use it via the command line or REST API. Models run entirely on your machine — nothing leaves your network. You can use the local model for chat, code generation, embeddings, or as a provider for Hermes Agent.



## Install



```bash

hermes skills install https://raw.githubusercontent.com/THEROCKSSS/hermes-skills-portfolio/main/skills/ollama-local/SKILL.md

```



## How to use



```

"Set up Ollama with a model that fits my 16GB RAM laptop"

```



The agent:

1. Installs Ollama (or verifies it's installed)

2. Recommends a model based on your RAM (llama3.1:8b for 16GB)

3. Pulls the model: `ollama pull llama3.1:8b`

4. Tests it: `ollama run llama3.1:8b "Hello"`

5. Shows you the REST API endpoint at http://localhost:11434



## Model selection



| Model | RAM needed | Best for |

|---|---|---|

| llama3.2:3b | 4 GB | Fast, simple tasks |

| llama3.1:8b | 8 GB | General purpose |

| qwen2.5-coder:7b | 8 GB | Code generation |

| llama3.1:70b | 64 GB | Complex reasoning |



## Example



```

User: "I want to use a local model for my Hermes agent instead of paying for API calls"



Agent:

  1. Checks RAM: 16 GB available

  2. Recommends: llama3.1:8b (fits in RAM, good quality)

  3. Pulls: ollama pull llama3.1:8b

  4. Configures Hermes:

     hermes config set model.provider custom

     hermes config set model.base_url http://localhost:11434/v1

     hermes config set model.default llama3.1:8b

  5. Returns: "Hermes is now using your local model. No API costs."

```