Ollama (Local LLM) Cheatsheet

Ollama allows you to get up and running with large language models locally.

Installation

# macOS / Linux (macOS also has a GUI app)
curl -fsSL https://ollama.com/install.sh | sh

Enable GPU Support (Linux)

If you have an NVIDIA GPU, install the CUDA toolkit and NVIDIA container toolkit. Ollama will automatically detect and use the GPU.

# Verify GPU is detected by Ollama logs
journalctl -u ollama | grep -i gpu

Service Management (Linux)

sudo systemctl enable --now ollama   # Start and enable on boot
sudo systemctl status ollama         # Check status

Running Models

# Run a model (downloads it automatically if not present)
ollama run llama3
 
# Run a specific version/quantization
ollama run llama3:70b
 
# Run a model in silent mode (no interactive prompt, useful for scripting)
ollama run llama3 "Write a python script to reverse a string"

Managing Models

ollama list                          # List all downloaded models
ollama pull mistral                  # Download a model without running it
ollama rm llama3                     # Delete a downloaded model
ollama show llama3                   # Show details about a model (license, parameters, etc.)

Custom Models (Modelfile)

You can create custom models (similar to a Dockerfile) by tweaking parameters or providing a system prompt.

Create a file named Modelfile:

FROM llama3
# Set creativity (higher is more creative)
PARAMETER temperature 0.8
# Set the system prompt
SYSTEM """
You are a helpful senior Linux system administrator.
Always provide commands in markdown blocks.
"""

Build and run the custom model:

ollama create my-linux-admin -f Modelfile
ollama run my-linux-admin

REST API Usage

Ollama provides a local REST API (default port 11434).

Generate a Completion

curl http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

Chat Interface

curl http://localhost:11434/api/chat -d '{
  "model": "llama3",
  "messages": [
    {
      "role": "user",
      "content": "Hello, how are you?"
    }
  ],
  "stream": false
}'

Embeddings API

Generate vector embeddings from text for RAG (Retrieval-Augmented Generation) applications.

curl http://localhost:11434/api/embeddings -d '{
  "model": "nomic-embed-text",
  "prompt": "The quick brown fox jumps over the lazy dog"
}'

Environment Variables

Configure the Ollama server by setting these environment variables before starting it (or in the systemd service file).

OLLAMA_HOST : IP and port to listen on (default 127.0.0.1:11434). Set to 0.0.0.0 to allow remote access.
OLLAMA_MODELS : Path to where models are stored.
OLLAMA_KEEP_ALIVE : How long to keep models in memory (default 5m). Set to -1 to keep forever.

Cheatsheets