Ollama — The Easiest Way to Run LLMs on Your Linux Server

Ollama has become the de-facto standard for running open-source LLMs on Linux servers. Version 0.1.30 brings significant performance improvements and a growing library of models.

What Ollama Supports in 2024

• Llama 3 — Meta's best open model (8B and 70B)
• Mistral 7B — excellent performance-to-size ratio
• Phi-3 — Microsoft's compact but capable model
• Code Llama — optimised for code generation
• Gemma — Google's open model
• 50+ models in the official library

Install and Run

curl -fsSL https://ollama.com/install.sh | sh

# Pull latest Llama 3
ollama pull llama3

# Run interactively
ollama run llama3

# List installed models
ollama list

REST API

# Chat completion
curl http://localhost:11434/api/chat -d '{
  "model": "llama3",
  "messages": [
    {"role": "user", "content": "Review this nginx config for security issues"}
  ]
}'

# Generate (single prompt)
curl http://localhost:11434/api/generate -d '{
  "model": "mistral",
  "prompt": "Write a bash script to backup MySQL databases",
  "stream": false
}'

Expose via Nginx with Authentication

# /etc/nginx/sites-available/ollama
server {
    listen 443 ssl;
    server_name ollama.yourdomain.com;

    # Basic auth protection
    auth_basic "Ollama API";
    auth_basic_user_file /etc/nginx/.htpasswd;

    # Rate limiting
    limit_req zone=general burst=10 nodelay;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_set_header Host $host;
        proxy_read_timeout 300s;
    }
}

Performance Tuning

# Set number of GPU layers (if GPU available)
OLLAMA_NUM_GPU=35 ollama run llama3

# Increase context window
ollama run llama3 --num-ctx 8192

# Run multiple models simultaneously
OLLAMA_MAX_LOADED_MODELS=3 ollama serve

Conclusion

Ollama makes self-hosted AI practical for any Linux server. Our team sets up and optimises AI server environments including Ollama deployments with secure Nginx proxying.