TrueNAS as an AI Ops Platform: Beyond File Storage to Inference, Images, and Pipelines

Core AI Services Layer

• Ollama: LLM inference server (qwen2.5-coder:14b, deepseek-r1:14b, etc.)
• Automatic1111 / ComfyUI: Stable Diffusion web UIs for image generation
• LiteLLM: Proxy/load balancer for multiple LLM providers with fallback
• Open-WebUI: Chat interface for interacting with LLMs
• Qdrant: Vector database for RAG and semantic search
• LangFuse: LLM observability and tracing
• N8n: Workflow automation and orchestration
• PostgresQL: Metadata storage for applications
• Redis: Caching and message brokering

Infrastructure & Support Services

• TrueNAS SCALE: Base OS with Kubernetes and Docker
• GPU Passthrough: Direct access to NVIDIA RTX 5060 Ti for acceleration
• NGINX Proxy Manager: SSL termination, routing, and load balancing
• Fail2ban + CrowdSec: Intrusion prevention and IP blocking
• Netdata: Real-time performance monitoring
• Uptime Kuma: Service monitoring and alerts
• Volumes: Fast NVMe for apps/models, Storage Pool for datasets
• Snapshots: ZFS snapshots for quick rollback and experimentation

Data & Model Management

• Datasets: Curated collections for training/fine-tuning
• Model Registry: Versioned LLM and diffusion models
• Embeddings Cache: Pre-computed vectors for fast retrieval
• Prompt Library: Tested and optimized prompt templates
• Output Gallery: Curated collections of generated content
• Experiment Tracking: Parameters, results, and metadata for ML runs

Deployment Options

• TrueNAS App: Official Ollama app from Community Apps
• Custom Container: docker run with GPU support (more/ollama or similar
• Kubernetes: Helm chart or manifest

Configuration for TrueNAS

# Example: docker run command for TrueNAS
docker run -d \
  --name ollama \
  --restart unless-stopped \
  -p 11434:11434 \
  -v /mnt/nvme-pool/apps/ollama:/root/.ollama \
  -v /mnt/Storage_Pool/models:/models \
  --gpus all \
  -e OLLAMA_MODELS=/models \
  -e OLLAMA_HOST=0.0.0.0 \
  ollama/ollama

# Alternative: TrueNAS SCALE app configuration
# In the TrueNAS UI when installing Ollama app:
# - Port: 11434
# - Volumes:
#   Host Path: /mnt/nvme-pool/apps/ollama → Container Path: /root/.ollama
#   Host Path: /mnt/Storage_Pool/models → Container Path: /models
# - Runtime: NVIDIA (enables GPU access)
# - Environment Variables:
#   OLLAMA_MODELS: /models
#   OLLAMA_HOST: 0.0.0.0

Model Management Strategy

• Storage Location: Use your Storage Pool for model persistence (/mnt/Storage_Pool/models)
• Organization: Create folders by type: llm/, embedding/, vision/
• Versioning: Use subfolders for different quantizations (q4_K_M, q5_K_M, etc.)
• Symlinks: Point Ollama's model directory to your organized storage
• Backup: Snapshots of your model dataset for experimentation safety

Why ComfyUI Over Automatic1111?

• Better resource control: Fine-grained VRAM management
• Workflow-based: Reproducible, shareable generation pipelines
• Lower overhead: Less complex than full WebUI
• API-first: Easy to integrate with n8n and other automation
• TrueNAS-friendly: Less prone to conflicts with system services

Deployment Configuration

# Example: docker run for ComfyUI with GPU
docker run -d \
  --name comfyui \
  --restart unless-stopped \
  -p 8188:8188 \
  -v /mnt/nvme-pool/apps/comfyui:/opt/comfyui \
  -v /mnt/Storage_Pool/comfyui-input:/opt/comfyui/input \
  -v /mnt/Storage_Pool/comfyui-output:/opt/comfyui/output \
  -v /mnt/Storage_Pool/comfyui-models:/opt/comfyui/models \
  --gpus all \
  -e NVIDIA_VISIBLE_DEVICES=all \
  comfyui/comfyui:latest

# Alternative: Using TrueNAS SCALE app
# In TrueNAS UI app installation:
# - Port: 8188
# - Volumes:
#   /mnt/nvme-pool/apps/comfyui → /opt/comfyui (app data)
#   /mnt/Storage_Pool/comfyui-input → /opt/comfyui/input
#   /mnt/Storage_Pool/comfyui-output → /opt/comfyui/output
#   /mnt/Storage_Pool/comfyui-models → /opt/comfyui/models
# - Runtime: NVIDIA
# - Environment: NVIDIA_VISIBLE_DEVICES=all

Model Organization for ComfyUI

• Checkpoints: /mnt/Storage_Pool/comfyui-models/checkpoints/
• LoRAs: /mnt/Storage_Pool/comfyui-models/loras/
• ControlNet: /mnt/Storage_Pool/comfyui-models/controlnet/
• VAEs: /mnt/Storage_Pool/comfyui-models/vae/
• Embeddings: /mnt/Storage_Pool/comfyui-models/embeddings/
• Upscalers: /mnt/Storage_Pool/comfyui-models/upscale/

Why Use LiteLLM?

• Provider fallback: If Ollama fails, try local Llama.cpp or remote API
• Load balancing: Distribute requests across multiple instances
• Cost tracking: Monitor usage and expenses for paid APIs
• Standardized interface: OpenAI-compatible endpoint for all providers
• Cache layers: Redis caching for repeated requests
• Security: API key management and rate limiting

Typical Configuration

# lite llm settings.yaml
model_list:
  - model_name: ollama_qwen
    litellm_provider: ollama
    model_name: qwen2.5-coder:14b
    api_base: http://localhost:11434
    num_retries: 3
    timeout: 120
    max_retries: 3

  - model_name: ollama_deepseek
    litellm_provider: ollama
    model_name: deepseek-r1:14b
    api_base: http://localhost:11434
    num_retries: 3
    timeout: 120

  - model_name: local_llama_cpp
    litellm_provider: llama_cpp
    model_path: /mnt/Storage_Pool/models/llama-3.1-8b-instruct.q4_K_M.gguf
    num_retries: 3
    timeout: 60

  - model_name: openai_backup
    litellm_provider: openai
    model: gpt-4o-mini
    api_key: ${OPENAI_API_KEY}
    num_retries: 3
    timeout: 60

# Routing strategy: try local first, then fall back
router:
  type: LLMSingleAgent
  model_list:
    - ollama_qwen
    - ollama_deepseek
    - local_llama_cpp
    - openai_backup

# Optional: Add Redis caching
cache:
  type: redis
  host: localhost
  port: 6379

# Start LiteLLM
litellm --config /path/to/settings.yaml --port 4000

Why Qdrant?

• TrueNAS-friendly: Official app available
• Performance: Optimized for vector search
• Scalability: Handles millions of vectors efficiently
• Features: Filtering, payload storage, hybrid search
• Easy integration: Works with LangChain, LlamaIndex, custom code
• Backup-friendly: Snapshots and exports work well

Deployment and Usage

# TrueNAS SCALE app deployment
# In TrueNAS UI:
# - App: Qdrant (from Community Apps)
# - Port: 6333 (HTTP) and 6334 (gRPC)
# - Volumes:
#   Host: /mnt/Storage_Pool/qdrant-storage → Container: /qdrant/storage
#   Host: /mnt/Storage_Pool/qdrant-snapshots → Container: /qdrant/snapshots
# - Resources: Limit RAM usage if needed (e.g., 4GB)

# Example: Using Qdrant with Ollama for RAG
import requests
from ollama import Client

# Initialize clients
ollama_client = Client(host='http://localhost:11434')
qdrant_url = "http://localhost:6333"

def embed_text(text):
    """Generate embedding using Ollama"""
    response = ollama_client.embeddings(
        model="nomic-embed-text",
        prompt=text
    )
    return response['embedding']

def store_document(doc_id, text, metadata=None):
    """Store document in Qdrant"""
    vector = embed_text(text)
    payload = {
        "text": text,
        "metadata": metadata or {}
    }

    requests.put(
        f"{qdrant_url}/collections/my_documents/points",
        json={
            "points": [
                {
                    "id": doc_id,
                    "vector": vector,
                    "payload": payload
                }
            ]
        }
    )

def search_documents(query, limit=5):
    """Search for similar documents"""
    query_vector = embed_text(query)

    response = requests.post(
        f"{qdrant_url}/collections/my_documents/points/search",
        json={
            "vector": query_vector,
            "limit": limit,
            "with_payload": true
        }
    )

    return response.json()['result']

# Usage example
store_document(
    "doc1",
    "How to optimize SQL queries in SAP S/4HANA",
    {"topic": "SAP", "type": "guide", "author": "AJ"}
)

results = search_documents("SAP performance tuning tips")
for result in results:
    print(f"Score: {result['score']:.3f}")
    print(f"Text: {result['payload']['text'][:100]}...")
    print("---")

Why Observe Your LLMs?

• Cost tracking: Monitor token usage and expenses
• Performance: Measure latency and throughput
• Quality: Track hallucinations, errors, and user feedback
• Debugging: Trace complex chains and agent workflows
• Optimization: Identify bottlenecks and inefficient patterns
• Compliance: Audit trails for regulated industries

Deployment Configuration

# docker run for LangFuse
docker run -d \
  --name langfuse \
  --restart unless-stopped \
  -p 3100:3000 \
  -v /mnt/nvme-pool/apps/langfuse:/langfuse \
  -e DATABASE_URL=postgresql://langfuse:langfuse@localhost:5432/langfuse \
  -e NEXTAUTH_SECRET=$(openssl rand -hex 32) \
  -e NEXTAUTH_URL=http://localhost:3100 \
  -e SIGNING_KEY=$(openssl rand -hex 32) \
  langfuse/langfuse:latest

# Optional: With external Postgres
# -v /mnt/Storage_Pool/postgres:/var/lib/postgresql/data
# -e POSTGRES_USER=langfuse
# -e POSTGRES_PASSWORD=langfuse
# -e POSTGRES_DB=langfuse

# Initialize in your application
from langfuse import Langfuse

langfuse = Langfuse(
    public_key="pk-lf-...",
    secret_key="sk-lf-...",
    host="http://localhost:3100"
)

# Track LLM calls
with langfuse.start_as_current_span(name="llm-call") as span:
    span.update_trace(
        input={"question": "How do I optimize this ABAP code?"},
        metadata={"user_id": "user123"}
    )

    response = ollama_client.generate(
        model="qwen2.5-coder:14b",
        prompt="How do I optimize this ABAP code for performance?"
    )

    span.update_trace(
        output={"answer": response['response']},
        metadata={"model": "qwen2.5-coder:14b", "tokens": response.get('eval_count', 0)}
    )

Why N8n for AI Workflows?

• Visual workflows: Easy to understand and modify
• Hundreds of integrations: Connect to LLMs, APIs, databases, files
• Built-in scheduling: Cron-like triggering of workflows
• Error handling: Retry policies, error workflows, alerts
• Custom nodes: Write JavaScript or Python for specialized tasks
• Webhook triggers: Start workflows from external events
• TrueNAS-proven: Well-established in the TrueNAS community

Example AI Workflows

# Workflow 1: Automated Content Generation Pipeline
# Trigger: Webhook (receive request via HTTP)
# Steps:
# 1. Receive request: { "topic": "SAP AI Units", "type": "article" }
# 2. Query knowledge base: Search Qdrant for relevant documents
# 3. Enhance with LLM: Ask Ollama to summarize findings
# 4. Generate outline: Use LLM to create article structure
# 5. Write sections: Loop through outline, generate each section
# 6. Create images: Use ComfyUI API to generate illustrations
# 7. Assemble: Combine text and images into final document
# 8. Notify: Send completion via email/n8n notification
# 9. Store: Save output to /mnt/Storage_Pool/published-content/

# Workflow 2: Model Maintenance and Updates
# Trigger: Schedule (weekly)
# Steps:
# 1. Check for new models: Query Hugging Face API for updates
# 2. Download: Fetch new versions to /mnt/Storage_Pool/models/
# 3. Test: Run inference tests on new models
# 4. Compare: Evaluate against benchmark suite
# 5. Deploy: Promote to production if passes tests
# 6. Notify: Send report on updates performed
# 7. Cleanup: Remove old versions based on retention policy

# Workflow 3: RAG Pipeline Maintenance
# Trigger: Schedule (daily) or webhook (when new docs added)
# Steps:
# 1. Scan for new documents: Check /mnt/Storage_Pool/documents/
# 2. Extract text: Use appropriate loader (PDF, DOCX, TXT, etc.)
# 3. Chunk text: Split into optimal segments for embedding
# 4. Generate embeddings: Use nomic-embed-text via Ollama
# 5. Store vectors: Add to Qdrant collection
# 6. Update metadata: Track when documents were last processed
# 7. Notify: Report on indexing progress

NVMe Pool (High Performance)

Use for:

• Operating system and TrueNAS SCALE
• Application containers and images
• Frequently accessed model files (active LLMs)
• Vector database working sets (Qdrant RAM cache overflow)
• Temporary processing space
• Logs and metrics that need fast writes

Example paths:

• /mnt/nvme-pool/apps/ - Container-app data
• /mnt/nvme-pool/models/active/ - Currently loaded models
• /mnt/nvme-pool/logs/ - Application logs
• /mnt/nvme-pool/cache/ - Redis, temporary files

Storage Pool (Capacity & Reliability)

Use for:

• Master copies of all models (quantized versions)
• Training datasets and archives
• Generated outputs (images, text, etc.)
• Long-term backups and snapshots
• Vector database persistent storage
• Media assets for ComfyUI workflows
• Document libraries for RAG systems

Example paths:

• /mnt/Storage_Pool/models/ - All model versions
• /mnt/Storage_Pool/datasets/ - Training data
• /mnt/Storage_Pool/generated/ - AI outputs
• /mnt/Storage_Pool/qdrant/ - Vector DB storage
• /mnt/Storage_Pool/documents/ - Source documents
• /mnt/Storage_Pool/comfyui/ - Input/output/media

SAP Development Assistant

Components: Ollama (Code Llama), Qdrant (SAP docs), n8n (workflow)

Capabilities:

• Explain complex ABAP code snippets
• Generate unit tests for SAP classes
• Search SAP documentation using natural language
• Suggest performance improvements for SQL queries
• Generate functional specs from user stories
• Explain error messages and suggest fixes

Content Creation Pipeline

Components: Ollama (writing), ComfyUI (images), n8n (orchestration), Qdrant (research)

Capabilities:

• Research topics and generate outlines
• Write draft articles and blog posts
• Create custom illustrations and diagrams
• Generate social media content variations
• Optimize content for SEO and readability
• Publish to multiple platforms automatically

Data Analysis assistant

Components: Ollama (analysis), Postgres (data), LangFuse (tracking)

Capabilities:

• Generate SQL queries from natural language
• Explain complex query execution plans
• Suggest optimizations for slow-running reports
• Generate insights from data trends
• Create data visualization specifications
• Explain statistical concepts in business terms

Automated Customer Support

Components: Ollama (LLM), Qdrant (knowledge base), n8n (workflow)

Capabilities:

• Answer common product questions automatically
• Route complex issues to appropriate human agents
• Generate troubleshooting steps for reported issues
• Create knowledge base articles from resolved tickets
• Analyze support trends and suggest improvements
• Provide 24/7 basic support coverage

TrueNAS AI Ops Platform (Annualized)

Equivalent Cloud Services (AWS/Azure/GCP)