Web Scraper Console — Complete Tutorial

Six containers cooperate over a shared Docker network. The browser only ever talks to the frontend (port 8000) and the API (port 8080). Everything else — the headless browser farm, S3-compatible storage, the vector DB, and the LLM — is reached only through the API.

The two main flows

File map

scraper/
├── docker-compose.yml          # 6-service orchestration
├── api/                        # Go HTTP API (port 8080)
│   ├── main.go                 # entrypoint, router, config
│   ├── models.go               # request/response/job types
│   ├── handlers.go             # HTTP handlers
│   ├── scraper.go              # BFS crawler
│   ├── storage.go              # S3/LocalStack client
│   ├── ai.go                   # ChromaDB + Ollama
│   ├── api_test.go             # mocked end-to-end tests
│   └── Dockerfile              # multi-stage build
├── playwright-service/         # JS-page renderer (port 3000)
│   ├── server.js               # Express + Playwright
│   ├── package.json
│   └── Dockerfile
├── frontend/                   # static UI (port 8000)
│   ├── index.html              # Tailwind + JS console
│   └── Dockerfile              # nginx
└── scripts/
    └── init-aws.sh             # creates the S3 bucket on LocalStack boot

The full file

Compose v2 syntax (no top-level version: key needed). Each service block defines an image (or build context), ports, env vars, dependencies, and any volumes or networks.

Block 1 — The Go API service

services:
  api:
    build: ./api
    container_name: scraper-api
    ports:
      - "8080:8080"
    environment:
      - PLAYWRIGHT_URL=http://playwright:3000
      - S3_ENDPOINT=http://localstack:4566
      - S3_BUCKET=scraper-data
      - AWS_ACCESS_KEY_ID=test
      - AWS_SECRET_ACCESS_KEY=test
      - AWS_REGION=us-east-1
      - CHROMA_URL=http://chromadb:8000
      - OLLAMA_URL=http://ollama:11434
      - OLLAMA_MODEL=llama3.2
      - OLLAMA_EMBED_MODEL=nomic-embed-text
    depends_on:
      - playwright
      - localstack
      - chromadb
      - ollama
    networks:
      - scraper-net

Block 2 — Playwright renderer

  playwright:
    build: ./playwright-service
    container_name: scraper-playwright
    ports:
      - "3000:3000"
    networks:
      - scraper-net

Block 3 — Frontend (nginx)

  frontend:
    build: ./frontend
    container_name: scraper-frontend
    ports:
      - "8000:80"
    depends_on:
      - api
    networks:
      - scraper-net

Block 4 — LocalStack

  localstack:
    image: localstack/localstack:latest
    container_name: scraper-localstack
    ports:
      - "4566:4566"
    environment:
      - SERVICES=s3
      - DEBUG=0
      - AWS_DEFAULT_REGION=us-east-1
    volumes:
      - ./scripts/init-aws.sh:/etc/localstack/init/ready.d/init-aws.sh
    networks:
      - scraper-net

Block 5 — ChromaDB

  chromadb:
    image: chromadb/chroma:latest
    container_name: scraper-chromadb
    ports:
      - "8001:8000"
    environment:
      - IS_PERSISTENT=TRUE
      - ANONYMIZED_TELEMETRY=FALSE
    volumes:
      - chroma-data:/data
    networks:
      - scraper-net

Block 6 — Ollama + the one-shot model puller

  ollama:
    image: ollama/ollama:latest
    container_name: scraper-ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    networks:
      - scraper-net

  ollama-init:
    image: ollama/ollama:latest
    container_name: scraper-ollama-init
    depends_on:
      - ollama
    entrypoint: >
      sh -c "
        sleep 5 &&
        OLLAMA_HOST=http://ollama:11434 ollama pull llama3.2 &&
        OLLAMA_HOST=http://ollama:11434 ollama pull nomic-embed-text
      "
    networks:
      - scraper-net
    restart: "no"

Block 7 — Volumes & network

volumes:
  chroma-data:
  ollama-data:

networks:
  scraper-net:
    driver: bridge

References

• Compose documentation
• Compose file reference
• LocalStack init hooks

scripts/init-aws.sh

LocalStack mounts /etc/localstack/init/ready.d/ and runs every script in there once the gateway is healthy. We use that hook to create our bucket.

#!/bin/bash
echo "Creating S3 bucket: scraper-data"
awslocal s3 mb s3://scraper-data
awslocal s3api put-bucket-cors --bucket scraper-data --cors-configuration '{
  "CORSRules": [{
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "PUT", "POST"],
    "AllowedOrigins": ["*"]
  }]
}'
echo "Bucket created successfully"

Why mount this script in ready.d/?

LocalStack supports four lifecycle hook directories: boot.d/ (before LocalStack starts), start.d/ (during startup), ready.d/ (after the gateway accepts traffic), and shutdown.d/. We use ready.d/ because we need the S3 service available to make the bucket call.

References

• LocalStack docs
• Initialization hooks
• AWS CLI s3 reference
• S3 CORS configuration

The full server.js

A single Express app with a /health probe and one real endpoint.

Imports & setup

const express = require('express');
const { chromium } = require('playwright');

const app = express();
app.use(express.json({ limit: '10mb' }));

let browser;

Lazy browser launch

async function getBrowser() {
  if (!browser || !browser.isConnected()) {
    browser = await chromium.launch({
      headless: true,
      args: ['--no-sandbox', '--disable-setuid-sandbox']
    });
  }
  return browser;
}

app.get('/health', (req, res) => {
  res.json({ status: 'ok' });
});

The /render handler

app.post('/render', async (req, res) => {
  const { url, screenshot = false, timeout = 30000 } = req.body;

  if (!url) {
    return res.status(400).json({ error: 'url is required' });
  }

  let context;
  let page;
  try {
    const b = await getBrowser();
    context = await b.newContext({
      userAgent: 'Mozilla/5.0 (compatible; SimpleScraper/1.0)',
      viewport: { width: 1280, height: 800 }
    });
    page = await context.newPage();

    await page.goto(url, { waitUntil: 'networkidle', timeout });

Extracting content

    const html = await page.content();
    const text = await page.evaluate(() => document.body ? document.body.innerText : '');
    const title = await page.title();

    const links = await page.evaluate(() => {
      const anchors = Array.from(document.querySelectorAll('a[href]'));
      return anchors.map(a => a.href).filter(h => h && (h.startsWith('http://') || h.startsWith('https://')));
    });

Screenshot & cleanup

    const result = {
      url, title, html, text,
      links: [...new Set(links)]
    };

    if (screenshot) {
      const buf = await page.screenshot({ fullPage: true, type: 'png' });
      result.screenshot = buf.toString('base64');
    }

    res.json(result);
  } catch (err) {
    console.error('Render error:', err.message);
    res.status(500).json({ error: err.message });
  } finally {
    if (page) await page.close().catch(() => {});
    if (context) await context.close().catch(() => {});
  }
});

Server bootstrap & graceful shutdown

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Playwright service listening on port ${PORT}`);
});

process.on('SIGTERM', async () => {
  if (browser) await browser.close();
  process.exit(0);
});

package.json

{
  "name": "playwright-service",
  "version": "1.0.0",
  "description": "Playwright-based page renderer",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.19.2",
    "playwright": "^1.47.0"
  }
}

Dockerfile

FROM node:20-bookworm-slim

WORKDIR /app

COPY package.json ./
RUN npm install --omit=dev

# Install the chromium build that matches whatever playwright version
# npm just resolved, plus all of its system library dependencies.
RUN npx playwright install --with-deps chromium \
 && rm -rf /var/lib/apt/lists/*

COPY server.js ./

EXPOSE 3000

CMD ["node", "server.js"]

References

• Playwright introduction
• Playwright Page API
• browser.newContext()
• Playwright Docker guide
• Express 4.x API

Reading a Go service is much easier when you start with its types. These structs flow through every layer: the HTTP handlers decode them, the scraper builds them, the storage layer reads from them, and the AI layer reads metadata from them.

Package & imports

package main

import "time"

The request types

// ScrapeRequest is the body for POST /api/scrape
type ScrapeRequest struct {
	URL    string `json:"url"`
	Depth  int    `json:"depth"`
	Format string `json:"format"` // "text", "html", or "screenshot"
}

The job & page types

// ScrapeJob represents one crawl operation
type ScrapeJob struct {
	ID        string       `json:"id"`
	SeedURL   string       `json:"seed_url"`
	Depth     int          `json:"depth"`
	Format    string       `json:"format"`
	Status    string       `json:"status"` // pending|running|done|failed
	Error     string       `json:"error,omitempty"`
	CreatedAt time.Time    `json:"created_at"`
	UpdatedAt time.Time    `json:"updated_at"`
	Pages     []ScrapePage `json:"pages"`
	Indexed   bool         `json:"indexed"`
}

// ScrapePage is a single page captured in a job
type ScrapePage struct {
	URL     string `json:"url"`
	Title   string `json:"title"`
	S3Key   string `json:"s3_key"`
	Depth   int    `json:"depth"`
	Format  string `json:"format"`
	Bytes   int    `json:"bytes"`
}

The query types

// QueryRequest is the body for POST /api/query
type QueryRequest struct {
	JobID  string `json:"job_id,omitempty"` // optional: scope to one job
	Prompt string `json:"prompt"`
	TopK   int    `json:"top_k,omitempty"`
}

// QueryResponse from POST /api/query
type QueryResponse struct {
	Answer  string   `json:"answer"`
	Sources []string `json:"sources"`
}

The Playwright response shape

// PlaywrightResp shape from the playwright service
type PlaywrightResp struct {
	URL        string   `json:"url"`
	Title      string   `json:"title"`
	HTML       string   `json:"html"`
	Text       string   `json:"text"`
	Links      []string `json:"links"`
	Screenshot string   `json:"screenshot,omitempty"` // base64 png
	Error      string   `json:"error,omitempty"`
}

References

• encoding/json package
• Struct type spec
• JSON and Go (blog)

Imports

package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"sync"
	"time"

	"github.com/go-chi/chi/v5"
	"github.com/go-chi/chi/v5/middleware"
	"github.com/go-chi/cors"
)

Config + env loading

type Config struct {
	PlaywrightURL    string
	S3Endpoint       string
	S3Bucket         string
	AWSRegion        string
	ChromaURL        string
	OllamaURL        string
	OllamaModel      string
	OllamaEmbedModel string
}

func loadConfig() Config {
	return Config{
		PlaywrightURL:    getEnv("PLAYWRIGHT_URL", "http://playwright:3000"),
		S3Endpoint:       getEnv("S3_ENDPOINT", "http://localstack:4566"),
		S3Bucket:         getEnv("S3_BUCKET", "scraper-data"),
		AWSRegion:        getEnv("AWS_REGION", "us-east-1"),
		ChromaURL:        getEnv("CHROMA_URL", "http://chromadb:8000"),
		OllamaURL:        getEnv("OLLAMA_URL", "http://ollama:11434"),
		OllamaModel:      getEnv("OLLAMA_MODEL", "llama3.2"),
		OllamaEmbedModel: getEnv("OLLAMA_EMBED_MODEL", "nomic-embed-text"),
	}
}

func getEnv(k, def string) string {
	if v := os.Getenv(k); v != "" {
		return v
	}
	return def
}

The Server struct — dependency injection

type Server struct {
	cfg     Config
	storage *Storage
	ai      *AI
	scraper *Scraper

	// In-memory job registry. Suitable for a simple demo;
	// for production swap with a real DB.
	mu   sync.RWMutex
	jobs map[string]*ScrapeJob
}

func newServer(cfg Config) (*Server, error) {
	storage, err := NewStorage(context.Background(), cfg)
	if err != nil {
		return nil, err
	}
	ai := NewAI(cfg)
	scraper := NewScraper(cfg, storage)

	return &Server{
		cfg:     cfg,
		storage: storage,
		ai:      ai,
		scraper: scraper,
		jobs:    make(map[string]*ScrapeJob),
	}, nil
}

Router configuration

func (s *Server) router() http.Handler {
	r := chi.NewRouter()
	r.Use(middleware.RequestID)
	r.Use(middleware.RealIP)
	r.Use(middleware.Logger)
	r.Use(middleware.Recoverer)
	r.Use(middleware.Timeout(5 * time.Minute))

	r.Use(cors.Handler(cors.Options{
		AllowedOrigins:   []string{"*"},
		AllowedMethods:   []string{"GET", "POST", "OPTIONS"},
		AllowedHeaders:   []string{"Content-Type", "Authorization"},
		AllowCredentials: false,
	}))

	r.Get("/health", s.handleHealth)

	r.Route("/api", func(r chi.Router) {
		r.Post("/scrape", s.handleScrape)
		r.Get("/scrapes", s.handleListScrapes)
		r.Get("/scrapes/{id}", s.handleGetScrape)
		r.Get("/scrapes/{id}/page", s.handleGetPage) // ?key=...
		r.Post("/scrapes/{id}/index", s.handleIndexScrape)
		r.Post("/query", s.handleQuery)
	})

	return r
}

main()

func main() {
	cfg := loadConfig()
	srv, err := newServer(cfg)
	if err != nil {
		log.Fatalf("startup failure: %v", err)
	}

	addr := ":8080"
	log.Printf("api listening on %s", addr)
	if err := http.ListenAndServe(addr, srv.router()); err != nil {
		log.Fatal(err)
	}
}

References

• chi router
• chi middleware
• net/http package
• 12-factor config

Six handlers. Each one is small because the heavy lifting lives in scraper.go, storage.go, and ai.go. The handler's job is: parse input, validate, dispatch, format output.

JSON helpers & /health

func writeJSON(w http.ResponseWriter, status int, v any) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(status)
	_ = json.NewEncoder(w).Encode(v)
}

func writeErr(w http.ResponseWriter, status int, msg string) {
	writeJSON(w, status, map[string]string{"error": msg})
}

func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
	writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
}

handleScrape — the entry point of every job

// POST /api/scrape  { url, depth, format }
func (s *Server) handleScrape(w http.ResponseWriter, r *http.Request) {
	var req ScrapeRequest
	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
		writeErr(w, http.StatusBadRequest, "invalid json: "+err.Error())
		return
	}

	req.URL = strings.TrimSpace(req.URL)
	if req.URL == "" {
		writeErr(w, http.StatusBadRequest, "url is required")
		return
	}
	if !strings.HasPrefix(req.URL, "http://") && !strings.HasPrefix(req.URL, "https://") {
		writeErr(w, http.StatusBadRequest, "url must start with http:// or https://")
		return
	}
	if req.Depth < 0 {
		req.Depth = 0
	}
	if req.Depth > 5 {
		req.Depth = 5 // safety cap
	}
	if req.Format == "" {
		req.Format = "text"
	}
	switch req.Format {
	case "text", "html", "screenshot":
	default:
		writeErr(w, http.StatusBadRequest, "format must be text|html|screenshot")
		return
	}

Job creation & async execution

	now := time.Now().UTC()
	job := &ScrapeJob{
		ID:        uuid.NewString(),
		SeedURL:   req.URL,
		Depth:     req.Depth,
		Format:    req.Format,
		Status:    "pending",
		CreatedAt: now,
		UpdatedAt: now,
	}

	s.mu.Lock()
	s.jobs[job.ID] = job
	s.mu.Unlock()

	// Run async; the caller polls /api/scrapes/{id}
	go func(jobID string) {
		ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
		defer cancel()

		s.mu.RLock()
		j := s.jobs[jobID]
		s.mu.RUnlock()
		if j == nil {
			return
		}
		s.scraper.Run(ctx, j)
	}(job.ID)

	writeJSON(w, http.StatusAccepted, job)
}

List & get

// GET /api/scrapes
func (s *Server) handleListScrapes(w http.ResponseWriter, r *http.Request) {
	s.mu.RLock()
	defer s.mu.RUnlock()

	out := make([]*ScrapeJob, 0, len(s.jobs))
	for _, j := range s.jobs {
		out = append(out, j)
	}
	writeJSON(w, http.StatusOK, out)
}

// GET /api/scrapes/{id}
func (s *Server) handleGetScrape(w http.ResponseWriter, r *http.Request) {
	id := chi.URLParam(r, "id")
	s.mu.RLock()
	job, ok := s.jobs[id]
	s.mu.RUnlock()
	if !ok {
		writeErr(w, http.StatusNotFound, "job not found")
		return
	}
	writeJSON(w, http.StatusOK, job)
}

handleGetPage — streaming from S3

// GET /api/scrapes/{id}/page?key=...
// Streams the stored object straight from S3 with its content-type.
func (s *Server) handleGetPage(w http.ResponseWriter, r *http.Request) {
	id := chi.URLParam(r, "id")
	key := r.URL.Query().Get("key")
	if key == "" {
		writeErr(w, http.StatusBadRequest, "key is required")
		return
	}
	// Make sure the key actually belongs to this job to prevent path tricks
	if !strings.HasPrefix(key, fmt.Sprintf("jobs/%s/", id)) {
		writeErr(w, http.StatusForbidden, "key does not belong to this job")
		return
	}

	data, ct, err := s.storage.Get(r.Context(), key)
	if err != nil {
		writeErr(w, http.StatusNotFound, err.Error())
		return
	}
	if ct == "" {
		ct = "application/octet-stream"
	}
	w.Header().Set("Content-Type", ct)
	w.WriteHeader(http.StatusOK)
	_, _ = w.Write(data)
}

handleIndexScrape — load to ChromaDB

// POST /api/scrapes/{id}/index
// Pulls every page's text from S3, chunks it, and indexes into ChromaDB.
func (s *Server) handleIndexScrape(w http.ResponseWriter, r *http.Request) {
	id := chi.URLParam(r, "id")
	s.mu.RLock()
	job, ok := s.jobs[id]
	s.mu.RUnlock()
	if !ok {
		writeErr(w, http.StatusNotFound, "job not found")
		return
	}
	if job.Status != "done" {
		writeErr(w, http.StatusConflict, "job not finished yet")
		return
	}

	indexed := 0
	for _, p := range job.Pages {
		// Always read the .txt sibling so we index clean text regardless
		// of what format the user asked for.
		textKey := strings.TrimSuffix(p.S3Key, ".png")
		textKey = strings.TrimSuffix(textKey, ".html")
		if !strings.HasSuffix(textKey, ".txt") {
			textKey = textKey + ".txt"
		}

		data, _, err := s.storage.Get(r.Context(), textKey)
		if err != nil {
			continue
		}
		text := string(data)
		chunks := ChunkText(text, 1500)

		for i, ch := range chunks {
			docID := fmt.Sprintf("%s::%s::%d", job.ID, p.URL, i)
			meta := map[string]string{
				"job_id": job.ID,
				"url":    p.URL,
				"title":  p.Title,
				"chunk":  fmt.Sprintf("%d", i),
			}
			if err := s.ai.IndexDoc(r.Context(), docID, ch, meta); err != nil {
				writeErr(w, http.StatusBadGateway, "index error: "+err.Error())
				return
			}
			indexed++
		}
	}

	s.mu.Lock()
	job.Indexed = true
	job.UpdatedAt = time.Now().UTC()
	s.mu.Unlock()

	writeJSON(w, http.StatusOK, map[string]any{
		"job_id":         job.ID,
		"chunks_indexed": indexed,
	})
}

handleQuery — RAG in 25 lines

// POST /api/query  { prompt, job_id?, top_k? }
func (s *Server) handleQuery(w http.ResponseWriter, r *http.Request) {
	var req QueryRequest
	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
		writeErr(w, http.StatusBadRequest, "invalid json: "+err.Error())
		return
	}
	req.Prompt = strings.TrimSpace(req.Prompt)
	if req.Prompt == "" {
		writeErr(w, http.StatusBadRequest, "prompt is required")
		return
	}

	docs, metas, err := s.ai.QueryChunks(r.Context(), req.Prompt, req.TopK, req.JobID)
	if err != nil {
		writeErr(w, http.StatusBadGateway, err.Error())
		return
	}

	prompt := BuildRAGPrompt(req.Prompt, docs)
	answer, err := s.ai.Generate(r.Context(), prompt)
	if err != nil {
		writeErr(w, http.StatusBadGateway, err.Error())
		return
	}

	sources := make([]string, 0, len(metas))
	seen := map[string]bool{}
	for _, m := range metas {
		if u, ok := m["url"].(string); ok && !seen[u] {
			sources = append(sources, u)
			seen[u] = true
		}
	}

	writeJSON(w, http.StatusOK, QueryResponse{Answer: answer, Sources: sources})
}

References

• json.NewDecoder
• context package
• google/uuid
• HTTP status codes (MDN)

Imports & constants

package main

import (
	"bytes"
	"context"
	"encoding/base64"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"net/url"
	"strings"
	"time"
)

const (
	maxPages       = 50
	requestTimeout = 60 * time.Second
)

Scraper struct & constructor

type Scraper struct {
	cfg     Config
	storage *Storage
	client  *http.Client
}

func NewScraper(cfg Config, storage *Storage) *Scraper {
	return &Scraper{
		cfg:     cfg,
		storage: storage,
		client: &http.Client{
			Timeout: requestTimeout + 5*time.Second,
		},
	}
}

Run — the BFS loop

type queueItem struct {
	URL   string
	Depth int
}

// Run is the main entrypoint. It updates the job in place.
func (s *Scraper) Run(ctx context.Context, job *ScrapeJob) {
	job.Status = "running"
	job.UpdatedAt = time.Now().UTC()

	seedHost, err := hostOf(job.SeedURL)
	if err != nil {
		job.Status = "failed"
		job.Error = "invalid seed url: " + err.Error()
		job.UpdatedAt = time.Now().UTC()
		return
	}

	visited := map[string]bool{}
	queue := []queueItem{{URL: job.SeedURL, Depth: 0}}

	for len(queue) > 0 && len(job.Pages) < maxPages {
		select {
		case <-ctx.Done():
			job.Status = "failed"
			job.Error = "cancelled"
			job.UpdatedAt = time.Now().UTC()
			return
		default:
		}

		item := queue[0]
		queue = queue[1:]

		if visited[item.URL] {
			continue
		}
		visited[item.URL] = true

Per-page fetch & store

		page, links, err := s.fetchAndStore(ctx, job, item.URL, item.Depth)
		if err != nil {
			// Soft-fail: log into job error but keep going for the rest
			if job.Error == "" {
				job.Error = fmt.Sprintf("page %s: %v", item.URL, err)
			}
			continue
		}
		job.Pages = append(job.Pages, page)
		job.UpdatedAt = time.Now().UTC()

		if item.Depth >= job.Depth {
			continue
		}

		for _, link := range links {
			if visited[link] {
				continue
			}
			h, err := hostOf(link)
			if err != nil || h != seedHost {
				continue // same-host scope
			}
			queue = append(queue, queueItem{URL: link, Depth: item.Depth + 1})
		}
	}

	job.Status = "done"
	if job.Error == "" || strings.HasPrefix(job.Error, "page ") {
		// Mark as done even if some pages failed
		job.Status = "done"
	}
	job.UpdatedAt = time.Now().UTC()
}

fetchAndStore — talking to Playwright + S3

func (s *Scraper) fetchAndStore(ctx context.Context, job *ScrapeJob, pageURL string, depth int) (ScrapePage, []string, error) {
	body, _ := json.Marshal(map[string]any{
		"url":        pageURL,
		"screenshot": job.Format == "screenshot",
		"timeout":    int(requestTimeout / time.Millisecond),
	})

	rctx, cancel := context.WithTimeout(ctx, requestTimeout+10*time.Second)
	defer cancel()

	req, err := http.NewRequestWithContext(rctx, "POST", s.cfg.PlaywrightURL+"/render", bytes.NewReader(body))
	if err != nil {
		return ScrapePage{}, nil, err
	}
	req.Header.Set("Content-Type", "application/json")

	resp, err := s.client.Do(req)
	if err != nil {
		return ScrapePage{}, nil, fmt.Errorf("playwright: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		buf, _ := io.ReadAll(resp.Body)
		return ScrapePage{}, nil, fmt.Errorf("playwright status %d: %s", resp.StatusCode, string(buf))
	}

	var pr PlaywrightResp
	if err := json.NewDecoder(resp.Body).Decode(&pr); err != nil {
		return ScrapePage{}, nil, err
	}

Storing artifacts to S3

	var data []byte
	var contentType string
	var ext string
	switch job.Format {
	case "html":
		data = []byte(pr.HTML)
		contentType = "text/html; charset=utf-8"
		ext = "html"
	case "screenshot":
		decoded, err := base64.StdEncoding.DecodeString(pr.Screenshot)
		if err != nil {
			return ScrapePage{}, nil, fmt.Errorf("base64 decode: %w", err)
		}
		data = decoded
		contentType = "image/png"
		ext = "png"
	default: // "text"
		data = []byte(pr.Text)
		contentType = "text/plain; charset=utf-8"
		ext = "txt"
	}

	safe := safeKeyFromURL(pageURL)
	key := fmt.Sprintf("jobs/%s/pages/%s.%s", job.ID, safe, ext)

	if err := s.storage.Put(ctx, key, data, contentType); err != nil {
		return ScrapePage{}, nil, err
	}

	// Also store a .txt sibling so AI indexing always has clean text
	if ext != "txt" && pr.Text != "" {
		txtKey := fmt.Sprintf("jobs/%s/pages/%s.txt", job.ID, safe)
		_ = s.storage.Put(ctx, txtKey, []byte(pr.Text), "text/plain; charset=utf-8")
	}

	return ScrapePage{
		URL:    pageURL,
		Title:  pr.Title,
		S3Key:  key,
		Depth:  depth,
		Format: job.Format,
		Bytes:  len(data),
	}, pr.Links, nil
}

URL helpers

func hostOf(rawURL string) (string, error) {
	u, err := url.Parse(rawURL)
	if err != nil {
		return "", err
	}
	return strings.ToLower(u.Host), nil
}

func safeKeyFromURL(u string) string {
	parsed, err := url.Parse(u)
	if err != nil {
		return strings.NewReplacer("/", "_", ":", "_", "?", "_").Replace(u)
	}
	host := parsed.Host
	path := strings.Trim(parsed.Path, "/")
	if path == "" {
		path = "index"
	}
	combo := host + "_" + path
	if parsed.RawQuery != "" {
		combo += "_" + parsed.RawQuery
	}
	combo = strings.NewReplacer("/", "_", ":", "_", "?", "_", "&", "_", "=", "-", " ", "_").Replace(combo)
	if len(combo) > 200 {
		combo = combo[:200]
	}
	return combo
}

References

• Breadth-first search (Wikipedia)
• net/url package
• http.Client
• Errors are values

Imports & type

package main

import (
	"bytes"
	"context"
	"fmt"
	"io"

	"github.com/aws/aws-sdk-go-v2/aws"
	awsconfig "github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

type Storage struct {
	cfg    Config
	client *s3.Client
	bucket string
}

Constructor — the LocalStack pivot

func NewStorage(ctx context.Context, cfg Config) (*Storage, error) {
	awsCfg, err := awsconfig.LoadDefaultConfig(ctx, awsconfig.WithRegion(cfg.AWSRegion))
	if err != nil {
		return nil, fmt.Errorf("aws config: %w", err)
	}

	client := s3.NewFromConfig(awsCfg, func(o *s3.Options) {
		if cfg.S3Endpoint != "" {
			o.BaseEndpoint = aws.String(cfg.S3Endpoint)
			o.UsePathStyle = true
		}
	})

	return &Storage{
		cfg:    cfg,
		client: client,
		bucket: cfg.S3Bucket,
	}, nil
}

Put & Get

func (s *Storage) Put(ctx context.Context, key string, data []byte, contentType string) error {
	_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
		Bucket:      aws.String(s.bucket),
		Key:         aws.String(key),
		Body:        bytes.NewReader(data),
		ContentType: aws.String(contentType),
	})
	if err != nil {
		return fmt.Errorf("s3 put %s: %w", key, err)
	}
	return nil
}

func (s *Storage) Get(ctx context.Context, key string) ([]byte, string, error) {
	out, err := s.client.GetObject(ctx, &s3.GetObjectInput{
		Bucket: aws.String(s.bucket),
		Key:    aws.String(key),
	})
	if err != nil {
		return nil, "", fmt.Errorf("s3 get %s: %w", key, err)
	}
	defer out.Body.Close()

	data, err := io.ReadAll(out.Body)
	if err != nil {
		return nil, "", err
	}

	ct := ""
	if out.ContentType != nil {
		ct = *out.ContentType
	}
	return data, ct, nil
}

References

• AWS SDK for Go v2
• s3 service package
• S3 request signing
• Path-style vs virtual-hosted style

This file is the largest in the API because it spans two systems (ChromaDB + Ollama) and implements the full retrieval-augmented generation pipeline. Two flows live here: indexing (turn pages into embeddings stored in Chroma) and querying (embed a question, retrieve nearest chunks, ask the LLM).

Imports & type

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"strings"
	"time"
)

// AI bundles ChromaDB + Ollama clients
type AI struct {
	cfg    Config
	client *http.Client
}

const (
	chromaTenant   = "default_tenant"
	chromaDatabase = "default_database"
	chromaCollName = "scraper_docs"
)

func NewAI(cfg Config) *AI {
	return &AI{
		cfg: cfg,
		client: &http.Client{
			Timeout: 5 * time.Minute, // LLM calls can be slow
		},
	}
}

ensureCollection — idempotent setup

func (a *AI) ensureCollection(ctx context.Context) error {
	url := fmt.Sprintf("%s/api/v2/tenants/%s/databases/%s/collections", a.cfg.ChromaURL, chromaTenant, chromaDatabase)

	body, _ := json.Marshal(map[string]any{
		"name":          chromaCollName,
		"get_or_create": true,
		"metadata":      map[string]string{"hnsw:space": "cosine"},
	})
	req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(body))
	if err != nil {
		return err
	}
	req.Header.Set("Content-Type", "application/json")
	resp, err := a.client.Do(req)
	if err != nil {
		return fmt.Errorf("chroma ensure: %w", err)
	}
	defer resp.Body.Close()
	if resp.StatusCode >= 300 {
		buf, _ := io.ReadAll(resp.Body)
		return fmt.Errorf("chroma ensure status %d: %s", resp.StatusCode, string(buf))
	}
	return nil
}

Embed — text → 768-d vector

func (a *AI) Embed(ctx context.Context, text string) ([]float32, error) {
	url := a.cfg.OllamaURL + "/api/embed"
	body, _ := json.Marshal(map[string]any{
		"model": a.cfg.OllamaEmbedModel,
		"input": text,
	})
	req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(body))
	if err != nil {
		return nil, err
	}
	req.Header.Set("Content-Type", "application/json")
	resp, err := a.client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("ollama embed: %w", err)
	}
	defer resp.Body.Close()
	if resp.StatusCode >= 300 {
		buf, _ := io.ReadAll(resp.Body)
		return nil, fmt.Errorf("ollama embed status %d: %s", resp.StatusCode, string(buf))
	}
	var er struct {
		Embeddings [][]float32 `json:"embeddings"`
	}
	if err := json.NewDecoder(resp.Body).Decode(&er); err != nil {
		return nil, err
	}
	if len(er.Embeddings) == 0 {
		return nil, fmt.Errorf("empty embedding response")
	}
	return er.Embeddings[0], nil
}

IndexDoc — store one chunk in Chroma

func (a *AI) IndexDoc(ctx context.Context, id, text string, meta map[string]string) error {
	if err := a.ensureCollection(ctx); err != nil {
		return err
	}
	vec, err := a.Embed(ctx, text)
	if err != nil {
		return err
	}

	// Look up the collection ID
	collID, err := a.collectionID(ctx)
	if err != nil {
		return err
	}

	url := fmt.Sprintf("%s/api/v2/tenants/%s/databases/%s/collections/%s/add", a.cfg.ChromaURL, chromaTenant, chromaDatabase, collID)

	metaIface := map[string]any{}
	for k, v := range meta {
		metaIface[k] = v
	}

	body, _ := json.Marshal(map[string]any{
		"ids":        []string{id},
		"embeddings": [][]float32{vec},
		"documents":  []string{text},
		"metadatas":  []map[string]any{metaIface},
	})
	req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(body))
	if err != nil {
		return err
	}
	req.Header.Set("Content-Type", "application/json")
	resp, err := a.client.Do(req)
	if err != nil {
		return fmt.Errorf("chroma add: %w", err)
	}
	defer resp.Body.Close()
	if resp.StatusCode >= 300 {
		buf, _ := io.ReadAll(resp.Body)
		return fmt.Errorf("chroma add status %d: %s", resp.StatusCode, string(buf))
	}
	return nil
}

QueryChunks — find most-similar text

// QueryChunks returns top-K most similar chunks to the prompt.
// If jobID is non-empty, results are filtered to that job.
func (a *AI) QueryChunks(ctx context.Context, prompt string, topK int, jobID string) ([]string, []map[string]any, error) {
	if topK <= 0 {
		topK = 5
	}
	if err := a.ensureCollection(ctx); err != nil {
		return nil, nil, err
	}
	vec, err := a.Embed(ctx, prompt)
	if err != nil {
		return nil, nil, err
	}
	collID, err := a.collectionID(ctx)
	if err != nil {
		return nil, nil, err
	}

	queryBody := map[string]any{
		"query_embeddings": [][]float32{vec},
		"n_results":        topK,
		"include":          []string{"documents", "metadatas"},
	}
	if jobID != "" {
		queryBody["where"] = map[string]string{"job_id": jobID}
	}

	body, _ := json.Marshal(queryBody)
	url := fmt.Sprintf("%s/api/v2/tenants/%s/databases/%s/collections/%s/query", a.cfg.ChromaURL, chromaTenant, chromaDatabase, collID)
	req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(body))
	if err != nil {
		return nil, nil, err
	}
	req.Header.Set("Content-Type", "application/json")
	resp, err := a.client.Do(req)
	if err != nil {
		return nil, nil, fmt.Errorf("chroma query: %w", err)
	}
	defer resp.Body.Close()
	if resp.StatusCode >= 300 {
		buf, _ := io.ReadAll(resp.Body)
		return nil, nil, fmt.Errorf("chroma query status %d: %s", resp.StatusCode, string(buf))
	}

	var qr struct {
		Documents [][]string         `json:"documents"`
		Metadatas [][]map[string]any `json:"metadatas"`
	}
	if err := json.NewDecoder(resp.Body).Decode(&qr); err != nil {
		return nil, nil, err
	}
	if len(qr.Documents) == 0 {
		return nil, nil, nil
	}
	return qr.Documents[0], qr.Metadatas[0], nil
}

Generate — call the LLM

func (a *AI) Generate(ctx context.Context, prompt string) (string, error) {
	url := a.cfg.OllamaURL + "/api/generate"
	body, _ := json.Marshal(map[string]any{
		"model":  a.cfg.OllamaModel,
		"prompt": prompt,
		"stream": false,
		"options": map[string]any{
			"temperature": 0.2,
		},
	})
	req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(body))
	if err != nil {
		return "", err
	}
	req.Header.Set("Content-Type", "application/json")
	resp, err := a.client.Do(req)
	if err != nil {
		return "", fmt.Errorf("ollama generate: %w", err)
	}
	defer resp.Body.Close()
	if resp.StatusCode >= 300 {
		buf, _ := io.ReadAll(resp.Body)
		return "", fmt.Errorf("ollama generate status %d: %s", resp.StatusCode, string(buf))
	}
	var gr struct {
		Response string `json:"response"`
	}
	if err := json.NewDecoder(resp.Body).Decode(&gr); err != nil {
		return "", err
	}
	return strings.TrimSpace(gr.Response), nil
}

ChunkText & BuildRAGPrompt

// ChunkText splits a long document into pieces near a target size,
// breaking on word boundaries.
func ChunkText(text string, maxLen int) []string {
	text = strings.TrimSpace(text)
	if len(text) <= maxLen {
		if text == "" {
			return nil
		}
		return []string{text}
	}
	var chunks []string
	for len(text) > 0 {
		if len(text) <= maxLen {
			chunks = append(chunks, text)
			break
		}
		end := maxLen
		// back off to last whitespace within the slice
		for end > maxLen/2 && !isSpace(text[end]) {
			end--
		}
		if end == maxLen/2 {
			end = maxLen
		}
		chunks = append(chunks, strings.TrimSpace(text[:end]))
		text = strings.TrimSpace(text[end:])
	}
	return chunks
}

func isSpace(b byte) bool {
	return b == ' ' || b == '\n' || b == '\t'
}

// BuildRAGPrompt assembles a context-aware prompt for the LLM.
func BuildRAGPrompt(question string, chunks []string) string {
	var b strings.Builder
	b.WriteString("You are a helpful assistant. Answer the user's question using ONLY the provided context. ")
	b.WriteString("If the context does not contain the answer, say you don't know. Be concise and cite which context block(s) you used.\n\n")
	b.WriteString("CONTEXT:\n")
	for i, c := range chunks {
		fmt.Fprintf(&b, "[%d] %s\n\n", i+1, c)
	}
	b.WriteString("\nQUESTION: ")
	b.WriteString(question)
	b.WriteString("\n\nANSWER:")
	return b.String()
}

References

• ChromaDB documentation
• Chroma collection API
• Ollama API reference
• nomic-embed-text model card
• What is RAG? (IBM)

Test setup

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
	"net/http/httptest"
	"strings"
	"sync"
	"testing"
)

// buildTestServer wires up a *Server backed by mock HTTP servers
// for Playwright, Chroma, Ollama, and an in-memory S3.
func buildTestServer(t *testing.T) (*Server, func()) {
	t.Helper()

	// Mock S3: trivial in-memory store hit by the AWS SDK
	s3Store := struct {
		sync.Mutex
		items map[string][]byte
	}{items: map[string][]byte{}}

	s3Mock := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// e.g. PUT /scraper-data/jobs/abc/pages/x.txt
		// or   GET /scraper-data/jobs/abc/pages/x.txt
		key := strings.TrimPrefix(r.URL.Path, "/scraper-data/")

		switch r.Method {
		case http.MethodPut:
			buf := new(bytes.Buffer)
			buf.ReadFrom(r.Body)
			s3Store.Lock()
			s3Store.items[key] = buf.Bytes()
			s3Store.Unlock()
			w.WriteHeader(http.StatusOK)
		case http.MethodGet:
			s3Store.Lock()
			data, ok := s3Store.items[key]
			s3Store.Unlock()
			if !ok {
				http.NotFound(w, r)
				return
			}
			w.Header().Set("Content-Type", "text/plain")
			w.Write(data)
		default:
			w.WriteHeader(http.StatusMethodNotAllowed)
		}
	}))

Mock Playwright, Chroma, Ollama

	playwrightMock := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		var body map[string]any
		json.NewDecoder(r.Body).Decode(&body)
		url := body["url"].(string)
		resp := map[string]any{
			"url":   url,
			"title": "Test Page",
			"html":  "<html><body>Hello from " + url + "</body></html>",
			"text":  "Hello from " + url,
			"links": []string{},
		}
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(resp)
	}))

	// Chroma + Ollama in one mux for convenience
	chromaCalls := 0
	mux := http.NewServeMux()
	mux.HandleFunc("/api/v2/", func(w http.ResponseWriter, r *http.Request) {
		chromaCalls++
		if strings.HasSuffix(r.URL.Path, "/collections") && r.Method == http.MethodPost {
			json.NewEncoder(w).Encode(map[string]any{"id": "test-coll-id"})
			return
		}
		if strings.HasSuffix(r.URL.Path, "/query") {
			json.NewEncoder(w).Encode(map[string]any{
				"documents": [][]string{{"matched chunk text"}},
				"metadatas": [][]map[string]any{{{"url": "http://example.com/", "job_id": "any"}}},
			})
			return
		}
		// add or other operations
		w.WriteHeader(http.StatusOK)
	})
	mux.HandleFunc("/api/embed", func(w http.ResponseWriter, r *http.Request) {
		json.NewEncoder(w).Encode(map[string]any{
			"embeddings": [][]float32{{0.1, 0.2, 0.3}},
		})
	})
	mux.HandleFunc("/api/generate", func(w http.ResponseWriter, r *http.Request) {
		json.NewEncoder(w).Encode(map[string]any{
			"response": "This is a mocked LLM answer.",
		})
	})
	chromaOllamaMock := httptest.NewServer(mux)

	// Build a Server with all upstreams pointing at our mocks
	cfg := Config{
		PlaywrightURL:    playwrightMock.URL,
		S3Endpoint:       s3Mock.URL,
		S3Bucket:         "scraper-data",
		AWSRegion:        "us-east-1",
		ChromaURL:        chromaOllamaMock.URL,
		OllamaURL:        chromaOllamaMock.URL,
		OllamaModel:      "test",
		OllamaEmbedModel: "test",
	}

	srv, err := newServer(cfg)
	if err != nil {
		t.Fatalf("newServer: %v", err)
	}

	cleanup := func() {
		s3Mock.Close()
		playwrightMock.Close()
		chromaOllamaMock.Close()
	}
	return srv, cleanup
}

Example test

func TestScrapeFlowEndToEnd(t *testing.T) {
	srv, cleanup := buildTestServer(t)
	defer cleanup()
	r := srv.router()

	// 1) POST /api/scrape — start a scrape
	body := strings.NewReader(`{"url":"http://example.com","depth":0,"format":"text"}`)
	req := httptest.NewRequest("POST", "/api/scrape", body)
	req.Header.Set("Content-Type", "application/json")
	w := httptest.NewRecorder()
	r.ServeHTTP(w, req)
	if w.Code != http.StatusAccepted {
		t.Fatalf("expected 202, got %d: %s", w.Code, w.Body.String())
	}
	var job ScrapeJob
	json.NewDecoder(w.Body).Decode(&job)

	// 2) Wait for completion (polling because run is async)
	deadline := time.Now().Add(5 * time.Second)
	for time.Now().Before(deadline) {
		srv.mu.RLock()
		j := srv.jobs[job.ID]
		srv.mu.RUnlock()
		if j != nil && (j.Status == "done" || j.Status == "failed") {
			job = *j
			break
		}
		time.Sleep(50 * time.Millisecond)
	}
	if job.Status != "done" {
		t.Fatalf("expected done, got %q (err=%q)", job.Status, job.Error)
	}
	if len(job.Pages) != 1 {
		t.Fatalf("expected 1 page, got %d", len(job.Pages))
	}

	// 3) POST /api/scrapes/{id}/index
	req = httptest.NewRequest("POST", "/api/scrapes/"+job.ID+"/index", nil)
	w = httptest.NewRecorder()
	r.ServeHTTP(w, req)
	if w.Code != http.StatusOK {
		t.Fatalf("index expected 200, got %d: %s", w.Code, w.Body.String())
	}

	// 4) POST /api/query
	body = strings.NewReader(fmt.Sprintf(`{"prompt":"hello","job_id":"%s"}`, job.ID))
	req = httptest.NewRequest("POST", "/api/query", body)
	req.Header.Set("Content-Type", "application/json")
	w = httptest.NewRecorder()
	r.ServeHTTP(w, req)
	if w.Code != http.StatusOK {
		t.Fatalf("query expected 200, got %d: %s", w.Code, w.Body.String())
	}
	var qr QueryResponse
	json.NewDecoder(w.Body).Decode(&qr)
	if !strings.Contains(qr.Answer, "mocked") {
		t.Fatalf("answer should contain 'mocked', got %q", qr.Answer)
	}
}

Other tests in the file

The file also contains five smaller tests that exercise specific units in isolation:

• TestHealthEndpoint — sanity check that /health returns 200.
• TestScrapeValidation — rejects empty URLs, bad schemes, illegal formats.
• TestGetPageAuthCheck — confirms the path-traversal guard rejects cross-job keys.
• TestChunkText — chunking respects max length, breaks on whitespace, handles tiny/empty inputs.
• TestBuildRAGPrompt — the assembled prompt contains the question, the rules, and numbered context blocks.

Run the suite with go test -v ./... from inside api/. All 6 tests should pass in <1 s.

References

• testing package
• httptest package
• Add a test (Go tutorial)
• Practical test pyramid

The full file

# --- Stage 1: build ---
FROM golang:1.22-alpine AS builder
WORKDIR /src

# Cache deps before copying full source
COPY go.mod go.sum ./
RUN go mod download

# Now copy source and build
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/api .

# --- Stage 2: run ---
FROM alpine:3.20
RUN apk add --no-cache ca-certificates
WORKDIR /app
COPY --from=builder /out/api ./api

EXPOSE 8080
CMD ["./api"]

References

• Multi-stage builds
• Build cache
• golang official image
• alpine official image

Document head — Tailwind via CDN

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Web Scraper Console</title>
  <script src="https://cdn.tailwindcss.com"></script>
</head>

Header & container

<body class="bg-slate-50 text-slate-900">
  <div class="max-w-6xl mx-auto p-6">
    <header class="mb-6">
      <h1 class="text-3xl font-bold">Web Scraper Console</h1>
      <p class="text-slate-600 text-sm">
        Crawl, store to S3, index in ChromaDB, ask the agent.
      </p>
    </header>

Scrape form

<section class="bg-white rounded-xl shadow p-5 mb-6">
  <h2 class="text-lg font-semibold mb-3">Start a scrape</h2>
  <div class="grid md:grid-cols-12 gap-3">
    <input id="urlInput" type="url" placeholder="https://example.com"
           class="md:col-span-6 px-3 py-2 border rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500">
    <input id="depthInput" type="number" min="0" max="5" value="0"
           class="md:col-span-2 px-3 py-2 border rounded-md">
    <select id="formatInput" class="md:col-span-2 px-3 py-2 border rounded-md">
      <option value="text">Text</option>
      <option value="html">HTML</option>
      <option value="screenshot">Screenshot</option>
    </select>
    <button id="scrapeBtn"
            class="md:col-span-2 bg-blue-600 hover:bg-blue-700 text-white font-medium rounded-md px-4 py-2">
      Scrape
    </button>
  </div>
  <p id="scrapeMsg" class="text-sm text-slate-500 mt-2"></p>
</section>

Jobs table

<section class="bg-white rounded-xl shadow p-5 mb-6">
  <div class="flex items-center justify-between mb-3">
    <h2 class="text-lg font-semibold">Jobs</h2>
    <button id="refreshBtn" class="text-sm text-blue-600 hover:underline">Refresh</button>
  </div>
  <div class="overflow-x-auto">
    <table class="w-full text-sm">
      <thead class="text-left bg-slate-100 text-slate-600">
        <tr>
          <th class="p-2">ID</th>
          <th class="p-2">Seed URL</th>
          <th class="p-2">Depth</th>
          <th class="p-2">Status</th>
          <th class="p-2">Pages</th>
          <th class="p-2">Indexed</th>
          <th class="p-2">Actions</th>
        </tr>
      </thead>
      <tbody id="jobsBody"></tbody>
    </table>
  </div>
</section>

AI agent panel

<section class="bg-white rounded-xl shadow p-5 mb-6">
  <h2 class="text-lg font-semibold mb-3">Ask the agent</h2>
  <div class="grid md:grid-cols-12 gap-3">
    <textarea id="promptInput" rows="3" placeholder="Summarize the site, list main sections..."
              class="md:col-span-9 px-3 py-2 border rounded-md"></textarea>
    <div class="md:col-span-3 flex flex-col gap-2">
      <select id="jobScope" class="px-3 py-2 border rounded-md">
        <option value="">All indexed jobs</option>
      </select>
      <button id="queryBtn" class="bg-emerald-600 hover:bg-emerald-700 text-white rounded-md px-4 py-2">
        Ask
      </button>
    </div>
  </div>
  <div id="answerBox" class="mt-4 hidden">
    <h3 class="text-sm font-semibold text-slate-700">Answer</h3>
    <p id="answerText" class="text-sm whitespace-pre-wrap"></p>
    <h4 class="text-sm font-semibold text-slate-700 mt-2">Sources</h4>
    <ul id="sourcesList" class="text-xs text-slate-600 list-disc pl-5"></ul>
  </div>
</section>

Modal viewer

<div id="modalBg" class="hidden fixed inset-0 bg-black/60 z-40"></div>
<div id="modal"
     class="hidden fixed inset-4 md:inset-10 bg-white rounded-xl shadow-2xl z-50 flex flex-col">
  <div class="flex items-center justify-between p-3 border-b">
    <h3 id="modalTitle" class="font-semibold">Page</h3>
    <button id="modalClose" class="text-slate-500 hover:text-slate-900 text-2xl leading-none">&times;</button>
  </div>
  <div id="modalBody" class="flex-1 overflow-auto p-4"></div>
</div>

Frontend Dockerfile

FROM nginx:alpine
COPY index.html /usr/share/nginx/html/index.html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

References

• Tailwind CSS docs
• Tailwind responsive design
• MDN HTML reference
• nginx docs

API helper & element shortcuts

const API = "http://localhost:8080";
const $ = (id) => document.getElementById(id);

async function api(path, opts = {}) {
  const res = await fetch(API + path, {
    headers: { 'Content-Type': 'application/json', ...(opts.headers || {}) },
    ...opts
  });
  if (!res.ok) {
    const txt = await res.text();
    throw new Error(`${res.status}: ${txt}`);
  }
  return res.json();
}

Loading & rendering jobs

async function loadJobs() {
  try {
    const jobs = await api('/api/scrapes');
    jobs.sort((a, b) => new Date(b.created_at) - new Date(a.created_at));
    renderJobs(jobs);
    refreshScopeOptions(jobs);
  } catch (e) {
    console.error(e);
  }
}

function renderJobs(jobs) {
  const tbody = $('jobsBody');
  if (!jobs.length) {
    tbody.innerHTML = '<tr><td colspan="7" class="p-4 text-slate-500">No jobs yet</td></tr>';
    return;
  }
  tbody.innerHTML = jobs.map(j => `
    <tr class="border-t">
      <td class="p-2 font-mono text-xs">${j.id.slice(0, 8)}</td>
      <td class="p-2 truncate max-w-xs"><a href="${j.seed_url}" target="_blank" class="text-blue-600 hover:underline">${j.seed_url}</a></td>
      <td class="p-2">${j.depth}</td>
      <td class="p-2"><span class="px-2 py-0.5 rounded text-xs ${statusClass(j.status)}">${j.status}</span></td>
      <td class="p-2">${(j.pages || []).length}</td>
      <td class="p-2">${j.indexed ? '✅' : '—'}</td>
      <td class="p-2 space-x-2">
        <button onclick="viewJob('${j.id}')" class="text-blue-600 hover:underline">View</button>
        ${j.status === 'done' && !j.indexed
          ? `<button onclick="indexJob('${j.id}')" class="text-emerald-600 hover:underline">Index</button>`
          : ''}
      </td>
    </tr>
  `).join('');
}

function statusClass(s) {
  return {
    pending: 'bg-slate-200 text-slate-700',
    running: 'bg-amber-100 text-amber-800',
    done:    'bg-emerald-100 text-emerald-800',
    failed:  'bg-rose-100 text-rose-800',
  }[s] || 'bg-slate-100';
}

Viewing pages

async function viewJob(id) {
  const job = await api(`/api/scrapes/${id}`);
  $('modalTitle').textContent = `Job ${id.slice(0, 8)} — ${job.seed_url}`;
  if (!job.pages || !job.pages.length) {
    $('modalBody').innerHTML = '<p class="text-slate-500">No pages yet.</p>';
  } else {
    $('modalBody').innerHTML = `
      <ul class="space-y-2">
        ${job.pages.map((p, i) => `
          <li class="border rounded p-2">
            <div class="text-xs text-slate-500">${p.url}</div>
            <div class="font-medium">${p.title || '(no title)'}</div>
            <button onclick="showPage('${id}', '${encodeURIComponent(p.s3_key)}', '${p.format}')"
                    class="text-blue-600 text-sm hover:underline">Open</button>
          </li>
        `).join('')}
      </ul>
    `;
  }
  showModal();
}

function showPage(jobId, encKey, format) {
  const url = `${API}/api/scrapes/${jobId}/page?key=${encKey}`;
  if (format === 'screenshot') {
    $('modalBody').innerHTML = `<img src="${url}" class="max-w-full" alt="screenshot">`;
  } else if (format === 'html') {
    $('modalBody').innerHTML = `<iframe src="${url}" class="w-full h-full border-0"></iframe>`;
  } else {
    fetch(url).then(r => r.text()).then(t => {
      $('modalBody').innerHTML = `<pre class="whitespace-pre-wrap text-sm">${escapeHtml(t)}</pre>`;
    });
  }
}

function escapeHtml(s) {
  return s.replace(/[&<>"']/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c]));
}

Index, scrape, query

async function indexJob(id) {
  $('scrapeMsg').textContent = `Indexing ${id.slice(0, 8)}…`;
  try {
    const r = await api(`/api/scrapes/${id}/index`, { method: 'POST' });
    $('scrapeMsg').textContent = `Indexed ${r.chunks_indexed} chunks.`;
    loadJobs();
  } catch (e) {
    $('scrapeMsg').textContent = 'Index error: ' + e.message;
  }
}

$('scrapeBtn').onclick = async () => {
  const url = $('urlInput').value.trim();
  if (!url) return;
  const body = {
    url,
    depth: parseInt($('depthInput').value || '0', 10),
    format: $('formatInput').value
  };
  $('scrapeMsg').textContent = 'Submitting…';
  try {
    const j = await api('/api/scrape', { method: 'POST', body: JSON.stringify(body) });
    $('scrapeMsg').textContent = `Job ${j.id.slice(0, 8)} created`;
    loadJobs();
  } catch (e) {
    $('scrapeMsg').textContent = 'Error: ' + e.message;
  }
};

$('queryBtn').onclick = async () => {
  const prompt = $('promptInput').value.trim();
  if (!prompt) return;
  const body = { prompt, job_id: $('jobScope').value, top_k: 5 };
  $('answerText').textContent = 'Thinking…';
  $('answerBox').classList.remove('hidden');
  try {
    const r = await api('/api/query', { method: 'POST', body: JSON.stringify(body) });
    $('answerText').textContent = r.answer;
    $('sourcesList').innerHTML = (r.sources || []).map(s =>
      `<li><a href="${s}" target="_blank" class="text-blue-600 hover:underline">${s}</a></li>`
    ).join('');
  } catch (e) {
    $('answerText').textContent = 'Error: ' + e.message;
  }
};

$('refreshBtn').onclick = loadJobs;
loadJobs();
setInterval(loadJobs, 4000);

References

• Fetch API (MDN)
• Template literals
• encodeURIComponent
• OWASP XSS

The four parallel arrays

Every Chroma operation works with parallel arrays — index i across all of them refers to one logical row.

{
  "ids":        ["jobA::url1::0", "jobA::url1::1", "jobB::url2::0"],
  "documents":  ["chunk text 1",  "chunk text 2",  "chunk text 3"],
  "embeddings": [[0.1, ...],      [0.2, ...],      [0.3, ...]],
  "metadatas":  [{"job_id":"A"},  {"job_id":"A"},  {"job_id":"B"}]
}

The three things you can do

Why client-side embeddings?

Chroma supports server-side embeddings via embedding functions, but our setup is the opposite: we compute embeddings in ai.go via Ollama and ship them to Chroma. Reasons:

• One embedding model, one place to upgrade it.
• Chroma stays a pure vector store; the AI complexity stays in the API.
• If Chroma changes pricing/operator, we just point at the new vector store with no model rewiring.

The v2 URL structure

POST /api/v2/tenants/{tenant}/databases/{db}/collections        # get-or-create
POST /api/v2/tenants/{tenant}/databases/{db}/collections/{id}/add
POST /api/v2/tenants/{tenant}/databases/{db}/collections/{id}/query

The tenant and database levels exist for multi-tenant deployments — a SaaS hosting embeddings for many customers would put each in its own tenant, hard-isolating their data. We use default_tenant / default_database.

References

• ChromaDB docs
• REST API reference
• Chroma source on GitHub
• HNSW algorithm

The two models

Ollama API endpoints we use

// Embed
POST /api/embed
{
  "model": "nomic-embed-text",
  "input": "text to embed"
}
→ { "embeddings": [[0.1, 0.2, ...]] }

// Generate
POST /api/generate
{
  "model": "llama3.2",
  "prompt": "...",
  "stream": false,
  "options": { "temperature": 0.2 }
}
→ { "response": "...", "done": true, ... }

The RAG prompt template, annotated

You are a helpful assistant. Answer the user's question using ONLY the
provided context. If the context does not contain the answer, say you
don't know. Be concise and cite which context block(s) you used.

CONTEXT:
[1] <chunk text from page A>
[2] <chunk text from page B>
[3] <chunk text from page C>

QUESTION: How do I do X?

ANSWER:

Three deliberate elements:

• "using ONLY the provided context" — discourages the model from drawing on its training data, which keeps answers grounded.
• "If the context does not contain the answer, say you don't know" — explicit fallback. Without it, models invent confidently.
• Numbered context blocks — gives the model a citation vocabulary ("see [1] and [3]").

References

• Ollama on GitHub
• Ollama model library
• Ollama API docs
• RAG prompting guide

First boot

cd scraper
docker compose up --build

# wait ~3 minutes the first time, ~30 s thereafter
# watch the ollama-init container exit cleanly when models are pulled
docker compose ps

Smoke tests

# API up?
curl -s http://localhost:8080/health
# → {"status":"ok"}

# Playwright up?
curl -s http://localhost:3000/health
# → {"status":"ok"}

# LocalStack bucket exists?
docker compose exec localstack awslocal s3 ls
# → 2025-XX-XX  scraper-data

# Ollama models present?
docker compose exec ollama ollama list
# → llama3.2, nomic-embed-text both listed

# Open the UI
open http://localhost:8000

A first scrape via curl

# Submit a job
curl -s -X POST http://localhost:8080/api/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","depth":1,"format":"text"}' | jq

# Poll until done
JOB_ID=<copy id from response>
curl -s http://localhost:8080/api/scrapes/$JOB_ID | jq '.status, .pages | length'

# Index it
curl -s -X POST http://localhost:8080/api/scrapes/$JOB_ID/index | jq

# Ask the agent
curl -s -X POST http://localhost:8080/api/query \
  -H 'Content-Type: application/json' \
  -d "{\"prompt\":\"summarize this site\",\"job_id\":\"$JOB_ID\"}" | jq

Resetting

# Stop containers (volumes survive)
docker compose down

# Stop AND wipe volumes (full reset — re-downloads Ollama models on next up)
docker compose down -v

# Just rebuild one service
docker compose build playwright
docker compose up -d playwright

References

• docker compose CLI
• jq (JSON CLI tool)

Common errors & fixes

docker compose build --no-cache playwright
docker compose up -d playwright

depends_on:
  playwright:
    condition: service_healthy

docker compose logs ollama-init
docker compose exec ollama ollama list   # should show both models

docker compose exec ollama ollama pull llama3.2
docker compose exec ollama ollama pull nomic-embed-text

chmod +x scripts/init-aws.sh
# convert CRLF if needed:
dos2unix scripts/init-aws.sh
docker compose down -v
docker compose up

docker compose exec localstack awslocal s3 mb s3://scraper-data

lsof -i :8080      # macOS/Linux
netstat -ano | findstr :8080   # Windows

Useful commands cheat sheet

# See all running containers
docker compose ps

# Tail one service's logs
docker compose logs -f api
docker compose logs -f ollama

# Shell into a running container
docker compose exec api sh
docker compose exec localstack bash

# Direct curl to inner services (via published ports)
curl http://localhost:3000/health                    # playwright
curl http://localhost:8001/api/v2/heartbeat          # chroma
curl http://localhost:11434/api/tags                 # ollama
curl http://localhost:4566/_localstack/health        # localstack

# Inspect S3 contents
docker compose exec localstack awslocal s3 ls s3://scraper-data --recursive

# Inspect ChromaDB
curl http://localhost:8001/api/v2/tenants/default_tenant/databases/default_database/collections | jq

# Run Go tests against the local source (no docker needed)
cd api && go test -v ./...

References

• docker compose logs
• docker compose exec
• Playwright troubleshooting
• LocalStack support