Skip to content

EXP Server Runbook

Host: exp.scottrecycling.com Role: Testing Odoo + production Django/Flask/FastAPI app host User: dan (sudo enabled) Last verified: 2026-04-14

This runbook is the single-page reference for operating the EXP server. It documents every service that matters, how to restart each one, where its config lives, common failure modes, and recovery steps. Keep this file current. When a new service is added, add its row to the tables below.

Critical reminder: EXP is a test server for Odoo. The sr_live_jan19_backup database and all other Odoo databases on EXP are neutralized clones — crons disabled, mail servers disabled, outgoing queues cancelled, companies marked [TEST CLONE]. Do not treat EXP Odoo as production. Real customer-facing production Odoo is on a separate host.

However, EXP does host real production Django/Flask/FastAPI apps (/assistant, /financial, /health, /life, /plantbiz, /deals, /tutor). These must stay up.

Service Inventory

Odoo instances (all TEST CLONES — neutralized)

Service Port DB Config Path served Service user
odoo.service 1819 sr_live_jan19_backup /etc/odoo/odoo.conf / and /stg/ odoo
odoo-superstore.service 1820 scott_superstore /etc/odoo/odoo-superstore.conf /superstore/ odoo
odoo-training.service 1821 sr_training /etc/odoo/odoo-training.conf training.scottrecycling.com odoo

Production Django / Flask / FastAPI apps

Service Port DB Directory Path served Framework
personal-assistant.service 8085 personal_assistant (pg) /opt/personal-assistant /assistant/ Django + Gunicorn
scott-financial.service 5000 scott_financial (pg) /opt/scott-financial /financial + financial.scottrecycling.com Flask + Gunicorn
health-erp.service 8080 health_erp (pg) /opt/health_erp /health/ Django + Gunicorn
life-erp.service 8081 life_erp (pg) /opt/life_erp /life/ Django + Gunicorn
plantbiz.service 5010 plantbiz (pg) /opt/plantbiz /plantbiz/ Flask + Gunicorn
deal-hunter.service 8090 config file /opt/deal-hunter /deals/ Flask + Gunicorn
deploy_api_1 (Docker) 8010 containerized pg (deploy_db_1) /opt/personal-tutor /tutor FastAPI in Docker Compose

Supporting infrastructure

Service Purpose Config
nginx Reverse proxy for all HTTP/HTTPS traffic /etc/nginx/sites-enabled/exp
postgresql (or postgresql@14-main) Database engine for all Odoo + production apps /etc/postgresql/14/main/
cron (root + dan) Nightly backup, monitoring, offsite staging crontab -l (per user)

Static documentation sites (no service — nginx aliases)

Path Source directory
/docs /var/www/docs/ (rsynced from ~/Desktop/Sr-docs/site/ via ./deploy.sh)
/sss-docs /var/www/sss-docs/
/scott-docs /var/www/scott-docs/
/plastics-docs /var/www/plastics-docs/

Restart Procedures

Odoo (test clones)

Restarting Odoo is SAFE because the clones are neutralized — no crons, no mail, no external side effects. But it briefly interrupts any active user session on the test UI.

# Port 1819 — main test clone
sudo systemctl restart odoo.service
sleep 5
curl -s -o /dev/null -w "1819: %{http_code}\n" http://localhost:1819/web/login

# Port 1820 — Superstore
sudo systemctl restart odoo-superstore.service
sleep 5
curl -s -o /dev/null -w "1820: %{http_code}\n" http://localhost:1820/web/login

# Port 1821 — Training
sudo systemctl restart odoo-training.service
sleep 5
curl -s -o /dev/null -w "1821: %{http_code}\n" http://localhost:1821/web/login

Expected: all three return HTTP 200 after restart.

Production Django/Flask apps

Restart ONE at a time unless you have a specific reason to restart all. Every restart briefly interrupts users of that specific app.

# Personal Assistant
sudo systemctl restart personal-assistant.service

# Scott Financial
sudo systemctl restart scott-financial.service

# Health ERP
sudo systemctl restart health-erp.service

# Life ERP
sudo systemctl restart life-erp.service

# PlantBiz
sudo systemctl restart plantbiz.service

# Deal Hunter
sudo systemctl restart deal-hunter.service

Verify after each restart:

sudo systemctl status <service-name> --no-pager
curl -s -o /dev/null -w "%{http_code}\n" https://exp.scottrecycling.com/<path>/

Personal Tutor (Docker)

cd /opt/personal-tutor
sudo docker compose restart
sudo docker compose ps
curl -s -o /dev/null -w "%{http_code}\n" https://exp.scottrecycling.com/tutor

Nginx

Nginx restart affects everything going through the reverse proxy. Only restart when nginx config changes.

# Check config first — never restart with a bad config
sudo nginx -t

# If config is good:
sudo systemctl reload nginx     # graceful, no dropped connections
# OR
sudo systemctl restart nginx    # brief interruption

PostgreSQL

DO NOT restart PostgreSQL lightly. It interrupts every database-backed service on the box at once. Only restart for: - Major PG version upgrades - Config changes requiring restart (pg_hba.conf typically only needs reload, not restart) - Emergency recovery

# Graceful reload (for pg_hba.conf changes, connection limits, etc.)
sudo systemctl reload postgresql

# Full restart (last resort)
sudo systemctl restart postgresql

If you restart PG, all 7+ services that depend on it will lose connections temporarily and may need a restart themselves.

Health Checking

Quick status check

/opt/monitoring/status.sh

This runs the same checks as the hourly cron and dumps the most recent alerts.

Detailed health check

/opt/monitoring/health-check.sh
tail -60 /opt/monitoring/health.log

Grep for alerts

grep ">>> ALERT:" /opt/monitoring/health.log | tail -20
grep ">>> CRITICAL:" /opt/monitoring/health.log | tail -20

Manual per-service checks

# All systemd services in one command
for svc in odoo odoo-superstore odoo-training personal-assistant scott-financial health-erp life-erp plantbiz deal-hunter nginx postgresql; do
    printf "%-25s %s\n" "$svc" "$(systemctl is-active $svc 2>/dev/null || echo 'not-found')"
done

# All HTTP endpoints
for url in \
    https://exp.scottrecycling.com/ \
    https://exp.scottrecycling.com/superstore/ \
    https://exp.scottrecycling.com/assistant/ \
    https://exp.scottrecycling.com/financial \
    https://financial.scottrecycling.com/ \
    https://exp.scottrecycling.com/health/ \
    https://exp.scottrecycling.com/life/ \
    https://exp.scottrecycling.com/plantbiz/ \
    https://exp.scottrecycling.com/deals/ \
    https://exp.scottrecycling.com/tutor \
    https://exp.scottrecycling.com/docs/ \
    https://training.scottrecycling.com/; do
    code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "$url")
    printf "%-50s %s\n" "$url" "$code"
done

Deployment Procedures

Deploy sr-docs

From Dan's local machine:

cd ~/Desktop/Sr-docs
git pull origin jan
./deploy.sh

The deploy.sh script builds with mkdocs build --strict, rsyncs site/ to /var/www/docs/ on EXP, and verifies HTTP 200. Takes ~10 seconds.

Verifies automatically — but manual sanity check:

curl -s -o /dev/null -w "%{http_code}\n" https://exp.scottrecycling.com/docs/
curl -s -o /dev/null -w "%{http_code}\n" https://exp.scottrecycling.com/docs/strategy/

Deploy SR-Odoo (test clone on EXP)

# On EXP:
cd /opt/odoo/custom_addons/Dan-Odoo
git pull origin jan

# For a MODULE upgrade (most common):
sudo systemctl stop odoo.service
sudo -u odoo bash -c 'cd /tmp && /opt/odoo/odoo17-venv/bin/python3 /opt/odoo/odoo17/odoo-bin \
    -c /etc/odoo/odoo.conf -d sr_live_jan19_backup -u <module_name> \
    --stop-after-init --logfile=/tmp/odoo_upgrade.log'
grep -iE "ERROR|CRITICAL|Traceback" /tmp/odoo_upgrade.log | tail -20
sudo systemctl start odoo.service

# For a FRESH INSTALL:
# Same as above but use -i <module_name> instead of -u

# Verify:
sleep 5
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:1819/web/login

IMPORTANT: EXP is a test clone. After any fresh clone refresh from production, re-run scripts/neutralize_clone.sql on the affected database to re-disable crons and cancel any stale outgoing mail. See the 2026-04-14 incident — 60,769 stale production mails were queued on an already-cloned DB.

Deploy a Django/Flask app

Each app has its own deployment procedure. General pattern:

# Example for scott-financial
cd /opt/scott-financial
sudo -u dan git pull
sudo -u dan /opt/scott-financial/venv/bin/pip install -r requirements.txt  # if deps changed
sudo systemctl restart scott-financial.service
sleep 3
curl -s -o /dev/null -w "%{http_code}\n" https://financial.scottrecycling.com/

Same pattern applies to the other Django/Flask apps. Replace the directory and service name.

Backup & Restore

Nightly backups (automated)

  • Who: /opt/backups/odoo/backup.sh runs at 02:00 via cron
  • What: pg_dump sr_live_jan19_backup | gzip
  • Retention: 14 days rolling
  • Location: /opt/backups/odoo/sr_live_jan19_backup_YYYYMMDD_HHMM.sql.gz
  • Log: /opt/backups/odoo/backup.log

Offsite staging (automated)

  • Who: /opt/backups/odoo-offsite-stage/stage.sh runs Sundays at 04:00 via cron
  • What: rotation to weekly / monthly / yearly tiers with automatic pruning
  • Retention: 8 weekly + 12 monthly + 5 yearly = 25 files × ~114 MB ≈ 2.9 GB total
  • Status: local staging only — not sending offsite yet (pending Dan's cloud credentials)
  • Docs: /opt/backups/odoo-offsite-stage/README.md

Restore (manual — for disaster or testing)

# 1. Identify the backup to restore
ls -lht /opt/backups/odoo/*.sql.gz | head -5

# 2. Create a target database (never overwrite live without explicit intent)
sudo -u postgres createdb <target_db> -O odoo

# 3. Restore
time gunzip -c /opt/backups/odoo/<backup_file>.sql.gz | sudo -u postgres psql <target_db>

# 4. Verify
sudo -u postgres psql <target_db> -c "SELECT count(*) FROM customer_pickups;"
sudo -u postgres psql <target_db> -c "SELECT id, name FROM res_company;"  # should say [TEST CLONE]

# 5. If restoring for a real recovery, re-run neutralize (in case the source is also a clone)
sudo -u postgres psql <target_db> < /opt/odoo/custom_addons/Dan-Odoo/scripts/neutralize_clone.sql

See /opt/backups/odoo/RESTORE_TEST.md for the quarterly restore-test procedure.

Common Failure Modes

"Odoo returns HTTP 500 or won't start"

  1. Check the service status: sudo systemctl status odoo.service
  2. Check the logs: sudo journalctl -u odoo.service --since "10 minutes ago" --no-pager | tail -60
  3. Look for ERROR or Traceback lines
  4. Common causes:
  5. Module upgrade failed — rollback with git reset --hard <prior-commit> in /opt/odoo/custom_addons/Dan-Odoo and restart
  6. Database connection refused — sudo systemctl status postgresql first
  7. Port in use — sudo ss -tlnp | grep :1819 to see if something else is holding the port
  8. Config file error — sudo -u odoo /opt/odoo/odoo17-venv/bin/python3 /opt/odoo/odoo17/odoo-bin -c /etc/odoo/odoo.conf --stop-after-init --test-enable to validate

"Django app returns 502"

502 means nginx can't reach the upstream (the gunicorn process).

  1. Check the service: sudo systemctl status <app>.service
  2. If it's inactive: restart it and check logs
  3. If it's active but gunicorn is crashing: sudo journalctl -u <app>.service --since "10 minutes ago" --no-pager | tail -40
  4. If the app won't start: check that its virtualenv is intact, dependencies are installed, database is reachable

"Nginx won't start"

  1. sudo nginx -t to validate the config
  2. If config error: fix the offending line in /etc/nginx/sites-enabled/exp
  3. sudo systemctl restart nginx
  4. If port 80 or 443 is in use: sudo ss -tlnp | grep -E ':80|:443'

"Everything is down at once"

Probably PostgreSQL. Check:

sudo systemctl status postgresql
sudo systemctl status postgresql@14-main
sudo journalctl -u postgresql@14-main --since "10 minutes ago" --no-pager | tail -40

If PG is running but connections fail, check: - /etc/postgresql/14/main/pg_hba.conf — authentication config - Connection limit: sudo -u postgres psql -c "SHOW max_connections;"

If PG is down and won't start, check disk space: df -h /var/lib/postgresql/. A full disk is the most common cause.

"Disk is full"

df -h
du -sh /var/log /var/lib /opt/backups /tmp /var/cache 2>&1 | sort -h

Likely culprits (in order): 1. /var/log/journal — systemd journal (use sudo journalctl --vacuum-time=7d) 2. /var/log/nginx — nginx access logs (rotate or truncate) 3. /var/lib/postgresql/14/main/pg_log — PG query logs (rotate) 4. /opt/backups/ — backup retention too long 5. /tmp — installer logs, temp files

"SSL certificate expired"

LetsEncrypt auto-renews via cron. If it failed:

sudo certbot renew --dry-run   # diagnose
sudo certbot renew             # force renewal
sudo systemctl reload nginx    # pick up the new cert

Check renewal logs: sudo tail -40 /var/log/letsencrypt/letsencrypt.log

"Monitoring alerts aren't firing"

  1. Check the hourly cron is running: crontab -l | grep health-check
  2. Check the log: tail -100 /opt/monitoring/health.log
  3. Check for stale timestamps: ls -la /opt/monitoring/health.log
  4. Check the email sink config: cat /opt/monitoring/alert.cfg (if SMTP configured)

Emergency Contacts

(Fill in with real data — see ~/Desktop/sr-drafts/succession-plan.md for the full list)

Role Who Phone When to call
Owner / primary Dan [[cell]] Any emergency
Backup coordinator [[name]] [[cell]] Dan unreachable
IT / systems Dan [[cell]] EXP server issues
Hosting provider [[if any]] [[phone]] Physical hardware, network, datacenter issues
DNS registrar [[provider]] [[phone]] Domain / DNS issues
Cloudflare or CDN [[if used]] [[phone]] CDN or DDoS issues

Update History

Date Change By
2026-04-14 Initial runbook created. Documents 3 Odoo + 7 Django/Flask + 4 static doc sites + supporting infrastructure. Dan (via master plan infrastructure hardening)