Disaster Recovery Plan — EXP Total Loss Recovery

Purpose: Step-by-step procedure for rebuilding everything on exp.scottrecycling.com if the server is completely lost (hardware failure, ransomware, datacenter incident, accidental rm -rf /).

Audience: Dan, Nisarg, any future technical staff, or a third-party IT consultant called in during an emergency

Why this exists: The EXP Runbook covers service-level recovery ("restart Odoo", "restart nginx"). This document covers total-loss recovery — you've lost the server, you have backups, what do you do next?

Related: - EXP Runbook — service inventory, day-to-day operations - Architecture — module dependencies - Deployment — module deployment procedures - /opt/backups/odoo/RESTORE_TEST.md on EXP — manual restore procedure

---

Recovery Objectives

RTO (Recovery Time Objective): 24-48 hours from incident detection to restored service on a new host

RPO (Recovery Point Objective): ≤24 hours of data loss — the nightly backup at 2 AM UTC is the recovery point for Odoo databases. Production Django/Flask app databases may have shorter RPOs depending on their individual backup strategy (see Section 8).

Business impact at stake: - Scott Recycling Odoo test instance (sr_live_jan19_backup) — low impact, this is a test clone - Scott SuperStore Odoo 19 (scott_superstore) — low impact, also a test clone - Scott Training Odoo (sr_training) — low impact, training data - Personal Assistant (Django) — production, Dan's personal productivity app - Scott Financial (Flask) — production, Scott Companies financial dashboard - Health ERP (Django) — production, Dan's health tracking - Life ERP (Django) — production - PlantBiz (Flask) — production, Stacie Scott's plant business app - Deal Hunter (Flask) — production, SuperStore deal discovery - Personal Tutor (FastAPI/Docker) — production

Customer-facing impact: moderate — financial.scottrecycling.com is customer-visible if used externally. training.scottrecycling.com is publicly accessible. None of the Scott Recycling actual production customer workflows run on EXP (real production Odoo is elsewhere — not on EXP).

---

Phase 0 — Incident Detection & Triage (first 30 minutes)

Detect

You'll know EXP is down when: - Hourly health check stops writing to /opt/monitoring/health.log (but you can't see that if EXP is down) - Services at exp.scottrecycling.com/* return 5xx, timeouts, or nothing - ssh dan@exp.scottrecycling.com fails to connect - Users report apps are down

Triage questions

1. Is it a network issue or a server issue?
2. Can you ping the IP? ping exp.scottrecycling.com
3. Does SSH work? ssh -v dan@exp.scottrecycling.com
4. If ping works but SSH fails → OS or services issue, try console access
5. If ping fails → network or host down
6. Is the host provider up?
7. Check the hosting provider's status page (InMotion / Linode / DigitalOcean / wherever EXP is hosted — fill in below)
8. Hosting provider: [[fill in]]
9. Status page: [[URL]]
10. Support phone: [[phone]]
11. Is it temporary or permanent?
12. Temporary (provider issue, routing, network blip) → wait with targeted communication
13. Permanent (hardware failure, filesystem corruption, ransomware) → activate DR

Communication during outage

If outage extends beyond 1 hour:

• Notify Dan immediately (Dan is the incident commander unless delegated)
• Post a status somewhere customers can see (if they'd notice) — e.g., a Twitter update, email to key customers, etc.
• Notify employees that EXP is down and what they should do (fall back to manual procedures)

---

Phase 1 — Secure Backup Access (first hour)

Before you do anything else: make sure you can access backups. Nothing else matters if you can't.

Where backups live

• Primary: /opt/backups/odoo/ on EXP (but if EXP is gone, this is gone too)
• Offsite staging: /opt/backups/odoo-offsite-stage/ on EXP (same problem — same box)
• Actually offsite: NONE YET as of 2026-04-14. The offsite staging area is prepared but not actively sending to cloud storage.

⚠️ This is a critical gap

If EXP is completely lost right now, there is no offsite backup. The offsite staging area is local to EXP.

Immediate action if this is still the state at time of disaster:

• Contact the hosting provider and request emergency filesystem access if possible
• Check any backups that might exist outside EXP:
• Manual backups Dan may have downloaded to his laptop?
• Any backups saved to cloud services (Google Drive, Dropbox, etc.)?
• The GitHub clone of the sr-docs and SR-Odoo repos (but these are CODE only, not data)
• Worst case: rebuild from GitHub code + manually re-entered recent data

Long-term fix (documented in /opt/backups/odoo-offsite-stage/README.md on EXP): wire up Backblaze B2 or AWS S3 so the offsite staging actually sends data offsite. Until that's done, this DR plan is aspirational for anything > what Dan has on his laptop.

If offsite backups exist (future state)

# Mount the offsite backup location (example for B2 via rclone)
rclone lsf scott-b2:scott-recycling-odoo-offsite/

# Find the most recent backup
rclone ls scott-b2:scott-recycling-odoo-offsite/weekly/ | sort | tail

# Download the most recent weekly snapshot
rclone copy scott-b2:scott-recycling-odoo-offsite/weekly/sr_live_YYYYMMDD_weekly.sql.gz .

Verify: - File size matches expected (~114 MB per the 2026-04-14 test backup) - gunzip -t passes - Header line reads "PostgreSQL database dump"

---

Phase 2 — Provision Replacement Server (hours 1-4)

Server specifications needed

Match or exceed current EXP specs:

Spec	Current EXP	Replacement
CPU	2 vCPU	2-4 vCPU
RAM	8 GB	8 GB minimum, 16 GB recommended
Disk	156 GB (47% used = ~75 GB used)	200 GB minimum SSD
OS	Ubuntu 24.04 LTS (Noble)	Ubuntu 24.04 LTS
Network	Public IPv4	Public IPv4
IPv6	Optional	Optional
Bandwidth	Adequate for normal traffic + backup restore	Unmetered or generous cap

Provider options (in order of preference)

1. Same hosting provider as before (easiest — DNS and networking already set up for that provider's conventions). Provider: [[fill in]]
2. Secondary hosting provider as failover. Candidates: Linode, DigitalOcean, Hetzner, Vultr, AWS EC2, Google Cloud Compute
3. On-premises box if available — least preferred unless existing

Provision steps

1. Create the new VM / server
2. Note the new IP address
3. Configure DNS (see Phase 6) — but wait, don't change DNS yet
4. SSH in as root initially

---

Phase 3 — Base OS Setup (hours 4-6)

System packages

# Update the system
sudo apt update && sudo apt upgrade -y

# Essential packages
sudo apt install -y \
    build-essential git curl wget vim htop \
    python3 python3-pip python3-venv python3-dev \
    python3-wheel python3-setuptools \
    postgresql postgresql-contrib postgresql-server-dev-all \
    nginx certbot python3-certbot-nginx \
    nodejs npm \
    libxml2-dev libxslt1-dev libldap2-dev libsasl2-dev \
    libtiff5-dev libjpeg8-dev libopenjp2-7-dev zlib1g-dev \
    libfreetype6-dev liblcms2-dev libwebp-dev \
    libharfbuzz-dev libfribidi-dev libxcb1-dev \
    libssl-dev libffi-dev libpq-dev \
    unzip zip tree jq \
    whois msmtp msmtp-mta \
    docker.io docker-compose

# Optional tools (install as needed)
sudo apt install -y iotop nethogs ncdu tmux

User accounts

# Create dan user
sudo useradd -m -s /bin/bash dan
sudo usermod -aG sudo,docker dan

# Copy Dan's SSH key (from somewhere — laptop backup, password manager)
sudo mkdir -p /home/dan/.ssh
sudo chmod 700 /home/dan/.ssh
# Paste authorized_keys content
sudo chown -R dan:dan /home/dan/.ssh
sudo chmod 600 /home/dan/.ssh/authorized_keys

# Create odoo system user (note: EXP uses 'odoo', not 'odoo17')
sudo useradd -m -d /opt/odoo -s /bin/bash odoo

# Disable root SSH login
sudo sed -i 's/^PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sudo systemctl reload ssh

Firewall (UFW)

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw --force enable

Hostname

sudo hostnamectl set-hostname exp.scottrecycling.com

---

Phase 4 — Install and Configure PostgreSQL (hour 6)

# Verify PostgreSQL is running
sudo systemctl status postgresql

# Set postgres user password (if needed)
sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'generate-strong-password';"

# Create the odoo PostgreSQL user
sudo -u postgres createuser -s odoo

# Apply performance tuning (based on 2026-04-14 profiling recommendations)
# Edit /etc/postgresql/14/main/postgresql.conf
sudo tee -a /etc/postgresql/14/main/postgresql.conf <<'EOF'

# Performance tuning for 8 GB RAM / 2 vCPU host
# Applied as part of DR recovery per 2026-04-14 profiling
shared_buffers = 2GB
effective_cache_size = 6GB
work_mem = 16MB
maintenance_work_mem = 512MB
random_page_cost = 1.1
checkpoint_timeout = 15min
max_wal_size = 4GB
wal_buffers = 16MB
log_min_duration_statement = 1000
shared_preload_libraries = 'pg_stat_statements'
autovacuum_vacuum_scale_factor = 0.05
autovacuum_max_workers = 4
EOF

sudo systemctl restart postgresql

---

Phase 5 — Restore Data (hours 6-12)

Create databases (structure only, before restore)

# List expected databases (from EXP inventory)
for db in sr_live_jan19_backup scott_superstore sr_training \
          dan_test_may dan_inventory_clean odoo_test \
          scott_plastics sr_backup_2march \
          personal_assistant scott_financial health_erp \
          life_erp plantbiz; do
    sudo -u postgres createdb -O odoo "$db"
done

Restore Odoo databases from offsite backups

# Primary test clone
gunzip -c /tmp/restore/sr_live_jan19_backup_YYYYMMDD_weekly.sql.gz | \
    sudo -u postgres psql sr_live_jan19_backup

# Repeat for other Odoo DBs that had their own backups
# (Note: only sr_live_jan19_backup was in the automated backup rotation as of 2026-04-14.
# Other Odoo test DBs may need to be recreated from scratch or skipped — they're test data.)

# Immediately re-run the neutralize script on restored Odoo clones
for db in sr_live_jan19_backup scott_superstore sr_training \
          dan_test_may dan_inventory_clean odoo_test \
          scott_plastics sr_backup_2march; do
    sudo -u postgres psql "$db" < /path/to/neutralize_clone.sql 2>/dev/null
done

Restore Django/Flask app databases

⚠️ CRITICAL GAP: As of 2026-04-14, only sr_live_jan19_backup is in the automated backup rotation (/opt/backups/odoo/backup.sh). The Django/Flask app databases (personal_assistant, scott_financial, health_erp, life_erp, plantbiz) are NOT automatically backed up.

Pre-disaster action needed: add each production Django/Flask database to a backup rotation. This is a gap to close before a disaster, not during.

During disaster if databases are gone: - Rebuild schema from application migrations (each Django app has migrations in source) - Restore data from any manual exports (Dan may have some) - Accept data loss for anything not manually exported

Verify restore

# For Odoo databases
sudo -u postgres psql sr_live_jan19_backup -c "\dt" | wc -l
sudo -u postgres psql sr_live_jan19_backup -c "SELECT count(*) FROM res_partner;"
sudo -u postgres psql sr_live_jan19_backup -c "SELECT count(*) FROM customer_pickups;"
sudo -u postgres psql sr_live_jan19_backup -c "SELECT count(*) FROM ir_module_module WHERE state='installed';"

Expected row counts (from 2026-04-14 baseline): - res_partner: ~149,736 - customer_pickups: ~31,395 - ir_module_module WHERE state='installed': ~207

---

Phase 6 — Restore Application Code (hours 12-18)

Clone repositories

# Create directory structure
sudo mkdir -p /opt/odoo /opt/personal-assistant /opt/scott-financial \
              /opt/health_erp /opt/life_erp /opt/plantbiz /opt/deal-hunter \
              /opt/personal-tutor
sudo chown dan:dan /opt/personal-assistant /opt/scott-financial \
                    /opt/health_erp /opt/life_erp /opt/plantbiz /opt/deal-hunter \
                    /opt/personal-tutor

# Odoo core 17
sudo -u odoo git clone --depth 1 --branch 17.0 https://github.com/odoo/odoo.git /opt/odoo/odoo17
sudo -u odoo python3 -m venv /opt/odoo/odoo17-venv
sudo -u odoo /opt/odoo/odoo17-venv/bin/pip install -r /opt/odoo/odoo17/requirements.txt
sudo -u odoo /opt/odoo/odoo17-venv/bin/pip install wheel psycopg2-binary

# Custom Odoo addons
sudo mkdir -p /opt/odoo/custom_addons
sudo chown -R odoo:odoo /opt/odoo/custom_addons
sudo -u odoo git clone https://github.com/ScottRecycling/SR-Odoo.git /opt/odoo/custom_addons/Dan-Odoo

# sr-docs
sudo -u dan git clone https://github.com/scottrecycling101/Sr-docs.git /home/dan/Desktop/Sr-docs

# Django/Flask apps — each has its own repo
# You need to know the repo URLs. Document them in the runbook or a private location.

Where to find repo URLs

If you don't remember the repo URLs, check: - Dan's laptop — most likely location - GitHub scottrecycling101 organization — scottrecycling101/Sr-docs confirmed, others unclear - Nisarg — may have copies - Dan's password manager — likely stored credentials for each

Pre-disaster action: write down all repo URLs in a document outside EXP.

Install Odoo 19 for SuperStore

sudo -u odoo git clone --depth 1 --branch 19.0 https://github.com/odoo/odoo.git /opt/odoo/odoo19
sudo -u odoo python3 -m venv /opt/odoo/odoo19-venv
# (assuming SS-Odoo custom addons are cloned separately)

---

Phase 7 — systemd Services (hours 18-20)

Recreate each systemd unit file. These are captured in /etc/systemd/system/ on current EXP.

Pre-disaster action: Back up /etc/systemd/system/*.service files and include them in the offsite backup. Currently NOT in the backup.

Example Odoo unit file:

# /etc/systemd/system/odoo.service
[Unit]
Description=Odoo 17
Requires=postgresql.service
After=network.target postgresql.service

[Service]
Type=simple
SyslogIdentifier=odoo
PermissionsStartOnly=true
User=odoo
Group=odoo
ExecStart=/opt/odoo/odoo17-venv/bin/python3 /opt/odoo/odoo17/odoo-bin -c /etc/odoo/odoo.conf
StandardOutput=journal+console
KillMode=mixed

[Install]
WantedBy=multi-user.target

Repeat for each service with its own config and port.

---

Phase 8 — nginx & SSL (hour 20)

# Copy the nginx config (backup from offsite, or recreate)
# /etc/nginx/sites-enabled/exp

# Test the config
sudo nginx -t

# Get SSL certs via LetsEncrypt
sudo certbot --nginx \
    -d exp.scottrecycling.com \
    -d financial.scottrecycling.com \
    -d training.scottrecycling.com \
    -d stg.scottrecycling.com \
    -d scottrecycling.com \
    --non-interactive --agree-tos --email dan@scottrecycling.com

# Start nginx
sudo systemctl restart nginx

⚠️ DNS must be updated BEFORE certbot runs (certbot validates by HTTP challenge requires DNS pointing at the new server).

---

Phase 9 — DNS Cutover (hour 20+)

What to update

All DNS records currently pointing to the old EXP IP need to point to the new IP.

Records to update: - exp.scottrecycling.com — A record - financial.scottrecycling.com — A record - training.scottrecycling.com — A record - stg.scottrecycling.com — A record (if in use) - *.scottrecycling.com — wildcard? Check DNS provider config

DNS provider

• Current DNS provider: [[fill in — e.g., Namecheap, Cloudflare, GoDaddy]]
• Admin login: [[store in password manager]]
• Support phone: [[provider support]]

TTL consideration

DNS has a TTL. If TTL is 300 seconds (5 min), changes propagate fast. If TTL is 86400 (1 day), expect slow propagation.

Pre-disaster action: Set all critical DNS records to a LOW TTL (300 seconds) as a default. Currently unknown what TTL is set.

---

Phase 10 — Restart Services & Verify (hour 21)

# Enable all services
for svc in odoo odoo-superstore odoo-training \
           personal-assistant scott-financial health-erp life-erp \
           plantbiz deal-hunter nginx; do
    sudo systemctl enable "$svc"
    sudo systemctl start "$svc"
    sleep 3
    sudo systemctl status "$svc" --no-pager | head -5
done

Verify each endpoint

for url in \
    https://exp.scottrecycling.com/ \
    https://exp.scottrecycling.com/superstore/ \
    https://exp.scottrecycling.com/assistant/ \
    https://exp.scottrecycling.com/financial \
    https://financial.scottrecycling.com/ \
    https://exp.scottrecycling.com/health/ \
    https://exp.scottrecycling.com/life/ \
    https://exp.scottrecycling.com/plantbiz/ \
    https://exp.scottrecycling.com/deals/ \
    https://exp.scottrecycling.com/docs/ \
    https://training.scottrecycling.com/; do
    code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "$url")
    printf "%-50s %s\n" "$url" "$code"
done

Restore monitoring

# Copy /opt/monitoring/* files from offsite backup
sudo -u dan cp -r /path/to/backup/monitoring /opt/monitoring
sudo chown -R dan:dan /opt/monitoring
chmod +x /opt/monitoring/*.sh

# Restore cron
(crontab -l 2>/dev/null; echo "0 * * * * /opt/monitoring/health-check.sh") | crontab -
(crontab -l 2>/dev/null; echo "0 2 * * * /opt/backups/odoo/backup.sh") | crontab -
(crontab -l 2>/dev/null; echo "0 4 * * 0 /opt/backups/odoo-offsite-stage/stage.sh") | crontab -

# Run first health check
/opt/monitoring/health-check.sh
tail -50 /opt/monitoring/health.log

---

Phase 11 — Communications (hour 22)

• Notify employees that services are back
• Notify customers if communication was needed during the outage
• Write up a post-mortem — what failed, what recovery looked like, what to improve

---

Pre-Disaster Action Items (THE POINT OF THIS DOCUMENT)

The DR plan above has several gaps that must be closed BEFORE a disaster, not during. These are the pre-disaster action items:

Critical gaps (high priority)

1. Offsite backups actually going offsite — currently staged locally on EXP. Wire up Backblaze B2 or AWS S3 per /opt/backups/odoo-offsite-stage/README.md. Without this, the whole DR plan is aspirational.
2. Django/Flask app database backups — currently only sr_live_jan19_backup is backed up nightly. Add the production Django app databases (personal_assistant, scott_financial, health_erp, life_erp, plantbiz) to the nightly rotation. ~30 minutes of work.
3. Backup of systemd unit files — /etc/systemd/system/*.service files are not in the current backup. Add them. 5 minutes of work.
4. Backup of nginx config — /etc/nginx/sites-enabled/exp and related files. Add them. 5 minutes.
5. Backup of PostgreSQL config — /etc/postgresql/14/main/postgresql.conf and pg_hba.conf. 5 minutes.
6. Backup of /opt/monitoring/ — the monitoring scripts and configs. 5 minutes.
7. List of all git repo URLs for the Django/Flask apps — not in any single place currently. Write it down. 10 minutes.
8. DNS provider credentials in password manager — verify they're current. 2 minutes.
9. Hosting provider credentials in password manager — verify. 2 minutes.
10. DNS TTL check — verify all critical records have short TTL (300s). 10 minutes.

Important gaps (medium priority)

1. Personal Tutor Docker config — the /opt/personal-tutor/docker-compose.yml and any environment files. Back these up.
2. Any .env files in Django/Flask app directories — these contain secrets (DB passwords, API keys, SMTP creds). Back these up separately with encryption.
3. Let's Encrypt renewal info — /etc/letsencrypt/ — certs are automatable but saves time in restore. Back it up.
4. Crontabs — dan's and root's crontabs. Back them up.
5. Public SSH keys for all users who need access — ~/.ssh/authorized_keys. Back them up.

Nice-to-have (low priority)

1. Automated DR test — annually, provision a temp VM and walk through Phases 2-10 to verify the procedure actually works. Half day once a year.
2. Secondary hosting account prep — sign up with a second hosting provider in advance so you don't waste time during a disaster. 30 minutes.
3. Runbook printouts — if EXP and your laptop are both unavailable, you need printed copies of this plan + credentials. 15 minutes.

---

Disaster Scenarios & Expected Recovery

Scenario	Data loss risk	Recovery time	Procedure
Single service crashes	None	Minutes	EXP runbook (not DR)
PostgreSQL corruption	≤24 hrs of data	Hours	Restore single DB from backup + neutralize
Full disk failure, hardware replacement on same host	None if hosting provider restores	Hours-Day	Coordinate with provider, no DR needed
Hosting provider outage >12 hours	None (their problem)	Variable	Wait, or activate DR if prolonged
Ransomware / OS compromise	≤24 hrs	2 days	Full DR, new host, restore from offsite
Datacenter disaster	≤24 hrs	2-3 days	Full DR, different provider
Accidental data destruction	≤24 hrs	Hours	Restore specific data from backup
Total loss (hardware, network, filesystem all destroyed)	≤24 hrs (IF offsite backups are working)	2-3 days (full DR)	This document

---

Review & Update

• Annually — full DR plan review + live test (provision a temp VM and walk through procedure)
• Triggered — any time service inventory changes (new app deployed, old app retired)
• Triggered — any time a major dependency changes (Ubuntu version, Python version, Odoo version)
• After any actual incident — update the plan based on lessons learned

Last review: 2026-04-14 (initial draft) Next review: annually, or when offsite backups go live (whichever is sooner) Owner: Dan

---

Related Documents

• EXP Runbook — day-to-day operations
• Architecture — module dependencies
• Deployment — module deployment
• ~/Desktop/sr-drafts/succession-plan.md — what happens to decision authority during an incident
• /opt/backups/odoo/RESTORE_TEST.md — manual restore procedure (on EXP)
• /opt/backups/odoo-offsite-stage/README.md — offsite backup setup instructions (on EXP)
• Master plan items A9 (backups), A10 (this doc), D1 (automation)