Farholm — Operations Runbook

Last reviewed: 2026-06-14. Owner: Jake (sole operator). Keep this short and actionable. For architecture, see README.md; for known risks, see REVIEW.md.

Quick reference

What	Where
Hosting / DNS / Worker / KV / R2 / Access	Cloudflare dashboard (one account)
Source + CI	GitHub → Cloudflare Workers Builds (deploy on push to `main`)
Booking data	KV namespace `farholm-bookings` (`booking:` / `trash:` keys)
Backups	R2 bucket `farholm-backups` (`snapshots/<date>/<stamp>.json`, ~90-day retention)
Contact inquiries	Web3Forms → email to `hello@farholm.com`
Privacy/cookie policy	Termly (embedded on `/privacy`, `/cookies`)
Health check	`GET https://trip.farholm.com/api/health` → `200 {ok:true}`
Admin	`https://trip.farholm.com/admin` (Cloudflare Access, admin email only)

Severity & first steps

SEV1 — booking portal down, suspected unauthorized access, or client-data loss.
SEV2 — degraded (contact form failing, a third-party lookup down).
SEV3 — cosmetic / non-urgent.

First, always: note the time, capture evidence (screenshots, curl -i output, Cloudflare Ray IDs, Worker logs) before changing anything. Then work the relevant scenario below.

Scenarios

1. Portal / site outage

Confirm scope: curl -I https://farholm.com/ and …/api/health (expect 200).
Cloudflare → Workers & Pages → farholm-site → Deployments. If a recent deploy correlates, roll back: open the previous good deployment → promote.
Check Workers Builds logs for a failed build; check the zone for an incident on Cloudflare Status.
If only dynamic routes fail but static pages serve, suspect a Worker error — check Worker Logs/real-time logs; redeploy last known-good.

2. Failed or missing backup

Signal: the scheduled cron logs {"evt":"snapshot","ok":false,...} or no new object appears under snapshots/<today>/ in R2.

Run a manual snapshot: admin → Snapshot now (or POST /api/admin/snapshot).
If it errors, check the R2 binding (BACKUPS) and bucket existence, then KV read access. Re-run.
Confirm a fresh object exists in farholm-backups.

3. Lost or corrupted booking data

Soft-deleted by mistake: read trash:<REF> in KV (Cloudflare dashboard) and re-save its JSON through the admin. (Trash holds 90 days.)
Corrupted/overwritten: download the most recent good snapshots/<date>/<stamp>.json from R2, find the booking, re-save via admin.
Bulk loss (namespace): restore from the latest R2 snapshot into a fresh KV namespace, repoint the BOOKINGS binding, redeploy. Test FH-9QXM4K7P.

4. Suspected unauthorized access

Cloudflare Zero Trust → Access → review the application logs for /admin and /api/admin/*; confirm only the admin email authenticated.
Rotate: Cloudflare account password + MFA, Access policy, and any exposed Worker secrets (GOOGLE_PLACES_KEY, AERODATABOX_KEY).
Take a snapshot for evidence; review recent booking:/trash: changes.
If client PII was exposed, see Client notification below.

5. Contact form failing

The form falls back to a visible mailto: link on failure, so inquiries can still arrive. Check Web3Forms status + the access key; confirm hello@farholm.com delivery (and spam folder). Consider enabling Web3Forms domain restriction / Turnstile if abuse (spam flood) is the cause.

6. Third-party outage

Termly down → /privacy /cookies render empty. Mitigation: keep a first-party copy to paste in (see REVIEW.md backlog). Not a SEV1.
Google Places / AeroDataBox / Frankfurter down → admin auto-fill / FX conversion degrade only; manual booking entry still works. No client impact.

Client notification (data events)

Notify affected clients without undue delay if their personal data was exposed or lost. Include what happened, what data, what you're doing, and what they should do. Check state breach-notification obligations for affected residents.

Recovery objectives (targets)

Portal restore (rollback): < 1 hour.
Booking data restore from snapshot: < 4 hours; acceptable data loss ≤ 24 hours (daily snapshot cadence + soft-delete).

Exercise schedule

Twice a year: (1) roll back to a previous Worker deployment on a quiet day and confirm the site serves; (2) restore one booking from an R2 snapshot into a scratch namespace. Record the date and result here.