---
tags: [servers]
---
# Server runbook

## Overview
The production host runs the web service, a worker and a PostgreSQL database behind a reverse proxy. All services start through systemd units named after the service, and logs go to the journal first and then to files under the var log directory. Keep this runbook short: when a procedure grows, move it to its own note.

## Deploying
Deployments are pulled by a script that fetches the newest tagged release, runs the migrations and restarts the web unit. Never deploy on Fridays. If the health check fails twice, the script rolls back to the previous tag and posts the result in the team channel.

## Monitoring
A small agent reports cpu, memory and disk every minute. The alert thresholds are eighty percent for cpu and ninety percent for disk. The dashboard lives on the internal wiki and the on call phone gets a page when two checks fail in a row.

## Log rotation
Logs rotate nightly with logrotate. The web service keeps fourteen compressed archives and the worker keeps seven. To force a rotation run logrotate with the force flag against the service file, then check that the archive appears next to the live file. If the disk is nearly full, delete the oldest archive by hand before forcing a rotation, because compression needs temporary space.

## Certificates
Certificates renew automatically sixty days before they expire. The renewal hook reloads the reverse proxy. When renewal fails, check that port eighty is reachable from outside and that the challenge directory is writable.

## Restore
To restore the database take the newest snapshot from the backup bucket, stop the worker, drop the damaged database, load the snapshot with pg_restore and start the worker again. Restoring a snapshot takes about twenty minutes for the current size. Always test the restore on the staging host first.

## Access
Only the platform team has shell access. Keys are rotated every quarter and removed the day someone leaves the team.