Project Phoenyx: Disaster Recovery (Vaultwarden Standby)
In IT, redundancy is not an option but a necessity. After establishing database replication in the previous article, we now turn to the application layer. The goal is to prepare the password manager Vaultwarden on a second server (Server 1) so that it can take over Server 2's work within minutes in case of a disaster.
1. The Concept: The Lifeboat
Imagine Server 2 as your cruise ship. It runs at full load, serving all requests and storing new data. Server 1 is the lifeboat on the side. It hangs on the davits (Docker container stopped), but is fully equipped (configured) and constantly maintained (database replication).
Server 2 (Master)
Status: LIVE
Traefik + Vaultwarden
MariaDB Master (Write)
Server 1 (Standby)
Status: SLEEPING
Caddy + Vaultwarden (Stopped)
MariaDB Slave (Read-Only)
We call this "Cold Standby" because the application on Server 1 is not actively running (Cold) but can be started at any time (Standby). The advantage over a "Hot Standby" (both running simultaneously) is the avoidance of data conflicts ("Split-Brain") and resource savings.
In an emergency, speed is key. Caddy is a web server that automatically obtains HTTPS certificates as soon as it starts. We save ourselves complex configurations for the "lifeline".
2. Prerequisites
- Server 1: IP
192.168.10.10(Internal VLAN). - Database: A functioning Master-Slave replication (see separate article) where the container
mariadb_replicaruns on Server 1. - Network: A Docker network named
database_internalexists and connects the DB to the outside world.
3. Installation on Server 1
Step 3.1: Directory Structure
We create a clean folder for our standby service. This is where all configuration files will reside later.
mkdir -p /opt/docker/vaultwarden_standby/vw-data
Step 3.2: Environment Variables (.env)
Create the file /opt/docker/vaultwarden_standby/.env. Here we define the secrets. Important: Since the database is replicated, username and password must match those of Server 2 exactly.
# --- GENERAL SETTINGS --- DOMAIN=vault.phoenyx-systems.de # Admin Token (Identical to Server 2, for emergency access) ADMIN_TOKEN=YourLongAdminTokenFromServer2 # --- DATABASE (Slave Connection) --- DB_USER=vaultwarden_user DB_PASSWORD=YourSecretDBPassword DB_NAME=vaultwarden # Hostname of the MariaDB Slave Container DB_HOST=mariadb_replica
Step 3.3: The Web Server (Caddyfile)
Caddy needs only a tiny configuration file. Create /opt/docker/vaultwarden_standby/Caddyfile. Caddy takes on the role Traefik plays on Server 2: It accepts requests from the internet (Port 80/443) and forwards them to Vaultwarden.
{$DOMAIN} {
log {
output file /var/log/caddy/access.log
}
# Forwards requests to the internal container
reverse_proxy vaultwarden-standby:80 {
header_up X-Real-IP {remote_host}
}
}
Step 3.4: Container Definition (Docker Compose)
The heart of the matter. Create /opt/docker/vaultwarden_standby/docker-compose.yml. Pay special attention to the setting restart: "no" – we don't want this container to start automatically by mistake.
services:
vaultwarden-standby:
image: vaultwarden/server:latest
container_name: vaultwarden_cold
restart: "no" # IMPORTANT: Does not start automatically!
env_file: .env
environment:
- DOMAIN=https://${DOMAIN}
- WEBSOCKET_ENABLED=true
- SIGNUPS_ALLOWED=false
- ADMIN_TOKEN=${ADMIN_TOKEN}
# Connection to replicated database
- DATABASE_URL=mysql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:3306/${DB_NAME}
volumes:
# Here land the synchronized attachments (images etc.)
- ./vw-data:/data
networks:
- database_internal
- caddy_net
caddy-standby:
image: caddy:latest
container_name: caddy_cold
restart: "no"
ports:
- "80:80"
- "443:443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- caddy_data:/data
- caddy_config:/config
environment:
- DOMAIN=${DOMAIN}
networks:
- caddy_net
depends_on:
- vaultwarden-standby
networks:
database_internal:
external: true
caddy_net:
driver: bridge
volumes:
caddy_data:
caddy_config:
4. Synchronization of Attachments
The database (passwords) is replicated automatically. But what about file attachments or icons? These reside as files on Server 2. We must copy them over regularly.
Create a cronjob or script on Server 2 that pushes the data via rsync:
rsync -avz --delete -e ssh \
/opt/containers/vaultwarden/vw-data/ \
[email protected]:/opt/docker/vaultwarden_standby/vw-data/
(This command should ideally run every few hours).
5. Disaster Recovery
Server 2 has failed (hardware defect, datacenter fire, etc.). You need your passwords immediately. Take a deep breath and follow these steps.
Step A: Switch DNS
Log in to your DNS provider (e.g., Cloudflare). Change the A-Record for vault.phoenyx-systems.de from Server 2's IP to Server 1's IP.
Step B: Open Database
The database on Server 1 is in "Read-Only" mode to prevent conflicts. For Vaultwarden to start, we must lift this protection.
docker exec -it mariadb_replica mariadb -u root -p -e "STOP SLAVE; SET GLOBAL read_only = 0;"
Step C: Start Lifeboat
Start the standby containers.
cd /opt/docker/vaultwarden_standby docker compose up -d
Step D: Wait
Caddy will notice upon startup that it is now responsible for the domain and automatically request certificates from Let's Encrypt. This takes about 30-60 seconds. After that, your password manager is back online.
6. Failback
When Server 2 is repaired, you cannot simply switch back. You must:
- Create a fresh database dump from Server 1.
- Import this dump on Server 2.
- Start Server 2 as Master again.
- Stop Server 1 and set it up as Slave (Read-Only) again.