Project Phoenyx: Disaster Recovery (Vaultwarden Standby)

Category: Advanced / Security | Documentation

In IT, redundancy is not an option but a necessity. After establishing database replication in the previous article, we now turn to the application layer. The goal is to prepare the password manager Vaultwarden on a second server (Server 1) so that it can take over Server 2's work within minutes in case of a disaster.

1. The Concept: The Lifeboat

Imagine Server 2 as your cruise ship. It runs at full load, serving all requests and storing new data. Server 1 is the lifeboat on the side. It hangs on the davits (Docker container stopped), but is fully equipped (configured) and constantly maintained (database replication).

Server 2 (Master)

Status: LIVE

Traefik + Vaultwarden

MariaDB Master (Write)

➡ Sync ➡

Server 1 (Standby)

Status: SLEEPING

Caddy + Vaultwarden (Stopped)

MariaDB Slave (Read-Only)

[Image of High Availability Architecture]

We call this "Cold Standby" because the application on Server 1 is not actively running (Cold) but can be started at any time (Standby). The advantage over a "Hot Standby" (both running simultaneously) is the avoidance of data conflicts ("Split-Brain") and resource savings.

Why Caddy instead of Traefik on Server 1?
In an emergency, speed is key. Caddy is a web server that automatically obtains HTTPS certificates as soon as it starts. We save ourselves complex configurations for the "lifeline".

2. Prerequisites

3. Installation on Server 1

Step 3.1: Directory Structure

We create a clean folder for our standby service. This is where all configuration files will reside later.

mkdir -p /opt/docker/vaultwarden_standby/vw-data

Step 3.2: Environment Variables (.env)

Create the file /opt/docker/vaultwarden_standby/.env. Here we define the secrets. Important: Since the database is replicated, username and password must match those of Server 2 exactly.

# --- GENERAL SETTINGS ---
DOMAIN=vault.phoenyx-systems.de

# Admin Token (Identical to Server 2, for emergency access)
ADMIN_TOKEN=YourLongAdminTokenFromServer2

# --- DATABASE (Slave Connection) ---
DB_USER=vaultwarden_user
DB_PASSWORD=YourSecretDBPassword
DB_NAME=vaultwarden

# Hostname of the MariaDB Slave Container
DB_HOST=mariadb_replica

Step 3.3: The Web Server (Caddyfile)

Caddy needs only a tiny configuration file. Create /opt/docker/vaultwarden_standby/Caddyfile. Caddy takes on the role Traefik plays on Server 2: It accepts requests from the internet (Port 80/443) and forwards them to Vaultwarden.

{$DOMAIN} {
    log {
        output file /var/log/caddy/access.log
    }

    # Forwards requests to the internal container
    reverse_proxy vaultwarden-standby:80 {
        header_up X-Real-IP {remote_host}
    }
}

Step 3.4: Container Definition (Docker Compose)

The heart of the matter. Create /opt/docker/vaultwarden_standby/docker-compose.yml. Pay special attention to the setting restart: "no" – we don't want this container to start automatically by mistake.

services:
  vaultwarden-standby:
    image: vaultwarden/server:latest
    container_name: vaultwarden_cold
    restart: "no"  # IMPORTANT: Does not start automatically!
    env_file: .env
    environment:
      - DOMAIN=https://${DOMAIN}
      - WEBSOCKET_ENABLED=true
      - SIGNUPS_ALLOWED=false
      - ADMIN_TOKEN=${ADMIN_TOKEN}
      # Connection to replicated database
      - DATABASE_URL=mysql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:3306/${DB_NAME}
    volumes:
      # Here land the synchronized attachments (images etc.)
      - ./vw-data:/data
    networks:
      - database_internal
      - caddy_net

  caddy-standby:
    image: caddy:latest
    container_name: caddy_cold
    restart: "no"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    environment:
      - DOMAIN=${DOMAIN}
    networks:
      - caddy_net
    depends_on:
      - vaultwarden-standby

networks:
  database_internal:
    external: true
  caddy_net:
    driver: bridge

volumes:
  caddy_data:
  caddy_config:

4. Synchronization of Attachments

The database (passwords) is replicated automatically. But what about file attachments or icons? These reside as files on Server 2. We must copy them over regularly.

Create a cronjob or script on Server 2 that pushes the data via rsync:

rsync -avz --delete -e ssh \
    /opt/containers/vaultwarden/vw-data/ \
    [email protected]:/opt/docker/vaultwarden_standby/vw-data/

(This command should ideally run every few hours).

Emergency Plan

5. Disaster Recovery

Server 2 has failed (hardware defect, datacenter fire, etc.). You need your passwords immediately. Take a deep breath and follow these steps.

Step A: Switch DNS

Log in to your DNS provider (e.g., Cloudflare). Change the A-Record for vault.phoenyx-systems.de from Server 2's IP to Server 1's IP.

Step B: Open Database

The database on Server 1 is in "Read-Only" mode to prevent conflicts. For Vaultwarden to start, we must lift this protection.

docker exec -it mariadb_replica mariadb -u root -p -e "STOP SLAVE; SET GLOBAL read_only = 0;"

Step C: Start Lifeboat

Start the standby containers.

cd /opt/docker/vaultwarden_standby
docker compose up -d

Step D: Wait

Caddy will notice upon startup that it is now responsible for the domain and automatically request certificates from Let's Encrypt. This takes about 30-60 seconds. After that, your password manager is back online.

6. Failback

Warning: As soon as you work on Server 1 (save new passwords), the data on Server 2 is outdated!

When Server 2 is repaired, you cannot simply switch back. You must:

  1. Create a fresh database dump from Server 1.
  2. Import this dump on Server 2.
  3. Start Server 2 as Master again.
  4. Stop Server 1 and set it up as Slave (Read-Only) again.