Skip to main content

Backing Up Longhorn to Backblaze B2 (and the Snapshots That Wouldn't Die)

Off-site Longhorn backups to Backblaze B2 for a few dollars a month, managed with Ansible. Plus how a cleanup job quietly deleted nothing for weeks and let 350 GiB of snapshots pile up.

Backing Up Longhorn to Backblaze B2 (and the Snapshots That Wouldn't Die)

Longhorn keeps two replicas of every volume on different nodes, which protects me from a dead NVMe drive. It does nothing if the whole cluster goes: a bad upgrade, a fire, or me running the wrong command. For that I needed copies somewhere else, so every Longhorn volume now backs up to Backblaze B2 every night.

This post covers the setup, the Ansible behind it, and a bug that let snapshots quietly fill two of my nodes.

Why Backblaze B2

  • S3-compatible. Longhorn's backup target speaks S3, and B2 has an S3 endpoint, so nothing special is needed.
  • Cheap. Storage costs a fraction of what the big clouds charge, and for a homelab the bill is a few dollars a month.
  • Off-site. It's somewhere other than my closet, which is the whole point.

Setting up the bucket

  1. Create a private bucket in B2.
  2. Create an application key that can only access that bucket. Don't use the master key. If this key ever leaks, the worst case is one bucket of backups, not the whole account.
  3. Note the bucket's S3 endpoint, in the form https://s3.<region>.backblazeb2.com.

Longhorn's backup target format is s3://<bucket>@<region>/<prefix>. I give each cluster its own prefix, so more than one cluster can share the bucket without stepping on each other.

One thing worth checking: by default B2 keeps old versions of files when they're deleted. When Longhorn prunes old backups, you want that space back, so look at the bucket's lifecycle settings.

The Ansible

The playbook is short. A secrets role loads the B2 key from my secrets manager at runtime, so it never lives in the repo, and then the Longhorn role does the rest:

---
- name: Install Longhorn on Talos Kubernetes Cluster
  hosts: localhost
  connection: local
  gather_facts: false

  vars_files:
    - ../group_vars/talos_cluster.yml

  roles:
    - role: secrets     # pulls the B2 key id/secret from the secrets manager
    - role: longhorn

The backup-related defaults in the role:

# S3 backup configuration
longhorn_backup_enabled: true
longhorn_backup_target: "s3://<bucket>@<region>/<cluster-name>"
longhorn_backup_secret_name: longhornbackup
longhorn_s3_endpoints: "https://s3.<region>.backblazeb2.com"

# Storage defaults
longhorn_default_replica_count: 2
longhorn_data_locality: best-effort

# Recurring jobs
longhorn_recurring_backup_cron: "30 2 * * *"
longhorn_recurring_backup_retain: 7
longhorn_recurring_system_backup_cron: "0 3 * * *"
longhorn_recurring_system_backup_retain: 7

1. The credentials secret

Longhorn reads the S3 credentials from a Kubernetes Secret. Setting AWS_ENDPOINTS is what points it at B2 instead of AWS:

- name: Create S3 backup secret
  when: longhorn_backup_enabled | bool
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: Secret
      metadata:
        name: "{{ longhorn_backup_secret_name }}"
        namespace: longhorn-system
      type: Opaque
      stringData:
        AWS_ACCESS_KEY_ID: "{{ longhorn_s3_access_key_id }}"
        AWS_SECRET_ACCESS_KEY: "{{ longhorn_s3_secret_access_key }}"
        AWS_ENDPOINTS: "{{ longhorn_s3_endpoints }}"

2. The backup target

The Helm install sets the default backup store, so the target exists the moment Longhorn comes up:

helm upgrade --install longhorn longhorn/longhorn \
  --namespace longhorn-system \
  ...
  --set defaultBackupStore.backupTarget="{{ longhorn_backup_target }}" \
  --set defaultBackupStore.backupTargetCredentialSecret={{ longhorn_backup_secret_name }}

3. The recurring jobs

Two Longhorn RecurringJobs do the actual work, declared in Ansible instead of clicked together in the UI:

  • daily-backup (02:30): backs up every volume's data to B2 and keeps 7.
  • system-backup (03:00): saves Longhorn's own configuration (volumes, settings, recurring jobs) so the whole storage layer can be rebuilt on a fresh cluster. It keeps 7.
- name: Declare Longhorn RecurringJobs
  kubernetes.core.k8s:
    state: present
    apply: true
    definition:
      apiVersion: longhorn.io/v1beta2
      kind: RecurringJob
      metadata:
        name: "{{ item.name }}"
        namespace: longhorn-system
      spec: "{{ item.spec }}"
  loop:
    - name: system-backup
      spec:
        name: system-backup
        task: system-backup
        cron: "{{ longhorn_recurring_system_backup_cron }}"
        retain: "{{ longhorn_recurring_system_backup_retain }}"
        concurrency: 0
        groups: []
        labels: {}
        parameters:
          volume-backup-policy: disabled
    - name: daily-backup-retain7
      spec:
        name: daily-backup-retain7
        task: backup
        cron: "{{ longhorn_recurring_backup_cron }}"
        retain: "{{ longhorn_recurring_backup_retain }}"
        concurrency: 2
        groups:
          - default
        labels: {}
  loop_control:
    label: "{{ item.name }}"

Skipping volumes you don't need

Not everything deserves an off-site copy. My Prometheus metrics volume is big, changes constantly, and is fine to lose. You'd think you could take a volume out of the backup by removing its default group label. You can't: Longhorn puts any volume with no recurring-job label straight back into default.

The trick is to put it in a different group that no job uses. I made an empty group called no-backup and label the Prometheus volume with it:

recurring-job-group.longhorn.io/no-backup: enabled

The snapshots that wouldn't die

This is the part that actually cost me something.

Every Longhorn backup starts with a local snapshot. Longhorn cleans up most of them on its own, but the snapshots left by my system backup job were flagged as user-created, and Longhorn never touches those. So every night, every volume got one more snapshot that would stay forever.

I already knew this, so I'd written a small CronJob to reap old ones. Every night it exited 0, the Job showed Complete, and everything looked fine. Then two nodes started running low on disk.

The logs told the story:

=== Done: deleted=0 skipped=2351 ===
SKIP [system-backup-...]: unparseable creationTime 2026-08-17T02:01:36Z (safety)

It had never deleted a single snapshot. The script converted timestamps with date -d, but the container image is Alpine, where date is BusyBox, and BusyBox's date can't parse an RFC 3339 timestamp. Each parse failed and fell back to 0. My "if the timestamp is unparseable, skip it to be safe" check then skipped every candidate. The safety check had quietly switched off the whole job.

By the time I found it there were 2,202 snapshots taking up 350 GiB. My Prometheus volume on its own had 100 GiB provisioned and 206 GiB actually on disk.

The fix was to parse the timestamp in jq, which was already reading the JSON anyway:

kubectl get snapshots.longhorn.io -n longhorn-system -o json \
  | jq -r '.items[]
      | [ .metadata.name,
          (.status.creationTime | try fromdateiso8601 catch 0) ]
      | @tsv'

I also capped how many snapshots it deletes per night. Deleting a snapshot makes Longhorn merge data on disk, and doing 2,000 at once would hammer every node. The backlog drained over about a week and a half.

Then I found the same date -d pattern in another cleanup job, the one that removes backups for volumes that no longer exist. There, the failed parse made every backup look ancient, so the 48-hour grace period never applied. That one failed in the dangerous direction, because it deletes from B2. Same fix.

Lessons

  • A green CronJob doesn't mean it worked. Have it log what it actually did, and read the logs now and then.
  • Know which shell tools your image has. BusyBox is not GNU coreutils.
  • Safety checks need an alarm. If one skips everything, that should be loud, not silent.
  • Exclude on purpose. Use an empty group to keep throwaway volumes out of off-site backups.
  • A backup you haven't restored is a guess. That's on my list next.

Next up: the Minecraft servers, which are now backed up every night along with everything else.

Get new posts delivered to your inbox