about

  • one man’s backup system
  • extreme basics of btrfs
  • a tale of optimism and naivety

caveats

memes/no_idea_dog.jpg

Warning btrfs is kinda experimental.
Warning don’t blindly trust ‘some guy from mlug’ with backup advice.

previous procedure

  • 2 portable HDDs
  • rdiff-backup
  • ad-hoc sync, ad-hoc rotation
  • surprisingly effective (?)

catalyst

  • backup HDD did not actually contain backup data

memes/long_neck.png

many issues

  • no sync in (at least) 2 years
  • dire warning against rdiff-backup in portage 
# Patrick Lauer <patrick@gentoo.org> (09 Apr 2014)
# Dead upstream, has known dataloss bugs.
# Please use something more sane: rsnapshot, backuppc, obnam, ...

current needs

systems
  • 2-5 desktops+laptops
  • 1 VPS
data
  • 30G development
  • 450G+ photos
  • 150G misc
ignore
  • 1-2TB replaceable media + games

requirements

  • automated everything, or it won’t happen
  • Gentoo and Ubuntu support
  • disconnected operation on remote systems via cron
  • local rollback
  • low network and CPU overhead for VPS

choices

  • rsnapshot
  • obnam
  • bacula
  • roll my own custom, error-prone, tool
    • also, use an experimental filesystem

btrfs features

  • device pooling
  • copy-on-write
  • scrubbing
  • online fsck
  • subvolumes
  • snapshots
  • transparent compression
  • checksums
  • seeding
  • send/receive
  • out-of-band dedup
  • RAID 1, 0, 10, 5, 6

btrfs features

memes/kitchen_sink.jpg

Note Google btrfs + ‘rampant layering violation’

useful btrfs features

  • subvolume
  • snapshot
  • send/receive

» subvolume

  • ‘namespaced’ child filesystem
    • a fancy directory with low-level knobs
  • directly mountable
    • not a block device (unlike LVM) 
btrfs sub create /home
mount -t btrfs -o subvol=home $DEV

» snapshot

  • sub-volume with shared data blocks
  • possibly read-only 
btrfs sub snap -r /home /backup/home
Important snapshots only work on subvolumes

» send/receive

  • print/read a stream of snapshot data
  • incremental diff from common parent(s) 
ssh $SERVER btrfs send -p $PARENT $SRC | btrfs receive $DST
Important snapshots must be read-only

approach

  • split the system by functionality:
    • creation of local snapshots
    • transfer of snapshots
  • periodically:
    • collect remote snapshots in a local cache
    • store local cache to external HDD

» storage

  • local cache of snapshots
    • by disk UUID
      • by subvolume path
        • by timestamp 
$PREFIX/backup/$UUID/path/to/subvol/_/$TIMESTAMP

setting up root fs

mkfs.btrfs /dev/sdb
mount /dev/sdb /newroot
btrfs sub create /newroot/{data,backup}
btrfs sub list # find /newroot/data id
btrfs sub set-default $ID /newroot

alternative setup

In place conversion of ext filesystem
btrfs-convert /dev/sda1

memes/itsatrap.gif

mountpoints

  • boot directly into the data volume
  • mount the real root volume somewhere accessible
  • allows us to hide the backup subvolume
/etc/fstab
/dev/vda  /               btrfs  subvol=data
/dev/vda  /var/lib/btrfs  btrfs  subvolid=0

snapshot generation

/usr/bin/python3 mason.py snapshot
  • for each local btrfs disk
    1. list all subvolumes: btrfs sub list -pcgouqR /
    2. ignore if basename is: tmp, portage, distfiles
    3. snapshot (readonly) into $FSROOT/backup/$UUID/$path/_/$TIMESTAMP
Note snapshots aren’t recursive. use this to help backup policy.

snapshot collection

/usr/bin/python3 mason.py pull $UUID $SERVER
local
  1. enumerate cached disks, volumes
  2. send local snapshot UUIDS and latest timestamp
remote
  1. find list of new snapshots from timestamp
  2. send path list
  3. send each subvolume data
local
  1. receive paths
  2. receive each subvolume

remote execution

  • only root can access btrfs-snapshot ioctls
  • run the backup script via sudo as a shell
/etc/sudoers.d/backup-send
backup ALL=(ALL) NOPASSWD: /home/backup/mason.py
/home/backup/.ssh/authorized_keys
command="/usr/bin/sudo /home/backup/mason.py $SSH_ORIGINAL_COMMAND"
ssh-rsa AAAAAAAA== danny@nerdcruft.net
Important the above isn’t optimal for security, but it is convenient…

usage

snapshot

mason.py snapshot

copy

mason.py pull e65c83cc-47e8-4937-a432-a36d8ac0b8b9 ssh://chronos

evaluation

  • btrfs feels to have higher latency, seeks, compared to ext4
  • send/receive network efficiency is ‘good enough’
    • not on par with rsync, but lower CPU usage
  • very effective automation, low disk usage
    • hourly local backups is entirely justifiable
  • writing your own backup scripts is error prone

todo

  • error handling… receive isn’t terrifically verbose
  • auto-remove unneeded snapshots
  • proper configs
  • UUID to human-readable names
  • more automation