btrfs backups in one afternoon
=============================
:author: Danny Robson
:copyright: nerdcruft
:backend: slidy
:maxwidth: 45em
:icons:

== about
* one man's backup system
* extreme basics of btrfs
* a tale of optimism and naivety


== caveats
image:memes/no_idea_dog.jpg[]

WARNING: btrfs is kinda experimental.

WARNING: don't blindly trust `some guy from mlug' with backup advice.


== previous procedure
* 2 portable HDDs
* rdiff-backup
* ad-hoc sync, ad-hoc rotation
* surprisingly effective (?)

== catalyst
* backup HDD did not actually contain backup data

image:memes/long_neck.png[height=400]


== many issues
* no sync in (at least) 2 years
* dire warning against rdiff-backup in portage
----
# Patrick Lauer <patrick@gentoo.org> (09 Apr 2014)
# Dead upstream, has known dataloss bugs.
# Please use something more sane: rsnapshot, backuppc, obnam, ...
----


== current needs
systems::
* 2-5 desktops+laptops
* 1 VPS
data::
* 30G development
* 450G+ photos
* 150G misc
ignore::
* 1-2TB replaceable media + games


== requirements
* automated everything, or it won't happen
* Gentoo and Ubuntu support
* disconnected operation on remote systems via cron
* local rollback
* low network and CPU overhead for VPS


== choices
* rsnapshot
* obnam
* bacula
* roll my own custom, error-prone, tool
** also, use an experimental filesystem


== btrfs features
* device pooling
* copy-on-write
* scrubbing
* online fsck
* subvolumes
* snapshots
* transparent compression
* checksums
* seeding
* send/receive
* out-of-band dedup
* RAID 1, 0, 10, 5, 6

== btrfs features
image:memes/kitchen_sink.jpg[]

NOTE: Google btrfs + `rampant layering violation'


== useful btrfs features
* subvolume
* snapshot
* send/receive

=== subvolume
* `namespaced' child filesystem
** a fancy directory with low-level knobs
* directly mountable
** not a block device (unlike LVM)

[source,bash]
----
btrfs sub create /home
mount -t btrfs -o subvol=home $DEV
----

=== snapshot
* sub-volume with shared data blocks
    * possibly read-only

----
btrfs sub snap -r /home /backup/home
----

IMPORTANT: snapshots only work on subvolumes


=== send/receive
* print/read a stream of snapshot data
* incremental diff from common parent(s)

[source,bash]
----
ssh $SERVER btrfs send -p $PARENT $SRC | btrfs receive $DST
----

IMPORTANT: snapshots must be read-only

== approach
* split the system by functionality:
** creation of local snapshots
** transfer of snapshots

* periodically:
** collect remote snapshots in a local cache
** store local cache to external HDD

=== storage
* local cache of snapshots
** by disk UUID
*** by subvolume path
**** by timestamp

[source,bash]
----
$PREFIX/backup/$UUID/path/to/subvol/_/$TIMESTAMP
----

== setting up root fs

[source,bash]
----
mkfs.btrfs /dev/sdb
mount /dev/sdb /newroot
btrfs sub create /newroot/{data,backup}
btrfs sub list # find /newroot/data id
btrfs sub set-default $ID /newroot
----


== alternative setup

.In place conversion of ext filesystem
----
btrfs-convert /dev/sda1
----

image:memes/itsatrap.gif[]


== mountpoints

* boot directly into the data volume
* mount the real root volume somewhere accessible
* allows us to hide the backup subvolume

./etc/fstab
----
/dev/vda  /               btrfs  subvol=data
/dev/vda  /var/lib/btrfs  btrfs  subvolid=0
----


== snapshot generation

----
/usr/bin/python3 mason.py snapshot
----

* for each local btrfs disk
    . list all subvolumes: `btrfs sub list -pcgouqR /`
    . ignore if basename is: tmp, portage, distfiles
    . snapshot (readonly) into `$FSROOT/backup/$UUID/$path/_/$TIMESTAMP`

NOTE: snapshots aren't recursive. use this to help backup policy.


== snapshot collection

----
/usr/bin/python3 mason.py pull $UUID $SERVER
----

local::
    . enumerate cached disks, volumes
    . send local snapshot UUIDS and latest timestamp
remote::
    . find list of new snapshots from timestamp
    . send path list
    . send each subvolume data
local::
    . receive paths
    . receive each subvolume


== remote execution

* only root can access btrfs-snapshot ioctls
* run the backup script via sudo as a shell

./etc/sudoers.d/backup-send
----
backup ALL=(ALL) NOPASSWD: /home/backup/mason.py
----

./home/backup/.ssh/authorized_keys
----
command="/usr/bin/sudo /home/backup/mason.py $SSH_ORIGINAL_COMMAND"
ssh-rsa AAAAAAAA== danny@nerdcruft.net
----

IMPORTANT: the above isn't optimal for security, but it _is_ convenient...

== usage

snapshot:: `mason.py snapshot`
copy:: `mason.py pull e65c83cc-47e8-4937-a432-a36d8ac0b8b9 ssh://chronos`

== evaluation

* btrfs feels to have higher latency, seeks, compared to ext4
* send/receive network efficiency is `good enough'
** not on par with rsync, but lower CPU usage
* very effective automation, low disk usage
** hourly local backups is entirely justifiable
* writing your own backup scripts is error prone

== todo
* error handling... receive isn't terrifically verbose
* auto-remove unneeded snapshots
* proper configs
* UUID to human-readable names
* more automation
