Backups and syncs¶

This mostly has to do with hydractl commands such as sync-{backups,media,tpc}.

Servers¶

Backups can be automatically generated and sent to other servers, including NAS instances.
Automated backup checking can be done.
But all this does not exempt a team from manually testing backups and to make their own offline copies in external drives.

Parameters¶

Start by running the following commands in your terminal, adjusting to your case:

export HYDRA="hydra-name"
export DISK="disk-name"
export DEVICE="/dev/sdc"
export USER="`whoami`"
export DOMAIN="`facter domain`"

if [ "$USER" != "root" ]; then
  export SUDO="sudo"
fi

New external drive¶

To add a new external drive into the pool, proceed as follows.

First make sure to create a passphrase for the new disk into your hydra's keyring:

keyringer $HYDRA pwgen disks/$DISK.$DOMAIN/luks/root

If you're using a USB drive, run this before plugging it on the computer:

hydractl usb-enable

With the disk plugged in, make a layout with single ${DEVICE}1 partition (example: a disk with only one partition like /dev/sdc1):

$SUDO cfdisk ${DEVICE}

Now create the LUKS encryption header in this new disk, using the passphrase generated above by keyringer:

$SUDO cryptsetup luksFormat ${DEVICE}1

If you want to make this volume to have a label (helpful for detecting the disk etc; downside: third parties can easily get the name), run this command:

$SUDO cryptsetup config ${DEVICE}1 --label ${DISK}

Open the encrypted volume and create the basic filesystem structure needed to sync content:

$SUDO cryptsetup luksOpen ${DEVICE}1 $DISK
$SUDO mkfs.ext4 /dev/mapper/$DISK
$SUDO mkdir /media/$DISK
$SUDO mount /dev/mapper/$DISK /media/$DISK
$SUDO mkdir /media/$DISK/media.$DOMAIN
$SUDO chown ${USER}: /media/$DISK/media.$DOMAIN

Finally, umount the drive:

hydractl umount-media $DISK

Regular sync¶

These commands should be enough to sync all media archives:

hydractl usb-enable  # run this for USB drives, then connect the disk
hydractl mount-media $DISK
hydractl sync-media  $DISK

As this should handle syncing all backups:

hydractl sync-backups $DISK

TPC¶

Chek the specific documentation for more information about TPCs.

When regularly syncing a TPC, use a procedure that guarantees minimal changes in the running TPC, to make sure syncing data won't create inconsistencies in the destination, by doing something like:

Reboot the machine
Turn off networking.
Log in through console.

Then do the following:

hydractl usb-enable  # run this for USB drives, then connect the disk
hydractl mount-media $TPC
hydractl sync-tpc    $TPC

If you don't want to do a full TPC sync, but just want to sync the home folders (which is faster), use the following instead of the last command above:

hydractl sync-home $TPC

To sync archives and remote backups, proceed with as explained in the NAS section.

You may also want to consider using borger (mirror) to have encrypted homedir backups in the external archive/backup volume.

Appliances¶

You might want to backup the whole SSD, M-SATA or microSD from your appliances. If so, proceed as follows with the appliance device connected in your TPC:

export APPLIANCE="appliance-name"
export DEST="/var/backups/remote/${APPLIANCE}.${DOMAIN}/image/`date +%Y%m%d`"
$SUDO mkdir -p $dest
dcfldd if=/dev/sdb | bzip2 > $DEST/${APPLIANCE}.img.bz2

Smartphone¶

Smartphones usually have their own way to be backed up. This is an example based on the android-backup utility:

android-backup <device-name>

Hardware rotation¶

It's recommended to rotate the current hardware in use in all places:

Backup disks.
Laptops, so they're not kept out of use (and/or especially the batteries).

Backup Kit¶

A Backup Kit is a box with the following items:

External encrypted archive/backup disk.
Case for SSD transportation.
Laptop power adapter and cables.
Dockstation SATA/USB/M.2/microSD/etc (with power adapter -- usually 12V).
Power adapter for external hard drives (usually 12V, and might be compatible with the dockstationadapter, so you might carry just one).
USB power adapter and cable (including USB 2, USB 3 and USB C).
USB cables (USB 2, USB 3 and USB C) for the Dockstation and the external drive.
TPC laptop with battery and TPS (SSD, M.2 etc), optionally with a UltraBase/Dockstation.
Philips screwdriver and other tools.
FCR-MG2 adaptor for microSD to USB.
Anything else your need (like eyeglasses).
Everything in a sealed box.

This may be the ultimate disaster recovery kit for your Hydra!

Restore¶

Having backup data leaked is a serious security issue, and that's why we encrypt backups. But losing access to the encrypted material is data loss, so it's important to make sure in advance that we can get back the material.

Procedures vary, and the following examples are sorted according to the software used to make the backup.

Duplicity¶

For duplicity:

duplicity collection-status file:///var/backups/duplicity
duplicity restore --file-to-restore home/$USER --time 2018-03-25 file:///var/backups/duplicity/ /home/$USER

Borg¶

For Borg:

export PATH_TO_RESTORE="path/to/be/restored"
mkdir ~/temp/misc/restore
cd    ~/temp/misc/restore
borg list    ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg
borg extract ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg::$USER-2018-06-11T17:07:39 mnt/crypt/home/$USER/$PATH_TO_RESTORE

Make sure to cleanup ~/temp/misc/restore after recovering what you need.

Note on backup keys:

In the past (before 2024), the Hydra Suite and it's companion Puppet modules used pre-generated Borg repository keys for the sake of automation. This is not possible anymore.
As it's important to keep copies of the borg repository key safely elsewhere, the managed configuration supports OpenPGP-encrypting the repository key and uploading it to the remote repository.
This OpenPGP-encrypted key file is named as keyfile.asc and is uploaded in the root folder of the remote repository.
This OpenPGP-encrypted key file is encrypted and signed with a provided OpenPGP keypair and passphrase (convention is to use the machines's OpenPGP general purpose key, or the machine's role key).
This allows the operators to fetch this encrypted keyfile and use their copy of the machine's OpenPGP key to extract the passphrase on their encrypted-storage workstations (recommendation is to not do this on the remote repository).

Just to be sure, let's emphasize Borg's own recommendation:

IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!

If you used a repokey mode, the key is stored in the repo, but you should back it up separately. Use "borg key export" to export the key, optionally in printable format. Write down the passphrase. Store both at safe place(s).

eCryptfs¶

For eCryptfs:

$SUDO ecryptfs-recover-private /media/$DISK/home/.ecryptfs/$USER/.Private