Backup & Restore

The backup and restore is a procedure for disaster-recovery scenario: it can be used to save the data of an installed module and restore it to a different node or cluster.

Backup & Restore

Design

The backup engine is restic which runs inside a container along with rclone used to inspect the backup repository contents.

Backups are saved inside backup repositories, remote network-accessible spaces where data are stored. No local storage backup is possible (eg. USB disks). A backup repository can contain multiple backup instances, each module instance has its own sub-directory to avoid conflicts.

The system implements the common logic for backup inside the agent with module-backup command. Each module can implement module-dump-state and module-cleanup-state to prepare/cleanup the data that has to be included in the backup. The state/environment file is always included inside the backup, as it is required by the cluster/restore-module action. The restore is implemented using a restore-module action inside the module agent, each module can extend it to implement specific restore steps. The basic 10restore step actually runs the Restic restore procedure in a temporary container.

All backups are scheduled by systemd timers. Given a backup with id 1, it is possible to retrieve the time status with:

rootless containers, eg. dokuwiki1, executed by dokuwiki1 user: systemctl --user status backup1.timer
rootfull containers, eg. samba1, executed by root user: systemctl status backup1-samba1.timer

Include and exclude files

Whenever possible, containers should always use volumes to avoid SELinux issues during backup an restore.

Includes can be added to the state-include.conf file saved inside AGENT_INSTALL_DIR/etc/. In the source tree, the file should be placed under <module>/imageroot/etc/state-include.conf. On installed modules, the file will appear on different paths:

rootless containers, eg. dokuwiki1, full path will be /home/dokuwiki1/.config/etc/state-include.conf
rootfull containers, eg. samba1, full path will be /var/lib/nethserver/samba1/etc/state-include.conf

Lines are interpreted as path patterns. Only patterns referring to volumes and the agent state/ directory are considered.

Lines starting with state/ refer to AGENT_STATE_DIR contents. Eg. to include mykey.dump under the AGENT_STATE_DIR add

state/mykey.dump

Lines starting with volumes/ will be mapped to a volume name. Eg. to include the whole dokuwiki-data volume add

volumes/dokuwiki-data

Internally, volumes will be mapped as:

<volume_name> (1-1) for rootless containers; eg. for module dokuwiki1, line prefix volumes/dokuwiki-data maps to volume name dokuwiki-data
<module_id>-<volume_name> for rootfull containers; eg. for module samba1, line prefix volumes/ data maps to volume name samba1-data

Volumes listed in state-include.conf are automatically mounted (and created if necessary) by the basic 10restore step of the restore-module action.

Excludes can be added to state-exclude.conf file saved inside the AGENT_INSTALL_DIR.

For a complete explanation of the patterns, like wildcard characters, see the official Restic documentation to include and exclude files. Note that include and exclude patterns have a slight different syntax.

Save and restore Redis keys

Dump key and state

To save a Redis key, you should:

dump the key inside the module-dump-state command
include the dump inside the backup

Given a module named mymodule, create the file mymodule/imageroot/bin/module-dump-state inside the module source tree:

#!/bin/bash
redis-dump module/mymodule1/mykey > mykey.dump

Make sure also module-dump-state is executable:

chmod a+x mymodule/imageroot/bin/module-dump-state

Then, add the key dump path to mymodule/imageroot/etc/state-include.conf:

state/mykey.dump

Cleanup state

As best practice, the dump should be removed when the backup has completed.

Given a module named mymodule, create the file mymodule/imageroot/bin/module-cleanup-state inside the module source tree:

#!/bin/bash
rm -f mykey.dump

Make sure also module-cleanup-state is executable:

chmod a+x mymodule/imageroot/bin/module-cleanup-state

Restore key

To restore a Redis key, you should add a step inside the restore-module action, after index 10.

Given a module named mymodule, create a file named mymodule/imageroot/actions/restore-module/20loadkey inside the module source tree:

#!/bin/bash
redis-restore mymodule1/mykey < mykey.dump

Execute a backup

Before executing a backup, you must create a backup repository. The flow to execute a backup will be something like:

create a backup repository and retrieve its UUID
retrieve the UUID of the repository
configure a backup using the repository as target
retrieve the backup ID
execute the backup

Create the repository:

api-cli run add-backup-repository --data '{"name":"BackBlaze repo1","url":"b2:backupns8","parameters":{"b2_account_id":"xxxxxxxxxxxxxxxxxxxxxxxxx","b2_account_key":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"},"provider":"backblaze","password":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}'

The output will be something like, please note the id field:

{"password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "id": "48ce000a-79b7-5fe6-8558-177fd70c27b4"}

Create a new daily backup named mybackup with a retention of 3 snapshots (3 days) which includes dokuwiki1 and samba1 instances:

api-cli run add-backup --data '{"repository":"48ce000a-79b7-5fe6-8558-177fd70c27b4","schedule":"daily","retention":3,"instances":["dokuwiki1","samba1"],"enabled":true, "name":"mybackup"}'

The output will the id of the backup:
```
1
```

Run the backup with id 1:

api-cli run run-backup --data '{"id":1}'

For debugging purposes, you can also launch systemd units:

rootless container, eg. dokuwiki1: ssh dokuwiki1@localhost "systemctl --user start backup1.service"
rootfull container, eg. samba1: systemctl start backup1-samba1.service

To remove the backup use:

api-cli run remove-backup --data '{"id":1}'

To remove the backup repository:

api-cli run remove-backup-repository --data '{"id":"c7a9cfea-303c-5104-8ab7-39ac9f9842bd"}'

Execute a restore

Before executing a restore, the backup repository should already be configured.

Restore the dokuwiki1 instance at node 1 from repository 48ce000a-79b7-5fe6-8558-177fd70c27b4:

api-cli run cluster/restore-module --data '{"node":1, "repository":"48ce000a-79b7-5fe6-8558-177fd70c27b4", "path":"dokuwiki/dokuwiki1@3792c7db-9450-4bd3-84a3-034cd0087839","snapshot":""}'

Cluster configuration backup

The cluster/download-cluster-backup API returns a random URL path where an encrypted archive is available for download.

curl -O http://127.0.0.1:9311/backup/$(api-cli run cluster/download-cluster-backup | jq -r .path)

If the previous command is successful a file dump.json.gz.gpg is created in the current directory.

The file with .gpg extension is encrypted with the sha256sum of the admin’s password. To decrypt it run a command like this:

echo -n "${ADMIN_PASSWORD:?}" | sha256sum | awk '{print $1}' | tr -d $'\n'  | \
    gpg --batch -d --passphrase-file /dev/stdin --pinentry-mode loopback -o dump.json.gz dump.json.gz.gpg

The restore procedure can be started from the UI of a new NS8 installation: upload the file and specify the password from the UI.