Reliable and efficient Nextcloud backups with deduplication, ransomware protection and zero downtimes.
In the previous post, I presented my criteria for good backups and outlined some implementation approaches. In this post I'll provide a step-by-step description for setting up a backup process for a Nextcloud Server that will fulfill all those criteria.
Feel free to chose and pick and adjust my approach to your own system.
You can also skip directly to the final code if you're fluent in bash and don't care for the reasoning behind the individual steps.
A typical Nextcloud backup will likely look like this:
Enabling maintenance mode serves to ensure, that all the things we are including in our backup are in sync and thus represent a consistent state of our service at a specific point in time. Otherwise, we might run into issues when restoring the backup - for instance files being present in the filesystem but not known to the database (and thus not visible to Nextcloud).
Unfortunately, we have ruled out using maintenance mode, because we don't want any downtime during our backup process, so we need another solution.
To accomplish consistent zero downtime backups, we will create synchronized
Popular modern filesystem types like BTRFS, ZFS, XFS, LVM and many VM storage solutions provide snapshot capabilities. Filesystem snapshots are read-only copies of a directory or volume that are usually created nearly instantly and take little to no additional storage space. This guide will give examples for BTRFS and LVM (more might be added in the future).
Most modern transactional database should provide options to do single-transaction copies of the database. I will give examples on how to achieve this for mariadb/mysql and postgresql.
Is any of this applicable to containerized deployments (e.g. Nextcloud All-in-One which is based on Docker)?
Yes, you just need to make sure that the volumes which contain your Nextcloud installation directory and your user files are backed by compatible storage - e.g. mounting a btrfs subvolume under the respective paths.
You will also have to prepend some of the commands in this guide with whatever is necessary to run a command in you
container. For example for docker, you would need: docker exec <name-of-your-container> <original-command>.
Other commands, especially filesystem operations, always need to be executed on the host system.
Let's walk us through the process of creating filesystem snapshots for our user files directory and Nextcloud installation directory.
This requires the nextcloud installation directory (assumed to be at /var/www/nextcloud) or respectively the user
files directory (assumed to be at /mnt/data) to be a BTRFS subvolume (created with btrfs subvolume create <path>).
# Path to the btrfs subvolume containing your NC files directory
USER_FILES_VOLUME=/mnt/data/nc_files
# Path where the (temporary) btrfs snapshot for the NC files directory should be created
USER_FILES_SNAPSHOT_PATH="/mnt/data/.nc_files_snapshot"
# Path to the btrfs subvolume containing your NC installation directory
NC_INSTALL_VOLUME=/var/www/nextcloud
# Path where the (temporary) btrfs snapshot for the NC installation directory should be created
NC_INSTALL_SNAPSHOT_PATH="/var/www/.nc_snapshot"
This requires the nextcloud installation directory or respectively the user files directory to be a LVM logical volume.
# Adjust to match the LVM volume group containing your nc installation directory
USER_FILES_VG="lvm-1"
# Adjust to match the logical volume name containing your nc installation directory
USER_FILES_LV="nc-files"
# Arbitrary name for the (temporary) lvm snapshot
USER_FILES_SNAPSHOT_NAME="nc-backup-files"
# Adjust to match the LVM volume group containing your nc installation directory
NC_INSTALL_VG="lvm-1"
# Adjust to match the logical volume name containing your nc installation directory
NC_INSTALL_VOLUME="nextcloud"
# Arbitrary name for the (temporary) lvm snapshot
NC_INSTALL_SNAPSHOT_NAME="nc-backup-nextcloud"
For creating our database snapshot, we will keep things simple and use the trusty sql dump utilities available for our reference databases mariadb and postgres. An sql dump is basically a set of SQL instructions that can be used to recreate the database in its current state.
Following you will find code snippets for creating those dumps:
SQL_DUMP_TARGET_DIR=/tmp/nc_backup/
DB_USER="root"
DB_NAME="nextcloud"
DB_PASSWORD="password"
DB_HOST="localhost"
DB_PORT=5432
SQL_DUMP_TARGET_DIR=/tmp/nc_backup/
DB_USER="root"
DB_NAME="nextcloud"
DB_PASSWORD="password"
DB_HOST="localhost"
DB_PORT=5432
Note: We need the parameter --single-transaction to ensure a "real" snapshot (a single transaction copy to be
precise) that is unaffected by ongoing writes to the DB. We add --skip-lock-tables as well, to improve performance
(not locking tables is okay here, because we're just reading from the DB).
Now that we can create both, filesystem snapshots and an sql dump, we are capable of creating a consistent point-in-time snapshot of our Nextcloud instance with zero downtime (well, actually 3 synchronized snapshots, but we'll get to that). Just make sure, to either trigger all three operations at the same time or perform the filesystem snapshots first (since they are super fast, that's basically the same thing). It's important that all 3 snapshots refer to the same point in time.
Right now, we have 3 individual snapshots lying around. They might even be located on separate disks. For the next steps we want one aggregated view on them, which we will achieve using bind mounts for btrfs (or other filesystems) and normal mounts for LVM. Bind mounts are a Linux feature which allows us to mount one directory to another path in our filesystem.
So, using the same variables as above, the code for this would look like follows:
The result is a directory (/tmp/nc_backup) which looks like it contains all filesystem snapshots and the sql dump:
/tmp/nc_backup
├── nextcloud # nextcloud installation directory
├── nc_files # user files directory
└── db.sql # sql dump of the database
As we have established, we need our backups to not be affected by a number of local hazards, including hardware failure or fires. Therefore, we need to store them somewhere else. Because we also want ransomware protection, we need a storage type with one of two options:
Either
These features (which I will explain in more detail later) will allow us to prevent attackers (e.g. ransomware) from destroying our backups.
In this guide, we will focus on object locking to ensure ransomware protection. If you want to use restricted access permissions, please refer to the Kopia documentation.
Because we need those features, we can't go with many of the storage options offered by Kopia and are left with something called "object storage" - basically a special kind of storage often found in cloud computing. Apart from big cloud providers, this kind of storage is so ubiquitous today, that you will find it offered by numerous providers and you can even self-host it. Because we will encrypt the backups you don't need to have a high level of trust in your provider, though (apart from trust in their capability of not losing your data, of course). Here are some options:
*Note: All options marked with
Setting up one of the self-hosting options is out of scope for this guide, however, I will provide instructions for AWS S3, Google Cloud Storage and Min-IO going forward (maybe I'll add Ceph at a later point).
For reference though: My personal storage server is formed by a RaspberryPi 5 8GB running Min-IO and coupled with a
So now, that you have chosen (or become!) a storage provider, you need to create a bucket (which is comparable to the concept of directories in traditional file systems, although they can't be nested). If you need object locking than you need to make sure it is supported by your provider and enabled for your bucket.
Now you need to create credentials for your bucket. Depending on your storage provider that can look differently.
For S3 compatible providers you need to create user and a policy and potentially an access key.
Assuming your bucket is named kopia, here is what a policy could look like:
If you want to use Google Cloud Storage, I recommend setting up a service account that you will then grant access to your google storage bucket. Finally, you will need to create a service account key for your service account that you will later use to connect the storage.
For this guide, I won't go into more detail on those steps. Please follow the linked official documentation instead.
By now, we have a snapshot of our instance and we have some remote storage for our backups. Now let's address getting our backup there.
To turn our instance snapshot into a backup that ticks all the boxes from our backup criteria, we will need a specialized backup tool. There are a number of options available, notably Restic, Borg and Kopia, which should all cover most of our needs. Because I have the most experience with it and use it for my own backups, we're going with Kopia in this guide (feel free to supply information on how to do the same with another tool and I'll consider adding it).
Why do we need a specialized tool? While we could implement some of its features ourselves, this would be more error-prone and very complex to achieve. Some features, like error correction and ransomware protection are especially infeasible to solve by hand.
For which features do we use Kopia? Notable features of Kopia that we rely on are:
Before proceeding, please make sure you have Kopia installed on your server as per the installation instructions.
Now, that we have our backup tool ready, we're going take our instance snapshot and write it to our object storage.
Let's first setup a
Use the following commands to initialize a Kopia repository. Adjust the variables
ENDPOINT,BUCKET_NAME, ACCESS_KEY, SECRET_ACCESS_KEY, PREFIX according to your S3 storage.
For all available paremeters, refer to the kopia documentation.
Use the following commands to initialize a Kopia repository. Adjust the variables
BUCKET_NAME, SERVICE_ACCOUNT_CREDENTIALS_FILES, PREFIX according to your setup.
For all available paremeters, refer to the kopia documentation.
Now we need to actually enable the use of object locks in Kopia:
And adjust our Kopia global policy (we could also do this per repository, but since we only have one, this is fine). Here we setup how many backups to retain and the type of compression to use. You can find the full list of parameters in the Kopia documentation.
All set! We're finally ready to actually transfer the snapshot to the backup storage:
Kopia does a lot of things for us. Apart from the things that we have configured explicitly (like retention, encryption, compression) we get very efficient deduplication, meaning we only need space for data that is actually new. That's also, why we don't compress our database dump - because that way we can make use of the more efficient compression and deduplication Kopia provides.
One particularly nice thing about Kopia backups is, that the progress of a partially transferred backup that got aborted due to e.g. network issues will be picked up during the next try, because Kopia still stores all the data blocks that were transferred and never transfers the same data block again (because of deduplication). In other words: If we need 10 attempts to create our initial backup, because we have a large amount of data, we will still make progress during every attempt and end up with a complete backup at the end.
Lastly, we can cleanup the snapshots, we don't need it anymore. Here's a script snippet to do that depending on the filesystem used:
Now that we know how to create a backup, let's automate the process and run it on a schedule.
For your convenience, here is a script, that you can drop in /etc/cron.hourly/ or use in a systemd service unit with
corresponding systemd timer.
If you haven't done it already, execute the following snippet (as root) to configure your Kopia Repository. Select the type of remote repository that matches your setup.
Following, configure the options to match your setup and then copy or download the backup script. Afterward, adjust the configuration section at the top of the script (mostly paths to your Nextcloud setup) and run it as root. The backup script will use your default Kopia repository (which you have initialized in the previous step).
Note: Make sure to save the script with permissions 0700 or 0500 to avoid other users from reading the credentials inside!
(Or move them outside the script, e.g. inside systemd credentials).
Now set up the script as cron job. If you want hourly backups (that's how I'm running it), you need to do nothing more than
drop it into /etc/cron.hourly/.
If your want a custom schedule, save it in a separate location instead (e.g. /usr/sbin/) and create an entry in /etc/crontab, e.g.:
which will run the script every 4 hours at every 2nd day at the half hour mark.
Make sure to adjust the script permissions so it is executable and not readable by other users:
Now, let's have a look at restoring a backup:
In order to restore a snapshot, you need a Nextcloud server, that is working fine (apart from missing your data). So either set one up or reuse your existing server, if you're confident that only the backed up data (i.e. Nextcloud installation directory, user files directory or database) was damaged. This server needs to match your previous one (at the time of the backup) as closely as possible (e.g. the database and PHP version should be the same or at least compatible).
The first thing to do is, to have a look at the list of snapshots and restore one of them. Alternatively to using the given commands, we can also use the Kopia GUI.
Now we need to restore the database from the restored /run/nc-restore-db.sql:
DB_USER="root"
DB_NAME="nextcloud"
DB_PASSWORD="password"
DB_HOST="localhost"
DB_PORT=5432
DB_USER="root"
DB_NAME="nextcloud"
DB_PASSWORD="password"
DB_HOST="localhost"
DB_PORT=5432
That's it! If things in your setup changed, you might still have to adjust /var/www/nextcloud/config/config.php (
e.g. db credentials, redis adress ... ). But other than that, your backup should be restored successfully at this point.
As becomes clear, producing great backups can be a bit involved. But in my opinion, they are more than worth it, because once set, they provide a quality of sleep that's otherwise hard to achieve as sysadmin. :D Jokes aside, having robust backups means a significant reduction in the "worst case" that we could endure with our service (in regard to data loss) and are therefore crucial to have.
Let's check how well we fared in terms of our criteria from part 1 of this blog series:
As you can see - nearly all boxes are ticked! The remaining box will be addressed in the upcoming 3rd part of this series where I talk about backup monitoring. So stay tuned and consider following me on mastodon or subscribing your feed reader to the RSS feed to make sure, you won't miss it. :)