I have a server that I own and host in a colocation data center that I host client projects on. This article will discuss how I have backups set up.
To make managing LAMP stack web apps easier (Wordpress, Classicpress, etc) easier, I use the open source Hestia Control Panel, which facilitates rolling out sites and automating things like Apache+Nginx virtual hosts etc.
It has a built in backup tool. However, the limitation of the backup tool is that it doesn't do incremental backups – it creates a tarball with everything.
Accordingly, I have created a custom workflow to optimize this.
The current storage structure of the server looks like the following:
- Two 2tb nvme drives in a raid 1 using Linux mdadm (I will likely add two 4tb drives in a ZFS mirror to store most data separate from the Linux boot drive. particularly web stuff as the client base grows)
- Four 12tb spinning HDDs in a ZFS raidZ2 ie. 24TB usable storage.
The nvme storage is used for anything performance sensitive such as the core files of web apps and the operating system. The HDDs are used for less performance sensitive data and large files, and projects other than web hosting that require hosting large files.
For remote backup storage, I primarily use rsync.net – I use rsync to upload the files to the remote server there, and I have ZFS snapshots set to be made daily with 32 days of retention. rsync.net is able to make the snapshots immutable, so, even if the server gets hit by ransomware etc its hard for a bad actor to wipe all of the snapshots (the server only can connect to the repo on rsync.net via an SSH key, it doesn't know the actual password, and the master account with the ability to change snapshot configuration is twofactored, the shell account can't mess up the ZFS snapshots). I also use their functionality to set up a georedundant backup of the backup.
For most data other than the main website data in the public_html folder, such as the database, the tarballs are fine. However, the public_html files are slow changing, but relatively large.
This is my shell script to run everything
https://gist.github.com/theopjones/3e989a2637f1506f11280847bb6f7171
I have it set to run every other hour with this cron entry
https://gist.github.com/theopjones/1003e48c7b7ba5c61b407f72fb159da5
I use Hestia's default backup with an exclude (set up separately) for every user set to exclude the public_html. This produces a tarball with everything managed by the Hestia software except the public_html data. This includes the database etc.
I have shut off the default backup cron job, instead using a shell script to trigger it via the command built into Hestia v-backup-users
Hestia's backup tool writes this to /backup, which I have a ZFS volume on the HDD pool mounted to. The backup script rsyncs this up. Hesta is configured to retain 12 snapshots, ie. a whole day's worth.
The script then runs a second script I have created to find the public_html files, and rsync them up with separation to be able to tell which users correspond to which domains
https://gist.github.com/theopjones/c6fc9c14b60e3bce2115bba66c3df080
For another layer of backups, I have a local ZFS volume on the HDDs that I rsync the HTML data to. I then create a snapshot of this volume, and the tarball backup volume with 32 day retention. This is the shell script
https://gist.github.com/theopjones/438c935717944a42c9efd98848c898c3
Restoration of all data from the remote server is easily done is parallel (downloading everything at once can be slow). This is an example command that downloads all of the data locally.
https://gist.github.com/theopjones/56447a1192f1be12ccb9860e61ad6c23
This is a script to fully restore the public_html for one user for all domains for that user. Multiple iterations of this script can be ran at once for each user. It assumes that the tarball backup has been restored using the standard script included with Hestia.
https://gist.github.com/theopjones/c07e9475c48cd94c6f9147487526f84c
For other client data on the server than website data, similar methods are used.