From 64e7ee67ac4c890a17c763f08409fed2ffb32a44 Mon Sep 17 00:00:00 2001 From: Thomas Schwery Date: Mon, 25 Nov 2019 08:28:49 +0100 Subject: [PATCH] Article on the migration of my backup archives to Borg --- articles/2019-11-21-borg-migration.md | 111 ++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 articles/2019-11-21-borg-migration.md diff --git a/articles/2019-11-21-borg-migration.md b/articles/2019-11-21-borg-migration.md new file mode 100644 index 0000000..18b0946 --- /dev/null +++ b/articles/2019-11-21-borg-migration.md @@ -0,0 +1,111 @@ +--- +title: Backup archives migration to Borg +date: 2019-11-21 18:30:00 +--- + +Last weekend I found a number of encrypted hard-drives that were used to do periodic +backups from 2006 to 2014. At the time, the backups were made using rsync +with hard-links to save only one occurrence of the file if it was not changed +since the last backup. + +I wanted to check that everything was still there and upgrade this to a +Borg repository so that I can profit from compression and deduplication +to reduce these backup size further down and store them in a more secure way. + +## Check the backups +The backups were made using hard-links with one backup corresponding to +one folder as follow : + +``` +$ ls backups/feronia/ +back-2014-06-19T19:05:10/ back-2014-10-10T07:30:00/ +back-2014-12-24T14:34:44/ current@ +``` + +To check that the backups were still readable, I listed the content of +the different folders and checked that some known configuration files were +present and matched what was expected. This worked until I processed some +backups done before I was using awesomewm, when I changed a lot of config +files to match my usage instead of using the default ones. + +All in all, the backups were still good and readable, I could use these +as a basis for the transition to a more robust and space-efficient system. + +I saw a number of freezes during the check I interpreted as signs of old +age for the spinning rust. + +## Initialize the Borg backup +The first step is to initialize the borg repository. We will put it on +one of the known good backup drive that still has some room. To estimate +the space needed for the backups, I took the size of the most recent backup +and multiplied by two, as I know that I did not delete a lot of files and +that the deduplication will reduce the size of the old backups that contained +a lot of checked-out subversion repositories. + +So, with a destination for my borg repository, I created a folder on the +disk and gave my user read-write rights on this folder. + +``` +$ sudo mkdir backups/borg-feronia +$ sudo chown 1000:1000 backups/borg-feronia -R +``` + +Then, the creation of the repository with borg : + +``` +$ borg init --encryption=repokey backups/borg-feronia +Enter new passphrase: +Enter same passphrase again: +Do you want your passphrase to be displayed for verification? [yN]: n +[...] +``` + +I decided to use the `repokey` encryption mode. This mode stores the key +in the repository, allowing me to only remember the passphrase and not having +to worry about backuping the key file. + +## Transfer the existing backups to Borg +The borg repository has been initialized, we can now start migrating the +backups from the hard-linked folders into borg. + +As borg does not care about hard-links, we can simply loop over the different +folders and create a new archive from it. It will take some time because +in each directory it will loop over the whole content, hash it, check whether +it changed, deduplicate it, compress it and then write it. Each backup +of approximately 70 GiB took one hour to migrate on my computer. It seems +that the process is limited by the single-thread performance of your CPU. + +``` +$ export BORG_PASSPHRASE=asdf +$ for i in back*; do \ + archivename=$(echo $i | cut -c 6-15); \ + pushd $i; \ + borg create --stats --progress ~/backups/borg-feronia::$archivename .; \ + popd; \ + done; +``` + +The env variable will allow us to walk away at this stage and let the computer +do its magic for some hours. + +## Check the migrated backups +Once the backups have been migrated, we need to check that everything is +in order before doing anything else. + +I did the same as before, using this time `borg list` and `borg extract` +to check whether the files are present and their content is correct. + + +## Archive these backups +Once the migrated backups have been tested, we can shred the old hard drives +that were showing signs of old age. + +Since storage is so cheap nowadays, I will also transfer an archive of +the Borg backup folder to an online storage service as to be able +to retrieve it in case the local storage supports are destroyed or otherwise +unreadable in the future. + +I choose to simply create a tar archive of the Borg folder and upload it +to AWS S3 since these backups will not be updated. Perhaps some day I will +add the more recent backups to this setup but for now they are a read-only +window into the laptop I had during my studies and during my first jobs.