Article on the migration of my backup archives to Borg

articles/2019-11-21-borg-migration.md

@@ -0,0 +1,111 @@
+---
+title: Backup archives migration to Borg
+date: 2019-11-21 18:30:00
+---
+
+Last weekend I found a number of encrypted hard-drives that were used to do periodic
+backups from 2006 to 2014. At the time, the backups were made using rsync
+with hard-links to save only one occurrence of the file if it was not changed
+since the last backup.
+
+I wanted to check that everything was still there and upgrade this to a
+Borg repository so that I can profit from compression and deduplication
+to reduce these backup size further down and store them in a more secure way.
+
+## Check the backups
+The backups were made using hard-links with one backup corresponding to
+one folder as follow :
+
+```
+$ ls backups/feronia/
+back-2014-06-19T19:05:10/  back-2014-10-10T07:30:00/
+back-2014-12-24T14:34:44/  current@
+```
+
+To check that the backups were still readable, I listed the content of
+the different folders and checked that some known configuration files were
+present and matched what was expected. This worked until I processed some
+backups done before I was using awesomewm, when I changed a lot of config
+files to match my usage instead of using the default ones.
+
+All in all, the backups were still good and readable, I could use these
+as a basis for the transition to a more robust and space-efficient system.
+
+I saw a number of freezes during the check I interpreted as signs of old
+age for the spinning rust.
+
+## Initialize the Borg backup
+The first step is to initialize the borg repository. We will put it on
+one of the known good backup drive that still has some room. To estimate
+the space needed for the backups, I took the size of the most recent backup
+and multiplied by two, as I know that I did not delete a lot of files and
+that the deduplication will reduce the size of the old backups that contained
+a lot of checked-out subversion repositories.
+
+So, with a destination for my borg repository, I created a folder on the
+disk and gave my user read-write rights on this folder.
+
+```
+$ sudo mkdir backups/borg-feronia
+$ sudo chown 1000:1000 backups/borg-feronia -R
+```
+
+Then, the creation of the repository with borg :
+
+```
+$ borg init --encryption=repokey backups/borg-feronia
+Enter new passphrase: 
+Enter same passphrase again: 
+Do you want your passphrase to be displayed for verification? [yN]: n
+[...]
+```
+
+I decided to use the `repokey` encryption mode. This mode stores the key
+in the repository, allowing me to only remember the passphrase and not having
+to worry about backuping the key file.
+
+## Transfer the existing backups to Borg
+The borg repository has been initialized, we can now start migrating the
+backups from the hard-linked folders into borg.
+
+As borg does not care about hard-links, we can simply loop over the different
+folders and create a new archive from it. It will take some time because
+in each directory it will loop over the whole content, hash it, check whether
+it changed, deduplicate it, compress it and then write it. Each backup
+of approximately 70 GiB took one hour to migrate on my computer. It seems
+that the process is limited by the single-thread performance of your CPU.
+
+```
+$ export BORG_PASSPHRASE=asdf
+$ for i in back*; do \
+    archivename=$(echo $i | cut -c 6-15); \
+    pushd $i; \
+    borg create --stats --progress ~/backups/borg-feronia::$archivename .; \
+    popd; \
+    done;
+```
+
+The env variable will allow us to walk away at this stage and let the computer
+do its magic for some hours.
+
+## Check the migrated backups
+Once the backups have been migrated, we need to check that everything is
+in order before doing anything else.
+
+I did the same as before, using this time `borg list` and `borg extract`
+to check whether the files are present and their content is correct.
+
+
+## Archive these backups
+Once the migrated backups have been tested, we can shred the old hard drives
+that were showing signs of old age.
+
+Since storage is so cheap nowadays, I will also transfer an archive of
+the Borg backup folder to an online storage service as to be able
+to retrieve it in case the local storage supports are destroyed or otherwise
+unreadable in the future.
+
+I choose to simply create a tar archive of the Borg folder and upload it
+to AWS S3 since these backups will not be updated. Perhaps some day I will
+add the more recent backups to this setup but for now they are a read-only
+window into the laptop I had during my studies and during my first jobs.