Move to Hetzner (2024-11-10; in progress)
=========================================

The combined cost of the bulk online storage at Backblaze and for the maintenance of the
managing server has increased beyond the prospected cost of a small Hetzner server with
an attached 5TB storage box. Backblaze currently houses 1.6TB of builds (generating about
3TB of monthly traffic) and 1.5TB of builds are stored locally (with an additional 2.4TB
in cold storage). Files excluding the cold storage fit the current 5TB storage expectation,
and the expected monthly traffic is within the 20TB limit set by Hetzner.

Additionally, my local storage (where builds are kept before being uploaded) has filled
to an alarming level and upgrading that storage immediately is not economically viable
(as it would involve replacing four 2TB disks and possibly one 4TB disk that otherwise
work just fine).

As planned, the new Hetzner server will host both the existing bulk online storage and
the managing software (including the previously offline-only local storage) going forward.
This also provides the potential for additional infrastructure improvements down the line,
especially regarding the "upload" speed and features such as torrent creation.
After a transition period, the existing Backblaze storage will be decomissioned.

A notable immediate downside is that the new Hetzner server will be a single point of failure
instead of being able to rely on Backblazes distributed network. The storage box is resistant
against individual hardware failure, but in the event of a catastrophe data loss may still occur.
Metadata is regularly backed up off-site, but a second bulk storage node is currently not planned.
This may change with increasing funds, and I plan to host a local backing storage node once
my local storage situation has improved again.

[X] Deploy MinIO (min.io) to a Hetzner server with attached storage box
      MinIO provides a self-hosted S3 API that improves integration as a Backblaze replacement.
      It additionally has features for site replication to provide redundancy in case of
      server failure or accesses from different regions.

[X] Change the default upload storage to b4.timschumi.net
      New builds will be uploaded to and accessible from b4.timschumi.net going forward.
      Already uploaded builds are not affected at this point in time.

[ ] Copy existing uploaded files from Backblaze to b4.timschumi.net
      Builds currently present on Backblaze (about 1.6TB of data) will be bulk-copied
      to b4.timschumi.net. Checksums will be tested on upload if MinIO allows for it.

[ ] Rewrite stored download links to point to b4.timschumi.net
      This is one of the two user-visible changes in the process.
      Remaining download links that reference b2.timschumi.net will now reference
      b4.timschumi.net instead.

[ ] Decommission Backblaze-hosted storage
      This is the second of the two user-visible changes in the process.
      The existing storage bucket at Backblaze will be emptied, and existing links to
      b2.timschumi.net will cease to work.
      This will only affect leftover arbitrary and archived links, as all links on
      the live site have been rewritten in the previous step.

[ ] Copy locally stored builds to the remote storage box
      Builds currently present in local storage (about 1.5TB of data) will
      be bulk-copied to the storage box, starting with the inactive branches and
      ending with the active branches.

[ ] Pause automatic downloading
      Downloading new builds will constantly replace already copied builds with new
      ones. Delay new downloads until after the process is finished.

[ ] Sync new builds to the storage box
      Catch up with file changes since the beginning of the copy operation.

[ ] Deploy lineage-archive to the Hetzner server
      This involves shutting down the old instance, copying over the metadata,
      changing DNS records, and starting the new instance.

[ ] Continue automatic downloading
      All state should now be consistent and the old copy is effectively decommissioned,
      so it should be safe to resume automated operations.