Restore storage node

This describes how to restore from a "blank" storage node, supposing the machine has been completely lost.

• order a machine
  • currently we use a EM-L110X-SATA (CPU Intel Xeon E5 1410 v2, RAM 96 GB, Disks 4 x 12 TB HDD) at Scaleway
• install latest Debian
  • RAID 1 for /boot (512M)
  • RAID 5 for / (the rest)
• follow standard machine installation procedure: https://github.com/kiwix/operations/wiki/Machine-and-k8s-node-Setup
• you can wait for RAID array synchronization to complete (it consumes lots of IOs, but Dec. 2024 incident seems to have proven it was not necessary to wait + taking a LOT of time)
  • mdadm --detail /dev/md1
• restore ZIMs from a Kiwix mirror
  • as of Dec. 2024, mirrors.dotsrc.org is our fastest mirror from France, and it has a 20Gps connection to the Internet, so it is capable to "fill" the classic 1Gbps our server typically has ; it also has the advantage of containing all our ZIMs and other data.
  • sample command to restore all wikipedia ZIMs in current folder: rsync -vrlptD -n rsync://mirrors.dotsrc.org/kiwix/zim/wikipedia/ .
  • to observe bandwidth consumption, you might use iptraf (go to Detailed interface statistics for summary per interface with a detail between incoming and outgoing rates)
• delete all cronjobs, statefulsets, deployments, pvcs and pvs linked to the lost node
  • you will need k patch pvc zim-receiver-pvc -n zim -p '{"metadata":{"finalizers":null}}' to force finalization of the PVC
• create nginx-logs folder structure:
  • mkdir -p /data/nginx-logs/{download-kiwix,download-openzim,library-kiwix}
  • mkdir -p /data/nginx-logs/archives/{download-kiwix,download-openzim,library-kiwix}
• change ownership of files for upload folders:
  • chown -R 1000:1000 /data/tmp/ci /data/download/release /data/download/nightly /data/openzim/release /data/openzim/nightly /data/openzim/wp1
  • chown -R 999 /data/zim-receiver/zim