SLOG Setup

[spflanze]

I seek to include a SLOG write through cache pool in an Ubuntu 22.04 OS I am preparing to install.

I have 128 GB RAM, and currently plan for 256 GB nvme SLOG partition. Is this size excessive?

Does a mount point need to be created for the SLOG's pool, and any other pool, the user does not normally access directly?

Should the SLOG pool be compressed?

The below Ubuntu installation command sequence snippet is the currently planned command sequence for setting up the SLOG. Please comment whether I have this right:

# These commands  are used to find out which logical name is associated with which hardware node
# in the /dev/disk/by-id/ directory:
lsscsi –device
lshw -C disk

# Assign variables to the hardware node paths. These hardware nodes are specific to my system.
# logical name: /dev/nvme0n1
SSD1=/dev/disk/by-id/nvme-eui.002538ba11517132
# logical name: /dev/nvme1n1
SSD2=/dev/disk/by-id/nvme-eui.002538ba11505423

# Create the cache partition in nvme memory:
sgdisk     -n6:0:+256G -t6:FD00  $SSD1
sgdisk     -n6:0:+256G -t6:FD00  $SSD2

# Create the cpool:
zpool create -f \
     -o ashift=12 \
    -o autotrim=on \
    -O encryption=on -O keylocation=prompt -O keyformat=passphrase \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
         -O canmount=off -O mountpoint=/ -R /mnt \
   cpool mirror ${SSD1}-part6  ${SSD2}-part6

# Assign a mount point (not sure this is necessary)
zfs create -o canmount=off -o mountpoint=none cpool/CACHE
zpool add cpool log mirror ${SSD1}-part6  ${SSD2}-part6
zpool add cpool cache mirror ${SSD1}-part6  ${SSD2}-part6

[KevinBuettner]

Take a close look at the lines adding the L2ARC and the SLOG. The devices and/or partitions being used for the log and cache need to be different from each other and also different from that used by the mirror vdev of the pool.

I think that your planned SLOG size is excessive. There are some good web resources which should help you come up with a more reasonable size.

[kftsehk]

A few comments from my experience on a small storage pool

L2ARC ~ Read Cache

• probably do not need mirror (declaration: I have not seen doc advise against mirroring cache)
  L2ARC [likely] survive disk crash or corrupt & [tested] missing any number of vdev.
  Cache is another copy of data retired from ARC into L2ARC, cache is not warm tier, original N copy of raided data is still on the pool, everything is checksumed so if L2ARC corrupted zfs can also pull another copy from storage pool
• Make L2ARC size > frequently used dataset size
  Any cache is subject to some degree of cache flush, no matter how smart replacement algo is, unless your cache is >= data size
  Had headache with ML workload requiring TBs of data randomly sampled for many epoch, flush cache no matter what tuning given to L2ARC
• => Mirroring L2ARC gives more read bandwidth, but less cache space, due to the cache flush more space is probably better
• Exclude special_small_vdev from L2ARC if special vdev is SSD, SSD is fast enough not to require a non-RAM cache (see module parameter)
• Just to remind you need to set both primarycache=all and secondarycache=all to use L2ARC
  Blocks has to be retired from ARC (primary cache) to be eligible to enter L2ARC (secondary cache), not sure if it is bug or feature.
  If setting primarycache=metadata, what ends up in L2ARC is only metadata even it is set to all.

SLOG ~ Write Buffer

[^slog]

• Mirroring increase transient data protection but not write buffer size or speed
  transient data writes = spiky write exceeding HDD write capacity
• Size according to your largest write spike
  Slog smooth out spike, but in long term however large buffer does not help if IO bandwidth constantly above HDD write bandwidth.
  The buffering time is SLOG size / Network BW, e.g. 10GB / 10GbE ~ 10s full load on write
  Use case Very good dealing with saving ML model checkpoints along training (few GBs of data over 10GbE < SLOG size)
  With SLOG: Few seconds into SLOG, program got write ack and continue, leaving HDD spinning for a minute to write that GB.
  Without SLOG: Training blocked for minutes until HDD finishes the write.
• Up to personal preference to mirror or not
  Failing slog device might have consequence similar to failing power without UPS, you lose few seconds of data

Separate SLOG / CACHE device?

Do some testing, do your SSD / NVMe device too cheap and suffer performance issue when simutaneously receive read and write commands, most model now does so little.

[^slog] https://www.servethehome.com/what-is-the-zfs-zil-slog-and-what-makes-a-good-one/

My pool for your ref:

        NAME                                 STATE     READ WRITE CKSUM
        brick                                ONLINE       0     0     0
          mirror-0                           ONLINE       0     0     0
            HDD1-16T           ONLINE       0     0     0
            HDD2-16T           ONLINE       0     0     0
            HDD3-16T           ONLINE       0     0     0
        special
          mirror-1                           ONLINE       0     0     0
            # throgh this 2 * SSD + 1 * NVMe raid1 I got ssd write speed but NVMe read speed on small files (my def: small <=1MB)
            # still expect smaller IOPS than raw NVMe before #10018 is merged
            # very important if you are python / js dev on a ZFS /home, 40K files each 128B-64KB .js / .py sucks on small enthusiast HDD ZFS pool
            SSD1-500G           ONLINE       0     0     0
            SSD2-500G           ONLINE       0     0     0
            NVMe1-500G  ONLINE       0     0     0 
        logs
          NVMe1-16G    ONLINE       0     0     0
        cache
          NVMe1~200G                          ONLINE       0     0     0

[IvanVolosyuk]

Interesting, why don't you have 'sync=disabled' instead of separate SLOG? This will achieve consistency in snapshots written without blocking the training. I think SLOG is useful for a network service, but for ML workloads it looks like an overkill.

[kftsehk]

It seems dangerous to disable fsync, if others might be running a whatever DB on it, like we are looking at vector db recently.

I also got several machines so some writes are from NFS, even with -o async set on the mount point, closing the file is one way to trigger an immediate write to NFS server.
it is the NFS server daemon that always write with fsync before ack.

[IvanVolosyuk]

fsync and 'zfs sync=disabled' are too different beasts. ZFS sync=disabled preserves write ordering, while the code without fsync does not. For databases which are exposed to network and NFS 'zfs sync=disabled' doesn't quite work as some other machine may rely on the data to be on disk after successful fsync() operation, while for LLM training where different machines are somewhat isolated 'sync=disabled' can be reasonable compromise between speed and resilience to crash/power failure. I thought that you had the LLM snapshots on the same machine as the training process, but I guess if it on separate machines connected via NFS it's a different story.

[kftsehk]

btw I've got a ansible playbook for my archlinux cluster settings, public repo.
In case discussion owner want an opinionated reference of some baseline kernel parameters or config, can have a look.
Also please kindly tell me if anyone catch some misconfig

https://github.com/kftsehk/ansible-archlinux-cluster

Although now works like single machine cuz I am not rich enough to buy a pair of infiniband card, mostly from a cluster scenario where MPI program will be run.