fix(csi): survive int reboots, pin image by digest, raise memory
All checks were successful
Deploy CSI Jobs / deploy (push) Successful in 36s

Both plugin jobs are pinned to int with count = 1 and had no restart or
reschedule stanza. When int goes away the allocations are marked Lost and
never return — csi-nfs-node showed 1 Complete / 3 Lost / 0 Failed, so these
were never application crashes, they were the node disappearing. The plugin
has been down since 2026-06-08 as a result. Adds restart + unlimited
reschedule with exponential backoff so they recover on their own.

The image was :latest, against our own rule for Nomad. The copy cached on
int is sha256:944d0e65 and roughly seven months old, so any fresh pull on
another node would get a different build — a good way to make failures
irreproducible. Pinned to the digest that has actually been running.

Memory was 128 MB for a Node.js process. Raised to 256 with memory_max so
mount and provisioning activity can burst without permanently reserving it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Henrik Jess Nielsen
2026-08-06 23:22:28 +02:00
parent c3a25a84e5
commit 53d696ea73
2 changed files with 51 additions and 6 deletions

View File

@@ -12,11 +12,32 @@ job "csi-nfs-node" {
group "node" {
count = 1
# int is both the NFS server and the only node running this plugin, so a
# reboot there takes it down. Without these the allocation is marked Lost
# and never comes back on its own — which is how it stayed dead from
# 2026-06-08 until someone noticed.
restart {
attempts = 5
interval = "10m"
delay = "30s"
mode = "delay"
}
reschedule {
unlimited = true
delay = "30s"
delay_function = "exponential"
max_delay = "5m"
}
task "plugin" {
driver = "docker"
config {
image = "democraticcsi/democratic-csi:latest"
# Pinned by digest rather than :latest. The tag moves upstream, so a
# fresh pull on a new node gets a different build than the one cached
# on int — this digest is what has actually been running there.
image = "democraticcsi/democratic-csi@sha256:944d0e65077efbd9c1fdf23997eec8fac4b4bfb7c3de400f63e33a0a849c5ced"
command = "/bin/democratic-csi"
network_mode = "host"
args = [
@@ -60,9 +81,13 @@ TMPL
destination = "${NOMAD_TASK_DIR}/driver-config-file.yaml"
}
# democratic-csi is a Node.js process; 128 MB is tight enough that mount
# activity can push it over. memory_max lets it burst without reserving
# the headroom permanently.
resources {
cpu = 100
memory = 128
cpu = 100
memory = 256
memory_max = 512
}
}
}