fix(csi): survive int reboots, pin image by digest, raise memory
All checks were successful
Deploy CSI Jobs / deploy (push) Successful in 36s

Both plugin jobs are pinned to int with count = 1 and had no restart or
reschedule stanza. When int goes away the allocations are marked Lost and
never return — csi-nfs-node showed 1 Complete / 3 Lost / 0 Failed, so these
were never application crashes, they were the node disappearing. The plugin
has been down since 2026-06-08 as a result. Adds restart + unlimited
reschedule with exponential backoff so they recover on their own.

The image was :latest, against our own rule for Nomad. The copy cached on
int is sha256:944d0e65 and roughly seven months old, so any fresh pull on
another node would get a different build — a good way to make failures
irreproducible. Pinned to the digest that has actually been running.

Memory was 128 MB for a Node.js process. Raised to 256 with memory_max so
mount and provisioning activity can burst without permanently reserving it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Henrik Jess Nielsen
2026-08-06 23:22:28 +02:00
parent c3a25a84e5
commit 53d696ea73
2 changed files with 51 additions and 6 deletions

View File

@@ -12,11 +12,28 @@ job "csi-nfs-controller" {
group "controller" {
count = 1
# Pinned to int, so an int reboot takes the controller down with it.
# Without these the allocation is marked Lost and never returns.
restart {
attempts = 5
interval = "10m"
delay = "30s"
mode = "delay"
}
reschedule {
unlimited = true
delay = "30s"
delay_function = "exponential"
max_delay = "5m"
}
task "plugin" {
driver = "docker"
config {
image = "democraticcsi/democratic-csi:latest"
# Pinned by digest rather than :latest — see node.nomad for why.
image = "democraticcsi/democratic-csi@sha256:944d0e65077efbd9c1fdf23997eec8fac4b4bfb7c3de400f63e33a0a849c5ced"
command = "/bin/democratic-csi"
args = [
"--csi-version=1.5.0",
@@ -60,9 +77,12 @@ TMPL
destination = "${NOMAD_TASK_DIR}/driver-config-file.yaml"
}
# The controller does the provisioning work (volume create, NFS export
# management), so it gets more headroom than the node plugin.
resources {
cpu = 100
memory = 128
cpu = 100
memory = 256
memory_max = 768
}
}
}