Yaroslav,
Another issue came up when searching through the logs (partially Claude assisted, but found myself in dmesg).
The LVM commands keep running after the cluster storage lock expires
These operations run their LVM commands inside cluster_lock_storage(), so that only one node changes the shared VG at a time:
- volume_resize (line 621)
- volume_snapshot (line 640)
- volume_snapshot_rollback (line 660)
- volume_snapshot_delete (line 676)
- rename_snapshot (line 690)
- StarLvmMonitor.pl (line 116)
PVE's cfs_lock sets alarm(60) around the locked code (pve-cluster, src/PVE/Cluster.pm: "fixed command timeout: cfs locks have a timeout of 120"). run_command (pve-common, src/PVE/Cmd.pm) only installs its own SIGALRM handler and kills the child when it is given a "timeout" parameter. The plugin never passes one. When the 60 s alarm fires, the Perl code dies and cfs_lock releases the lock, but the lvcreate, lvremove or lvextend child keeps running. From then on, other nodes can take the storage lock and write VG metadata at the same time as the still-running command. Without lvmlockd, this can corrupt the metadata of the whole VG, not just one pool.
Thin pool operations easily take longer than 60 s under load, because each thin volume and the pool are suspended in turn.
Evidence
On 2026-10-10 on idkstbabshv05, StarLvmMonitor.pl started "lvextend --use-policies standard/lvmth-1033". It stayed in D state in thin_presuspend/thin_postsuspend for about 30 minutes. By the code above, the lock was released after 60 s, so the extend ran unlocked for about 29 minutes.
Expected
The lock is held until the LVM command has finished, or the operation fails without leaving a running LVM command behind. Alternatively, document and support lvmlockd for the shared VG.
Claude's suggested fix for this issue is to wrap every LVM command that runs under the lock in a helper. The 60 second abort is replaced by a timer that renews the lock every 30 seconds and only would returns when the command is really finished.
Code: Select all
# Run an LVM command inside cluster_lock_storage() and keep the cfs lock
# alive until the command has exited, instead of losing it after 60 s.
sub run_lvm_locked {
my ($storeid, $cmd, %param) = @_;
my $lockfile = "/etc/pve/priv/lock/storage-$storeid";
local $SIG{ALRM} = sub {
# same lock renewal the HA manager uses (PVE::HA::Env::PVE2)
utime(0, time(), $lockfile)
or warn "cannot renew cfs lock 'storage-$storeid': $!\n";
alarm(30);
};
alarm(30);
my $res = eval { run_command($cmd, %param) };
my $err = $@;
alarm(0);
die $err if $err;
return $res;
}
