Big issues with the StarLVM plugin and data corruption after a snapshot rollback

Software-based VM-centric and flash-friendly VM storage + free version
Post Reply
nvanaert
Posts: 9
Joined: Tue Jul 14, 2026 6:39 am

Thu Oct 15, 2026 12:24 pm

Dear sir/miss,

StarLVM plugin (starwind-proxmox-plugin-pve9 1.1.41, PVE 9.2.21): snapshot rollback recreates the disk LVs with lvcreate -kn -n vm-<id>-disk-<n> -s standard/snap_vm-<id>-disk-<n>_<snapname>, which clears ACTIVATION_SKIP. All other plugin operations use -ky. With the shared VG, LVM event autoactivation (vgchange -aay --autoactivation event) on any node that boots then activates these thin LVs and their pool while the VM runs on another node. On 2026-10-05 this happened on two nodes during a rolling reboot, the pool was active on two nodes at once, and the thin pool metadata was corrupted (thin_check status 64 on next activation). We have the LVM archives, journals and the original tmeta (as an image file) available. Please fix the rollback to use -ky, and confirm whether vgchange --setautoactivation n on the shared VG is supported with your plugin.
yaroslav (staff)
Staff
Posts: 4507
Joined: Mon Nov 18, 2019 11:11 am

Thu Oct 15, 2026 1:47 pm

Hi,

Can you please pull the plugin logs from both nodes?
Use /opt/starwind/starwind-proxmox-plugin/collect_pve_logs.sh

Upload them here https://www.starwindsoftware.com/support-form. 1560299 - is your reference
nvanaert
Posts: 9
Joined: Tue Jul 14, 2026 6:39 am

Thu Oct 15, 2026 2:36 pm

Yaroslav,

I've uploaded the logs as requested. I think the code below could solve the issue, be careful, as I might've made mistakes though

Code: Select all

--- StarLvmPlugin.pm.1.1.41.orig	2026-10-15 14:37:49.340779120 +0200
+++ StarLvmPlugin.pm	2026-10-15 14:37:49.361779194 +0200
@@ -661,7 +661,7 @@
         # remove the thin volume (it is a snapshot too)
         run_command(['/sbin/lvremove', '-f', "$vg/$volname"], errmsg => "lvremove '$vg/$volname' error");
         # create a copy of the specified snapshot with the volume's name
-        run_command(['/sbin/lvcreate', '-kn', '-n', $volname, '-s', "$vg/$snapvol"], errmsg => "lvm rollback '$vg/$snapvol' error");
+        run_command(['/sbin/lvcreate', '-ky', '-an', '-n', $volname, '-s', "$vg/$snapvol"], errmsg => "lvm rollback '$vg/$snapvol' error"); # local patch: -ky -an (was -kn), keep ACTIVATION_SKIP on shared VG
     });
 }
nvanaert
Posts: 9
Joined: Tue Jul 14, 2026 6:39 am

Thu Oct 15, 2026 2:42 pm

Yaroslav,

Another issue came up when searching through the logs (partially Claude assisted, but found myself in dmesg).

The LVM commands keep running after the cluster storage lock expires
These operations run their LVM commands inside cluster_lock_storage(), so that only one node changes the shared VG at a time:
- volume_resize (line 621)
- volume_snapshot (line 640)
- volume_snapshot_rollback (line 660)
- volume_snapshot_delete (line 676)
- rename_snapshot (line 690)
- StarLvmMonitor.pl (line 116)

PVE's cfs_lock sets alarm(60) around the locked code (pve-cluster, src/PVE/Cluster.pm: "fixed command timeout: cfs locks have a timeout of 120"). run_command (pve-common, src/PVE/Cmd.pm) only installs its own SIGALRM handler and kills the child when it is given a "timeout" parameter. The plugin never passes one. When the 60 s alarm fires, the Perl code dies and cfs_lock releases the lock, but the lvcreate, lvremove or lvextend child keeps running. From then on, other nodes can take the storage lock and write VG metadata at the same time as the still-running command. Without lvmlockd, this can corrupt the metadata of the whole VG, not just one pool.

Thin pool operations easily take longer than 60 s under load, because each thin volume and the pool are suspended in turn.

Evidence
On 2026-10-10 on idkstbabshv05, StarLvmMonitor.pl started "lvextend --use-policies standard/lvmth-1033". It stayed in D state in thin_presuspend/thin_postsuspend for about 30 minutes. By the code above, the lock was released after 60 s, so the extend ran unlocked for about 29 minutes.

Expected
The lock is held until the LVM command has finished, or the operation fails without leaving a running LVM command behind. Alternatively, document and support lvmlockd for the shared VG.

Claude's suggested fix for this issue is to wrap every LVM command that runs under the lock in a helper. The 60 second abort is replaced by a timer that renews the lock every 30 seconds and only would returns when the command is really finished.

Code: Select all

  # Run an LVM command inside cluster_lock_storage() and keep the cfs lock
  # alive until the command has exited, instead of losing it after 60 s.
  sub run_lvm_locked {
      my ($storeid, $cmd, %param) = @_;

      my $lockfile = "/etc/pve/priv/lock/storage-$storeid";
      local $SIG{ALRM} = sub {
          # same lock renewal the HA manager uses (PVE::HA::Env::PVE2)
          utime(0, time(), $lockfile)
              or warn "cannot renew cfs lock 'storage-$storeid': $!\n";
          alarm(30);
      };
      alarm(30);

      my $res = eval { run_command($cmd, %param) };
      my $err = $@;
      alarm(0);
      die $err if $err;
      return $res;
  }
yaroslav (staff)
Staff
Posts: 4507
Joined: Mon Nov 18, 2019 11:11 am

Thu Oct 15, 2026 3:08 pm

AFAIK, a timeout extension for the plugin is already on the to-do list. Please wait for the next build.
Please tell me more about the system
1. storage provider
2. protocol over which the storage is connected.
3. Proxmox version.
nvanaert
Posts: 9
Joined: Tue Jul 14, 2026 6:39 am

Fri Oct 16, 2026 9:57 am

yaroslav (staff) wrote: ↑
Thu Oct 15, 2026 3:08 pm
AFAIK, a timeout extension for the plugin is already on the to-do list. Please wait for the next build.
Please tell me more about the system
1. storage provider
2. protocol over which the storage is connected.
3. Proxmox version.
1. Fujitsu Siemens Eternus with a 4x Dell PowerEdge R740 and 6x Fujitsu servers (2 separate clusters with 2 separate volumes)
2. We're using iSCSI 10Gbps with Multipath to 2x Cisco Nexus 9300 series switches.
3. Proxmox version on all hosts is 9.2.21

Another few issues that I've encountered based on the logs
- During live migration, the thin pool is active on two nodes
- LVM commands keep running after the cluster storage lock expires
- StarLvmMonitor.pl reports success when the extend did not run
- Pools left active on other nodes are not detected, and their space is reused

Evidence of these bugs can be found in the logs
- We hit the same pattern with a longer window. On 2026-10-05, shv02 activated lvmth-1005 at boot (autoactivation, 11:24:35) while VM 1005 was running on shv05. VM 1005 was live-migrated onto shv02 at 11:29-11:30. That VM had had a snapshot since 2026-09-17, so most of its writes were copy-on-write. Days later, the pool failed thin_check with status 64.

- After lvmth-1005 was removed and recreated on shv05 (2026-10-15), shv03 still had live dm mappings of the old lvmth-1005 pool and disks. The VM was not running there. We removed them by hand. An earlier case on shv02 left orphaned dm mappings of a deleted pool (lvmth-1001), with "Failed to suspend ... with queued messages" when the VMID was reused.

- On 2026-10-10 on shv05, StarLvmMonitor.pl started "lvextend --use-policies standard/lvmth-1033". It stayed in D state in thin_presuspend/thin_postsuspend for about 30 minutes, during a heavy initial write into a new VM. By the code above, the lock was released after 60 s, so the extend ran unlocked for about 29 minutes.

I've also put these details in the ticket via an e-mail update.
nvanaert
Posts: 9
Joined: Tue Jul 14, 2026 6:39 am

Mon Oct 19, 2026 9:33 am

After doing some digging myself (assisted by Claude), I had Claude write a few patches (which I reviewed) for the current StarLVM version (untested in Production, but they compile and passed testing and seem to work), I'll see if I can test them here on an older test cluster:

Of course use these patches at your own risk and check / review them first (as I'm familiar with Perl, but certainly no expert).

0001-rollback-keep-activation-skip.patch

Code: Select all

--- a/usr/share/perl5/PVE/Storage/Custom/StarLvmPlugin.pm
+++ b/usr/share/perl5/PVE/Storage/Custom/StarLvmPlugin.pm
@@ -661,7 +661,9 @@
         # remove the thin volume (it is a snapshot too)
         run_command(['/sbin/lvremove', '-f', "$vg/$volname"], errmsg => "lvremove '$vg/$volname' error");
         # create a copy of the specified snapshot with the volume's name
-        run_command(['/sbin/lvcreate', '-kn', '-n', $volname, '-s', "$vg/$snapvol"], errmsg => "lvm rollback '$vg/$snapvol' error");
+        # keep activation skip and leave it inactive like every other volume on the shared vg,
+        # '-kn' let boot autoactivation on other nodes activate the thin pool
+        run_command(['/sbin/lvcreate', '-ky', '-an', '-n', $volname, '-s', "$vg/$snapvol"], errmsg => "lvm rollback '$vg/$snapvol' error");
     });
 }
0002-disable-lv-autoactivation.patch

Code: Select all

--- a/usr/share/perl5/PVE/Storage/Custom/StarLvmPlugin.pm
+++ b/usr/share/perl5/PVE/Storage/Custom/StarLvmPlugin.pm
@@ -111,6 +111,35 @@
     }
 };
 
+# Boot or iSCSI-login autoactivation must never activate a volume of the shared vg on
+# another node. Like PVE 9 LvmThinPlugin, disable it per LV in addition to activation
+# skip. lvcreate on trixie does not accept --setautoactivation for thin LVs, so use lvchange.
+sub set_lv_autoactivation_off {
+    my ($vg, $lv) = @_;
+
+    eval { run_command(['/sbin/lvchange', '--setautoactivation', 'n', "$vg/$lv"]); };
+    warn "could not disable autoactivation for '$vg/$lv': $@" if $@;
+}
+
+my $vg_autoactivation_checked = {};
+
+sub warn_vg_autoactivation {
+    my ($vg) = @_;
+
+    # activate_storage runs often, check once an hour per process
+    return if time() - ($vg_autoactivation_checked->{$vg} // 0) < 3600;
+    $vg_autoactivation_checked->{$vg} = time();
+
+    my $autoactivation = '';
+    eval {
+        run_command(['/sbin/vgs', '--noheadings', '-o', 'autoactivation', $vg],
+            outfunc => sub { $autoactivation = trim(shift); },
+            errfunc => $ignore_no_medium_warnings);
+    };
+    print "WARNING: autoactivation is enabled on shared vg '$vg', disable it with 'vgchange --setautoactivation n $vg'\n"
+        if $autoactivation eq 'enabled';
+}
+
 sub lvm_vg_pv_dm_name {
     my ($vg) = @_;
 
@@ -278,6 +307,7 @@
 	    # create new thin pool
         $cmd = ['/sbin/lvcreate', '--type', 'thin-pool', '-an', '-ky', '-Zy', '-n', $pool, '-L', "${psize}k", '--addtag', $sidtag, "$vg"];
         run_command($cmd, errmsg => "creation of thin pool '$pool' in $vg error");
+        set_lv_autoactivation_off($vg, $pool);
     }
 
     my $msize = $size / 1024;
@@ -285,6 +315,7 @@
 
     $cmd = ['/sbin/lvcreate', '--type', 'thin', '-an', '-ky', '-V', "${size}k", '-n', $name, '--addtag', "pve-vm-$vmid", "$vg/$pool"];
     run_command($cmd, errmsg => "lvcreate thin lv '$vg/$name' error");
+    set_lv_autoactivation_off($vg, $name);
 
     my $cache = {};
     activate_volume($class, $storeid, $scfg, $name, undef, $cache);
@@ -399,6 +430,7 @@
 
     my $vg = $scfg->{vgname};
     print "unsupported storage of vg '$vg'\n" unless chk_stor($vg);
+    warn_vg_autoactivation($vg);
 
     $class->SUPER::activate_storage($storeid, $scfg, $cache);
 }
@@ -536,6 +568,7 @@
         push @$cmd, '--addtag', "pve-sid-$storeid";
         push @$cmd, "$vg";
         run_command($cmd, errmsg => "creation of pool '$pool' in $vg error");
+        set_lv_autoactivation_off($vg, $pool);
     }
 
     if ($lvs->{$vg}->{$lv} && $lvs->{$vg}->{$lv}->{lv_state} eq 'a') {
@@ -546,6 +579,7 @@
     my $cmd = ['/sbin/lvcreate', '-s', '-an', '-ky', '-n', $name, '-prw', '--thinpool', $pool,
             '--addtag', "pve-vm-$vmid", "$vg/$volname"];
     run_command($cmd, errmsg => "clone image '$vg/$volname' into $name with pool $pool error");
+    set_lv_autoactivation_off($vg, $name);
 
     return $name;
 }
@@ -664,6 +698,8 @@
         # keep activation skip and leave it inactive like every other volume on the shared vg,
         # '-kn' let boot autoactivation on other nodes activate the thin pool
         run_command(['/sbin/lvcreate', '-ky', '-an', '-n', $volname, '-s', "$vg/$snapvol"], errmsg => "lvm rollback '$vg/$snapvol' error");
+        # PVE 9 LvmThinPlugin does the same after its rollback
+        set_lv_autoactivation_off($vg, $volname);
     });
 }
0003-plugin-renew-cfs-lock.patch

Code: Select all

--- a/usr/share/perl5/PVE/Storage/Custom/StarLvmPlugin.pm
+++ b/usr/share/perl5/PVE/Storage/Custom/StarLvmPlugin.pm
@@ -121,6 +121,32 @@
     warn "could not disable autoactivation for '$vg/$lv': $@" if $@;
 }
 
+# PVE cfs_lock() aborts the locked code after 60 s and releases the lock, but a running
+# LVM command is not killed and goes on changing the shared vg without the lock (a thin
+# pool suspend can take minutes under load). Renew the lock every 30 s while the command
+# runs, like PVE::HA::Env::PVE2 does, so it is held until the command has exited.
+# Only for commands that run while the storage lock is held.
+sub run_command_locked {
+    my ($storeid, $scfg, $cmd, %param) = @_;
+
+    # not shared: PVE uses a local flock, which does not expire
+    return run_command($cmd, %param) if !$scfg->{shared};
+
+    my $lockid = "storage-$storeid";
+    local $SIG{ALRM} = sub {
+        utime(0, time(), "/etc/pve/priv/lock/$lockid")
+            or print STDERR "WARNING: can't renew cfs lock '$lockid': $!\n";
+        alarm(30);
+    };
+    alarm(30);
+    my $res = eval { run_command($cmd, %param) };
+    my $err = $@;
+    alarm(0);
+    die $err if $err;
+
+    return $res;
+}
+
 my $vg_autoactivation_checked = {};
 
 sub warn_vg_autoactivation {
@@ -295,18 +321,18 @@
         if ($free < $psize) {
             $psize = $psize - $free;
             $cmd = ['/sbin/lvresize', '-L', "+${psize}k", "$vg/$pool"];
-            run_command($cmd, errmsg => "lvresize thin pool '$vg/$pool' to ${psize}k error");
+            run_command_locked($storeid, $scfg, $cmd, errmsg => "lvresize thin pool '$vg/$pool' to ${psize}k error");
     	}
 
 	    if (!$lvs->{$vg}->{$pool}->{tags}) {
             # add our tag with storeid to use it in monitor's callbacks for proper vg locking
             $cmd = ['/sbin/lvchange', '--addtag', $sidtag, "$vg/$pool"];
-            run_command($cmd, errmsg => "add tag '$sidtag' for thin pool '$vg/$pool' failed");
+            run_command_locked($storeid, $scfg, $cmd, errmsg => "add tag '$sidtag' for thin pool '$vg/$pool' failed");
         }
     } else {
 	    # create new thin pool
         $cmd = ['/sbin/lvcreate', '--type', 'thin-pool', '-an', '-ky', '-Zy', '-n', $pool, '-L', "${psize}k", '--addtag', $sidtag, "$vg"];
-        run_command($cmd, errmsg => "creation of thin pool '$pool' in $vg error");
+        run_command_locked($storeid, $scfg, $cmd, errmsg => "creation of thin pool '$pool' in $vg error");
         set_lv_autoactivation_off($vg, $pool);
     }
 
@@ -314,7 +340,7 @@
     print " creating lv $name (size ${msize}M) from pool $vg/$pool\n";
 
     $cmd = ['/sbin/lvcreate', '--type', 'thin', '-an', '-ky', '-V', "${size}k", '-n', $name, '--addtag', "pve-vm-$vmid", "$vg/$pool"];
-    run_command($cmd, errmsg => "lvcreate thin lv '$vg/$name' error");
+    run_command_locked($storeid, $scfg, $cmd, errmsg => "lvcreate thin lv '$vg/$name' error");
     set_lv_autoactivation_off($vg, $name);
 
     my $cache = {};
@@ -350,13 +376,13 @@
         foreach my $lv (keys %$dat) {
             next if $lv !~ m/^snap_${volname}_${PVE::JSONSchema::CONFIGID_RE}$/;
 
-            run_command(['/sbin/lvremove', '--yes', "$vg/$lv"], errmsg => "lvremove snapshot '$vg/$lv' error");
+            run_command_locked($storeid, $scfg, ['/sbin/lvremove', '--yes', "$vg/$lv"], errmsg => "lvremove snapshot '$vg/$lv' error");
             undef $dat->{$lv};
         }
 
         # remove lv itself (if exists)
         if ($dat->{$volname}) {
-            run_command(['/sbin/lvremove', '--yes', "$vg/$volname"], errmsg => "lvremove '$vg/$volname' error");
+            run_command_locked($storeid, $scfg, ['/sbin/lvremove', '--yes', "$vg/$volname"], errmsg => "lvremove '$vg/$volname' error");
             undef $dat->{$volname};
         }
 
@@ -378,7 +404,7 @@
                 my $p = "$vg/$1";
             print "path $p tainted!\n" if Scalar::Util::tainted($p);
 
-            run_command(['/sbin/lvremove', '--yes', "$p"], errmsg => "lvremove '$p' error");
+            run_command_locked($storeid, $scfg, ['/sbin/lvremove', '--yes', "$p"], errmsg => "lvremove '$p' error");
         } else {
             #print "$vg/$pool still has $num ref(s)\n";
         }
@@ -567,18 +593,18 @@
         my $cmd = ['/sbin/lvcreate', '--type', 'thin-pool', '-an', '-ky', '-Zy', '-n', $pool, '-L', '100M'];
         push @$cmd, '--addtag', "pve-sid-$storeid";
         push @$cmd, "$vg";
-        run_command($cmd, errmsg => "creation of pool '$pool' in $vg error");
+        run_command_locked($storeid, $scfg, $cmd, errmsg => "creation of pool '$pool' in $vg error");
         set_lv_autoactivation_off($vg, $pool);
     }
 
     if ($lvs->{$vg}->{$lv} && $lvs->{$vg}->{$lv}->{lv_state} eq 'a') {
-        run_command(['/sbin/lvchange', '-an', "$vg/$lv"], errmsg => "deactivate_volume '$vg/$lv' error");
+        run_command_locked($storeid, $scfg, ['/sbin/lvchange', '-an', "$vg/$lv"], errmsg => "deactivate_volume '$vg/$lv' error");
     }
 
     # create a new snapshot LV with external readonly base LV
     my $cmd = ['/sbin/lvcreate', '-s', '-an', '-ky', '-n', $name, '-prw', '--thinpool', $pool,
             '--addtag', "pve-vm-$vmid", "$vg/$volname"];
-    run_command($cmd, errmsg => "clone image '$vg/$volname' into $name with pool $pool error");
+    run_command_locked($storeid, $scfg, $cmd, errmsg => "clone image '$vg/$volname' into $name with pool $pool error");
     set_lv_autoactivation_off($vg, $name);
 
     return $name;
@@ -612,7 +638,7 @@
     }
 
     if ($lvs->{$vg}->{$pool} && $lvs->{$vg}->{$pool}->{lv_state} eq 'a') {
-        run_command(['/sbin/lvchange', '--monitor', 'n', "$vg/$pool"], errmsg => "monitor=n for thin pool '$pool' failed");
+        run_command_locked($storeid, $scfg, ['/sbin/lvchange', '--monitor', 'n', "$vg/$pool"], errmsg => "monitor=n for thin pool '$pool' failed");
     }
 
     if ($lvs->{$vg}->{$volname} && $lvs->{$vg}->{$volname}->{lv_state} eq 'a') {
@@ -622,11 +648,11 @@
     my $newname = $name;
     $newname =~ s/^vm-/base-/;
     my $cmd = ['/sbin/lvrename', $vg, $volname, $newname];
-    run_command($cmd, errmsg => "lvrename '$vg/$volname' => '$vg/$newname' error");
+    run_command_locked($storeid, $scfg, $cmd, errmsg => "lvrename '$vg/$volname' => '$vg/$newname' error");
 
     # set read-only flag for the thin lv
     $cmd = ['/sbin/lvchange', '-pr', "$vg/$newname"];
-    eval { run_command($cmd); };
+    eval { run_command_locked($storeid, $scfg, $cmd); };
     warn $@ if $@;
 
     return $newname;
@@ -654,7 +680,7 @@
 
     $class->cluster_lock_storage($storeid, $scfg->{shared}, undef, sub {
         # Resize only the LV, not its thin pool. Thin pools are auto extended with dmeventd
-    	run_command(['/sbin/lvextend', '-L', $size, $p], errmsg => "error resizing volume '$p'");
+    	run_command_locked($storeid, $scfg, ['/sbin/lvextend', '-L', $size, $p], errmsg => "error resizing volume '$p'");
     	#run_command(['/sbin/lvextend', '-L', $size, $pname], errmsg => "error resizing thin pool '$pname'");
     });
 
@@ -672,7 +698,7 @@
     my $cmd = ['/sbin/lvcreate', '-s', '-an', '-n', $snapvol, '-ky', '-pr', "$vg/$volname"];
 
     $class->cluster_lock_storage($storeid, $scfg->{shared}, undef, sub {
-        run_command($cmd, errmsg => "lvcreate snapshot '$vg/$snapvol' error");
+        run_command_locked($storeid, $scfg, $cmd, errmsg => "lvcreate snapshot '$vg/$snapvol' error");
     });
 }
 
@@ -693,11 +719,11 @@
 
     $class->cluster_lock_storage($storeid, $scfg->{shared}, undef, sub {
         # remove the thin volume (it is a snapshot too)
-        run_command(['/sbin/lvremove', '-f', "$vg/$volname"], errmsg => "lvremove '$vg/$volname' error");
+        run_command_locked($storeid, $scfg, ['/sbin/lvremove', '-f', "$vg/$volname"], errmsg => "lvremove '$vg/$volname' error");
         # create a copy of the specified snapshot with the volume's name
         # keep activation skip and leave it inactive like every other volume on the shared vg,
         # '-kn' let boot autoactivation on other nodes activate the thin pool
-        run_command(['/sbin/lvcreate', '-ky', '-an', '-n', $volname, '-s', "$vg/$snapvol"], errmsg => "lvm rollback '$vg/$snapvol' error");
+        run_command_locked($storeid, $scfg, ['/sbin/lvcreate', '-ky', '-an', '-n', $volname, '-s', "$vg/$snapvol"], errmsg => "lvm rollback '$vg/$snapvol' error");
         # PVE 9 LvmThinPlugin does the same after its rollback
         set_lv_autoactivation_off($vg, $volname);
     });
@@ -712,7 +738,7 @@
     print " deleting snapshot $snapvol of $vg/$volname...\n";
 
     $class->cluster_lock_storage($storeid, $scfg->{shared}, undef, sub {
-        run_command(['/sbin/lvremove', '-f', "$vg/$snapvol"], errmsg => "lvremove snapshot '$vg/$snapvol' error");
+        run_command_locked($storeid, $scfg, ['/sbin/lvremove', '-f', "$vg/$snapvol"], errmsg => "lvremove snapshot '$vg/$snapvol' error");
     });
 }
 
@@ -727,7 +753,7 @@
 
     $class->cluster_lock_storage($storeid, $scfg->{shared}, undef, sub {
         my $cmd = ['/sbin/lvrename', $vg, $src_snapvol, $tgt_snapvol];
-        run_command($cmd, errmsg => "lvrename '$vg/$src_snapvol' => '$vg/$tgt_snapvol' error");
+        run_command_locked($storeid, $scfg, $cmd, errmsg => "lvrename '$vg/$src_snapvol' => '$vg/$tgt_snapvol' error");
     });
 }
0004-monitor-report-failures-renew-lock.patch

Code: Select all

--- a/usr/share/perl5/PVE/Storage/Custom/StarLvmMonitor.pl
+++ b/usr/share/perl5/PVE/Storage/Custom/StarLvmMonitor.pl
@@ -76,13 +76,42 @@
     return $lvs;
 }
 
+# Same lock renewal as run_command_locked() in StarLvmPlugin.pm: cfs_lock() gives the
+# code 60 s and then releases the lock while lvextend keeps running. Renew it every
+# 30 s until lvextend has exited.
+sub run_command_locked {
+    my ($storeid, $cmd, %param) = @_;
+
+    my $lockid = "storage-$storeid";
+    local $SIG{ALRM} = sub {
+        utime(0, time(), "/etc/pve/priv/lock/$lockid")
+            or print STDERR "WARNING: can't renew cfs lock '$lockid': $!\n";
+        alarm(30);
+    };
+    alarm(30);
+    my $res = eval { run_command($cmd, %param) };
+    my $err = $@;
+    alarm(0);
+    die $err if $err;
+
+    return $res;
+}
+
 # main code to extend thin pools if needed
 
-my $percent = 80; #FIXME: we use the hardcoded 80% now, but it should be read from lvmlocal.conf thin_pool_autoextend_threshold
+# the threshold from lvm.conf/lvmlocal.conf (the package installs 80), 100 disables autoextend
+my $percent = 80;
+eval {
+    run_command(['/sbin/lvmconfig', '--typeconfig', 'full', 'activation/thin_pool_autoextend_threshold'],
+        outfunc => sub { $percent = $1 if $_[0] =~ m/=\s*(\d+)/; },
+        errfunc => $ignore_no_medium_warnings);
+};
+print STDERR "can't read thin_pool_autoextend_threshold, using $percent: $@" if $@;
 
 my $vol = $ARGV[0] || "";
-my $datap = $ENV{DMEVENTD_THIN_POOL_DATA} || '?';
-my $metap = $ENV{DMEVENTD_THIN_POOL_METADATA} || '?';
+# an unknown usage counts as 100, lvextend --use-policies checks the real usage itself
+my $datap = ($ENV{DMEVENTD_THIN_POOL_DATA} // '') =~ m/^(\d+(?:\.\d+)?)/ ? $1 : 100;
+my $metap = ($ENV{DMEVENTD_THIN_POOL_METADATA} // '') =~ m/^(\d+(?:\.\d+)?)/ ? $1 : 100;
 
 die "volume name not specified.\n" if !$vol;
 
@@ -98,6 +127,8 @@
 if ($pool =~ m/^lvmth-\d+$/) {
     # it's our thin pool, we have to use the cluster lock to extend it
 
+    exit 0 if $percent >= 100;
+
     if ($percent > $datap && $percent > $metap) {
         print STDERR "called for vol '$vg/$pool' ($percent > $datap:$metap)\n";
         exit 0;
@@ -113,9 +144,16 @@
 
     print STDERR " calling 'lvextend --use-policies $vol' with cluster lock '$storeid' ($datap:$metap)\n";
 
-    $res = PVE::Cluster::cfs_lock_storage($storeid, 10, sub {
-        return run_command(['/sbin/lvextend', '--use-policies', $vol], errmsg => "storeid $storeid - locked extending of vol '$vol' failed");
+    # wait up to 60 s, a snapshot or another extend may hold the lock for a while
+    $res = PVE::Cluster::cfs_lock_storage($storeid, 60, sub {
+        return run_command_locked($storeid, ['/sbin/lvextend', '--use-policies', $vol], errmsg => "storeid $storeid - locked extending of vol '$vol' failed");
     });
+    # cfs_lock_storage() does not die, it returns undef and sets $@ (no lock, no quorum,
+    # lvextend failed). Exit non-zero so dmeventd logs it and calls us again.
+    if (my $err = $@) {
+        print STDERR "extending '$vol' failed: $err";
+        exit 1;
+    }
 } else {
     # it's some other thin pool, it should be safe to extend it directly (default behaviour for LVM) 
0005-guard-pools-active-elsewhere.patch

Code: Select all

--- a/usr/share/perl5/PVE/Storage/Custom/StarLvmPlugin.pm
+++ b/usr/share/perl5/PVE/Storage/Custom/StarLvmPlugin.pm
@@ -12,7 +12,10 @@
 use IO::File;
 use Scalar::Util;
 
+use PVE::Cluster;
+use PVE::INotify;
 use PVE::JSONSchema qw(get_standard_option);
+use PVE::SSHInfo;
 use PVE::Tools qw(run_command trim);
 use PVE::Storage::Plugin;
 use PVE::Storage::LVMPlugin;
@@ -166,6 +169,52 @@
         if $autoactivation eq 'enabled';
 }
 
+# Without lvmlockd a node only knows its own activations. A thin pool that is still
+# active on another node keeps device-mapper mappings to extents that LVM hands out
+# again once the LVs are removed, and that node later writes its stale metadata there.
+# Ask every other online node before removing or recreating LVs of the pool.
+sub check_pool_inactive_elsewhere {
+    my ($vg, $pool, $action) = @_;
+
+    my $members = PVE::Cluster::get_members() // {};
+    my $nodename = PVE::INotify::nodename();
+
+    # device-mapper names double the '-' inside vg and lv names
+    my $dmname = join('-', map { (my $n = $_) =~ s/-/--/g; $n } ($vg, $pool));
+    my $re = qr/^\Q$dmname\E(?:[-_]|$)/;
+
+    my (@active, @failed);
+    foreach my $node (sort keys %$members) {
+        next if $node eq $nodename;
+
+        if (!$members->{$node}->{online}) {
+            print "WARNING: node '$node' is offline, can't check it for active thin pool '$vg/$pool'\n";
+            next;
+        }
+
+        # free_image runs under the storage lock (120 s lifetime): keep the worst case short
+        my $ssh = PVE::SSHInfo::ssh_info_to_command(PVE::SSHInfo::get_ssh_info($node), '-o', 'ConnectTimeout=5');
+        my @devs;
+        eval {
+            run_command([@$ssh, '/sbin/dmsetup', 'ls'], timeout => 10, outfunc => sub {
+                my ($dev) = split(/\s+/, trim(shift));
+                push @devs, $dev if $dev && $dev =~ $re;
+            });
+        };
+        if (my $err = $@) {
+            chomp $err;
+            push @failed, "$node: $err";
+            next;
+        }
+        push @active, "$node: " . join(' ', @devs) if @devs;
+    }
+
+    die "thin pool '$vg/$pool' is active on other node(s), refusing to $action\n  "
+        . join("\n  ", @active) . "\n" if @active;
+    die "can't check other node(s) for thin pool '$vg/$pool', refusing to $action\n  "
+        . join("\n  ", @failed) . "\n" if @failed;
+}
+
 sub lvm_vg_pv_dm_name {
     my ($vg) = @_;
 
@@ -362,6 +411,8 @@
     my $pool = $lvs->{$vg}->{$volname}->{pool_lv} ? $lvs->{$vg}->{$volname}->{pool_lv} : "lvmth-$vmid";
     die "no such thin pool $vg/$pool\n" if !$lvs->{$vg}->{$pool};
 
+    check_pool_inactive_elsewhere($vg, $pool, "delete $volname");
+
     print " deleting $volname from $vg/$pool\n";
 
     if (my $dat = $lvs->{$vg}) {
@@ -717,6 +768,9 @@
 
     print " rolling back snapshot $snapvol of $vg/$volname...\n";
 
+    die "Invalid volname $volname" if $volname !~ m/^(vm|base)-(\d+)-/;
+    check_pool_inactive_elsewhere($vg, "lvmth-$2", "roll back $volname");
+
     $class->cluster_lock_storage($storeid, $scfg->{shared}, undef, sub {
         # remove the thin volume (it is a snapshot too)
         run_command_locked($storeid, $scfg, ['/sbin/lvremove', '-f', "$vg/$volname"], errmsg => "lvremove '$vg/$volname' error");
0006-optional-exclusive-activation.patch

Code: Select all

--- a/usr/share/perl5/PVE/Storage/Custom/StarLvmPlugin.pm
+++ b/usr/share/perl5/PVE/Storage/Custom/StarLvmPlugin.pm
@@ -548,6 +548,11 @@
         }
     } else {
         if ($v->{$pool}->{lv_state} ne 'a') {
+            # Opt-in: a pool activated here while another node still has it active works with
+            # stale metadata. This also refuses online migration (the target activates the pool
+            # while the source still runs), so migrate offline while the file exists.
+            check_pool_inactive_elsewhere($vg, $pool, "activate it on this node")
+                if -e '/etc/pve/starlvm-exclusive-activation';
             run_command(['/sbin/lvchange', '-aly', '-K', '--monitor', 'y', $vol], errmsg => "activating thin pool '$vol' failed");
             $v->{$pool}->{lv_state} = 'a'; # update cached state
         }
Side Notes:
- Patch 0005 can still miss two cases:
- Another node activates the pool in the short window between the check and the delete.
- An offline node that isn't fenced (like hv01 on its test bench) still has stale mappings. Offline nodes only get a warning.
- Without quorum the lock can't be renewed. The bug race is then possible again, and wouldn't know how this could be prevented (can this be prevented?).
- Patch 0003 doesn't cover the activation step inside disk creation.
- Patch 0006 only avoids it by refusing live migration, and whether the bug happens at all still depends on a migration test.
Post Reply