We've recently migrated from ESXi to Proxmox. We are using the same storage we used at ESXI EV, it's FC SAN mounted as usual in such case.
We have two disks 15Tb each. We use Proxmox as cluster with multuple nodes, 4 nodes now active.
One is LVM - pure, no volume-chains, no other tricks, not using snapshots on it. No issues.
Another one is starlvm - for dev VMs where we need to use snapshot functionality. This is the problematic one.
For development purposes we need to use snapshots as a tool to roll-back VM changes fast and reliably.
But sometimes, randomly, during snapshot revert procedure some VMs are losing disks:
Code: Select all
unsupported storage of vg 'star_vg_fc_san_3738'
activating vm-13472-disk-0...
deactivating vm-13472-disk-0...
Use of uninitialized value in string ne at /usr/share/perl5/PVE/Storage/Custom/StarLvmPlugin.pm line 481.
TASK ERROR: no such logical volume star_vg_fc_san_3738/vm-13472-disk-0shutdown VM (gracefully)
get list of snapshots
revert to the latest snapshot (have no parent)
start VM
I'm running this routine sequentially for each VM group(6 VMs in a group), but there are multiple VM groups at the cluster (alpha, beta, each with 6 VMs in it), and sometimes they might execute this routine for different VMs in parallel. I'm using some randomization and time shift to run snapshot and power-related tasks as random as possible. I'm not sure if this may be the cause, but it's maybe worth mentioning.
Here is the problematic VM config:
- all vms are built similarly, only name, tags, mac changes for each vm version and group
Code: Select all
#Clone made from SOME VM
agent: enabled=1,fstrim_cloned_disks=1
boot: order=scsi0;net0
cores: 2
cpu: host
machine: q35
memory: 4096
meta: creation-qemu=11.0.0,ctime=1780414247
name: VM-NAME-delta
net0: virtio=some_MAC,bridge=somebridge,firewall=1,queues=1
onboot: 0
ostype: l26
parent: phase_2
scsi0: swsan3738lv:vm-13472-disk-0,aio=native,cache=none,detect_zeroes=1,discard=on,iothread=1,queues=2,size=70G
scsihw: virtio-scsi-single
smbios1: uuid=UUID
tags: SOME-TAGS
vmgenid: 896cef72-1765-41dd-8b93-89f4ea668e09
[phase_2]
#Cloned VM Reverting this snapshot when needed
agent: enabled=1,fstrim_cloned_disks=1
boot: order=scsi0;net0
cores: 2
cpu: host
machine: q35
memory: 4096
meta: creation-qemu=11.0.0,ctime=1780414247
name: VM-NAME-delta
net0: virtio=some_MAC,bridge=somebridge,firewall=1,queues=1
onboot: 0
ostype: l26
scsi0: swsan3738lv:vm-13472-disk-0,aio=native,cache=none,detect_zeroes=1,discard=on,iothread=1,queues=2,size=70G
scsihw: virtio-scsi-single
smbios1: uuid=UUID
snaptime: 1780436041
tags: SOME-TAGS
vmgenid: 1f6ec138-c221-497e-ae8f-23d69c5c92d8I'm checking two VMs, one is already affected by this issue 13472 and another one is OK 13649.
And there are some checks I run after this fail to get the full view:
Code: Select all
#
# Affected VM 13472
#
pvesm status
unsupported storage of vg 'star_vg_fc_san_3738'
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
local dir disabled 0 0 0 N/A
local-lvm lvmthin disabled 0 0 0 N/A
san3739lv lvm active 16106123264 10011447296 6094675968 62.16%
swsan3738lv starlvm active 16106123264 4272119808 11834003456 26.52%
#
pvesm list swsan3738lv --vmid 13472
unsupported storage of vg 'star_vg_fc_san_3738'
Volid Format Type Size VMID
#
sudo /usr/sbin/lvscan | grep /star_vg_fc_san_3738 | grep 13472
inactive '/dev/star_vg_fc_san_3738/lvmth-13472' [70.00 GiB] inherit
inactive '/dev/star_vg_fc_san_3738/snap_vm-13472-disk-0_phase_2' [70.00 GiB] inherit
#
vgs; lvs -a -o vg_name,lv_name,lv_attr,lv_size,pool_lv,data_percent,metadata_percent | grep 13472
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- 277.87g 16.00g
star_vg_fc_san_3738 1 167 0 wz--n- <15.00t <10.96t
vg_fc_san_3739 1 55 0 wz--n- <15.00t <5.68t
star_vg_fc_san_3738 lvmth-13472 twi---tz-k 70.00g
star_vg_fc_san_3738 [lvmth-13472_tdata] Twi------- 70.00g
star_vg_fc_san_3738 [lvmth-13472_tmeta] ewi------- 72.00m
star_vg_fc_san_3738 snap_vm-13472-disk-0_phase_2 Vri---tz-k 70.00g lvmth-13472
#
ls -lah /dev/star_vg_fc_san_3738/ | grep 13472
-- none ---
#
ls -l /dev/mapper/ | grep 13472
-- none ---
#
vgchange -a y star_vg_fc_san_3738
18 logical volume(s) in volume group "star_vg_fc_san_3738" now active
#
lvs | grep 13472
lvmth-13472 star_vg_fc_san_3738 twi---tz-k 70.00g
snap_vm-13472-disk-0_phase_2 star_vg_fc_san_3738 Vri---tz-k 70.00g lvmth-13472
# It does not help me to check the disk itself, yet
mount /dev/star_vg_fc_san_3738/lvmth-13472 /mnt/pve/
mount /dev/star_vg_fc_san_3738/snap_vm-13472-disk-0_phase_2 /mnt/pve/
#
# Example of healthy VM 13649
#
pvesm list swsan3738lv --vmid 13649
unsupported storage of vg 'star_vg_fc_san_3738'
Volid Format Type Size VMID
swsan3738lv:vm-13649-disk-0 raw images 69793218560 13649
#
sudo /usr/sbin/lvscan | grep /star_vg_fc_san_3738 | grep 13649
ACTIVE '/dev/star_vg_fc_san_3738/lvmth-13649' [65.00 GiB] inherit
inactive '/dev/star_vg_fc_san_3738/snap_vm-13649-disk-0_phase_2' [65.00 GiB] inherit
ACTIVE '/dev/star_vg_fc_san_3738/vm-13649-disk-0' [65.00 GiB] inherit
#
vgs; lvs -a -o vg_name,lv_name,lv_attr,lv_size,pool_lv,data_percent,metadata_percent | grep 13649
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- 277.87g 16.00g
star_vg_fc_san_3738 1 167 0 wz--n- <15.00t <10.96t
vg_fc_san_3739 1 55 0 wz--n- <15.00t <5.68t
star_vg_fc_san_3738 lvmth-13649 twi-aotz-k 65.00g 30.48 20.12
star_vg_fc_san_3738 [lvmth-13649_tdata] Twi-ao---- 65.00g
star_vg_fc_san_3738 [lvmth-13649_tmeta] ewi-ao---- 68.00m
star_vg_fc_san_3738 snap_vm-13649-disk-0_phase_2 Vri---tz-k 65.00g lvmth-13649
star_vg_fc_san_3738 vm-13649-disk-0 Vwi-aotz-k 65.00g lvmth-13649 28.84
#
ls -l /dev/mapper/ | grep 13649
lrwxrwxrwx 1 root root 8 Jun 12 08:34 star_vg_fc_san_3738-lvmth--13649 -> ../dm-45
Code: Select all
star_vg_fc_san_3738 vm-13649-disk-0 Vwi-aotz-k 65.00g lvmth-13649 28.84So, if I understand correctly, using example of unaffected VM:
1 VM main disk as volume lvmth-13649 twi-aotz-k 65.00g
2 VM disk mounted volume as disk vm-13649-disk-0 Vwi-aotz-k 65.00g lvmth-13649
3 VM snapshot separate volume, mounted as disk? snap_vm-13649-disk-0_phase_2 Vri---tz-k 65.00g lvmth-13649
Affected VM somehow lost the mount #2 during the procudure of snapshot reverting:
1 VM main disk as volume lvmth-13472 twi---tz-k 70.00g
2 VM disk mounted LOST? no such logical volume star_vg_fc_san_3738/vm-13472-disk-0
3 VM snapshot separate volume, mounted as disk? snap_vm-13472-disk-0_phase_2 Vri---tz-k 70.00g lvmth-13472
Questions:
What am I doing wrong?
Is there a way to mount lost volume again, since I still have a snapshot and original disk of affected VM in place?
UPD: I'm still investigating the issue, trying to find a cause and to blame some network issues or improper setup, or concurrency, but I cannot find a strong evidence or a proper error in host logs.
It happens randomly and I cannot recreate the issue running snapshot reverting routine in loop for 100 times in parallel to normal routines. I tried switching this routine in thread-running case or in sequential execution: one vm after another - and it still happens.
UPD2: Trying to revert snapshot for affected VM throw error:
This is new vm, new ID, but the same issue.
Code: Select all
unsupported storage of vg 'star_vg_fc_san_3738'
TASK ERROR: lvremove 'star_vg_fc_san_3738/vm-19582-disk-0' error: Failed to find logical volume "star_vg_fc_san_3738/vm-19582-disk-0"This is something I'm trying to find workaround for too: how can I skip this error or make a dummy mount to fool Proxmox into thinking it actually "deleted" an old disk-0 and replaced it with snapshot version of VM disk?
UPD3: Just for fun I've created a fake volume to trick Proxmox into thinking it can delete this fake volume and assign snapshot disk back. But it does not work:
Code: Select all
qm unlock 19582
lvcreate -L 1M -n vm-19582-disk-0 star_vg_fc_san_3738
sudo /usr/sbin/lvscan | grep vm-19582-disk-0
inactive '/dev/star_vg_fc_san_3738/snap_vm-19582-disk-0_phase_2' [65.00 GiB] inherit
ACTIVE '/dev/star_vg_fc_san_3738/vm-19582-disk-0' [4.00 MiB] inheritSnapshot revert task returned:
Code: Select all
unsupported storage of vg 'star_vg_fc_san_3738'
Consider pruning star_vg_fc_san_3738 VG archive with more than 992 MiB in 7774 files (see archiving settings in lvm.conf).
Logical volume "vm-19582-disk-0" successfully removed.
Thin pool star_vg_fc_san_3738-lvmth--19582-tpool (252:90) transaction_id is 10, while expected 9.
TASK ERROR: lvm rollback 'star_vg_fc_san_3738/snap_vm-19582-disk-0_phase_2' error: Aborting. Failed to locally activate thin pool star_vg_fc_san_3738/lvmth-19582.Checking:
Code: Select all
lvconvert --repair star_vg_fc_san_3738/lvmth-19582
Consider pruning star_vg_fc_san_3738 VG archive with more than 992 MiB in 7779 files (see archiving settings in lvm.conf).
Consider pruning star_vg_fc_san_3738 VG archive with more than 992 MiB in 7780 files (see archiving settings in lvm.conf).
WARNING: LV star_vg_fc_san_3738/lvmth-19582_meta0 holds a backup of the unrepaired metadata. Use lvremove when no longer required.Thanks in advance