daknetworks.com

Task already in progress:

vim-cmd vmsvc/getallvms
vim-cmd vmsvc/power.getstate 1234 (vm_id)
vim-cmd vmsvc/get.tasklist 1234 (vm_id)
vim-cmd vimsvc/task_info haTask-1234-vim.task-here
(vim-cmd vimsvc/task_info haTask-1234-vim.VirtualMachine.removeAllSnapshots-869472401)

You can try to shutdown or power-off the VM, but this never worked for me:
vim-cmd vmsvc/power.shutdown VMID
vim-cmd vmsvc/power.off VMID

=================================
You can try to kill the VM, but this never worked for me:

esxcli vm process list
esxcli vm process kill -t=soft -w=WorldID
esxcli vm process kill -t=hard -w=WorldID
esxcli vm process kill -t=force -w=WorldID

Soft is the most graceful
Hard performs an immediate shutdown
Force should be used as a last resort

============================================
There might be a lock on one of the files:

cd ~
cd /vmfs/volumes/DATASTORE-NAME-HERE/VM-NAME-HERE/
(ie: cd /vmfs/volumes/MDL_64TB_0/DC-FL-02)

Or if you want the actual volume identifier:
find -iname VM-NAME-HERE
cd /vmfs/volumes/5f241452-2001c64a-3959-1c721d715751/VM-NAME-HERE

lsof |grep -i "VM-NAME-HERE"
ls |while read x; do vmfsfilelockinfo -p $x |grep -i "is locked"; done

If there is a lock from a MAC address, try to return the VM to the host that has the MAC address and consolidate the snapshots. Check to see if the lock is removed.

In some cases, I can log into the vm and gracefully shutdown the guest OS and that removes the lock.

If 2 mac addresses show, find that hosts that are the culprit but note that HA can handle multiple mac addresses gracefully:
esxcli network ip neighbor list

You can try to restart the following services, but this never worked for me:
/etc/init.d/vpxa stop
/etc/init.d/hostd stop
/etc/init.d/vpxa start
/etc/init.d/hostd start

Find any deltas:
ls -la /dev/deltadisks
(There should be no vmdk's here).

Shows mappings from device to uuid:
esxcli storage vmfs extent list

cd /var/log
less vmkernel.log |grep -i "VM-NAME-HERE"

cd /var/run/log
less vmkernel.log |grep -i "VM-NAME-HERE"

============================================
Finally just hard reboot the host via iDRAC.

Eventually found an error message:
There is no more space for virtual disk 'VM-NAME-HERE1_1-000003.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking _Retry. Click Cancel to terminate this session.

What is strange is that there is 40TB free on the Datastore-0.
Migrated vm data files to independent datastore; Datastore-4 with 123TB free.
Removed oldest snapshot.

In some situations, Veeam Backup software will create a snapshot prior to backing a VM. However, it fails. The snapshot will not show in the snapshot-manager.

Try to create a new snapshot and then choose the delete all. This should consolidate the snapshot chain.

However, if it keeps failing the consolidation, you might find that there is a .lck on it (see the instructions above).

Rebooting the host should unlock it. If it does not, stop all services associated to the backup software on the VM and try again.

Veeam locks the file when it fails. So, once the file is no longer locked, it should create a snapshot and then delete all.

NOTES:
https://kb.vmware.com/s/article/10051
https://kb.vmware.com/s/article/1004340
https://kb.vmware.com/s/article/84475

cd /vmfs/volumes/datastore_name/virtual_machine_name/