Fixing an Issue On Windows Server 2019 Hyper-V with Uneven Distribution of Available CPU Cores

Issue:

When guest-VMs are being migrated between Hyper-V Hosts within a cluster, CPU core scheduling seems to have been disproportionately distributed toward even-numbered cores (0,2,4,6,8, etc.)

Cause:

There appears to be a new type of processor scheduling named ‘core scheduler’ in Windows Server 2019 that succeeds the previous versions of Windows of ‘classic scheduler’. The difference between those two types of scheduling would affect how migrating VMs would be pinned toward certain numbered CPUs. In the screenshot above, odd-numbered CPU cores seem to be excluded from those migrated VMs.

Resolution:

Check VMHost Supported Versions (notice the IsDefault field):

PS C:\Windows\system32> Get-VMHostSupportedVersion

Name Version IsDefault
---- ------- ---------
Microsoft Windows 8.1/Server 2012 R2 5.0 False
Microsoft Windows 10 1507/Server 2016 Technical Preview 3 6.2 False
Microsoft Windows 10 1511/Server 2016 Technical Preview 4 7.0 False
Microsoft Windows Server 2016 Technical Preview 5 7.1 False
Microsoft Windows 10 Anniversary Update/Server 2016 8.0 False
Microsoft Windows 10 Creators Update 8.1 False
Microsoft Windows 10 Fall Creators Update/Server 1709 8.2 False
Microsoft Windows 10 April 2018 Update/Server 1803 8.3 False
Microsoft Windows 10 October 2018 Update/Server 2019 9.0 True

Check Guest VMs Configured Versions:

PS C:\Windows\system32> Get-VM | FT Name,Version,State

Name Version State
---- ------- -----
TESTVM01 8.0 Off
TESTVM02 8.0 Running
TESTVM03 9.0 Off
TESTVM04 9.0 Running

Perform ‘whole-sale’ Updates:
Note: only ‘offline’ guest VMs will be able to update. Any running VMs will throw errors upon invoking the Update-VMVersion command

PS C:\WINDOWS\system32> Get-VM | Update-VMVersion

Confirm
Are you sure you want to perform this action?
Performing a configuration version update of "TESTVM01" will prevent it from being migrated to or imported on previous
versions of Windows. This operation is not reversible.

[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): A
Update-VMVersion : The operation cannot be performed while the virtual machine is in its current state.
At line:1 char:10
+ Get-VM | Update-VMVersion
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (VirtualMachine ...-5cb7879f5c4f']:VirtualMachine) [Update-VMVersion],
VirtualizationException
+ FullyQualifiedErrorId : InvalidState,Microsoft.HyperV.PowerShell.Commands.UpdateVMVersion

Check VM Processor’s Hardware Thread Counts:

PS C:\WINDOWS\system32> Get-VM | Get-VMProcessor | FT VMName,HwThreadCountPerCore

VMName HwThreadCountPerCore
------ --------------------
TESTVM01 1
TESTVM01 1
TESTVM01 1
TESTVM01 1

Fix the CPU Core Scheduling Affinity:
Note: the below command only affects off-lined guest VMs

PS C:\WINDOWS\system32> Get-VM | Set-VMProcessor -HwThreadCountPerCore 0
Set-VMProcessor : Failed to modify device 'Processor'.
Cannot change the processor functionality of a virtual machine now.
'TESTVM02' failed to modify device 'Processor'. (Virtual machine ID C99B-4469-A0E7)
Cannot change the processor functionality of virtual machine 'TESTVM02' while it is running. (Virtual machine ID
C99B-4469-A0E7)
At line:1 char:10
+ Get-VM | Set-VMProcessor -HwThreadCountPerCore 0
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [Set-VMProcessor], VirtualizationException
+ FullyQualifiedErrorId : InvalidState,Microsoft.HyperV.PowerShell.Commands.SetVMProcessor

Generalized Approach:

To fix all guest VMs that have a lowered version of VM Host CPU support, each of the VM must be turned off. If that is not convenient due to production impacts, one could run this command to only target off-lined VMs:

Get-VM |?{$_.Version -lt 9.0}|Update-VMVersion -Force -EA SilentlyContinue|?{$_.State -eq 'Off'}|Set-VMProcessor -HwThreadCountPerCore 0

In our environment, the VM update command has no adverse effects on the performance of the machines thereafter. More importantly, the guest VMs have powered on without errors triggered by these changes. Also, it appears that version 9.0 guest VMs may have been patched to optimally allocate CPU resources. Hence, the Set-VMProcessor -HwThreadCountPerCore 0 may be unnecessary as of this writing. Still, setting that value as zero is recommended for consistency.

Result:

Screenshot of a more balanced CPU Core Scheduling

Leave a Reply

Your email address will not be published. Required fields are marked *