Virtual Machine Networking Error 15011

Creating New Logical Network

$logicalNetworkID="somehash-hash-hash"
$newNetworkName='Test Network'
$subnet="192.168.500.0/24"
$logicalNetwork = Get-SCLogicalNetwork -ID $logicalNetworkID

$vmNetwork = New-SCVMNetwork -Name $newNetworkName -LogicalNetwork $logicalNetwork -IsolationType "WindowsNetworkVirtualization" -CAIPAddressPoolType "IPV4" -PAIPAddressPoolType "IPV4"
Write-Output $vmNetwork
$subnet = New-SCSubnetVLan -Subnet $subnet
New-SCVMSubnet -Name $newNetworkName -VMNetwork $vmNetwork -SubnetVLan $subnet -EnableEncryption $false

Remove Logical Network in VMM

$scvmNetworkName='Test Network'
Get-SCVMNetwork -Name $scvmNetworkName |Remove-SCVMNetwork

Error Message

Remove-SCVMNetwork : VMM is unable to delete the VMNetwork 'Test Network' because other objects, such as VMSubnets, Load balancer templates and  Virtual network adapters depend on it. (Error ID: 15011)

Remove the VMNetwork association with all dependent resources, and then remove the VMNetwork.

To restart the job, run the following command:
PS> Restart-Job -Job (Get-VMMServer localhost | Get-Job | where { $_.ID -eq "{somehash-hash-hash}"})
At line:1 char:47
+ Get-SCVMNetwork -Name "Test Network"|Remove-SCVMNetwork
+                                               ~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ReadError: (:) [Remove-SCVMNetwork], CarmineException
    + FullyQualifiedErrorId : 15011,Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.RemoveSCVMNetworkCmdlet

Resolve Error 15011

Run Virtual Machine Manager Console > Navigate to 'VMs and services' >  'VM Networks' > right-click the Logical Network > Properties > Depedencies > Delete all subnet dependencies > OK > right-click the Logical Network again > Delete > OK

Virtual Machine Manager (VMM) Error ID: 1730

Symptom:
$vmName='bad-guestvm'
$vm = Get-SCVirtualMachine -Name $vmName
Read-SCVirtualMachine -VM $vm

Read-SCVirtualMachine : The selected action could not be completed because the virtual machine is not in a state in
which the action is valid. (Error ID: 1730)

Check the state of the virtual machine, and verify that the selected job can be run on a virtual machine in that state.

To restart the job, run the following command:
PS> Restart-Job -Job (Get-VMMServer localhost | Get-Job | where { $_.ID -eq "{442cfeda-2df8-4ee8-9c3d-f36390653124}"})
At line:1 char:1
+ Read-SCVirtualMachine -VM $vm
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ReadError: (:) [Read-SCVirtualMachine], CarmineException
    + FullyQualifiedErrorId : 1730,Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.RefreshVmCmdlet

Resolution: remove guest VM clustered role an then re-add it to the cluster
Resolution:
remove guest VM clustered role an then re-add it to the cluster

1. Remove Clustered Role

# Experimental: remove clustered roles via PowerShell
# Note: currently, these commands don't remove the roles properly; hence, the GUI method is necessary
$vmName='bad-guestvm0001'
$clusteredRoles=Get-ClusterResource -name *$vmName*
foreach($role in $clusteredRoles){
    $ownerNode=$role.OwnerNode
    $roleName=$role.Name
    write-warning "Remove-ClusterResource -Name $roleName -Force"
    pause
    if($env:computername -ne $role.OwnerNode){
        $session=new-pssession $ownerNode
        if($session.State -eq 'Opened'){            
            invoke-command -session $session {param($roleName)Remove-ClusterResource -Name $roleName -Force} -Args $roleName
            remove-pssession $session
        }
    }else{
        Remove-ClusterResource -Name $roleName -Force
    }
}

2. Re-add Guest VM to clustered role

# Required: console or RDP session onto the owner node

# Experimental: running this command via Remote WinRM has failed with this error
# WARNING: If you are running Windows PowerShell remotely, note that some failover clustering cmdlets do not work
# remotely. When possible, run the cmdlet locally and specify a remote computer as the target. To run the cmdlet
# remotely, try using the Credential Security Service Provider (CredSSP). All additional errors or warnings from this
# cmdlet might be caused by running it remotely.
# WARNING: You do not have administrative privileges on the cluster. Contact your network administrator to request
# access.
#     Access is denied
$vmName='bad-guestvm'
function addVmToCluster{
    param($vmNames,$targetCluster)
    $results=@()
    foreach ($vmName in $vmNames){
        try{
            #Start-VM -Name $vmName -EA Stop
            if(!$targetCluster){$targetCluster=(get-cluster -ea SilentlyContinue).Name}
            if($targetCluster){
                Add-ClusterVirtualMachineRole -Cluster $targetCluster -VirtualMachine $vmName -EA Stop
                $results+=[hashtable]@{$vmName=$true}
            }else{
                write-host "No clusters defined."
                $results+=[hashtable]@{$vmName=$false}
                }
        }catch{
            write-warning "$($error[0])"
            $results+=[hashtable]@{$vmName=$false}
            }
    }
    return $results
}

addVmToCluster $vmName

Repair with the Ignore Option:

Kubernetes Broken Due To Unknown Reasons

Problem 1: Admin User Unable to Login to Cluster via Controller (Master Node)

# SSL Error:
The connection to the server x.x.x.x:6443 was refused - did you specify the right host or port?

# Resolution to the SSL problem:
sudo -i
swapoff -a
exit
strace -eopenat kubectl version

# User privilege error:
kim@controller01:~$ kubectl cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
error: You must be logged in to the server (Unauthorized)

kim@controller01:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.10", GitCommit:"8152330a2b6ca3621196e62966ef761b8f5a61bb", GitTreeState:"clean", BuildDate:"2021-08-11T18:06:15Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
error: You must be logged in to the server (the server has asked for the client to provide credentials)

# Resolution to Admin user privilege error:
# Grant current user admin privileges on Kubernetes
# mkdir -p $HOME/.kube # this was done during previous setup
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config # this was done during previous setup

Problem 2: SSL Certificates Not Automatically Renewed

# Error message with kimconnect.com SSL Cert
Warning  Failed   84m (x328 over 13d)  cert-manager  The certificate request has failed to complete and will be retried: Failed to wait for order resource "kimconnect-cert-qlnl9-1800784958" to become ready: order is in "invalid" state:

kim@controller01:~$ k get certificaterequests.cert-manager.io
NAME                          READY   AGE
kimconnect-cert-jqlvf         True    90d
kimconnect-cert-qlnl9         False   30d

# Try to delete cert requests and secrets, and wait for cert to regenerate
k delete certificaterequests kimconnect-cert-qlnl9
k delete secret kimconnect-cert

# Try to force cert to renew before 1440 hours (immediately)
kubectl patch certificate kimconnect-cert --patch '
- op: replace
  path: /spec/renewBefore
  value: 1440h
' --type=json

# Wait for cert to become ready, then reverse the change
kubectl patch certificate kimconnect-cert --patch '
- op: remove
  path: /spec/renewBefore
' --type=json

# Worst case scenario, delete the cert and recreate it
k delete cert kimconnect-cert
cat <<EOF > kimconnect-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kimconnect-cert
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    acme.cert-manager.io/http01-edit-in-place: "true"
    kubernetes.io/tls-acme: "true"
spec:
  dnsNames:
    - kimconnect.com
    - www.kimconnect.com
  secretName: kimconnect-cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
EOF
kubectl apply -f kimconnect-cert.yaml

PowerShell: Move Virtual Machine Storage Using VMM

# moveVmStorageUsingVmm.ps1
# Version 0.01

$vmNames=@(
  'TESTVM0001',  
  'TESTVM0002',
  'TESTVM0003'
)
$storageLocations=@(
  'C:\ClusterStorage\BLOB001',
  'C:\ClusterStorage\BLOB002',
  'C:\ClusterStorage\BLOB003'
)
$storageMaxPercent=79
$confirmation=$true

function moveVmStorageUsingVmm($vmName,$newStorage,$storageMaxPercent=80){  
  try{
    $vmHosts=Get-SCVMHost
    $vm=Get-SCVirtualMachine -Name $vmName
    if($vm.count -eq 1){
      $currentHost=$vm.Hostname
      $storage=Get-SCStorageVolume -VMHost $currentHost|?{$_.Name -eq $newStorage}
      $capacity=$storage.Capacity
      $freespace=$storage.FreeSpace
      $storageUtilizedPercent=[math]::round(($capacity-$freespace)/$capacity*100,2)
      $totalSize=0
      $disks = Get-SCVirtualDiskDrive -VM $vmname
      $disks.VirtualHardDisk.Size|%{$totalSize+=$_}      
      $projectedPercent=[math]::round(($capacity-$freespace+$totalSize)/$capacity*100,2)
      write-host "$newStorage current utilization percentage: $storageUtilizedPercent`% and projected: $projectedPercent`% after adding $([math]::round($totalSize/1GB,2))GB's"
      $storageFeasible=if($projectedPercent -lt $storageMaxPercent){$true}else{$false}
      if($storageFeasible){
        $vmHost=$vmHosts|?{$_.Name -eq $currentHost}
        Move-SCVirtualMachine -VM $vm -VMHost $vmHost -Path $newStorage -UseLAN -UseDiffDiskOptimization # -RunAsynchronously
        return $true
      }else{
        write-warning "Infeasible storage location: Available storage volume is $storageMaxPercent`% and projected is $projectedPercent`%"
        return $false
      }
    }else{
      write-warning "$vmName matches more than 1 guest VM's; hence, this item is skipped."
      return $null
    }
  }catch{
    write-warning $_
    return $false
  }
}

function moveStorage($vmNames,$storageLocations,$storageMaxPercent,$confirmation){
  $storageIndex=0
  $useSameStorage=$true
  if($vmNames.count -gt 1){
    for($i=0;$i -lt $vmNames.count;$i++){
      $vmName=if($useSameStorage){$vmNames[$i]}else{$vmNames[--$i]}
      try{
        $null=Get-SCVirtualMachine -Name $vmName|Read-SCVirtualMachine -ea Stop
        $storageLocation=if($useSameStorage){$storageLocations[$storageIndex]}else{$storageLocations[++$storageIndex]}
        if($storageLocation){
          if($confirmation){
            write-host "Move $vmName storage to $storageLocation`?"
            pause
          }else{
            write-host "Moving $vmName storage to $storageLocation ..."
          }
          $useSameStorage=moveVmStorageUsingVmm $vmName $storageLocation $storageMaxPercent
        }else{
          write-warning "Exhausted storage locations to move VM's"
          return $false
        }
      }catch{
        write-warning $_
        # Get-SCVirtualMachine -Name $vmName|Repair-SCVirtualMachine -force # I haven't tested this
        return $false
      }
    }
  }else{
    [string]$vmname=$vmNames
    try{
      $null=Get-SCVirtualMachine -Name $vmname|Read-SCVirtualMachine -ea Stop
      foreach($storageLocation in $storageLocations){
        if($storageLocation){
          if($confirmation){
            write-host "Move $vmname storage to $storageLocation`?"
            pause
          }else{
            write-host "Moving $vmname storage to $storageLocation ..."
          }
          $success=moveVmStorageUsingVmm $vmName $storageLocation $storageMaxPercent
          if($true -eq $success){
            return $true
          }else{
            write-warning "$vmname cannot be moved to $storageLocation"
          }
        }else{
          write-warning "Exhausted storage locations to move VM's"
          return $false
        }
      }      
    }catch{
      write-warning $_
      return $false
    }
  }
}

moveStorage $vmNames $storageLocations $storageMaxPercent $confirmation

Get Hyper-V Cluster Automatic Balancing Configurations

# getHyperVLoadBalanceMode.ps1

$clustername=(Get-Cluster).Name
function getHyperVLoadBalanceMode($clustername=(Get-Cluster).Name){

    $AutoBalancerLevel=[hashtable]@{
        '1'='Low | Move when host is more than 80% loaded (default)'
        '2'='Medium | Move when host is more than 70% loaded'
        '3'='High | Average nodes and move when host is more than 5% above average'
    }
    
    $AutoBalancerMode=[hashtable]@{
        '0'='Disabled'
        '1'='Load balance on node joins'
        '2'='Load balance on node joins and every 30 minutes (default)'
    }
    try{
        $level=(Get-Cluster $clustername).AutoBalancerLevel.toString()
        $levelDescription=$AutoBalancerLevel[$level]
        $mode=(Get-Cluster $clustername).AutoBalancerMode.toString()
        $modeDescription=$AutoBalancerMode[$mode]
        write-host "Clustername: $clusterName`r`n - AutoBalanceLevel: $level $levelDescription`r`n - AutoBalanceMode: $mode $modeDescription"
    }catch{
        write-warning $_
    }

}

getHyperVLoadBalanceMode $clustername
# Sample output
PS C:\Windows\system32> getHyperVLoadBalanceMode
Clustername: cluster1
 - AutoBalanceLevel: 1 (default) Low Move when host is more than 80% loaded
 - AutoBalanceMode: 2 (default) Load balance on node join and every 30 minutes

PowerShell: Create Hyper-V Guest VM From Virtual Disk (VHDX)

Part 1: Creating Hyper-V Guest VM From a Virtual Disk

# createHyperVGuestVmFromDisk.ps1
# Version 0.02
# The intent of this script is to create a Hyper-V Guest VM basing on existing backup VHDX file(s)

$sourceVhdx='\\FILESERVER008\_Images\Windows2019_Image.vhdx'
$destinationFolder='\\VIRTUALMACHINES\STAGE'
$network='External-Connection'
$vlan='3000'

$newVmName='TESTVM009'
$memory='8GB'
$cpus=4
$secureBoot=$true # Windows: True, Linux: False
$extraDisks=@($null) # $extraDisks=@('test')
$generation=2
$deleteSourceDisks=$true
$onlineVm=$true

function createHyperVGuestVmFromDisk{
  param(
      $sourceVhdx,
      $newVmName,
      $destinationFolder,
      $memory='4GB',
      $cpus=2,
      $network,
      $vlan,
      $extraDisks=$null,
      $generation=2,
      $secureBoot=$false,
      $onlineVm=$true,
      $deleteSourceDisks=$false
  )
  $ErrorActionPreference = 'stop'
  
  function confirmation($content,$testValue="I confirm",$maxAttempts=3){
    $confirmed=$false
    $cancelCondition=@('cancel','no','exit','nope')
    $attempts=0
    clear-host 
    write-host $($content|out-string).trim()
    write-host "`r`nPlease review this content for accuracy.`r`n"
    while ($attempts -le $maxAttempts){
        if($attempts++ -ge $maxAttempts){
            write-host "A maximum number of attempts have reached. No confirmations received!`r`n"
            break;
            }
        $userInput = Read-Host -Prompt "Please type in this value => $testValue <= to confirm. Input CANCEL to skip this item";
        if ($userInput.ToLower() -eq $testValue.ToLower()){
            $confirmed=$true;
            write-host "Confirmed!`r`n";
            break;                
        }elseif($userInput.tolower() -in $cancelCondition){
            write-host 'Cancel command received.'
            $confirmed=$false
            break
        }else{
            clear-host
            $content|write-host
            write-host "Attempt number $attempts of $maxAttempts`: $userInput does not match $testValue. Try again or Input CANCEL to skip this item`r`n"
            }
        }
    return $confirmed
  }

  try{
      $newFolder="$destinationFolder\$newVmName"
      if(!(test-path $newFolder)){new-item -ItemType Directory -Path $newFolder -force}              
      $newVhdx="$newFolder\$newVmName`_disk0.vhdx"
      if(!(test-path $newVhdx)){
          New-Item -ItemType File -Path $newVhdx -Force # touch before copying contents
          Copy-Item -Path $sourceVhdx -Destination $newVhdx
      }else{
          write-warning "Volume $newVhdx already exists. Thus, that VMDK will be used instead of a clone."
      }
      New-VM -Name $newVmName `
              -MemoryStartupBytes $($memory/1) `
              -BootDevice VHD `
              -VHDPath $newVhdx `
              -Path $newFolder `
              -Generation $generation `
              -Switch $network         
      if($vlan){Set-VMNetworkAdapterVlan -VMName $newVmName -Access -VlanId $vlan}
              if($cpus -gt 1){Set-VMProcessor $newVmName -Count $cpus}
      Set-VMProcessor $newVmName -CompatibilityForMigrationEnabled $true
      if(!$secureBoot){Set-VMFirmware -VMName $newVmName -DisableSecureBoot}
      # Adding disks (optional)
      if($extraDisks){
          for($i=0;$i -lt $extraDisks.count;$i++){
            $extraDisk=$extraDisks[$i]
            $isvalidDisk=if(test-path $extraDisk){$extraDisk}else{$null}
            if($isvalidDisk){
              $newDiskPath=(join-path $destinationFolder $vmName) + "\$vmName`_disk$($i+1).vmdk"
              if(!(test-path $newDiskPath)){
                New-Item -ItemType File -Path $newDiskPath -Force # touch before copying contents
                Copy-Item -Path $extraDisk -Destination $newDiskPath
              }else{
                  write-warning "Volume $newDiskPath already exists. Thus, that VMDK will be used instead of a copy."
              }              
              Add-VMHardDiskDrive -VMName $newVmName -Path $newDiskPath 
            }else{
              write-warning "Disk path '$extraDisk' in invalid."
            }
          }
      }else{
        write-host "$newVmName has no extra disks to attach."
      }
      $disksToRemove=[array]$extraDisks+$sourceVhdx|?{$_} # join string to array and remove empty entries
      foreach($diskToRemove in $disksToRemove){
        $confirmed=confirmation "Remove source disk $diskToRemove"
        if($confirmed){
          remove-item $diskToRemove -force
        }else{
          write-host "$diskToRemove NOT removed."
        }
      }
      if($onlineVm){
          start-vm $newVmName
      }
      return $true
  }catch{
      write-warning "$($error[0])"
      return $false
  }
}

createHyperVGuestVmFromDisk $sourceVhdx `
  $newVmName `
  $destinationFolder `
  $memory `
  $cpus `
  $network `
  $vlan `
  $extraDisks `
  $generation `
  $secureBoot `
  $onlineVm `
  $deleteSourceDisk

Part 2: Adding New VM to Cluster

function addVmToCluster{
    param($vmNames,$targetCluster)
    $results=@()
    foreach ($vmName in $vmNames){
        try{
            #Start-VM -Name $vmName -EA Stop
            if(!$targetCluster){$targetCluster=(get-cluster -ea SilentlyContinue).Name}
            if($targetCluster){
                $null=Add-ClusterVirtualMachineRole -Cluster $targetCluster -VirtualMachine $vmName -EA Stop
                $results+=[hashtable]@{$vmName=$true}
            }else{
                write-host "No clusters defined."
                $results+=[hashtable]@{$vmName=$false}
                }
            $moved=if(get-cluster -ea SilentlyContinue){Move-ClusterVirtualMachineRole $newVmName}else{$false}
            if($moved){write-host "'$newVmname' has been moved to $($moved.OwnerNode)"}
        }catch{
            write-warning "$($error[0])"
            $results+=[hashtable]@{$vmName=$false}
            }
    }
    return $results
}

addVmToCluster $newVmName

Possible Error Message:

Microsoft Hyper-V UEFI

Virtual Machine Boot Summary

1.SCSI Disk (0,0) the boot loader did not load an operating system
2. Network adapter (00155D406142) a boot image was not found

No operating system was loaded. Your virtual machine may be configured incorrectly. Exit and rec-configure your VM or click restart to retry the current boot sequence again.

Resolution:

The case where this error has been thrown has been associated with an incorrect virtual machine generation type. Hence, the resolution has been:

A. Convert Generation 2 machine type back to Generation 1 as the original source disk VM must match its re-creation.
B. The misconfigured VM must be ‘deleted’ and re-created as a Generation 1 VM.

# Converting Generation 2 virtual disk to Gen 1
$diskFile='\\VIRTUALMACHINES\HyperV\originalDisk_gen2.vhdx'
$fixedFile='\\VIRTUALMACHINES\HyperV\originalDisk_gen1.vhdx'
Convert-VHD -Path $diskFile -DestinationPath $fixedFile -VHDType Dynamic

PowerShell: Find Guest VMs Associated with a Certain Storage Path

# findGuestMvsByStorage.ps1

$storagePath='\\SMBSERVER009'

function getAllGuestVms($clusterName){
  try{
    Import-Module FailoverClusters
    $clusterName=if($clusterName){
      $clustername
    }else{
      (get-cluster).name
    }
    $allHyperVHosts={(Get-ClusterNode -Cluster $clusterName|?{ $_.State -eq "Up" }).Name | %{$_.ToLower()}}.Invoke()
    $allVms=foreach ($hyperVHost in $allHyperVHosts){
      invoke-command -computername $hyperVHost -scriptblock{
        write-host "Getting VM List on $env:computername";
        Get-VM |select-object Name,State,Status,Path
      }|select-object * -ExcludeProperty RunspaceId,PSShowComputerName
    }
    if($allVms){return $allVms}else{return $null}
  }catch{
    write-warning $_
    return $false
  }
}
function findGuestMvsByStorage{
  param (
    $storagePath,
    $clusterName=$null
    )
  try{
      $allVms=getAllGuestVms $clusterName
      if(!$allVms){return $null}
      $matchedStorage=$allVms|?{$_.Path -like "*$storagePath*"}
      $online=$matchedStorage|?{$_.State.Value -like 'Running*'}
      $offline=$matchedStorage|?{$_.State.Value -notlike 'Running*'}
      if($online){
        write-host "There are $($online.count) online VMs associated with '$storagePath'"
      }
      if($offline){
        write-host "There are $($offline.count) offlined VMs associated with '$storagePath'"
      }
      if($matchedStorage){
        $matchedStorage
        return $matchedStorage
      }else{
        write-host "'$storagePath' is not being associated with any VM in cluster '$((get-cluster).Name)'"
        return $null
      }
  }catch{
      write-warning $_
      return $false
  }
}

$result=findGuestMvsByStorage $storagePath
$result|select Name,State,PSComputerName
# Sample Output
PS C:\Windows\system32> $result=findGuestMvsByStorage $storagePath
Getting VM List on HYV01
Getting VM List on HYV02
Getting VM List on HYV03
Getting VM List on HYV04
Getting VM List on HYV05
Getting VM List on HYV06
Getting VM List on HYV07
Getting VM List on HYV08
Getting VM List on HYV09
Getting VM List on HYV10
Getting VM List on HYV11
Getting VM List on HYV12
Getting VM List on HYV13
Getting VM List on HYV14
Getting VM List on HYV15
Getting VM List on HYV16
Getting VM List on HYV17
Getting VM List on HYV18
Getting VM List on HYV19
There are 6 online VMs associated with '\\FILESERVER009'
There are 16 offlined VMs associated with '\\FILESERVER009'

PS C:\Windows\system32> $result|select Name,State
Name                           State
----                           -----
TESTVM01                       OffCritical
TESTVM02                       Off
TESTVM03                       OffCritical
TESTVM04                       OffCritical
TESTVM05                       RunningCritical

PowerShell: Restart a Service on All Hyper-V Hosts of a Cluster

$serviceName='vmms'
$clusterName='HyperV-cluster001'

function restartServiceAllClusterNodes($service='vmms',$clusterName){
  function restartService($serviceName){
    $waitSeconds=40
    $isValidProcess=try{[bool](get-service $serviceName -EA Stop)}catch{$false}
    if($isValidProcess){
        try{
            $process=Start-Process -FilePath powershell.exe -ArgumentList "-Command Restart-Service $serviceName" -PassThru -NoNewWindow
            $process|Wait-Process -Timeout $waitSeconds -ErrorAction Stop
            return $true
        }catch{
            write-warning $_
            $process|Stop-Process -Force
            $processId=(get-process $serviceName).Id
            if($processId){
                write-host "Program now forcefully kills PID $processId of process $serviceName"
                $null=$processId|%{taskkill /f /pid $_} # works more reliably than Stop-Process $processName -Force
                Start-Service $serviceName -ErrorAction Ignore
                $started=$(try{get-service $serviceName}catch{$false})
                if($started){
                    write-host "'serviceName' status is now $($started.Status)"
                    return $true
                }else{
                    write-warning "'serviceName' status is $($started.Status)"
                    return $false
                }
            }else{
                write-warning "Service '$serviceName' PID not found."
                return $false        
            }
        }
    }
  }

  $results=@()
  try{
    Import-Module FailoverClusters
    $clusterName=if($clusterName){
      invoke-command -computername $clustername {(get-cluster).name}
    }else{
      (get-cluster).name
    }
    $allHyperVHosts={(Get-ClusterNode -Cluster $clusterName|?{ $_.State -eq "Up" }).Name | %{$_.ToLower()}}.Invoke()
    foreach ($hyperVHost in $allHyperVHosts){
      $result=invoke-command -computername $hyperVHost -EA SilentlyContinue -scriptblock {
        param($restartService,$serviceName)
        [scriptblock]::create($restartService).invoke($servicename)
      } -Args ${function:restartService},$serviceName
      write-host "$hypervHost=$result"
      $results+=[pscustomobject]@{$hypervHost=$result}
    }
  }catch{
    write-warning $_    
  }
}

$results=restartServiceAllClusterNodes $serviceName $clusterName
$results

PowerShell: Find Hyper-V Host by Guest VM Name

# findVmHostByGuestName.ps1

$vmName='TESTVM'

function findVmHostByGuestName($vmName){
    try{
        Import-Module Hyper-V
        Import-Module FailoverClusters
        $allHyperVHosts={(Get-ClusterNode | Where { $_.State -eq "Up" }).Name | %{$_.ToLower()}}.Invoke()
        $allVms=foreach ($hyperVHost in $allHyperVHosts){invoke-command -computername $hyperVHost -scriptblock{write-host "Getting VM List on $env:computername";Get-VM |select Name,Path}|select-object * -ExcludeProperty RunspaceId,PSShowComputerName}
        $matchedHost=$allVms|?{$_.Name -like "*$vmName*"}
        if($matchedHost){
            return $matchedHost
        }else{
            write-host "'$vmName' is not found in cluster '$((get-cluster).Name)'"
            return $null
        }
    }catch{
        write-warning $_
        return $false
    }
}

findVmHostByGuestName $vmName

PowerShell: Get All Hyper-V Host Spectre Patch Versions

# getAllVmSpectrePatchVersions.ps1

function getHyperVHostsInForest{
    function includeRSAT{
        $ErrorActionPreference='stop'
        [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
        #$rsatWindows7x32='https://download.microsoft.com/download/4/F/7/4F71806A-1C56-4EF2-9B4F-9870C4CFD2EE/Windows6.1-KB958830-x86-RefreshPkg.msu'
        $rsatWindows7x64='https://download.microsoft.com/download/4/F/7/4F71806A-1C56-4EF2-9B4F-9870C4CFD2EE/Windows6.1-KB958830-x64-RefreshPkg.msu'
        $rsatWindows81='https://download.microsoft.com/download/1/8/E/18EA4843-C596-4542-9236-DE46F780806E/Windows8.1-KB2693643-x64.msu'
        $rsat1709 = "https://download.microsoft.com/download/1/D/8/1D8B5022-5477-4B9A-8104-6A71FF9D98AB/WindowsTH-RSAT_WS_1709-x64.msu"
        $rsat1803 = "https://download.microsoft.com/download/1/D/8/1D8B5022-5477-4B9A-8104-6A71FF9D98AB/WindowsTH-RSAT_WS_1803-x64.msu"
        $rsatWs2016 = "https://download.microsoft.com/download/1/D/8/1D8B5022-5477-4B9A-8104-6A71FF9D98AB/WindowsTH-RSAT_WS2016-x64.msu"
   
        # This command does not work on Windows 2012R2
        #$releaseId=(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion" -Name ReleaseId).ReleaseId
        #Get-ItemProperty : Property ReleaseId does not exist at path HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows
        #NT\CurrentVersion.
        #At line:1 char:2
        #+ (Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion" -Na ...
        #+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        #    + CategoryInfo          : InvalidArgument: (ReleaseId:String) [Get-ItemProperty], PSArgumentException
        #    + FullyQualifiedErrorId : System.Management.Automation.PSArgumentException,Microsoft.PowerShell.Commands.GetItemPropertyCommand
   
        $releaseId=(Get-Item "HKLM:SOFTWARE\Microsoft\Windows NT\CurrentVersion").GetValue('ReleaseID')
        $osVersion=[System.Environment]::OSVersion.Version
        [double]$osVersionMajorMinor="$($osVersion.Major).$($osVersion.Minor)" 
        $osName=(Get-WmiObject Win32_OperatingSystem).Name
        #$osType=switch ((Get-CimInstance -ClassName Win32_OperatingSystem).ProductType){
        #    1 {'client'}
        #    2 {'domaincontroller'}
        #    3 {'memberserver'}
        #    }
   
        $windowsVersion=(Get-CimInstance Win32_OperatingSystem).Version
   
        switch ($releaseId){
            1607{write-host 'Windows Server 2016 Release 1607 detected';$link=$rsatWs2016;break}
            1709{write-host 'Windows Server 2016 Release 1709 detected';$link=$rsat1709;break}
            1803{write-host 'Windows Server 2016 Release 1803 detected';$link=$rsat1803}
        }
       
        switch ($osVersionMajorMinor){
            {$_ -eq 6.0}{write-host 'Windows Server 2008 or Windows Vista detected';$link=$rsat1709;break}
            {$_ -eq 6.1}{write-host 'Windows Server 2008 R2 or Windows 7 detected';$link=$rsatWindows7x64;break}
            {$_ -eq 6.2}{write-host 'Windows Server 2012 or Windows 8.1 detected';$link=$rsatWindows81;break}
            {$_ -eq 6.3}{write-host 'Windows Server 2012 R2 detected';$link=$rsatWindows81}
        }
  
        if (!(Get-Module -ListAvailable -Name ActiveDirectory -EA SilentlyContinue)){
            Write-host "Prerequisite checks: module ActiveDirectory NOT currently available on this system. Please wait while the program adds that plugin..."
            try{
                # If OS is Windows Server, then install RSAT using a different method
                if ($osName -match "^Microsoft Windows Server") {
                    # This sequence has confirmed to be valid on Windows Server 2008 R2 and above
                    Write-Verbose "Importing Windows Feature: RSAT-AD-PowerShell"
                    Import-Module ServerManager
                    Add-WindowsFeature RSAT-AD-PowerShell
                    }
                else{
                    Write-Verbose "This sequence targets Windows Client versions"
                    $destinationFile= ($ENV:USERPROFILE) + "\Downloads\" + (split-path $link -leaf)
                    Write-Host "Downloading RSAT from $link..."
                    Start-BitsTransfer -Source $link -Destination $destinationFile
                    $fileCheck=Get-AuthenticodeSignature $destinationFile
                    if($fileCheck.status -ne "valid") {write-host "$destinationFile is not valid. Please try again...";break}
                    $wusaCommand = $destinationFile + " /quiet"
                    Write-host "Installing RSAT - please wait..."
                    Start-Process -FilePath "C:\Windows\System32\wusa.exe" -ArgumentList $wusaCommand -Wait
                    }
                return $true
                }
            catch{
                write-warning "$($error[0].Exception)"
                return $false
                }
        }else{
            Write-host "Prerequisite checks: module ActiveDirectory IS currently available on this system." -ForegroundColor Green
            return $true
            }
    }
     function listAllHyperVNodes($verbose=$true){
        try{
            $timer=[System.Diagnostics.Stopwatch]::StartNew()
            $domains=(Get-ADForest).Name|%{(Get-ADForest -Identity $_).Name}
            foreach ($domain in $domains){
                #[string]$dc=(get-addomaincontroller -DomainName "$domain" -Discover -NextClosestSite).HostName
                write-host "Collecting all Hyper-V Clusters in $domain. This may take a while, depending on cluster sizes."
                $allClusters=(get-cluster -domain $domain).Name
                if($verbose){
                    $elapsed=[math]::round($timer.Elapsed.TotalMinutes,2)
                    write-host "Minutes elapsed $elapsed`: cluster names collected"
                    }

                $allHyperVNodes=@()
                foreach ($cluster in $allClusters){
                    $nodes=.{$x=Get-ClusterNode -Cluster $cluster -ea SilentlyContinue
                            if($x){
                                $x|Where-Object{$_.State -eq 'Up'}|Select-Object Name,@{name='Cluster';e={$cluster}}
                            }else{
                                $false
                            }
                            }
                    if($nodes){$allHyperVNodes+=$nodes}
                }
                if($verbose){
                    $elapsed=[math]::round($timer.Elapsed.TotalMinutes,2)
                    write-host "Minutes elapsed $elapsed`: Hyper Node names collected..."
                    }
                }                
            return $allHyperVNodes
        }catch{
            Write-Error $_
            return $false
            }
    }
 
    try{
        $null=includeRSAT;
        $hyperVHosts=listAllHyperVNodes
        #$hyperVHostNames=sortArrayStringAsNumbers $hyperVHosts
        $hyperVHostNames=$hyperVHosts|sort -property Cluster
        return $hyperVHostNames
    }catch{
        Write-Error $_
        return $false
        }
    }

function pickList($list){
    # Although it's more efficient to obtain the index and set it as display,
    # humans prefer see a list that starts with 1, instead of 0
    $display=for ($i=0;$i -lt $list.count;$i++){
        "$($i+1)`:`t$($list[$i])`r`n";
        }
    $lines=($display | Measure-Object -Line).Lines
    write-host $($display)
    $maxAttempts=3
    $attempts=0;
    while ($attempts -le $maxAttempts){
        if($attempts++ -ge $maxAttempts){
            write-host "Attempt number $maxAttempts of $maxAttempts. Exiting loop..`r`n"
            break;
            }
        $userInput = Read-Host -Prompt 'Please pick a number from the list above';
        try {
            $value=[int]$userInput;
            }catch{
                $value=-1;
                }
        if ($value -lt 1 -OR $value -gt $lines){
            cls;
            write-host "Attempt number $attempts of $maxAttempts`: $userInput is an invalid value. Try again..`r`n"
            write-host $display
            }else{
                $item=$list[$value-1];
                write-host "$userInput corresponds to $item`r`n";
                return $item
                }
        }     
    }
function selectCluster($clusters){
    # Requires function named pickList
    $uniqueClusters=$clusters|select -unique
    $pickedCluster=$(pickList $uniqueClusters)
    if($pickedCluster){
        return $pickedCluster
    }else{
        write-warning 'No clusternames were picked.'
        return $false
    }
}

function pickHost($hosts){
    # Requires function named pickList
    $pickedHost=$(pickList $hosts)
    if($pickedHost){
        return $pickedHost
    }else{
        write-warning 'No clusternames were picked.'
        return $false
    }
}
function sortArrayStringAsNumbers([string[]]$names){
    $hashTable=@{}
    foreach ($name in $names){
        #[int]$x=.{[void]($name -match '(?:.(\d+))+$');$matches[1]}
        #$x=.{[void]($name -match '(?:.(\d+)+)$');@($name.substring(0,$name.length-$matches[1].length),$matches[1])}
        $x=.{[void]($name -match '(?:.(\d+)+)$');($name.substring(0,$name.length-$matches[1].length))+$matches[1].PadLeft(8,'0')}
        $hashTable.Add($name,$x)
        }
    $sorted=foreach($item in $hashTable.GetEnumerator() | Sort Value){$item.Name}
    return $sorted
}

write-host "Obtaining cluster names and associated hosts..."
$hyperVHostsInForest=getHyperVHostsInForest
$pickedCluster=selectCluster $hyperVHostsInForest.Cluster
$pickedHosts=$hyperVHostsInForest|?{$_.Cluster -eq $pickedCluster}
$pickedHyperVHosts=sortArrayStringAsNumbers $pickedHosts.Name

function getSpectrePatchingVersions($hyperVHosts){
    write-host "Now obtaining Spectre Patching versions of Hyper-V Hosts..."
    $results=@{}
    foreach ($hyperVHost in $hyperVHosts){
        $spectrePatchingVersion=(Get-WmiObject -ComputerName $hyperVHost -ClassName Win32_BIOS).SMBIOSBIOSVersion
        write-host "$hyperVHost`: $spectrePatchingVersion"
        $results+=@{$hyperVHost=$spectrePatchingVersion}
    }
    return $results
}
getSpectrePatchingVersions $pickedHyperVHosts
function setVmMigrationPerformanceRemote($hyperVHosts){
    function setVmMigrationPerformance{
        param($performanceOption='TCPIP') # TCP is most compatible, and the other options are SMB and Compression
        $ErrorActionPreference='stop'
        try{
            Set-VMHost -VirtualMachineMigrationPerformanceOption $performanceOption
            return $true
        }catch{
            return $false
        }
    }
    $results=@{}
    $performanceOption='TCPIP'
    foreach ($hyperVHost in $hyperVHosts){
        try{
            $result=invoke-command -ComputerName $hyperVHost -ScriptBlock {
                param($setVmMigrationPerformance,$performanceOption)
                [scriptblock]::create($setVmMigrationPerformance).invoke($performanceOption)
            } -Args ${function:setVmMigrationPerformance},$performanceOption
            if($result){
                write-host "$hyperVHost`: $performanceOption"
                $results+=@{$hyperVHost=$performanceOption}
            }
        }catch{
            write-warning $_
        }
    }
    return $results
}
setVmMigrationPerformanceRemote $pickedHyperVHosts

PowerShell: Fix All VMs CPU Compatibility Setting

# fixAllVmCpuCompatibility.ps1
# version 0.01
function getHyperVHostsInForest{
    function includeRSAT{
        $ErrorActionPreference='stop'
        [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
        #$rsatWindows7x32='https://download.microsoft.com/download/4/F/7/4F71806A-1C56-4EF2-9B4F-9870C4CFD2EE/Windows6.1-KB958830-x86-RefreshPkg.msu'
        $rsatWindows7x64='https://download.microsoft.com/download/4/F/7/4F71806A-1C56-4EF2-9B4F-9870C4CFD2EE/Windows6.1-KB958830-x64-RefreshPkg.msu'
        $rsatWindows81='https://download.microsoft.com/download/1/8/E/18EA4843-C596-4542-9236-DE46F780806E/Windows8.1-KB2693643-x64.msu'
        $rsat1709 = "https://download.microsoft.com/download/1/D/8/1D8B5022-5477-4B9A-8104-6A71FF9D98AB/WindowsTH-RSAT_WS_1709-x64.msu"
        $rsat1803 = "https://download.microsoft.com/download/1/D/8/1D8B5022-5477-4B9A-8104-6A71FF9D98AB/WindowsTH-RSAT_WS_1803-x64.msu"
        $rsatWs2016 = "https://download.microsoft.com/download/1/D/8/1D8B5022-5477-4B9A-8104-6A71FF9D98AB/WindowsTH-RSAT_WS2016-x64.msu"
   
        # This command does not work on Windows 2012R2
        #$releaseId=(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion" -Name ReleaseId).ReleaseId
        #Get-ItemProperty : Property ReleaseId does not exist at path HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows
        #NT\CurrentVersion.
        #At line:1 char:2
        #+ (Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion" -Na ...
        #+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        #    + CategoryInfo          : InvalidArgument: (ReleaseId:String) [Get-ItemProperty], PSArgumentException
        #    + FullyQualifiedErrorId : System.Management.Automation.PSArgumentException,Microsoft.PowerShell.Commands.GetItemPropertyCommand
   
        $releaseId=(Get-Item "HKLM:SOFTWARE\Microsoft\Windows NT\CurrentVersion").GetValue('ReleaseID')
        $osVersion=[System.Environment]::OSVersion.Version
        [double]$osVersionMajorMinor="$($osVersion.Major).$($osVersion.Minor)" 
        $osName=(Get-WmiObject Win32_OperatingSystem).Name
        #$osType=switch ((Get-CimInstance -ClassName Win32_OperatingSystem).ProductType){
        #    1 {'client'}
        #    2 {'domaincontroller'}
        #    3 {'memberserver'}
        #    }
   
        $windowsVersion=(Get-CimInstance Win32_OperatingSystem).Version
   
        switch ($releaseId){
            1607{write-host 'Windows Server 2016 Release 1607 detected';$link=$rsatWs2016;break}
            1709{write-host 'Windows Server 2016 Release 1709 detected';$link=$rsat1709;break}
            1803{write-host 'Windows Server 2016 Release 1803 detected';$link=$rsat1803}
        }
       
        switch ($osVersionMajorMinor){
            {$_ -eq 6.0}{write-host 'Windows Server 2008 or Windows Vista detected';$link=$rsat1709;break}
            {$_ -eq 6.1}{write-host 'Windows Server 2008 R2 or Windows 7 detected';$link=$rsatWindows7x64;break}
            {$_ -eq 6.2}{write-host 'Windows Server 2012 or Windows 8.1 detected';$link=$rsatWindows81;break}
            {$_ -eq 6.3}{write-host 'Windows Server 2012 R2 detected';$link=$rsatWindows81}
        }
  
        if (!(Get-Module -ListAvailable -Name ActiveDirectory -EA SilentlyContinue)){
            Write-host "Prerequisite checks: module ActiveDirectory NOT currently available on this system. Please wait while the program adds that plugin..."
            try{
                # If OS is Windows Server, then install RSAT using a different method
                if ($osName -match "^Microsoft Windows Server") {
                    # This sequence has confirmed to be valid on Windows Server 2008 R2 and above
                    Write-Verbose "Importing Windows Feature: RSAT-AD-PowerShell"
                    Import-Module ServerManager
                    Add-WindowsFeature RSAT-AD-PowerShell
                    }
                else{
                    Write-Verbose "This sequence targets Windows Client versions"
                    $destinationFile= ($ENV:USERPROFILE) + "\Downloads\" + (split-path $link -leaf)
                    Write-Host "Downloading RSAT from $link..."
                    Start-BitsTransfer -Source $link -Destination $destinationFile
                    $fileCheck=Get-AuthenticodeSignature $destinationFile
                    if($fileCheck.status -ne "valid") {write-host "$destinationFile is not valid. Please try again...";break}
                    $wusaCommand = $destinationFile + " /quiet"
                    Write-host "Installing RSAT - please wait..."
                    Start-Process -FilePath "C:\Windows\System32\wusa.exe" -ArgumentList $wusaCommand -Wait
                    }
                return $true
                }
            catch{
                write-warning "$($error[0].Exception)"
                return $false
                }
        }else{
            Write-host "Prerequisite checks: module ActiveDirectory IS currently available on this system." -ForegroundColor Green
            return $true
            }
    }
     function listAllHyperVNodes($verbose=$true){
        try{
            $timer=[System.Diagnostics.Stopwatch]::StartNew()
            $domains=(Get-ADForest).Name|%{(Get-ADForest -Identity $_).Name}
            foreach ($domain in $domains){
                #[string]$dc=(get-addomaincontroller -DomainName "$domain" -Discover -NextClosestSite).HostName
                write-host "Collecting all Hyper-V Clusters in $domain. This may take a while, depending on cluster sizes."
                $allClusters=(get-cluster -domain $domain).Name
                if($verbose){
                    $elapsed=[math]::round($timer.Elapsed.TotalMinutes,2)
                    write-host "Minutes elapsed $elapsed`: cluster names collected"
                    }

                $allHyperVNodes=@()
                foreach ($cluster in $allClusters){
                    $nodes=.{$x=Get-ClusterNode -Cluster $cluster -ea SilentlyContinue
                            if($x){
                                $x|Where-Object{$_.State -eq 'Up'}|Select-Object Name,@{name='Cluster';e={$cluster}}
                            }else{
                                $false
                            }
                            }
                    if($nodes){$allHyperVNodes+=$nodes}
                }
                if($verbose){
                    $elapsed=[math]::round($timer.Elapsed.TotalMinutes,2)
                    write-host "Minutes elapsed $elapsed`: Hyper Node names collected..."
                    }
                }                
            return $allHyperVNodes
        }catch{
            Write-Error $_
            return $false
            }
    }
 
    try{
        $null=includeRSAT;
        $hyperVHosts=listAllHyperVNodes
        #$hyperVHostNames=sortArrayStringAsNumbers $hyperVHosts
        $hyperVHostNames=$hyperVHosts|sort -property Cluster
        return $hyperVHostNames
    }catch{
        Write-Error $_
        return $false
        }
    }

function pickList($list){
    # Although it's more efficient to obtain the index and set it as display,
    # humans prefer see a list that starts with 1, instead of 0
    $display=for ($i=0;$i -lt $list.count;$i++){
        "$($i+1)`:`t$($list[$i])`r`n";
        }
    $lines=($display | Measure-Object -Line).Lines
    write-host $($display)
    $maxAttempts=3
    $attempts=0;
    while ($attempts -le $maxAttempts){
        if($attempts++ -ge $maxAttempts){
            write-host "Attempt number $maxAttempts of $maxAttempts. Exiting loop..`r`n"
            break;
            }
        $userInput = Read-Host -Prompt 'Please pick a number from the list above';
        try {
            $value=[int]$userInput;
            }catch{
                $value=-1;
                }
        if ($value -lt 1 -OR $value -gt $lines){
            cls;
            write-host "Attempt number $attempts of $maxAttempts`: $userInput is an invalid value. Try again..`r`n"
            write-host $display
            }else{
                $item=$list[$value-1];
                write-host "$userInput corresponds to $item`r`n";
                return $item
                }
        }     
    }
function selectCluster($clusters){
    # Requires function named pickList
    $uniqueClusters=$clusters|select -unique
    $pickedCluster=$(pickList $uniqueClusters)
    if($pickedCluster){
        return $pickedCluster
    }else{
        write-warning 'No clusternames were picked.'
        return $false
    }
}

function pickHost($hosts){
    # Requires function named pickList
    $pickedHost=$(pickList $hosts)
    if($pickedHost){
        return $pickedHost
    }else{
        write-warning 'No clusternames were picked.'
        return $false
    }
}
function sortArrayStringAsNumbers([string[]]$names){
    $hashTable=@{}
    foreach ($name in $names){
        #[int]$x=.{[void]($name -match '(?:.(\d+))+$');$matches[1]}
        #$x=.{[void]($name -match '(?:.(\d+)+)$');@($name.substring(0,$name.length-$matches[1].length),$matches[1])}
        $x=.{[void]($name -match '(?:.(\d+)+)$');($name.substring(0,$name.length-$matches[1].length))+$matches[1].PadLeft(8,'0')}
        $hashTable.Add($name,$x)
        }
    $sorted=foreach($item in $hashTable.GetEnumerator() | Sort Value){$item.Name}
    return $sorted
}

function enableCpuCompatibility($vmName){
    $compatibilityForMigration=(Get-VMProcessor $vmName).CompatibilityForMigrationEnabled
    if(!$compatibilityForMigration){    
        $vmIsRunning=(get-vm $vmname).State -eq 'Running'
        if($vmIsRunning){stop-vm $vmName}
        Set-VMProcessor $vmName -CompatibilityForMigrationEnabled 1
        if($vmIsRunning){start-vm $vmName}
    }else{
        write-host "$vmName already has CPU CompatibilityForMigrationEnabled set to True"
    }
}

write-host "Obtaining cluster names and associated hosts..."
$hyperVHostsInForest=getHyperVHostsInForest
$pickedCluster=selectCluster $hyperVHostsInForest.Cluster
$pickedHosts=$hyperVHostsInForest|?{$_.Cluster -eq $pickedCluster}
$pickedHyperVHosts=sortArrayStringAsNumbers $pickedHosts.Name
$results=@()
foreach ($hyperVHost in $pickedHyperVHosts){
    write-host "Performing discovery on $hyperVhost..."
    $getVMCpuCompatiblity={
        Get-VMProcessor *|Select-Object VMName,CompatibilityForMigrationEnabled,@{name='Online';e={if((get-vm $_.VMName).State -eq 'Running'){$true}else{$false}}}
    }
    $result=invoke-command -ComputerName $hyperVHost -ScriptBlock $getVMCpuCompatiblity|Select-Object -Property * -ExcludeProperty RunspaceId
    write-host ($result|out-string).trim()
    $results+=$result
}
$negatives=$results|?{$_.CompatibilityForMigrationEnabled -eq $false -and $_.Online -eq $true}

# Fix all items
foreach ($item in $negatives){
    $vmHost=$item.PSComputerName
    $vmName=$item.VMName
    invoke-command -computername $vmHost -scriptblock {
        param($enableCpuCompatibility,$vmName)
        [scriptblock]::create($enableCpuCompatibility).invoke($vmName)
    } -Args ${function:enableCpuCompatibility},$vmName
}
# Fix each item
$vmName='TESTVM008'
$vmHost=($negatives|?{$_.VMName -eq $vmName}).PSComputerName
if($vmHost){
    invoke-command -computername $vmHost -scriptblock {
        param($vmName)$vmIsRunning=(get-vm $vmname).State -eq 'Running'
        if($vmIsRunning){stop-vm $vmName -Force}
        Set-VMProcessor $vmName -CompatibilityForMigrationEnabled 1
        if($vmIsRunning){start-vm $vmName}
    } -Args $vmName
}else{
    write-host "VMName $vmName doesn't match an existing record."
}

PowerShell: Search for Hyper-V Guest VM That Has Not Been Registered In Cluster

# findVmHost.ps1

$vmName='TESTVM01'

function findVmHost($vmName){
    try{
        Import-Module Hyper-V
        Import-Module FailoverClusters
        $allHyperVHosts={(Get-ClusterNode | Where { $_.State –eq "Up" }).Name | %{$_.ToLower()}}.Invoke()
        $allVms=foreach ($hyperVHost in $allHyperVHosts){invoke-command -computername $hyperVHost -scriptblock{write-host "Getting VM List on $env:computername";Get-VM |select Name,Path}|select-object * -ExcludeProperty RunspaceId,PSShowComputerName}
        $matchedHost=$allVms|?{$_.Name -like "*$vmName*"}
        if($matchedHost){
            return $matchedHost
        }else{
            write-host "'$vmName' is not found in cluster '$((get-cluster).Name)'"
            return $null
        }
    }catch{
        write-warning $_
        return $false
    }
}

findVmHost $vmName

Resolving Guest Virtual Machine Critical Status in Hyper-V Manager

Part A: Validating problem as VM in Critical Status

# Check VM Status in Hyper-V
$vmName='TESTVM.kimconnect.com'
get-vm $vmName

# Sample output
Name                      State       CPUUsage(%) MemoryAssigned(M) Uptime   Status
----                      -----       ----------- ----------------- ------   ------
TESTVM.kimconnect.com     OffCritical 0           0                 00:00:00 Cannot connect to virtual machine confi...

# Check storage locations
Get-VM $vmName | fl *Location, Path

# Sample output
CheckpointFileLocation : \\NAS\VMS\TESTVM.kimconnect.com
ConfigurationLocation  : \\NAS\VMS\TESTVM.kimconnect.com\config
SnapshotFileLocation   : \\NAS\VMS\TESTVM.kimconnect.com
Path                   : \\NAS\VMS\TESTVM.kimconnect.com\config

Part B: Refresh Hyper-V Virtual Machine Management Service

# Restart Virtual Machine Management Service from VMM Server
$vmName='TESTVM.kimconnect.com'
$vmInfo=Get-SCVirtualMachine -Name $vmName
#$badVmConfig=$vmInfo|?{$_.Status -eq 'IncompleteVMConfig'}
$currentHosts=$vmInfo.VMHost.FullyQualifiedDomainName

function restartService($serviceName){
    $waitSeconds=40
    $isValidProcess=try{[bool](get-service $serviceName -EA Stop)}catch{$false}
    if($isValidProcess){
        try{
            $process=Start-Process -FilePath powershell.exe -ArgumentList "-Command Restart-Service $serviceName" -PassThru -NoNewWindow
            $process|Wait-Process -Timeout $waitSeconds -ErrorAction Stop
            return $true
        }catch{
            write-warning $_
            $process|Stop-Process -Force
            $processId=(get-process $serviceName).Id
            if($processId){
                write-host "Program now forcefully kills PID $processId of process $serviceName"
                $null=$processId|%{taskkill /f /pid $_} # works more reliably than Stop-Process $processName -Force
                Start-Service $serviceName -ErrorAction Ignore
                $started=$(try{get-service $serviceName}catch{$false})
                if($started){
                    write-host "'serviceName' status is now $($started.Status)"
                    return $true
                }else{
                    write-warning "'serviceName' status is $($started.Status)"
                    return $false
                }
            }else{
                write-warning "Service '$serviceName' PID not found."
                return $false         
            }
        }
    }
}

$currentHosts|%{invoke-command -computername $_ {param($restartService,$serviceName);write-host "$env:computername";[scriptblock]::create($restartService).invoke($serviceName)}} -Args ${function:restartService},'vmms'

Part C: Guest VM shows as online

# Checking VM while login to Hyper-V Host
PS C:\Windows\system32> get-vm $vmName
get-vm : Hyper-V was unable to find a virtual machine with name "TESTVM.kimconnect.com".
At line:1 char:1
+ get-vm $vmName
+ ~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (TESTVM.kimconnect.com:String) [Get-VM], VirtualizationException
    + FullyQualifiedErrorId : InvalidParameter,Microsoft.HyperV.PowerShell.Commands.GetVM

# Checking VM while login to VMM
PS C:\Windows\system32> (Get-SCVirtualMachine -Name $vmName).VMHost.FullyQualifiedDomainName
HYPERV01
HYPERV07 (bad record)

# Find bad VM config on VMM Server
$vmName='TESTVM.kimconnect.com'
$vmInfo=Get-SCVirtualMachine -Name $vmName
$badVmConfig=$vmInfo|?{$_.Status -eq 'IncompleteVMConfig'}
$badVmConfig|write-host

# Example of failed repair
Repair-SCVirtualMachine : This action is not valid because virtual machine TESTVM.kimconnect.com is in state
Incomplete VM Configuration. The repair action can only be used on a virtual machine in a failed state. (Error ID: 693)

At line:1 char:1
+ Repair-SCVirtualMachine -VM $badRecord -Retry
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ReadError: (:) [Repair-SCVirtualMachine], CarmineException
    + FullyQualifiedErrorId : 693,Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.RepairVmCmdlet

PS C:\Windows\system32> refresh-vm $badRecord
refresh-vm : An internal error has occurred trying to contact the 'HYPERV01' server: NO_PARAM: NO_PARAM.
WinRM: URL: [http://HYPERV01:5985], Verb: [ENUMERATE], Resource:
[http://schemas.microsoft.com/wbem/wsman/1/wmi/root/microsoft/windows/storage/MSFT_Volume], Filter: []
 (Error ID: 2912, Detailed Error: The requested resource is in use (0x800700AA))

Check that WS-Management service is installed and running on server 'HYPERV01'. For more information
use the command "winrm helpmsg hresult". If 'HYPERV01' is a host/library/update server or a PXE server
role then ensure that VMM agent is installed and running. Refer to http://support.microsoft.com/kb/2742275 for more
details.

To restart the job, run the following command:
PS> Restart-Job -Job (Get-VMMServer localhost | Get-Job | where { $_.ID -eq "{26ce1506-da92-42a7-a400-705fe1008188}"})
At line:1 char:1
+ refresh-vm $badRecord
+ ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ReadError: (:) [Read-SCVirtualMachine], CarmineException
    + FullyQualifiedErrorId : 2912,Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.RefreshVmCmdlet

Fixing an Issue On Windows Server 2019 Hyper-V with Uneven Distribution of Available CPU Cores

Issue:

When guest-VMs are being migrated between Hyper-V Hosts within a cluster, CPU core scheduling seems to have been disproportionately distributed toward even-numbered cores (0,2,4,6,8, etc.)

Cause:

There appears to be a new type of processor scheduling named ‘core scheduler’ in Windows Server 2019 that succeeds the previous versions of Windows of ‘classic scheduler’. The difference between those two types of scheduling would affect how migrating VMs would be pinned toward certain numbered CPUs. In the screenshot above, odd-numbered CPU cores seem to be excluded from those migrated VMs.

Resolution:

Check VMHost Supported Versions (notice the IsDefault field):

PS C:\Windows\system32> Get-VMHostSupportedVersion

Name Version IsDefault
---- ------- ---------
Microsoft Windows 8.1/Server 2012 R2 5.0 False
Microsoft Windows 10 1507/Server 2016 Technical Preview 3 6.2 False
Microsoft Windows 10 1511/Server 2016 Technical Preview 4 7.0 False
Microsoft Windows Server 2016 Technical Preview 5 7.1 False
Microsoft Windows 10 Anniversary Update/Server 2016 8.0 False
Microsoft Windows 10 Creators Update 8.1 False
Microsoft Windows 10 Fall Creators Update/Server 1709 8.2 False
Microsoft Windows 10 April 2018 Update/Server 1803 8.3 False
Microsoft Windows 10 October 2018 Update/Server 2019 9.0 True

Check Guest VMs Configured Versions:

PS C:\Windows\system32> Get-VM | FT Name,Version,State

Name Version State
---- ------- -----
TESTVM01 8.0 Off
TESTVM02 8.0 Running
TESTVM03 9.0 Off
TESTVM04 9.0 Running

Perform ‘whole-sale’ Updates:
Note: only ‘offline’ guest VMs will be able to update. Any running VMs will throw errors upon invoking the Update-VMVersion command

PS C:\WINDOWS\system32> Get-VM | Update-VMVersion

Confirm
Are you sure you want to perform this action?
Performing a configuration version update of "TESTVM01" will prevent it from being migrated to or imported on previous
versions of Windows. This operation is not reversible.

[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): A
Update-VMVersion : The operation cannot be performed while the virtual machine is in its current state.
At line:1 char:10
+ Get-VM | Update-VMVersion
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (VirtualMachine ...-5cb7879f5c4f']:VirtualMachine) [Update-VMVersion],
VirtualizationException
+ FullyQualifiedErrorId : InvalidState,Microsoft.HyperV.PowerShell.Commands.UpdateVMVersion

Check VM Processor’s Hardware Thread Counts:

PS C:\WINDOWS\system32> Get-VM | Get-VMProcessor | FT VMName,HwThreadCountPerCore

VMName HwThreadCountPerCore
------ --------------------
TESTVM01 1
TESTVM01 1
TESTVM01 1
TESTVM01 1

Fix the CPU Core Scheduling Affinity:
Note: the below command only affects off-lined guest VMs

PS C:\WINDOWS\system32> Get-VM | Set-VMProcessor -HwThreadCountPerCore 0
Set-VMProcessor : Failed to modify device 'Processor'.
Cannot change the processor functionality of a virtual machine now.
'TESTVM02' failed to modify device 'Processor'. (Virtual machine ID C99B-4469-A0E7)
Cannot change the processor functionality of virtual machine 'TESTVM02' while it is running. (Virtual machine ID
C99B-4469-A0E7)
At line:1 char:10
+ Get-VM | Set-VMProcessor -HwThreadCountPerCore 0
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [Set-VMProcessor], VirtualizationException
+ FullyQualifiedErrorId : InvalidState,Microsoft.HyperV.PowerShell.Commands.SetVMProcessor

Generalized Approach:

To fix all guest VMs that have a lowered version of VM Host CPU support, each of the VM must be turned off. If that is not convenient due to production impacts, one could run this command to only target off-lined VMs:

Get-VM |?{$_.Version -lt 9.0}|Update-VMVersion -Force -EA SilentlyContinue|?{$_.State -eq 'Off'}|Set-VMProcessor -HwThreadCountPerCore 0

In our environment, the VM update command has no adverse effects on the performance of the machines thereafter. More importantly, the guest VMs have powered on without errors triggered by these changes. Also, it appears that version 9.0 guest VMs may have been patched to optimally allocate CPU resources. Hence, the Set-VMProcessor -HwThreadCountPerCore 0 may be unnecessary as of this writing. Still, setting that value as zero is recommended for consistency.

Result:

Screenshot of a more balanced CPU Core Scheduling

PowerShell: Add New Virtual Disk to Existing Guest VM in Hyper-V

# Adding disks (optional)
$newVMNames='TestWindows2019'
$extraDiskSize='200GB'

if($extraDiskSize){
    foreach($newVmName in $newVMNames){
        STOP-VM -vmname $newVmName
        $diskFile=(join-path $destinationFolder $newVmName) + "\$newVmName`_disk1.vhdx"
        NEW-VHD -Fixed $diskFile -SizeBytes (Invoke-Expression $extraDiskSize) -ea Stop
        # Preempt error by adding (Invoke-Expression $sizeBytes)
        # New-VHD : Cannot bind parameter 'SizeBytes'. Cannot convert value "200GB" to type "System.UInt64". Error: "Input
        # string was not in a correct format."
        # At line:1 char:45
        # +         NEW-VHD -Fixed $diskFile -SizeBytes $extraDiskSize -ea Stop
        # +                                             ~~~~~~~~~~~~~~
        #     + CategoryInfo          : InvalidArgument: (:) [New-VHD], ParameterBindingException
        #     + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.Vhd.PowerShell.Cmdlets.NewVhd
        Add-VMHardDiskDrive -VMName $newVmName -Path $diskFile
    }
}

How To Upgrade NextCloud 22.1.1 to 22.2.0 When Deployed with Kubernetes & Helm

Step 1:

Navigate to nextcloud > html > edit version.php

<?php 
$OC_Version = array(22,1,1,2);
$OC_VersionString = '22.1.1';
$OC_Edition = '';
$OC_Channel = 'stable';
$OC_VersionCanBeUpgradedFrom = array (
  'nextcloud' => 
  array (
    '21.0' => true,
    '22.0' => true,
    '22.1' => true,
    '22.2' => true,   # Add this line 
  ),
  'owncloud' => 
  array (
    '10.5' => true,
  ),
);
$OC_Build = '2021-08-26T13:27:46+00:00 1eea64f2c3eb0e110391c24830cea5f8d9c3e6a1';
$vendor = 'nextcloud';

Step 2: Run the ‘helm upgrade…’ command with the desired NextCloud version

# Example:
helm upgrade nextcloud nextcloud/nextcloud \
  --set image.tag=22.2.0-fpm \
  --set nginx.enabled=true \
  --set nextcloud.host=dragoncoin.com \
  --set nextcloud.username=dragon,nextcloud.password=SOMEVERYCOMPLEXANDVERYVERYLONGPASSWORD \
  --set internalDatabase.enabled=false \
  --set externalDatabase.existingSecret.enabled=true \
  --set externalDatabase.type=postgresql \
  --set externalDatabase.host='nextcloud-db-postgresql.default.svc.cluster.local' \
  --set persistence.enabled=true \
  --set persistence.existingClaim=nextcloud-claim \
  --set persistence.size=100Ti \
  --set livenessProbe.enabled=false \
  --set readinessProbe.enabled=false \
  --set nextcloud.phpConfigs.upload_max_size=40G \
  --set nextcloud.phpConfigs.upload_max_filesize=40G \
  --set nextcloud.phpConfigs.post_max_size=40G \
  --set nextcloud.phpConfigs.memory_limit=80G

Step 3: Check the logs and wait for the upgrading process to complete

Previous pods terminated to make way for new pods

admin@controller:~$ k get pod
NAME                                              READY   STATUS        RESTARTS   AGE
nextcloud-67855fc94c-lc2xr                        0/2     Terminating   0          74m
nextcloud-db-postgresql-0                         1/1     Running       0          91m
admin@controller:~$ k get pod
NAME                                              READY   STATUS    RESTARTS   AGE
nextcloud-79b5b775fd-2s4bj                        2/2     Running   0          56s
nextcloud-db-postgresql-0                         1/1     Running   0          92m

Expected 502 errors during pod upgrades

admin@controller:~$ k logs nextcloud-79b5b775fd-2s4bj nextcloud-nginx
2021/11/01 05:36:49 [error] 32#32: *24 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.95, server: , request: "GET /status.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "dragoncoin.com"
10.10.0.95 - dragon [01/Nov/2021:05:36:49 +0000] "GET /status.php HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Linux) mirall/3.2.2git (build 5903) (Nextcloud, linuxmint-5.4.0-89-generic ClientArchitecture: x86_64 OsArchitecture: x86_64)" "192.168.0.164"

Logs showing that the upgrading process has progressed… and eventually completed

admin@controller:~$ kubectl logs nextcloud-79b5b775fd-2s4bj nextcloud

Initializing nextcloud 22.2.0.2 ...
Upgrading nextcloud from 22.1.1.2 ...
Initializing finished
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Setting log level to debug
Turned on maintenance mode
Updating database schema
Updated database
Updating <lookup_server_connector> ...
Updated <lookup_server_connector> to 1.10.0
Updating <oauth2> ...
Updated <oauth2> to 1.10.0
Updating <files> ...
Updated <files> to 1.17.0
Updating <cloud_federation_api> ...
Updated <cloud_federation_api> to 1.5.0
Updating <dav> ...
Fix broken values of calendar objects

 Starting ...

Updated <dav> to 1.19.0
Updating <files_sharing> ...
Updated <files_sharing> to 1.14.0
Updating <files_trashbin> ...
Updated <files_trashbin> to 1.12.0
Updating <files_versions> ...
Updated <files_versions> to 1.15.0
Updating <sharebymail> ...
Updated <sharebymail> to 1.12.0
Updating <workflowengine> ...
Updated <workflowengine> to 2.4.0
Updating <systemtags> ...
Updated <systemtags> to 1.12.0
Updating <theming> ...
Updated <theming> to 1.13.0
Updating <accessibility> ...
Migrate old user config

    0/0 [>---------------------------]   0% Starting ...
    0/0 [->--------------------------]   0%
 Starting ...

Updated <accessibility> to 1.8.0
Updating <contactsinteraction> ...
Updated <contactsinteraction> to 1.3.0
Updating <federatedfilesharing> ...
Updated <federatedfilesharing> to 1.12.0
Updating <provisioning_api> ...
Updated <provisioning_api> to 1.12.0
Updating <settings> ...
Updated <settings> to 1.4.0
Updating <twofactor_backupcodes> ...
Updated <twofactor_backupcodes> to 1.11.0
Updating <updatenotification> ...
Updated <updatenotification> to 1.12.0
Updating <user_status> ...
Updated <user_status> to 1.2.0
Updating <weather_status> ...
Updated <weather_status> to 1.2.0
Checking for update of app accessibility in appstore
Checked for update of app "accessibility" in App Store
Checking for update of app activity in appstore
Checked for update of app "activity" in App Store
Checking for update of app audioplayer in appstore
Checked for update of app "audioplayer" in App Store
Checking for update of app breezedark in appstore
Checked for update of app "breezedark" in App Store
Checking for update of app bruteforcesettings in appstore
Checked for update of app "bruteforcesettings" in App Store
Checking for update of app camerarawpreviews in appstore
Checked for update of app "camerarawpreviews" in App Store
Checking for update of app cloud_federation_api in appstore
Checked for update of app "cloud_federation_api" in App Store
Checking for update of app cms_pico in appstore
Checked for update of app "cms_pico" in App Store
Checking for update of app contactsinteraction in appstore
Checked for update of app "contactsinteraction" in App Store
Checking for update of app dav in appstore
Checked for update of app "dav" in App Store
Checking for update of app documentserver_community in appstore
Checked for update of app "documentserver_community" in App Store
Checking for update of app drawio in appstore
Checked for update of app "drawio" in App Store
Checking for update of app external in appstore
Checked for update of app "external" in App Store
Checking for update of app federatedfilesharing in appstore
Checked for update of app "federatedfilesharing" in App Store
Checking for update of app files in appstore
Checked for update of app "files" in App Store
Checking for update of app files_antivirus in appstore
Checked for update of app "files_antivirus" in App Store
Checking for update of app files_markdown in appstore
Checked for update of app "files_markdown" in App Store
Checking for update of app files_mindmap in appstore
Checked for update of app "files_mindmap" in App Store
Checking for update of app files_pdfviewer in appstore
Checked for update of app "files_pdfviewer" in App Store
Checking for update of app files_rightclick in appstore
Checked for update of app "files_rightclick" in App Store
Checking for update of app files_sharing in appstore
Checked for update of app "files_sharing" in App Store
Checking for update of app files_trashbin in appstore
Checked for update of app "files_trashbin" in App Store
Checking for update of app files_versions in appstore
Checked for update of app "files_versions" in App Store
Checking for update of app files_videoplayer in appstore
Checked for update of app "files_videoplayer" in App Store
Checking for update of app forms in appstore
Checked for update of app "forms" in App Store
Checking for update of app logreader in appstore
Checked for update of app "logreader" in App Store
Checking for update of app lookup_server_connector in appstore
Checked for update of app "lookup_server_connector" in App Store
Checking for update of app maps in appstore
Checked for update of app "maps" in App Store
Checking for update of app music in appstore
Checked for update of app "music" in App Store
Checking for update of app news in appstore
Checked for update of app "news" in App Store
Checking for update of app notifications in appstore
Checked for update of app "notifications" in App Store
Checking for update of app oauth2 in appstore
Checked for update of app "oauth2" in App Store
Checking for update of app password_policy in appstore
Checked for update of app "password_policy" in App Store
Checking for update of app photos in appstore
Checked for update of app "photos" in App Store
Checking for update of app privacy in appstore
Checked for update of app "privacy" in App Store
Checking for update of app provisioning_api in appstore
Checked for update of app "provisioning_api" in App Store
Checking for update of app quicknotes in appstore
Checked for update of app "quicknotes" in App Store
Checking for update of app recommendations in appstore
Checked for update of app "recommendations" in App Store
Checking for update of app registration in appstore
Checked for update of app "registration" in App Store
Checking for update of app richdocuments in appstore
Checked for update of app "richdocuments" in App Store
Checking for update of app serverinfo in appstore
Checked for update of app "serverinfo" in App Store
Checking for update of app settings in appstore
Checked for update of app "settings" in App Store
Checking for update of app sharebymail in appstore
Checked for update of app "sharebymail" in App Store
Checking for update of app spreed in appstore
Checked for update of app "spreed" in App Store
Checking for update of app support in appstore
Checked for update of app "support" in App Store
Checking for update of app survey_client in appstore
Checked for update of app "survey_client" in App Store
Checking for update of app systemtags in appstore
Checked for update of app "systemtags" in App Store
Checking for update of app tasks in appstore
Checked for update of app "tasks" in App Store
Checking for update of app text in appstore
Checked for update of app "text" in App Store
Checking for update of app theming in appstore
Checked for update of app "theming" in App Store
Checking for update of app twofactor_backupcodes in appstore
Checked for update of app "twofactor_backupcodes" in App Store
Checking for update of app updatenotification in appstore
Checked for update of app "updatenotification" in App Store
Checking for update of app user_status in appstore
Checked for update of app "user_status" in App Store
Checking for update of app video_converter in appstore
Checked for update of app "video_converter" in App Store
Checking for update of app viewer in appstore
Checked for update of app "viewer" in App Store
Checking for update of app weather_status in appstore
Checked for update of app "weather_status" in App Store
Checking for update of app workflowengine in appstore
Checked for update of app "workflowengine" in App Store
Starting code integrity check...

After about 5 minutes (depending on the system hardware), NextCloud should be rendered back online. At this point, the upgrade has completed.

Kubernetes Ingress Error 502 Upon NextCloud Upgrades

Issue:
Just the other day, I’ve attempted to run a ‘helm upgrade…’ command on my NextCloud application. I’ve taken care to ensure that the container’s version matches that of the persistent storage’s marking (e.g. image.tag=22.1-fpm) as a variance in that would cause NextCloud not to start. However, there’s another issue that has puzzled me: a 502 Error upon navigating to the URL of the application.

Resolution:
– Check the logs
– Review Kubernetes Ingress documentation
– Realize that this specific issue requires no fixing

Checking the logs:

admin@controller:~$ k logs nextcloud-67855fc94c-lc2xr nextcloud-nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2021/11/01 04:18:37 [error] 34#34: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.16.90.192, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "dragoncoin.com"
... Truncated for brevity ...
2021/11/01 04:34:20 [error] 34#34: *155 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.100.95, server: , request: "GET /apps/photos/service-worker.js HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "dragoncoin.com", referrer: "https://dragoncoin.com/apps/photos/service-worker.js"
172.16.100.95 - - [01/Nov/2021:04:34:20 +0000] "GET /apps/photos/service-worker.js HTTP/1.1" 502 559 "https://dragoncoin.com/apps/photos/service-worker.js" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36" "172.16.100.164"
admin@controller:~$ k logs nextcloud-67855fc94c-lc2xr nextcloud
Initializing nextcloud 22.1.1.2 ...
Upgrading nextcloud from 22.1.0.1 ...
Initializing finished
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Setting log level to debug
Turned on maintenance mode
Updating database schema
Updated database
Updating <workflowengine> ...
Updated <workflowengine> to 2.3.1
Checking for update of app accessibility in appstore
Checked for update of app "accessibility" in App Store
Checking for update of app activity in appstore
Checked for update of app "activity" in App Store
Checking for update of app audioplayer in appstore
Update app audioplayer from App Store
Checked for update of app "audioplayer" in App Store
Checking for update of app breezedark in appstore
Update app breezedark from App Store
Checked for update of app "breezedark" in App Store
Checking for update of app bruteforcesettings in appstore
Checked for update of app "bruteforcesettings" in App Store
Checking for update of app camerarawpreviews in appstore
Checked for update of app "camerarawpreviews" in App Store
Checking for update of app cloud_federation_api in appstore
Checked for update of app "cloud_federation_api" in App Store
Checking for update of app cms_pico in appstore
Update app cms_pico from App Store
Repair warning: Replacing Pico CMS config file "config.yml.template"
Repair warning: Replacing Pico CMS system template "empty"
Repair warning: Replacing Pico CMS system template "sample_pico"
Repair warning: Replacing Pico CMS system theme "default"
Repair warning: Replacing Pico CMS system plugin "PicoDeprecated"
Checked for update of app "cms_pico" in App Store
Checking for update of app contactsinteraction in appstore
Checked for update of app "contactsinteraction" in App Store
Checking for update of app dav in appstore
Checked for update of app "dav" in App Store
Checking for update of app documentserver_community in appstore
Checked for update of app "documentserver_community" in App Store
Checking for update of app drawio in appstore
Checked for update of app "drawio" in App Store
Checking for update of app external in appstore
Checked for update of app "external" in App Store
Checking for update of app federatedfilesharing in appstore
Checked for update of app "federatedfilesharing" in App Store
Checking for update of app files in appstore
Checked for update of app "files" in App Store
Checking for update of app files_antivirus in appstore
Update app files_antivirus from App Store
Checked for update of app "files_antivirus" in App Store
Checking for update of app files_markdown in appstore
Checked for update of app "files_markdown" in App Store
Checking for update of app files_mindmap in appstore
Checked for update of app "files_mindmap" in App Store
Checking for update of app files_pdfviewer in appstore
Checked for update of app "files_pdfviewer" in App Store
Checking for update of app files_rightclick in appstore
Checked for update of app "files_rightclick" in App Store
Checking for update of app files_sharing in appstore
Checked for update of app "files_sharing" in App Store
Checking for update of app files_trashbin in appstore
Checked for update of app "files_trashbin" in App Store
Checking for update of app files_versions in appstore
Checked for update of app "files_versions" in App Store
Checking for update of app files_videoplayer in appstore
Checked for update of app "files_videoplayer" in App Store
Checking for update of app forms in appstore
Checked for update of app "forms" in App Store
Checking for update of app logreader in appstore
Checked for update of app "logreader" in App Store
Checking for update of app lookup_server_connector in appstore
Checked for update of app "lookup_server_connector" in App Store
Checking for update of app maps in appstore
Checked for update of app "maps" in App Store
Checking for update of app music in appstore
Update app music from App Store
Checked for update of app "music" in App Store
Checking for update of app news in appstore
Update app news from App Store
Checked for update of app "news" in App Store
Checking for update of app notifications in appstore
Checked for update of app "notifications" in App Store
Checking for update of app oauth2 in appstore
Checked for update of app "oauth2" in App Store
Checking for update of app password_policy in appstore
Checked for update of app "password_policy" in App Store
Checking for update of app photos in appstore
Checked for update of app "photos" in App Store
Checking for update of app privacy in appstore
Checked for update of app "privacy" in App Store
Checking for update of app provisioning_api in appstore
Checked for update of app "provisioning_api" in App Store
Checking for update of app quicknotes in appstore
Checked for update of app "quicknotes" in App Store
Checking for update of app recommendations in appstore
Checked for update of app "recommendations" in App Store
Checking for update of app registration in appstore
Checked for update of app "registration" in App Store
Checking for update of app richdocuments in appstore
Update app richdocuments from App Store
Checked for update of app "richdocuments" in App Store
Checking for update of app serverinfo in appstore
Checked for update of app "serverinfo" in App Store
Checking for update of app settings in appstore
Checked for update of app "settings" in App Store
Checking for update of app sharebymail in appstore
Checked for update of app "sharebymail" in App Store
Checking for update of app spreed in appstore
Update app spreed from App Store
Checked for update of app "spreed" in App Store
Checking for update of app support in appstore
Checked for update of app "support" in App Store
Checking for update of app survey_client in appstore
Checked for update of app "survey_client" in App Store
Checking for update of app systemtags in appstore
Checked for update of app "systemtags" in App Store
Checking for update of app tasks in appstore
Checked for update of app "tasks" in App Store
Checking for update of app text in appstore
Checked for update of app "text" in App Store
Checking for update of app theming in appstore
Checked for update of app "theming" in App Store
Checking for update of app twofactor_backupcodes in appstore
Checked for update of app "twofactor_backupcodes" in App Store
Checking for update of app updatenotification in appstore
Checked for update of app "updatenotification" in App Store
Checking for update of app user_status in appstore
Checked for update of app "user_status" in App Store
Checking for update of app video_converter in appstore
Update app video_converter from App Store
Checked for update of app "video_converter" in App Store
Checking for update of app viewer in appstore
Checked for update of app "viewer" in App Store
Checking for update of app weather_status in appstore
Checked for update of app "weather_status" in App Store
Checking for update of app workflowengine in appstore
Checked for update of app "workflowengine" in App Store
Starting code integrity check...

Reviewing Documentation:
According to the kubernetes ingress requirements (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-loadbalancing/glbc#prerequisites) the application must return a 200 status code at ‘/’. It’s a known behavior that when an application is not in a ‘ready’ state, it would return a 302 (redirect to login). If health checks are configured to run, failing results would cause the ingress resource returns 502. Even if health checks are skipped, the container would that are still in a ‘starting code integrity check…’ state would still relay non-200 statuses, which leads to Ingress to return 502 to the users.

Kubernetes: Cert-Manager x509 ECDSA verification failure

Symptoms

Error from server (InternalError): error when creating "kimconnect-cert.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "cert-manager-webhook-ca")
Warning: resource certificates/kimconnect-cert is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
Error from server (InternalError): error when applying patch:

Check Cert-Manager Pods

# Example showing multiple restarts of 'cainjector' pod
admin@controller:~$ kubectl get pod -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-756bb56c5-zc7sb               1/1     Running   3          77d
cert-manager-cainjector-86bc6dc648-2txgt   1/1     Running   9          77d
cert-manager-webhook-66b555bb5-t2fds       1/1     Running   1          77d

Check the logs of ‘cainjector’

# command to view the logs of 'cainjector'
kubectl logs -f -n cert-manager cert-manager-cainjector
# Sample output
W1030 18:29:32.199890       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 18:29:33.198383       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 18:30:22.202957       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 18:32:04.182442       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 18:32:15.301538       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 18:32:29.169059       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 18:32:57.294345       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 18:33:33.192557       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 18:36:11.190905       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 18:37:19.167013       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 18:38:24.202423       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 18:38:29.325957       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 18:39:07.203502       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 18:39:29.204680       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 18:42:19.189894       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 18:42:42.331353       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 18:44:34.313338       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 18:44:45.185535       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 18:45:53.167541       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 18:46:22.156119       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 18:46:42.146226       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 18:47:41.127856       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 18:50:15.340261       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 18:51:14.322043       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 18:51:15.200657       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 18:51:45.253603       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 18:52:08.236342       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 18:52:35.244945       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 18:55:26.228485       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 18:55:39.226791       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 18:56:53.331235       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 18:57:06.330913       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 18:58:01.184241       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 18:58:17.187563       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 19:01:37.188133       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 19:02:04.190773       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 19:02:29.178814       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 19:03:16.190983       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 19:03:26.323601       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 19:03:28.195785       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 19:03:31.206823       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 19:03:53.331321       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 19:08:06.209166       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 19:08:28.209724       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 19:10:34.314565       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 19:11:04.308148       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 19:11:05.171881       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 19:11:55.193059       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 19:12:10.211088       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 19:12:15.206870       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 19:13:34.212396       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 19:14:27.196581       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 19:16:13.320215       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 19:17:20.215518       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 19:18:29.330780       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 19:18:30.210758       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 19:20:00.175053       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 19:20:41.195374       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 19:22:08.337737       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 19:22:33.210625       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 19:24:24.180391       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 19:25:43.343594       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 19:25:45.216737       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 19:26:28.222022       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 19:27:02.195941       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 19:27:50.198613       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 19:28:33.178871       1 warnings.go:67] apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
W1030 19:29:59.215813       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 19:30:05.341900       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 19:33:12.205251       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
W1030 19:33:27.338509       1 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W1030 19:34:50.217154       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W1030 19:35:39.197589       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration

Resolutions

Option 1: delete ‘CainJector’ Pod so that it would automatically recreate

# Command to delete pod
kubectl delete pod cert-manager-cainjector-86bc6dc648-2txgt -n cert-manager 

Option 2: patch cert-manager

# Shared by wutz: https:// github.com/jetstack/cert-manager/issues/3338

patchesJson6902:
  - target:
      kind: ClusterRole
      name: cert-manager-cainjector
      version: v1
      group: rbac.authorization.k8s.io
    patch: |-
      - op: add
        path: /rules/-
        value:
            apiGroups:
              - ""
            resources:
              - configmaps
            verbs:
              - get
              - create
              - update

Option 3: add snippet to helm file

# Shared by sullerandras (https:// github.com/jetstack/cert-manager/issues/3338)
releases:
  - name: cert-manager
    namespace: kube-system
    chart: jetstack/cert-manager
    version: v1.0.2
    values:
      - installCRDs: true
      - cainjector:
          enabled: false

Kubernetes: Cert-Manager Certificate Request YAML Example

# Set variables
certPrefix=kimconnect
domainName=kimconnect.com
domainName2=www.kimconnect.com
serviceName=kimconnectblog-wordpress
servicePort=8080

# Create a yaml file and create cert with it
cat <<EOF > $certPrefix-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: $certPrefix-cert
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    acme.cert-manager.io/http01-edit-in-place: "true"
    kubernetes.io/tls-acme: "true"
spec:
  dnsNames:
    - $domainName
    - $domainName2
  secretName: $certPrefix-cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
EOF
kubectl create -f $certPrefix-cert.yaml