Category: Systems Administration

Symptom:

Event Logs with repeated entries…

Log Name: Application
Source: MSCRMAsyncService
Date: 11/15/2021 01:55:49 AM
Event ID: 25349
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: CRM02.kimconnect.com
Description:
Async backlog is detected in the Queue: AsyncOperation for organization. OrganizationId: {SomeID} Sample 1: Number of Jobs Backlogged: 1, Max Latency Observed: 25847 seconds Sample 2: Number of Jobs Backlogged: 1, Max Latency Observed: 26748 seconds Sample 3: Number of Jobs Backlogged: 1, Max Latency Observed: 27648 seconds Sample 4: Number of Jobs Backlogged: 1, Max Latency Observed: 28548 seconds
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSCRMAsyncService" />
<EventID Qualifiers="49152">25349</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-11-16T01:55:49.056151900Z" />
<EventRecordID>102675119</EventRecordID>
<Channel>Application</Channel>
<Computer>CRM02.kimconnect.com</Computer>
<Security />
</System>
<EventData>
<Data>AsyncOperation</Data>
<Data>SOMEID</Data>
<Data>1</Data>
<Data>25847</Data>
<Data>1</Data>
<Data>26748</Data>
<Data>1</Data>
<Data>27648</Data>
<Data>1</Data>
<Data>28548</Data>
</EventData>
</Event>

Resolution:

Option A: Update the status of all the emails with a status of ‘Pending Send’ to ‘Cancelled’
Option B: Ensure that the email router account is valid and sending emails smoothly
Option C: Reboot Frontend (Web Tier) and Backend (SQL, Async) servers

Sometimes, I sit back and watch the mind games in politics, companies, churches, and even small families… Here’re my observations, abstracted from a Movie:

Besides Thor and Captain America, there is one other entity that could lift Thor’s hammer (Mjolnir?), and his name is Vision. When watching that scene, I’ve thought to myself, ‘how does one determine the worthiness of a person?’ … If it were up to me, I would devise a structured approach in measuring a person’s multi-faceted character thresholds as meta-psychological evals are hard to grasp without objective data. Here are 10 items that I doubt Marvel Comics writers have had considered:

1. Moral compass: bring up very hard moral topics (e.g. abortion due to rape, sacrifice one child to save several)
2. Compassion: observe facial expression of the subject when confronted with someone else’s tragedy
3. Incorruptibility: test on whether person would cheat, steal, or abuse of power if given an opportunity
4. Trustworthiness: check the reliability of timeliness & delivery basing on his/her own commitments
5. Flexibility: force one to come up with an impossible solution such as the ‘Kobayashi Maru’ test.
6. Resiliency: how does the subject behave when defeated in a game, in an logical argument, or in a fight? Another method would be to temporarily restrict blood circulation to this person’s brain while he performs a difficult test, under duress (e.g. place tourniquets on the man’s limbs, then make him solve math problems or dodge rubber bullets)
7. Cleverness: how about a standardized IQ test? If not, present a problem that requires intelligence to solve (what’s the distance of 2 vehicles after 30 minutes if the difference of their speed is 50% and their directions are opposite? No scratch papers allowed as the solution would be a math equation rather than a static value)
8. Subconscious personality: give this person plenty of alcohol (or sodium thiopental)
9. Decisiveness: how long does one require to make a simple, complex, and very complex decision of high quality?
10. Vices: does the person have addiction to any of these 4 dopamine abuses (a) gambling (b) drugs/alcohol (c) sex (d) video games

Step 1: Create a New Cloud Instance

Preparation:

Create a new Active Directory Group (‘Test Cloud Administrators’) and a new user (‘vmmtest’)

$groupName='Test Cloud Administrators'
$samAccountName='TestCloudAdministrators'
$container="CN=Users,DC=Intranet,DC=KIMCONNECT,DC=Com"

New-ADGroup -Name $groupName -GroupCategory Security -GroupScope Global -DisplayName $groupName -Path $container -Description $groupName # -SamAccountName $samAccountName

$groupName='Test Cloud Administrators'
$newUsername='vmmtest'
$encryptedPassword=Read-Host -AsSecureString "Input User Password for account $newUsername"
New-ADUser -Name $newUsername -Enabled $True -AccountPassword $encryptedPassword
Add-ADGroupMember -Identity $groupName -Members $newUsername

$groupName='Test VMM Read-only Admins'
$samAccountName=$groupName -replace ' ','_'
$container="CN=Users,DC=Intranet,DC=KIMCONNECT,DC=Com"

New-ADGroup -Name $groupName -GroupCategory Security -GroupScope Global -DisplayName $groupName -Path $container -Description $groupName  # -SamAccountName $samAccountName

$groupName='Test VMM Read-only Admins'
$newUsername='VMM_Test_Admin_RO'
$encryptedPassword=Read-Host -AsSecureString "Input User Password for account $newUsername"
New-ADUser -Name $newUsername -Enabled $True -AccountPassword $encryptedPassword
Add-ADGroupMember -Identity $groupName -Members $newUsername

Grant ‘Test Cloud Administrators’ Group RDP access to VMM Server:

$groupEntity='Intranet\Test Cloud Administrators'
Add-LocalGroupMember -Group 'Remote Desktop Users' -Member $groupEntity

$groupEntity='Intranet\Test VMM Read-only Admins'
Add-LocalGroupMember -Group 'Remote Desktop Users' -Member $groupEntity

Use VMM To Create New Clouds

Start Virtual Machine Manager > right-click Clouds > select ‘Create Cloud’ to initiate the Create Cloud Wizard > Input a name for this new cloud (e.g. ‘Private Cloud 1’ or ‘Test Cloud’) > Next

Put a check mark next to the appropriate container > Next

Select the appropriate Network > Next

If necessary, select the appropriate NLB > Click Next

If necessary, select appropriate template > Next

If necessary, select the appropriate port classification > Next

Select the appropriate storage > Next

Click Browse to select an appropriate Stored VM Path > if necessary, click Add to select a read-only library shares (this must be a unique path)

Review the storage path and library shares > click Next when ready

Set appropriations of CPU, Memory, and Storage resources > Next

Select the available capability profile(s) > Next

If necessary, select the replication groups > Next

Pick an appropriate QoS policy > Next

Review the summary > click Finish when done

Possible Error:

---------------------------
Virtual Machine Manager
---------------------------
The specified path '\\FILESERVER\MSSCVMMLibrary' is not unique.

Ensure that the path or part of the path that you provided is not used as a writable library share path on a private cloud, a read-only share path on a private cloud, or a user role data path on a self-service user role.

ID: 23505
---------------------------
OK 
---------------------------

Workaround: removed read-only library shares

Observe the Jobs window for the Cloud Creation progress

When the wizard has completed, a new Cloud item would appear as an icon under the Clouds tab

Performing the same steps via Scripting (obtained from ‘view script’ button):

Set-SCCloudCapacity -JobGroup "74b6-462e-877e" -UseCustomQuotaCountMaximum $true -UseMemoryMBMaximum $false -UseCPUCountMaximum $false -UseStorageGBMaximum $false -UseVMCountMaximum $true -MemoryMB 524288 -CPUCount 50 -StorageGB 6000

$resources = @()
$resources += Get-SCLogicalNetwork -ID "92d8-4678-a429"

$resources += Get-SCStorageClassification -ID "f9f9-4d3f-80c6"


$addCapabilityProfiles = @()
$addCapabilityProfiles += Get-SCCapabilityProfile -Name "Hyper-V"

Set-SCCloud -JobGroup "74b6-462e-877e" -RunAsynchronously -ReadWriteLibraryPath "\\VMMSERVER\MSSCVMMLibrary\Templates" -AddCloudResource $resources -AddCapabilityProfile $addCapabilityProfiles

$hostGroups = @()
$hostGroups += Get-SCVMHostGroup -ID "fa00-47f0-a451"
New-SCCloud -JobGroup "74b6-462e-877e" -VMHostGroup $hostGroups -Name "Test Cloud" -Description "" -RunAsynchronously

Step 2: Create a Role Based Access Control

Please note that this section is to create a ‘VM Administrator’ role. This is only available in Windows 2019 Server’s Virtual Machine Manager (VMM). This role has a broader scope of access as compared to ‘Tenant Administrator, which may be more fitting to grant limited self-service guest VM administrator level access to ‘virtual clouds’ without full visibility into the cluster. Therefore, these steps should only be observed as informational as it is more advisable to peruse the ‘Tenant Administrator’ RBAC in most scenarios.

To create an RBAC role for VM administrator, go to Settings > right-click  User Roles > Create User Role

Type in the name as ‘Test Cloud Administrator’ > Next

Select ‘Virtual Machine Administrator’ > Next

Click Add > select Active Directory Users or Groups > OK > Next

Narrow down the scope (e.g. ‘Test Cloud’) > Next

Put a check mark to each desired permissions (as listed below) > Next

Role Based Access Controlled Virtual Machine Administrator Permissions:
- Checkpoint: Create and manage virtual machine checkpoints
  - Checkpoint (Restore only): Restore to but cannot create virtual machine checkpoints
- Deploy: Create virtual machines and service from VHDs or templates
  - Deploy (From template only): Create virtual machines and services form templates only
- Deploy shielded: Create shielded vitual machines
- Local Administrator: Grants local administrator rights on virtual machines
- Manage Azure Profiles: Create and Manage Azure Profiles
- Migrate virtual Machine and Storage: Migrate Virtual Machine acress Hosts and Clouds and storage of Virtual Machines
- Pause and resume: Pause and Resume virtual machines and services
- Receive: Receive resources from other self-service users
- Remote connection: Remotely connect to virtual machines
- Remove: Remove virtual machines and services
- Save: Save virtual machines and services
- Share: Share resources with other self-service users
- Shutdown: Shut down virtual machines
- Start: Start virtual machines and services
- Stop: Stop virtual machines and services
- Store and re-deploy: Store virtual machines in the library, and re-deploy those virtual machines
- Update VM functional level: Update Functional Level of the Virtual Machines

Add Library Servers (if required) > Next > Add ‘Run As Accounts’ (if required) > Next > Finish

Creating VM Administrator RBAC via Scripting:

$cloudsToAdd_0 = Get-SCCloud -ID "4cbb-4643-9bf9"
Add-SCUserRolePermission -Cloud $cloudsToAdd_0 -JobGroup "37f5-4362-84c8"
$scopeToAdd = @()
$scopeToAdd += Get-SCCloud -ID "4cbb-4643-9bf9"
Set-SCUserRole -JobGroup "37f5-4362-84c8" -AddMember @("INTRANET\TestAdmins") -AddScope $scopeToAdd -Permission @("Checkpoint", "CheckpointRestoreOnly", "CreateFromVHDOrTemplate", "Create", "AllowLocalAdmin", "MigrateVM", "PauseAndResume", "Shutdown", "Start", "Stop", "UpdateVMFunctionalLevel")
New-SCUserRole -Name "Test Cloud Administrator" -UserRoleProfile "VMAdmin" -Description "" -JobGroup "37f5-4362-84c8"

Step 3: Associating Guest VMs to Virtual Clouds

Note: assigning VMs into individual clouds are only possible if the Cloud entity has been associated with a Host Group that contains online Hyper-V Servers or Clusters.

To associate individual virtual machines (VM’s) toward a particular ‘cloud’, one would run Virtual Machine Manager (VMM) > select VMs and Services > locate a desired VM > right-click that VM > Properties > select General Tab > pick the correct cloud name in the drop-down menu > OK to save

Bonus Materials: VMM User Roles Summary

Source: Microsoft

This is a quick note to myself so that I wouldn’t repeat the same mistake (due to lack of knowledge) when upgrading a Synology SSD cache on a particular volume.

Apparently, the proper procedure to change SSD caching drives would be:

1. Remove the existing SSD caching: 
   Open Storage Manager > Storage > expand the desired volume (e.g. volume 1) > click on the three dots ‘…’ associated with the SSD Cache volume > select Remove > wait until the removal process completes
   
2. Turn off Synology
3. Physically replace the old SSD drives with new ones
4. Turn Synology back on
5. Assign the new caching SSDs toward the desired volumes with one of these options:
   – Read-only: optimized for websites
   – Read-write: optimized for databases and virtual machines

Here are some sample error messages when the above procedure has been executed out of order:

The read-write cache of Volume 1 on SYNOLOGY02 is missing. Please power off your Synology NAS first, and make sure that the SSD used by the SSD Cache is plugged in and then reboot your device.

From SYNOLOGY02

Volume 1 on SYNOLOGY02 has crashed. It is possible that more files may be corrupted if this volume is still used. Please go to Storage Manager > Storage for more information.

From SYNOLOGY02

1. Migrate application or services of old server to new server
2. Obtain approval from application owner or business unit to decommission machine
3. Shutdown old server and monitor the process for 30 days
4. Validate that original application or service is running properly on the new server without dependency of old server
5. Delete old server’s hostname(s) and DNS record(s) from active directory
6. Delete IP reservation for old server from DHCP reservations, if necessary
7. If server or application class falls within retention policy definition, make backups of data and/or machine files as necessary
8. Remove physical hardware, or delete virtual machine (VM)
9. Validate that the machine instance has been purged in DNS, DHCP, AD

Error message:

Unable to read data from the transport connection: net_io_connectionclosed.

Troubleshooting steps:

1. Ensure that TSL1.2 is being used as Windows 2016 & older may default to TLS1.1
   
   

   [Net.ServicePointManager]::SecurityProtocol=[Net.SecurityProtocolType]::Tls12

2. This new error message resulted

   The SMTP server requires a secure connection or the client was not authenticated. The server response was: 5.7.57
   Client not authenticated to send mail. Error: 535 5.7.139 Authentication unsuccessful, SmtpClientAuthentication is
   disabled for the Tenant. Visit https://aka.ms/smtp_auth_disabled for more information.
   [SJ0PR03CA0167.namprd03.prod.outlook.com]

3. Use a browser to login and check the email account

Notice that 2nd-factor advisories are being displayed…

Attempted to navigate to https:// admin.microsoft.com/Adminportal/Home to realize that the provided account is not a member of the Global Administrators…

4. Advise client to have the organization administrator perform this task:

1. Open the Microsoft 365 admin center (https:// admin.microsoft.com/) and go to Users > Active users > Select the user, and in the flyout that appears, click Mail > In the Email apps section, click Manage email apps > Verify the Authenticated SMTP setting: unchecked = disabled, checked = enabled (preferred)

2. Sign in to the Microsoft 365 admin center (https:// admin.microsoft.com/adminportal) using a security administrator , Conditional Access administrator, or Global admin credentials > In the left pane, select Show All > under Admin centers, select Azure Active Directory > In the left pane of the Azure Active Directory admin center, select Azure Active Directory > From the left menu of the Dashboard, in the Manage section, select Properties > Manage Security defaults > set Enable Security defaults = No > Save changes
   
   This is to bypass this prompt: “Microsoft has enabled Security Defaults to keep your account secure. Learn more about the benefits of Security Defaults
   Skip for now (14 days until this is required) Use a different account”
   
   

   Before SMTP option has been changed from disabled to enabled:

   The SMTP server requires a secure connection or the client was not authenticated. The server response was: 5.7.57
   Client not authenticated to send mail. Error: 535 5.7.139 Authentication unsuccessful, SmtpClientAuthentication is
   disabled for the Tenant. Visit https://aka.ms/smtp_auth_disabled for more information.

   After:

   The SMTP server requires a secure connection or the client was not authenticated. The server response was: 5.7.57
   Client not authenticated to send mail. Error: 535 5.7.139 Authentication unsuccessful, the request did not meet the
   criteria to be authenticated successfully. Contact your administrator. [BYAPR05CA0042.namprd05.prod.outlook.com]