SCCM Applications randomly not showing after SCCM upgrade

After a recent SCCM Site upgrade we had a seemingly random issue with some Applications periodically not arriving on computers for some users. The issue was the perfect culmination of “not all Applications, not all users, and the list of affected users and Applications would change every day”, making it especially difficult to track down or even know if a specific fix had solved the issue or this user and Application just wasn’t affected anymore.

I still haven’t found concrete evidence of what causes the issue or how to replicate it, however, I have worked out enough information from logs and searching on the internet to diagnose the issue and apply a fix.

Diagnose

Logs to check on the client:

PolicyAgent.log — the client will be receiving policy from the Management Point successfully just like normal. No errors should show in this log.

AppDiscovery.log — application detection will have begun, however, the log will have paused/not progressed passed a specific Application. The following screenshot was taken after I had already resolved the issue:

Application discovery does not continue to check all Applications deployed to the device/user

You can see an Oracle Application detection ran and it was not detected. No other messages were logged until about 50 minutes later when Application Discovery started again. If Application deployments had been working correctly I would have normally seen other Applications be detected/not detected after the Oracle one.

Note: The problematic Application is not the Oracle Application. It was able to successfully evaluate the detection rule. The problematic Application is the one SCCM tried to evaluate next.

CIDownloader.log — this log will show an error cycle like so at the same timestamp as the last detection that ran in AppDiscovery.log (above):

Part of an Application’s metadata fails to be found on the Management Point

I don’t know enough about how SCCM’s internals work at these specific points to know with 100% certainty what is failing, however, the issue is that the client is trying to download some part of the SCCM Application metadata (delivered as Configuration Items (CI)) and is not able to find at least one part on the Management Point.

The CIDownloader component stops processing any other Applications that were queued and goes to sleep for 1 hour before it tries again:

This cycle will then repeat every hour, failing at the same Application (or Applications) every time.

The frustrating part about trying to pinpoint the exact Application(s) that are failing is made complex by the fact that the client will process the Applications deployed to it in any order. This is where the “randomness” comes from with which users are affected and the Applications that appear to be affected: the actual Application that is causing the client to stop processing further deployments may be the 1st Application that the client processes today. Later on today, that same Application may then be the 5th — meaning the 4 Applications that are processed before it will deploy correctly. This cycle repeats on each machine, for each user, with the problematic Application halting each client at a different point.

The fix

I wouldn’t really say this is “fixing” the underlying cause but rather fixing the Applications so they appear on devices again.

The fix is to create a new revision of an affected Application. I assume this works because it forces SCCM to essentially “reprocess” everything to do with the Application in the database, therefore fixing whatever issue has occurred during the Site upgrade.

In my specific scenario, we began to find each individual Application that was causing issues and fixed each one manually. We soon realised that fixing one Application just uncovered another Application. The process was tedious and going to take many weeks to do manually.

Instead, I wrote a small piece of PowerShell to reach out to each Application and perform an action on it that wouldn’t actually result in any change to the Application, but it would trigger SCCM to create a new revision:

#Get all Application that AREN’T ‘retired’
$Apps = Get-CMApplication -Fast | Where-Object {$_.IsExpired -eq $false } | Sort-Object -Property LocalizedDisplayName
foreach ($App in $Apps)
{
      # Write the Application name to the console window, purely for progress monitoring
      $App.LocalizedDisplayName 
      
      # Trigger a new revision by trying to remove a Support Contact that won’t exist anyway
      $App | Set-CMApplication -RemoveSupportContact “somethingthatdoesntexist” 
}

750 Application revisions later and all Applications are now deploying as quickly as they used to.