This issue:  https://mnscug.org/blogs/sherry-kissinger/477-configmgr-current-branch-topic-id-611-swd-state-messages-flood, where Console admins accidentally checking the box for "Use Server Groups", where all members of that collection then start sending state messages every 1 minute, failing to patch, and potentially affecting State Message processing ... that occurred again in our environment.  We got lax in interactively checking for whether or not a collection accidentally had that done.  Since we have System Center Operations Manager for monitoring, and SCOM can trigger on EventIDs, we made a custom SQL Job to run multiple times a day, and if any collections (not already exempted/expected to have that setting) were to get that setting, that job will drop an Error into the Application Event Log on the Server.  A custom SCOM rule, and we'll now get alerted if this happens again.

In case someone else might find this useful, here is what we did. 

1) TEST first.  Create a collection with 1 member in it, and set that checkbox (so that you can test).  In SSRS itself, as a new query run this...modifying both of the "c3.name not in " entries for any known exceptions your environment already has.  Use Server Groups might be a valid and expected setting for servers in a Server Cluster, for example.  This is only to help assist when that setting is accidentally checked.

if (select case when count(c1.UseCluster) > 0 then 0 else 1 end as [result]
from CEP_CollectionExtendedProperties c1
join collections_g c2 on c2.collectionid=c1.collectionid
join v_collection c3 on c3.collectionid=c2.siteid
where c1.UseCluster=1
and c3.name not in ('Known Collection Name that Should Have that option checked','Another Collection that should be OK'))=0
BEGIN
 DECLARE @CollectionIDAndName nvarchar(200) = (
  select top 1 c2.siteid + ', ' + c3.name as [ValueToLog] from CEP_CollectionExtendedProperties c1
   join collections_g c2 on c2.collectionid=c1.collectionid
   join v_collection c3 on c3.collectionid=c2.siteid
   where c1.UseCluster=1
   and c3.name not in ('Known Collection Name that Should Have that option checked','Another Collection that should be OK')
)
 DECLARE @VALUE nvarchar(MAX) = ('At least 1 collection has Use Server Group Enabled: ' + @CollectionIDAndName + ' To resolve, either edit the SQL Job on the server to include this collection as a known exception, or edit that collection and uncheck Use Server Group. A possible consequence of that setting is machines in that collection may be unable to patch, and may also cause a state message backlog due to submitting state messages every minute')
 RAISERROR (@Value, 16, 1) with LOG
END

When you run that against your cm_xxx database in SSRS, because you created a test collection with that option checked, you should get an Error message in the Application Event Log on the server holding the cm_xxx database.  To test again, edit the properties of the collection and uncheck the box.  Then re-run the query to confirm it does NOT create an Application Event Log error message.  Once you have confirmed that an Event Log Error entry is created and works as you expect it to work, you can continue.

2) In SQL Server Management Studio, Create a new job, give it any name; but make it one you might understand when you or a coworker looks at sql jobs and is trying to make sense of it a year from now.  The Owner is your known standard owner, if you have a standard.  If you do not have a specific standard, try using sa or NT Authority\System.  (You just don't want to use your own individual personal ID--that is poor practice) You may also want to input as much of a description as possible.  For Steps, there will be only 1 step.
Type:  Transact-SQL script (T-SQL)
Run as:  <blank>
Database:  Your CM_xxx database
into the Command: area, input your entire tested sql script you tested successfully.

Under Advanced--use whatever Actions are your standards.  If you don't have a specific standard, we have "on success, quit the job reporting success", on Failure "Quit the job reporting failure".  0 retries, no output file, no log to table, run as user is blank.

Save the job (with no schedules for now).

3) test your job by editing the properties on a test collection and setting the "use Server Group" enabled.  Then in SQL SSRS, Jobs, right-click your job and  Run the Job from Step... and confirm you do get a new EventLog entry.  Remove the setting, and test again--ensuring you do NOT get a new EventLogEntry

4) Once you are satisfied it works as expected, using your favorite monitoring tool, whatever that might be (for us it was SCOM), set it up to monitor for Application Event Log, SOURCE: MSSQLSERVER  EventID: 17063  (that's what we got... confirm that is what you get in Application Event Log--I assume that is universal... but I've been wrong before, many many times)

5) Confirm your monitoring tool correctly reports on any new events, by again enabling Use Server Group on a test collection, run the job, removing, run the job.

6) Once confirmed; edit the job and add a Schedule or Schedules.  How frequently you would like this to run, and potentially alert on the issue, is up to your own standards and discretion.

Done!  With this in place, we hope we will get monitoring alerts about this situation... instead of alerts about a state message backlog.

Issue:  Leveraging SrsResources.dll using ConfigMgr Reporting Services Reports used to work... but after an upgrade or reconfiguration any reports using the expression "=SrsResources.Localization.GetErrorMessage(Fields!ErrorCode.Value, User!Language) "  in the report just says #Error

Resolution:

I'll give the quick resolution here, and the long explanation later.

  1. Get the absolute latest version of SrsResources.dll you have, by looking at any CM Administration Console installation, in the \bin folder.
  2. On the Server which has your ConfigMgr Reporting Services Role, on the same drive where you have that role, make a directory (call it whatever you like, but for this example, I'm calling it CMSrsResources, and for me, the drive was S:.  Copy that latest SrsResources.dll to S:\CMSrsResources folder.
  3. On the Server which has your ConfigMgr Reporting Services role, you will have had to install SQL Reporting Services.  Find that installed location, it might be on C:, D:, or elsewhere.  The folder will likely be "something like" ....\MSRS13.MSSQLServer\Reporting Services\ReportServer.  In that location will be a rssrvpolicy.config file.  Edit rssrvpolicy.config in notepad, and look for the reference to SrsResources.dll.  It is most likely pointing to ...\MSRS13.MSSQLServer\Reporting Services\ReportServer\bin\SrsResources.dll.  CHANGE that to point instead to what you did in Step 2.  In my case, it would be S:\CMSrsResources\SrsResources.dll.  Save the config file.  (if it won't LET you save the config file, go to Services, and stop the SQL Reporting Services 'service', then save it.)
  4. Restart the service for SQL Reporting Services, or Restart the Server.

Done.

The long Explanation...

We have multiple custom reports, especially for Application Deployments, where knowing what an errorcode number means in a localized language (in our case, usually English) is very handy, instead of looking at that information in the console.  SQL reports are preferred in many instances.  In order to get that localization, according to multiple sources, the way to do that using ConfigMgr is 3 steps:

  1. rssrvpolicy.config for your SQL Reporting Services needs to have the SCCM Assembly referenced, pointing to the location of the SrsResources.dll file.
  2. that SrsResources.dll file has to exist in that location identified in the .config file.
  3. for any individual Report for ConfigMgr, for the report properties, on the 'References' tab, add the SrsResources assembly. Inside that report, presuming an error code is one of the values of your dataset, you can use the expression =SrsResources.Localization.GetErrorMessage(Fields!ErrorCode.Value, User!Language to get that information displayed in legible english (if you are english speaking, french if you're french, etc. etc.)

I noticed after CM 1610, and then again after CM 1702, apparently what is done "for me" is either permissions are reset, or something else fun is happening--maybe it's only when you are using SQL 2016, I don't know (all my labs and production are SQL 2016 latest).  But the .config file, if it's pointing to the Reporting Services\ReportServer\Bin location... It just won't read it and use it.  What the report displays instead of a nice handy english-y message is just #Error .  If I instead copy the .dll elsewhere, and edit the .config file to say "go find it over here" -- it uses it just fine.

Here's hoping this might help others if they like leveraging the SrsResources.dll within ConfigMgr reports, and they are doing everything by the book... and yet it still doesn't work.  You might just need to copy the .dll elsewhere and edit the config file.

Issue:  ConfigMgr Clients which should be reporting via hardware inventory "CCMRecentlyUsedApps" have nothing to report.  An analysis of the client indicates there is nothing in WMI root\cimv2\sms\ccm_recentlyusedapps TO report, and mtrmgr.log on the client contains lines like "StartPrepDriver - OpenService Failed with Error".  See --> KB3213242 <-- at Microsoft for more details.

Remediation: What worked for us was to re-register the 'prepdrv.inf', and then restart SMS Agent Host (aka ccmexec)

Before you do anything suggested below--confirm this will fix the issue you are seeing.  Login to a box or two with the issue, and from an elevated cmd prompt, run the "remediation" powershell script below.  Watch mtrmgr.log; and manually check that root/cimv2/sms select * from ccm_recentlyusedapps gets information.  Once you see info, do a hardware inventory, and confirm that box now reports information up to your database, as you expect it to.  If manually remediating works, then you can look to completing the steps below to automate the fix across your environment.

What we did to remediate the 'StartPrepDriver - OpenService Failed with Error' using a Configuration Item and Baseline was this:

It is assumed that you are using a "custom client agent setting" for enabling CCM_RecentlyUsedApps, since you normally do NOT want to target "every client".  Usually you don't want to target heavily used Application Servers, or Citrix Servers--since hundreds of userIDs can 'launch' applications, the results of CCMRUA on those types of machines often won't process (the MIF file will be bigger than 50mb) and isn't that useful anyway.  So go check, in your Custom Client Agent Settings, which collection you are targeting to enable 'ccm recently used apps'.  Once you know that, then continue.

1) first, determine with SQL query how big of an issue it might be; maybe it was only a couple of boxes; and you can address them manually. Depending upon the count returned, it'll be up to you if you want to pursue this as a Configuration Item, or manually address.

   Declare @CollID as nvarchar(8) = (Select collectionid from v_collection c where c.name = 'The Collection Name You figured out has CCMRUA as a HINV rule')
   select count(ws.resourceid) [Count]
   from v_gs_Workstations_Status ws
   where ws.LastHWScan is not null
   and ws.resourceid not in (select resourceid from v_gs_ccm_recently_used_apps)
   and ws.resourceid in (select resourceid from v_fullcollectionmembership_valid fcm where [email protected])

2) Create the Collection Query, to target.

   - Create a new Collection, using the "limit to collection" as the collection you use for targeting when CCM Recently Used Apps information should be reported via Hardware Inventory. (the one you figured out above)

   - The collection query rule should have two conditions... where:
      SMS_G_System_WORKSTATION_STATUS.LastHardwareScan is not null
      and
      SMS_R_SYSTEM.ResourceId not in (Select ResourceId from SMS_G_System_CCM_RECENTLY_USED_APPS)

3) If the count of the SQL query, and the count of the Collection query are the same (or close enough); then you can continue to creating the ConfigItem, and deploying the Baseline to remediate the issue.

4) Create the Configuration Item, where the Detection and Remediation Logic are at the end of this blog post.  Both are Powershell scripts. for "what means compliant", it will be a String, equals "Compliant". Make sure you check the box about Remediate if non-compliant.

5) After you create the Configuration Item, create a Baseline, add the CI to the baseline, and deploy the baseline, with remediation, to the collection you created above. (you may want to do a pilot, to a subset of 2 or 3 specific boxes, just to be sure it will all work as you expect it to work)

6) That's it... then it's just a waiting game.  You are waiting for the Baseline to run, remediate the issue by doing the rundll routine, restart ccmexec.  Once that is done, in a minute or two mtrmgr.log will start to begin recording information into the root\cimv2\ccm_recentlyusedapps WMI location.  Once there is information there, then at the next hardware inventory, presuming this client is asked via hinv policy to report that information, it will have something to report.  So how long it takes completely depends upon YOUR schedules, that you defined.  How frequently the Baseline evaluates, and how frequently your clients do the scheduled HINV action.

PS: you might be tempted to "let me just add a line to the remediation script, to trigger a hinv right at the end".  I wouldn't bother.  It takes a few minutes for the client, after the ccmexec restart, to populate WMI.  It'll all shake out quickly enough.

######### DETECTION Powershell Script for the CI #####################
<#
.SYNOPSIS
   This Script is for Detection of a known condition for the inability of CM Client to initialize the StartPrepDriver
.DESCRIPTION
   This script finds the CM mgrmgr.log file, and reads it for a known good, or known error, condition.
   "what means good" =  mtrmgr.log contains lines like this
        PREP driver successfully initialized 
        or
        Termination event received for process
   "what means bad" = mtrmgr.log contains lines like this
        StartPrepDriver - OpenService Failed with error
.NOTES
   Steps are to
    1) read the regkey for CM Client to find the correct log file location
    2) Look for good or Bad entries in the file
    3) Exit with 'Compliant' or 'Non-Compliant' depending upon the results
.VERSION
    1.0 Sherry Kissinger  2017-03-30
#>
$ErrorActionPreference = 'SilentlyContinue'
#Get the LogDirectory for CM Client
$CMLogDir = (Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\CCM\Logging\@Global" -Name LogDirectory).LogDirectory
#Define the LogFile to Read
$CMLogFile = $CMLogDir + '\mtrmgr.log'
#Read the Log file for the Error Phrase
$GotErrors = (Get-Content -Path $CMLogFile | where-object {$_ -like '*StartPrepDriver - OpenService Failed with error*'})
#Read the Log File for a known good condition phrases
$GotGoodEntry = (Get-Content -Path $CMLogFile | where-object {($_ -like '*PREP driver successfully initialized*') -or ($_ -like '*Termination event received for process*')})
if ($GotGoodEntry) {
  write-host 'Compliant'
  }
else {
  if ($GotErrors) {
     write-host 'Non-Compliant'
  }
}

############## REMEDIATION Powershell Script for the CI  ##########################
<#
.SYNOPSIS
   This Script is for Detection and Remediation of a known condition for the inability of CM Client to initialize the StartPrepDriver
.DESCRIPTION
   This script finds the CM mgrmgr.log file, and reads it for a known good, or known error, condition.
   "what means good" =  mtrmgr.log contains lines like this
        PREP driver successfully initialized 
        or
        Termination event received for process
   "what means bad" = mtrmgr.log contains lines like this
        StartPrepDriver - OpenService Failed with error
.NOTES
   Steps are to
    1) read the regkey for CM Client to find the correct log file location
    2) Look for good or Bad entries in the file
    3) Exit with 'Compliant' or if 'Non-Compliant', to run the fix
       3a) the fix is two steps
            RUNDLL32.EXE SETUPAPI.DLL,InstallHinfSection DefaultInstall 128 C:\WINDOWS\CCM\prepdrv.inf
            Restart-Service ccmexec
.VERSION
    1.0 Sherry Kissinger  2017-03-30
#>
$ErrorActionPreference = 'SilentlyContinue'
#Get the LogDirectory for CM Client
$CMLogDir = (Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\CCM\Logging\@Global" -Name LogDirectory).LogDirectory
#Define the LogFile to Read
$CMLogFile = $CMLogDir + '\mtrmgr.log'
#Read the Log file for the Error Phrase
$GotErrors = (Get-Content -Path $CMLogFile | where-object {$_ -like '*StartPrepDriver - OpenService Failed with error*'})
#Read the Log File for a known good condition phrases
$GotGoodEntry = (Get-Content -Path $CMLogFile | where-object {($_ -like '*PREP driver successfully initialized*') -or ($_ -like '*Termination event received for process*')})
if ($GotGoodEntry) {
  write-host 'Compliant'
  }
else {
if ($GotErrors) {
 Try { Set-ExecutionPolicy -ExecutionPolicy 'Bypass' -Scope 'Process' -Force -ErrorAction 'Stop'}
 Catch {}
 $CMClientDIR = (Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\SMS\Client\Configuration\Client Properties" -Name 'Local SMS Path').'Local SMS Path'
 $ExePath = $env:windir + '\system32\RUNDLL32.EXE'
 $CLine = ' SETUPAPI.DLL,InstallHinfSection DefaultInstall 128 ' + $CMClientDIR + 'prepdrv.inf'
 #Parse the Parameters
 $Prms = $Cline.Split(" ")
 #Execute the fix with parameters
 & "$Exepath" $Prms
 #Restart ccmexec service
 #CCMExec should be restarted; If you'd rather wait for a natural client reboot, comment out this line.
 #Note that this CI will continue to attempt to remediate until a ccmexec restart or reboot
 restart-service ccmexec
  }
}

 

 

Recently we internally had a need for technical people, but not necessarily people extremely versed in reading Task Sequence log files, to be able to tell via a report only why some step in a Task Sequence failed to trigger.

Pre-requisites and lessons learned during testing:

  1. Package/Program(s) defined with the source files containing 1 file:
     --> CreateMIF.ps1 <--
  2. The mif file has some specific structural requirements.  All Fields have to have a value, a returned value of “” (nothing) is not acceptable and won’t be picked up and forwarded.  So hard coding “ENU” for language and “NIL” for Serial Number is done in-script. (You can change the language if you like; but they have to match; in the script and in the Package Language)
  3. In order for CM client to match the .mif file correctly, several elements need to “match”; the filename, the Manufacturer, the PackageName, the Version, and ENU.
  4. In order for CM client to ‘find’ the mif file to forward to the MP, the mif file needs to be in the %windir% folder.

The remainder of this document is illustrating it’s use using a simple example:  Where a pre-requisite check for “is there enough hard drive space”, failed.  This is useful in reports for a verbose status description to help people via reporting only (without reading a long local log file on a client) determine why a step failed.

The steps of the script are essentially this:

  1. Grab the 6 required parameters

  2. Create the MIF in %windir%

Package Properties, note the values for 1, 2, 3, 4. NOTE that the script itself hard codes the value for “language” as ENU—so that has to be ENU on the General Tab here, item #4.

For Status MIF Matching on the Package, note values 1, 2, 3, and 5

In the program Command Line, note the values for 1, 2, 3, and 5; how they have to match the 1, 2, 3 in Package General Tab, and Reporting Tab.

Here’s a sample Task Sequence, using that package.

 

If it’s working, you’ll see this in execmgr.log on a client:

 

 

Using this additional SQL line in some specific reports, you’ll be able to see the “Description”, whatever it may be:
, StatusMifDescription= (select sstring.InsStrValue3 from vStatusMessagesWithStrings sstring where sstring.messageid in (10007,10009) and sstring.recordid=stat.recordid)
in "__History of a Task Sequence Deployment on a Computer"

 

For the curious... why did I use such an odd number for "enough free disk"; because that's about what was free on the test box.  so I could easily test for that, over and over again.

 

If for whatever reason you can't get the attached .zip file, below is the script:

<#
.SYNOPSIS
 This script creates a System Center Configuration Manager compliant status.mif file
.DESCRIPTION
 This script creates a specifically formated text file ending in '.mif' in the $windir folder.
.PARAMETER MIFFileName
 The Name to give the .mif filename, excluding the .mif extension.  This must match the filename in ConfigMgr package status mif matching.
.PARAMETER Company
 The Company or Manufacturer must match the Manufacturer as defined in the Package Properties in ConfigMgr
.PARAMETER Package
 This must match the Package name as defined in the Package Properties in ConfigMgr
.PARAMETER Version
 This must match the version as defined in the Package Properties in ConfigMgr
.PARAMETER Description
 This can be any sentance or phrase, up to 254 characters
.PARAMETER Status
 The only acceptable values are SUCCESS or FAILURE
.EXAMPLE
 Create-MIF.ps1 -MIFFileName ABC12345 -Company 'Acme Corporation' -Package 'Widgets' -Version '3.0' -Description 'Requirement of enough disk space not met' -Status 'Failure'
.EXAMPLE
 Create-MIF.ps1 -MIFFileName ABC12345 -Company 'Acme Corporation' -Package 'Widgets' -Version '3.0' -Description 'Requirement enough disk space met' -Status 'Success'
.NOTES
 On Package Properties in ConfigMgr, the Language of ENU must exist
 The resulting MIF file created needs be in the %windir% folder for ConfigMgr to utilize it.
.VERSION
 1.0 Sherry Kissinger 03-27-2017
#>
[CMDletBinding()]
Param (
 [Parameter(Mandatory=$true)]
 [string]$MIFFileName,
 [Parameter(Mandatory=$true)]
 [string]$Company,
 [Parameter(Mandatory=$true)]
 [string]$Package,
 [Parameter(Mandatory=$true)]
 [string]$Version,
 [Parameter(Mandatory=$true)]
 [string]$Description,
 [Parameter(Mandatory=$true)]
 [ValidateSet('Failed','Success')]
 [string]$Status  
)
$Locale = "ENU"
$SerNum = "NIL"
$FileName = $Env:WinDir+'\'+$MIFFileName + '.mif'
$DoubleQuote = '"'
"START COMPONENT" | Set-Content $FileName
"NAME = " + $DoubleQuote + "WORKSTATION" + $DoubleQuote | Add-Content $FileName
"  START GROUP" | Add-Content $FileName
"    NAME = " + $DoubleQuote + "ComponentID" + $DoubleQuote | Add-Content $FileName
"    ID = 1"  | Add-Content $FileName
"    CLASS = " + $DoubleQuote + "DMTF|ComponentID|1.0" + $DoubleQuote | Add-Content $FileName
"    START ATTRIBUTE" | Add-Content $FileName
"      NAME = " +$DoubleQuote + "Manufacturer" + $DoubleQuote | Add-Content $FileName
"      ID = 1" | Add-Content $FileName
"      ACCESS = READ-ONLY" | Add-Content $FileName
"      STORAGE = SPECIFIC" | Add-Content $FileName
"      TYPE = STRING(64)" | Add-Content $FileName
"      VALUE = " + $DoubleQuote + $Company + $DoubleQuote | Add-Content $FileName
"    END ATTRIBUTE" | Add-Content $FileName
"    START ATTRIBUTE" | Add-Content $FileName
"      NAME = " + $DoubleQuote + "Product" + $DoubleQuote | Add-Content $FileName
"      ID = 2" | Add-Content $FileName
"      ACCESS = READ-ONLY" | Add-Content $FileName
"      STORAGE = SPECIFIC" | Add-Content $FileName
"      TYPE = STRING(64)" | Add-Content $FileName
"      VALUE = " + $DoubleQuote + $Package + $DoubleQuote | Add-Content $FileName
"    END ATTRIBUTE" | Add-Content $FileName
"    START ATTRIBUTE" | Add-Content $FileName
"      NAME = " + $DoubleQuote + "Version" + $DoubleQuote  | Add-Content $FileName
"      ID = 3"  | Add-Content $FileName
"      ACCESS = READ-ONLY"  | Add-Content $FileName
"      STORAGE = SPECIFIC" | Add-Content $FileName
"      TYPE = STRING(64)" | Add-Content $FileName
"      VALUE = " + $DoubleQuote + $Version + $DoubleQuote | Add-Content $FileName
"    END ATTRIBUTE" | Add-Content $FileName
"    START ATTRIBUTE" | Add-Content $FileName
"      NAME = " + $DoubleQuote + "Locale" + $DoubleQuote  | Add-Content $FileName
"      ID = 4" | Add-Content $FileName
"      ACCESS = READ-ONLY" | Add-Content $FileName
"      STORAGE = SPECIFIC" | Add-Content $FileName
"      TYPE = STRING(16)" | Add-Content $FileName
"      VALUE = " + $DoubleQuote + $Locale + $DoubleQuote  | Add-Content $FileName
"    END ATTRIBUTE" | Add-Content $FileName
"    START ATTRIBUTE" | Add-Content $FileName
"      NAME = " + $DoubleQuote + "Serial Number" + $DoubleQuote  | Add-Content $FileName
"      ID = 5"  | Add-Content $FileName
"      ACCESS = READ-ONLY" | Add-Content $FileName
"      STORAGE = SPECIFIC" | Add-Content $FileName
"      TYPE = STRING(64)" | Add-Content $FileName
"      VALUE = " + $DoubleQuote + $SerNum + $DoubleQuote  | Add-Content $FileName
"    END ATTRIBUTE" | Add-Content $FileName
"    START ATTRIBUTE" | Add-Content $FileName
"      NAME = " + $DoubleQuote + "Installation" + $DoubleQuote | Add-Content $FileName
"      ID = 6" | Add-Content $FileName
"      ACCESS = READ-ONLY" | Add-Content $FileName
"      STORAGE = SPECIFIC" | Add-Content $FileName
"      TYPE = STRING(64)" | Add-Content $FileName
"      VALUE = " + $DoubleQuote + "DateTime" +$DoubleQuote | Add-Content $FileName
"    END ATTRIBUTE" | Add-Content $FileName
"  END GROUP" | Add-Content $FileName
"  START GROUP" | Add-Content $FileName
"    NAME = " + $DoubleQuote + "InstallStatus" + $DoubleQuote  | Add-Content $FileName
"    ID = 2" | Add-Content $FileName
"    CLASS = " + $DoubleQuote + "MICROSOFT|JOBSTATUS|1.0" + $DoubleQuote | Add-Content $FileName
"    START ATTRIBUTE" | Add-Content $FileName
"      NAME = " + $DoubleQuote + "Status" + $DoubleQuote | Add-Content $FileName
"      ID = 1"  | Add-Content $FileName
"      ACCESS = READ-ONLY" | Add-Content $FileName
"      STORAGE = SPECIFIC" | Add-Content $FileName
"      TYPE = STRING(32)" | Add-Content $FileName
"      VALUE = " + $DoubleQuote + $Status + $DoubleQuote | Add-Content $FileName
"    END ATTRIBUTE" | Add-Content $FileName
"    START ATTRIBUTE" | Add-Content $FileName
"      NAME = " + $DoubleQuote + "Description" + $DoubleQuote | Add-Content $FileName
"      ID = 2" | Add-Content $FileName
"      ACCESS = READ-ONLY" | Add-Content $FileName
"      STORAGE = SPECIFIC" | Add-Content $FileName
"      TYPE = STRING(128)" | Add-Content $FileName
"      VALUE = " + $DoubleQuote + $Description + $DoubleQuote | Add-Content $FileName
"    END ATTRIBUTE" | Add-Content $FileName
"  END GROUP" | Add-Content $FileName
"END COMPONENT" | Add-Content $FileName
Exit 0
 

 

Issue:  Clients are flooding the inboxes\auth\statesys.box\incoming with state messages similar to the below.  There isn't much there to go on. The only promising thing to look for was that it's "Topic ID="611"" and Type="611".  Researching those topicid of 611 and we found nothing public, and opening a case with Microsoft provided some clues, but this was really a new issue, at least at this scale.

<?xml version="1.0" encoding="UTF-16"?>
<Report>
<ReportHeader>
<Identification>
  <Machine>
   <ClientInstalled>1</ClientInstalled>
   <ClientType>1</ClientType>
   <ClientID>GUID:0EE65490-9075-41B1-B64D-AAAAAAAAAAAA</ClientID>
   <ClientVersion>5.00.8412.1006</ClientVersion>
   <NetBIOSName>CLIENTNAMEHERE</NetBIOSName>
   <CodePage>437</CodePage>
   <SystemDefaultLCID>1033</SystemDefaultLCID>
   <Priority>5</Priority>
  </Machine>
</Identification>
<ReportDetails>
   <ReportContent>State Message Data</ReportContent>
   <ReportType>Full</ReportType>
   <Date>20170314180133.967000+000</Date>
   <Version>1.0</Version>
   <Format>1.0</Format>
</ReportDetails>
</ReportHeader>
<ReportBody>
<StateMessage MessageTime="20170314180133.857000+000" SerialNumber="18823">
  <Topic ID="611" Type="611" IDType="0" User="" UserSID=""/>
  <State ID="100" Criticality="0"/>
  <UserParameters Flags="0" Count="2">
   <Param>GUID:0EE65490-9075-41B1-B64D-AAAAAAAAAAAA</Param>
   <Param>0</Param>
  </UserParameters>
</StateMessage>
</ReportBody>
</Report>

Cause: In a ConfigMgr 1610 environment, and in 1602, 1606 versions of this were available as well, the cause of these messages is because that client is a member of a collection, where either by accident or design, that collection has the setting for "All devices are part of the same server group".  The collection contained 40% of all clients in the environment--and in our case checking the box was NOT by design--it was accidental.

What that does, is two things we observed (and others have documented 1 of them, see links below). 
1) as per one of the links below, machines in that collection may not ever patch as expected, again.  that's because it thinks it's part of a cluster, and if it's not... it's waiting for it's "turn" to patch. 
2) Every single device in that collection, once per minute, locally does two "schedule triggers", for two different things:
{00000000-0000-0000-0000-000000000111} -- which is for "Send Unsent State Message"
{00000000-0000-0000-0000-000000000116} -- which is for "State system policy bulk send low"

You'll see that over and over and over again locally on the client in SMSClientMethodProvider.log

and that apparently ends up as .swd files in the statesys box to be processed by the database, with TopicID 611, Type 611.  If it's enough devices, and it's hotfix Tuesday and lots of state messages anyway (*cough* for example *cough*)--the auth\statesys.box inbox may become backlogged, and never catch up.

Remediation:

I'm a SQL person, so using this sql query, identify the collections which have that checkbox checked--and confirm you really meant it (If you are here reading this blog post... you likely didn't mean it).  If not, uncheck the box for "All devices are part of the same server group" on the collections listed.

;with UseCluster as (select c.SiteID as [CollectionID] from CEP_CollectionExtendedProperties CEP
join collections_g c on CEP.Collectionid=c.Collectionid
where usecluster=1)
select c.*
from UseCluster
join v_collection c on c.collectionid=UseCluster.CollectionID

Monitoring for if this happens again:

https://mnscug.org/blogs/sherry-kissinger/503-example-of-custom-sql-job-to-log-to-application-event-viewer-for-configmgr

Links:
https://blogs.technet.microsoft.com/enterprisemobility/2016/05/16/update-1605-for-configuration-manager-technical-preview-available-now/
https://docs.microsoft.com/en-us/sccm/core/servers/deploy/install/release-notes
https://docs.microsoft.com/en-us/sccm/core/get-started/capabilities-in-technical-preview-1605#BKMK_ServerGroups
https://social.technet.microsoft.com/Forums/en-US/86783d86-0e38-4cb4-acf8-6110acc76c0e/configmgr-1602-error-0x87d006662016410010-while-installing-update?forum=configmanagersecurity