KB article about Agent Health Tip and Fixes

Microsoft has published an updated KB article (KB2616936) about SCOM Agent Health Tips & Fixes.

The SCOM Agent plays an important role in a SCOM environment, so it’s important to run healthy Agents.
This KB article will tell you a lot about what to do when certain issues arise.
Important hotfixes for WMI, Windows Scripting Host, a.o. are mentioned and referred to.

KB article about gray agent states in SCOM

Microsoft has published an KB article (KB2288515) about troubleshooting gray SCOM Agents.

This article contains a lot of good information, starting from easy troubleshooting to taking a deep dive into your SCOM environment.

Blues Brother.... at last on BluRay

Blues Brothers is here.... on BluRay... GREAT

Here is a link to a great article on "The Making of The Blues Brothers"

And, how about a little trivia about Blues Brother ?

The Blues Brothers, Part 1TriviaQuiz

The Blues Brothers, Part II Trivia Quiz

CU5 for SCOM 2007 R2 available

Cumulative Update 5 for SCOM 2007 R2 has just been released ....

The KB article describing the fixes, changes, and instructions:
http://support.microsoft.com/kb/2495674
Get it from the download Center:
http://www.microsoft.com/download/en/details.aspx?id=26938
List of all OpsMgr R2 Cumulative Updates:
http://support.microsoft.com/kb/2453149



Cumulative Update 5 for Operations Manager 2007 R2 resolves the following issues:

• Restarts of non-Operations Manager services when the agent is updated is resolved.
• UI hang caused by SDK locking.
• Web console is timing out while opening the left navigation tree.
• Reports - Drill-through fails due to rsParameterTypeMismatch in the EnterpriseManagementChartControl.
• Reports - Edit Schedule button is disabled with SQL Server 2008 R2.
• Reports - Scheduled Reports view for Windows Server 2003 and SQL SRS 2005 SP3 CU9 - returns System.IndexOutOfRangeException: Index was outside the bounds of the array.
• ACS - Event log message is truncated or corrupted in SCDW.
• ACS Filter fails for certain wildcard queries.
• ACS - Updated ACS reports
• Workflows - TCP Port Probe incorrectly reports negative ping latency.
• Workflows - MissingEvent Manual Reset Monitor does not work as expected.
• Workflows - Signed MPs cannot be imported when new attributes are added to existing classes. 

Cross Platform CU5 for Operations Manager 2007 R2 resolves the following issues:

• Performance data for LVM managed partitions is not available
• Process monitor does not retain name if run via symbolic link
• AIX with large number of processes crashes with bad alloc

Cross Platform CU5 for Operations Manager 2007 R2 adds the following features:

• Support for Red Hat 6

Note The new agent for Red Hat 6 is included in Cumulative Update 5. The management pack for Red Hat 6 can be downloaded by visiting the following Microsoft webpage:  System Center Operations Manager 2007 R2 Cross Platform Monitoring Management Packs (http://www.microsoft.com/downloads/details.aspx?FamilyID=b15fef5c-e331-4006-8913-be376bb0e0c1)

Recommended installation order:

1. Backup the Operations and Warehouse databases, and all unsealed MP’s.
2. Apply the hotfix to the RMS
3. Run the SQL script(s) update against the OpsMgr DB AND OpsMgrDW DB.
4. Import the updated management packs provided.
5. Apply the hotfix to all secondary Management Servers.
6. Apply the hotfix to Gateway Servers.
7. Apply the hotfix to my agents by approving them from pending
8. Apply the hotfix to dedicated consoles (Terminal servers, desktop machines, etc…)
9. Apply the hotfix to Web Console server
10. Apply the hotfix to Audit collection servers
11. Update manually installed agents…. well, manually. 

1. Backup
Ok... do the backup...

2. Apply the hotfix to the RMS
Download the CU... (only 950 MB this time)
Restart the server
My RMS are running Windows server 2008 R2 so I will open an elevated command prompt and launch the setup.
When I get the splash screen, I select "Run Server Update".
A setup will run, and when clicking Finish, another Setup is kicked off. This is by design. There should be three actual setups running consecutively (once for the core update, one for the localization, and one for Xplat).
When finished, you are asked to restart the server.... don't ... yet...

3. SQL scripts
Very important....
There are 2 scripts, located on the RMS, in the \Program Files (x86)\System Center 2007 R2 Hotfix Utility\KB2449679\SQLUpdate folder:

• CU5_Database.sql
• CU5_DataWarehouse.sql

Let’s start with CU5_Database.sql.
Open this file in SQL management studio and excute it against the OperationsManager Database. It will return a single string of output stating MPLastModified with a timestamp, upon success.
Open CU5_DataWarehouse.sql and execute it against the OperationsManagerDW database. It will return “Command(s) completed successfully”.

4. Import the new Managment Packs
They are located at \Program Files (x86)\System Center 2007 R2 Hotfix Utility\KB2495674\ManagementPacks\ and are named:

• Microsoft.SystemCenter.DataWarehouse.Report.Library
• Microsoft.SystemCenter.WebApplication.Library.mp
• Microsoft.SystemCenter.WSManagement.Library.mp

These will upgrade existing MP’s in your environment. They take a few minutes each to import.
I also uses cross platform monitoring, so I have downloaded the new XPlat updates, and will at this point execute the  msi and then import the updated management packs.

Now I will restart my RMS.

5. Management server
On all secondary Management Servers, launch the update and choose "Run Server Update".

6. Gateway server
Again... Launch the update .. now choose "Run Gateway Update"

7. Agents
Update the Agents by Approving them in Pending Management.
XPlat: Use the Discovery Wizard to rediscover the Linux and UNIX computers that you monitor. After you rediscover the computers, the agents on those computers upgrade automatically.

8. Consoles
On console machine, kick off the installer and select "Run Server Update".

9. Web Console servers
On Web console server, kick off the installer and select "Run Server Update".
The update the Web.Config file:
Insert just below the line:
The server will need a restart after this.


10. ACS servers
The management server part of the is all ready updated (part 5), so kick off the installer and choose "Run ACS Server update".

11. Manually installed Agents
Well... on all manually installed agents, kick off the installer and choose "Run Agent Update"

Huge amount of files fills up harddrive

I was called to a customer, because their Management Server was running out of space on drive C:
After some investigation, I found that one folder held millions of files taking up 10 GB of space
The folder holding all these files was:

C:\Document And Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

The huge number of files did that the GUI did not work, so..
how to delete old files from the command line ??

The I stumbled over an internal utility / command: FORFILES
Forfiles has the option of locating files ie on dates or "files older than xx days" and then fire a command
- just what I needed..

So - this command did the trick:
forfiles /D -180 /C "cmd /c del /A:S /Q @path"
(find files older than 180 days and delete... /A:S because these files are system files)

I navigated to the MachineKeys folder and fired the command...
- well... it runs very slow... but it works.

Now I have freed up some space, so now I will try to locate the application that fills the folder.
As I suspected, it was the Quest QMX tool that throw all these files..
And to be more specific... it was the Oracle extension. I will now have a dialog with the Quest team about this.

Well, it turned out that this was a known bug in that version of the Oracle extension.
It is fixed in newer versions.

Performance Monitor corrupted

At a customer site, I was setting up performance monitoring using PerfMon on several servers, when I at one server encountered the following prolem…
NO performance counter “names” – just numbers…

Running just one tiny command solved the problem
- and that without interferring with production…

%systemdrive%\windows\system32

lodctr.exe /r

observe that I use a “r” and not “R” as documented

Quest QMX are not inserting data in SCOM Datawarehouse by default

I was at a customer, who wanted to generate performance report - and some of these performance reports was regarding VMware, which is monitored using Quest QMX.
But I found no perf data regarding the VMware.

It turned out, that a stiny little checkmark was missing in the QMX Configuration Tool:
g_boolWriteOpsMgrPerfMetricsToDWDB (OpsMgrGlobalVariables)