Monitoring servers in DMZ using SCOM

I know there is a lot of information on this subject out there, and I spended a lot of time reading blogs and KBs, but still I was left with unanswered questions.
So – taking some bits here and there, putting them together, done some trying, and trying - this is the steps I will have to take next time I have a server in DMZ, that should be monitored by SCOM.

---

At my company, we have a lot of servers in DMZ, we wanted to monitor in our SCOM (still on SCOM 2007 R2, cu5).

We have a functioning CA-server and I have a SCOM Gateway server.

---

First – we need a Trusted Root Certificate.

This certificate has to be imported on all involved servers (RMS, GTW, DMZ servers).

• Browse to http://CA/certsrv
• Download a CA certificate, certificate chain, or CRL
• Download CA certificate chain
• Save certnew.p7b in a folder for your certs (ie. c:\certs)

This certificate needs to be copied to the servers, and imported.

• Open MMC with Certificates (Local Computer) snap-in
• Import the certificate under “Trusted Root Certification Authorities”

---

Second – we need a certificate for every server.

• Browse to http://CA/certsrv
• Request a certificate
• “Or, submit an advanced certificate request”
• Create and submit a request to this CA
• Name: FQDN of server
• Type of Certificate Needed: Other…
• OID: 1.3.6.1.5.5.7.3.1,1.3.6.1.5.5.7.3.2
• Create new key set
• CSP: Microsoft Enhanced Cryptographic Provider v1.0
• Select: “Mark keys as exportable”
• Store certificate in the local computers certificate store.
• Friendly Name: FQDN of server
• Submit

Now, the certificate request have to be issued.

• On CA server, open Certification Authority console
• In “Pending Requests”, right-click the certificate > All-Tasks > “Issue”

Save the certificate

• Browse to http://CA/certsrv
• View the status of a pending certificate request
• Select your sertificate
• Install this certificate
• Open MMC with Certificate (Local Computer) snap-in
• Personal > Certificates
• right-click your certificate > All Tasks > Export
• Yes, export the private key
• Personal Information Exchange > Enable strong protection
• Type a password (remember.. remember.. you will need it later)
• Save it in your cert-folder as FQDN.pfx

Do this for all servers, and copy the file to the servers.

---

On the server in DMZ

Hosts – file
Can your DMZ server resolve the Gateway or RMS ?
Else – put these into the hosts file (C:\Windows\System32\Drivers\etc)

Manually install scom agent
Copy the installation files to your server (also copy MOMCertImport.exe, we will need it) and launch MOMAgent.msi
You will need to supply the name of your Management Group and the FQDN of your GTW.

MOMCertImport
Ok, you copied MOMCertImport, and the certificate file is here too, then:
MOMCertImport <path>\<certificate file>
- and here you will need your password for the certificate.

Now you need to bounch the “System Center Management”-service
and go to Pending Management in your OpsMgr console and approve.

Install CU
Go back to your DMZ server and install current cu.

Manageable
Your agent is now manually installed, which means that it won’t get updated automatically.
Fortunately, using an query on your OpsMgrDB, you can alter a bit, and in this way make the agent manageable.

Windows could not start the System Center Management on Local Computer

Today, checking on my scom health, I found several grey servers.
I thought that this was just another "Stop System Center Management service - delete Health Service State - start System Center Management service"...
BUT.... no.... when I started the service, I got a pop-up:

...and the service didn't start.
hmmm....
Further investigation - and a little help from this article:
http://blogs.technet.com/b/smsandmom/archive/2008/04/30/opsmgr-2007-healthservice-service-fails-to-start-with-25362-warning.aspx
send me in this direction:
check the WindowsAccountLockDownSD key in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService\Parameters\Management Groups\.
And.. sure... the key was not present on these grey machines.
So, I found a healthy machine, exported the WindowsAccountLockDownSD key, copied the reg-file to the "defect" machines and merged it.

And... VOILA.... the service started nicely again.

BUT.... the service started nicely, but the server was still grey.
Looking into the eventlog I found an error 7005 with the following text:
The Health Service was unable to publish its public key to management group [MyMG] and will be unable to receive secure messages until this key is published. Attempts to publish the key will continue.
As long as the agent can't publish its public key it will not communicate with the SCOM management server.
It turned out that two more keys was missing in the registry.
In the following location there should be two keys with a long coded name (string of about 30 characters):
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService\Parameters\Management Groups\mymgmtgroup\SSDB\References\
If it is not there you can pick it up from another machine in the same management group and merge it.
Then restart the System Center Management service and ... Voila... the servers go green.

VMPD - it's finally arrived

Finally - The System Center 2012 Visio MP Designer—VMPD— has arrived.
VMPD is an add-in for Visio 2010 Premium that allows you to visually design Management Packs for  System Center Operations Manager.

You can download it from here: http://www.microsoft.com/en-us/download/details.aspx?id=30170

I will install it right away, and can't wait to get started.

Installing a new Management Server in SCOM R2

I installed a new management server in my SCOM R2 environment but got an error when it was started up.
In the Operations Manager log on the new management server, I received a HealthService 7022 Error.

As I was sure I did nothing wrong during the setup, I called for Google and came up with a solution.
Microsoft KB2027535: New Management Server unable to get configuration in System Center Operations Manager 2007.
As stated in the KB article, I changed the “Default Action Account” runas profile for my new server to “Local System Action Account” and everything was running again.
After a while I would change the”Default Action Account” back to the domain account (as stated in the article).
But I got an error, and I couldn’t save the profile. I then went to properties for the Action Account – retyped the password (yes… typed in the same password !!!) and now I was able to change the runas profile for the server and save it.

Health Service Store has reached its maximum...

Error
Source: Health Service ESE Store
Event ID: 623
Task Category: Transaction Manager

HealthService (5664) Health Service Store: The version store for this instance (0) has reached its maximum size of 60Mb. It is likely that a long-running transaction is preventing cleanup of the version store and causing it to build up in size. Updates will be rejected until the long-running transaction has been completely committed or rolled back.

Possible long-running transaction:

SessionId: 0x0000000000E020C0

Session-context: 0x00000000

Session-context ThreadId: 0x0000000000000F0C

Cleanup: 1
--------------------------------
I have installed Veeam nWorks for VMware, and on the 2 agents, used as collectors, this error came up.
Deleting the Health Service Store and restarting only helped for a short while.

Then - a little change in registry - and a restart of the service did the trick..
This is the registry updates:

1 - Update ‘Version Store Size’ (the Ops Mgr Agent queue/cache Db)
"HKLM\System\CurrentControlSet\Services\HealthService\Parameters\"Persistence Version Store Maximum".
Value should be 5120 (decimal) (equates to 80MB).

2 - Update value for ‘MaximumQueueSizeKb’ HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService\Parameters\Management Groups\
Value should be 102400 (decimal)

3 - Create DWORD value (if not exist) for State Queue Items
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService\Parameters Value should be 4096 (decimal)

Cannot start "System Center Management" service

When starting the System Center Management service you get an error that says that the
"service terminated with service-specific error %%-2130771964".



After diggin' in to this, I found that it was the Healt Service store (or the cache) that was corrupted.
I renamed the Health Service State-folder (c:\program files\system center operations manager 2007),
and then I could again start the service.

SCOM versions

This is a list of SCOM versions...
I thought it might be helpfull...


SCOM 2007 RTM = 6.0.5000 23 Mar 2007

SCOM SP1 RC1 = 6.0.6246 5 Nov 2007

SCOM 2007 SP1 = 6.0.6278 22 Feb 2008

SCOM 2007 SP1 Update Rollup = 6.0.6278.100 (KB 971541) 13 Nov 2009

SCOM 2007 SP1 CU1 = 6.0.6278.112 (KB 2028594) 30 Jul 2010

SCOM 2007 R2 Beta = 6.1.6407 20 Nov 2008

SCOM 2007 R2 RC = 6.1.7043 25 Mar 2009

SCOM 2007 R2 RTM = 6.1.7221 23 May 2009

SCOM 2007 R2 CU1 = 6.1.7221.13 (KB 974144) 30 Jan 2010

SCOM 2007 R2 CU2 = 6.1.7221.15 (KB 979257) 29 Jun 2010

SCOM 2007 R2 CU3 = 6.1.7221.49 (KB 2251525) 1 Oct 2010

SCOM 2007 R2 CU4 = 6.1.7221.61 (KB 2449679) 31 Jan 2011

SCOM 2007 R2 CU5 = 6.1.7221.81 (KB 2495674) 2 Aug 2011


---------------------------------------------------------------------
SCOM 2012 Public Beta = 7.0.8289.0 – July 2011

SCOM: Standard Dataset Maintenance troubleshooting

The System Center Operations Manager Support Team has published a great article on

Standard Dataset Maintenance troubleshooter for System Center Operations Manager 2007

 

KB article about troubleshooting the installation ot SCOM agents

Microsoft published a KB article (KB2566152) about troubleshooting the installation of the SCOM Agent.  If you experience issues with installing the SCOM Agent, this article is a good starting point in order to get it solved.

KB article about Agent Health Tip and Fixes

Microsoft has published an updated KB article (KB2616936) about SCOM Agent Health Tips & Fixes.

The SCOM Agent plays an important role in a SCOM environment, so it’s important to run healthy Agents.
This KB article will tell you a lot about what to do when certain issues arise.
Important hotfixes for WMI, Windows Scripting Host, a.o. are mentioned and referred to.

KB article about gray agent states in SCOM

Microsoft has published an KB article (KB2288515) about troubleshooting gray SCOM Agents.

This article contains a lot of good information, starting from easy troubleshooting to taking a deep dive into your SCOM environment.

CU5 for SCOM 2007 R2 available

Cumulative Update 5 for SCOM 2007 R2 has just been released ....

The KB article describing the fixes, changes, and instructions:
http://support.microsoft.com/kb/2495674
Get it from the download Center:
http://www.microsoft.com/download/en/details.aspx?id=26938
List of all OpsMgr R2 Cumulative Updates:
http://support.microsoft.com/kb/2453149



Cumulative Update 5 for Operations Manager 2007 R2 resolves the following issues:

• Restarts of non-Operations Manager services when the agent is updated is resolved.
• UI hang caused by SDK locking.
• Web console is timing out while opening the left navigation tree.
• Reports - Drill-through fails due to rsParameterTypeMismatch in the EnterpriseManagementChartControl.
• Reports - Edit Schedule button is disabled with SQL Server 2008 R2.
• Reports - Scheduled Reports view for Windows Server 2003 and SQL SRS 2005 SP3 CU9 - returns System.IndexOutOfRangeException: Index was outside the bounds of the array.
• ACS - Event log message is truncated or corrupted in SCDW.
• ACS Filter fails for certain wildcard queries.
• ACS - Updated ACS reports
• Workflows - TCP Port Probe incorrectly reports negative ping latency.
• Workflows - MissingEvent Manual Reset Monitor does not work as expected.
• Workflows - Signed MPs cannot be imported when new attributes are added to existing classes. 

Cross Platform CU5 for Operations Manager 2007 R2 resolves the following issues:

• Performance data for LVM managed partitions is not available
• Process monitor does not retain name if run via symbolic link
• AIX with large number of processes crashes with bad alloc

Cross Platform CU5 for Operations Manager 2007 R2 adds the following features:

• Support for Red Hat 6

Note The new agent for Red Hat 6 is included in Cumulative Update 5. The management pack for Red Hat 6 can be downloaded by visiting the following Microsoft webpage:  System Center Operations Manager 2007 R2 Cross Platform Monitoring Management Packs (http://www.microsoft.com/downloads/details.aspx?FamilyID=b15fef5c-e331-4006-8913-be376bb0e0c1)

Recommended installation order:

1. Backup the Operations and Warehouse databases, and all unsealed MP’s.
2. Apply the hotfix to the RMS
3. Run the SQL script(s) update against the OpsMgr DB AND OpsMgrDW DB.
4. Import the updated management packs provided.
5. Apply the hotfix to all secondary Management Servers.
6. Apply the hotfix to Gateway Servers.
7. Apply the hotfix to my agents by approving them from pending
8. Apply the hotfix to dedicated consoles (Terminal servers, desktop machines, etc…)
9. Apply the hotfix to Web Console server
10. Apply the hotfix to Audit collection servers
11. Update manually installed agents…. well, manually. 

1. Backup
Ok... do the backup...

2. Apply the hotfix to the RMS
Download the CU... (only 950 MB this time)
Restart the server
My RMS are running Windows server 2008 R2 so I will open an elevated command prompt and launch the setup.
When I get the splash screen, I select "Run Server Update".
A setup will run, and when clicking Finish, another Setup is kicked off. This is by design. There should be three actual setups running consecutively (once for the core update, one for the localization, and one for Xplat).
When finished, you are asked to restart the server.... don't ... yet...

3. SQL scripts
Very important....
There are 2 scripts, located on the RMS, in the \Program Files (x86)\System Center 2007 R2 Hotfix Utility\KB2449679\SQLUpdate folder:

• CU5_Database.sql
• CU5_DataWarehouse.sql

Let’s start with CU5_Database.sql.
Open this file in SQL management studio and excute it against the OperationsManager Database. It will return a single string of output stating MPLastModified with a timestamp, upon success.
Open CU5_DataWarehouse.sql and execute it against the OperationsManagerDW database. It will return “Command(s) completed successfully”.

4. Import the new Managment Packs
They are located at \Program Files (x86)\System Center 2007 R2 Hotfix Utility\KB2495674\ManagementPacks\ and are named:

• Microsoft.SystemCenter.DataWarehouse.Report.Library
• Microsoft.SystemCenter.WebApplication.Library.mp
• Microsoft.SystemCenter.WSManagement.Library.mp

These will upgrade existing MP’s in your environment. They take a few minutes each to import.
I also uses cross platform monitoring, so I have downloaded the new XPlat updates, and will at this point execute the  msi and then import the updated management packs.

Now I will restart my RMS.

5. Management server
On all secondary Management Servers, launch the update and choose "Run Server Update".

6. Gateway server
Again... Launch the update .. now choose "Run Gateway Update"

7. Agents
Update the Agents by Approving them in Pending Management.
XPlat: Use the Discovery Wizard to rediscover the Linux and UNIX computers that you monitor. After you rediscover the computers, the agents on those computers upgrade automatically.

8. Consoles
On console machine, kick off the installer and select "Run Server Update".

9. Web Console servers
On Web console server, kick off the installer and select "Run Server Update".
The update the Web.Config file:
Insert just below the line:
The server will need a restart after this.


10. ACS servers
The management server part of the is all ready updated (part 5), so kick off the installer and choose "Run ACS Server update".

11. Manually installed Agents
Well... on all manually installed agents, kick off the installer and choose "Run Agent Update"

Huge amount of files fills up harddrive

I was called to a customer, because their Management Server was running out of space on drive C:
After some investigation, I found that one folder held millions of files taking up 10 GB of space
The folder holding all these files was:

C:\Document And Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

The huge number of files did that the GUI did not work, so..
how to delete old files from the command line ??

The I stumbled over an internal utility / command: FORFILES
Forfiles has the option of locating files ie on dates or "files older than xx days" and then fire a command
- just what I needed..

So - this command did the trick:
forfiles /D -180 /C "cmd /c del /A:S /Q @path"
(find files older than 180 days and delete... /A:S because these files are system files)

I navigated to the MachineKeys folder and fired the command...
- well... it runs very slow... but it works.

Now I have freed up some space, so now I will try to locate the application that fills the folder.
As I suspected, it was the Quest QMX tool that throw all these files..
And to be more specific... it was the Oracle extension. I will now have a dialog with the Quest team about this.

Well, it turned out that this was a known bug in that version of the Oracle extension.
It is fixed in newer versions.

Removing Override MPs from monitoring view

Override MPs are showing in the Monitoring view as empty folders.

You can remove these folders, using “Show or Hide Views”, but this only applies to your profile.

For an entreprise, with many consoles, directly removing the folders in the MP will be the right solution.

1. Export your override MP to a location (ie. d:\MP) for editing.

2. Open the xml file in Notepad

3. Remove the whole <Presentation> part.

4. Save the xml file and import the MP into OpsMgr again.

Now – the Override folder will be gone from the Monitoring view on all consoles.

OpsMgr Agent upgrade fails

I have just installed OpsMgr CU4 on my OpsMgr server, but when Approving agent upgrade in Pending Management, one server comes up with:

"The MOM Server could not start the MOMAgentInstaller service on computer "name" in the time."

This was the only server having having the problem. This server is running Windows 2008 with firewall enabled, but I have other servers with same configuration (also W2k8 w. FW enabled).
BUT... it turned out that IT WAS the firewall that was the problem.

Deleting the rule "MOM Agent Installer Service" solved the problem.

Cumulative Update 4 for OpsMgr 2007 R2

It seems just like a few days ago I installed CU3 - and now CU4 is available.

Read about it here:
http://support.microsoft.com/kb/2449679
You don't have to install CU3 before installing CU4, but if you have not installed CU3, you will have to download and install latest management packs for monitoring Linux and Unix computers.


Kevin Holman has done a great blog about installing CU4
http://blogs.technet.com/b/kevinholman/archive/2011/02/01/opsmgr-2007-r2-cu4-rollup-hotfix-ships-and-my-experience-installing-it.aspx

Great Heath Check report management pack

At SystemCenterCentral you can get the "SCC HealthCheck Reports v.2" Management Pack.
This MP contains 27 great reports regarding the health of the system.

Go get this MP - but be sure to follow the instructions for installation.

Nice PING tool for OpsMgr 2007

We have all created a ping tool, to be used inside of OpsMgr2007, but OpsLogix have created one with a nice GUI interface, to configure directly from the Monitoring pane in the OpsMrg console.

Download free from opslogix.com

Installing ACS reports

After installing ACS, the reports have to be manually installed on the report server.

Follow these simple steps:

1. Log on to the Management Server with an account that is a member of the OpsMgr administrators group

2. Open explorer and navigate to the ReportModels folder in your Operations Manager install files

3. Copy the folder "acs" to a locally drive, i.e. d:\acs

4. Open a Command prompt and navigate to the folder d:\acs

5. Run the command:
UploadAuditReports ReportingServerName http://ReportingServerName>/ReportServer d:\acs
note: you may see some warning errors as below on uploading the reports, these are expected and can be ignored

6. Open Internet Explorer and enter the following address to view the SQL Reporting Services Home page. http://ReportingServerName/Reports

7. Click Audit Reports in the body of the page and then click Show Details in the upper right part of the page.

8. Click the Db Audit data source.
In the Connect Using section, select Windows Integrated Security (if not already checked) and click Apply.

Starting SCOM console with cache cleared

Fire the following command to launch SCOM Console UI with a cleared cache:


Run: "C:\Program Files\System Center Operations Manager 2007\Microsoft.mom.ui.console.exe" /ClearCache