How to configure Microsoft System Center Operations Manager (SCOM) 2012 R2 as SNMP trap receiver for VMware vCenter on MS Windows?

One of the most interesting and very often asked things is how do I configure SCOM 2012 as a SNMP trap receiver? The solution isn't really easy and a lot of SNMP basics are needed.

Again Microsoft need to do some homework to make some things much more easier and improve SCOM regarding SNMP traps. But first I would like to thank some people for their good documentation which pushed me to the right direction:

I've done some investigations of our System Center Operations Manager 2012 R2 update rollup 2 installation and here are my findings:

  1. If your SNMP trap sending device isn't able to act as an typical SNMP agent, means that there can be an SNMP daemon running, you never ever get this device discovered by SCOM! That's the fact!

  2. You should force a discovery for a specific SNMP version. If the device is discovered as version 2(c) device, SCOM ignores every trap from this device if it is not a version 2 trap!

To get more familiar with the SCOM SNMP network device discovery I build following scenario:

Provisioning of a RedHat Enterprise Linux 5 VM and installation of the net-snmp suite. This software package delivered a full featured SNMP daemon (snmpd) and additionally a snmptrap daemon (snmptrapd). All daemons (aka services) are highly configurable and there is very good documentation.

  1. Enable discovering as SNMP v1 device
    To act as a SNMP version 1 device is easy to configure. Just comment the v2c line.
    [root@<hostname> ~]# vi /etc/snmp/snmpd.conf 
    ...
    #       groupName      securityModel securityName
    group   notConfigGroup v1           notConfigUser
    #group   notConfigGroup v2c           notConfigUser
    
    ####
    # Third, create a view for us to let the group have rights to:
    ...
    
  2. Running the SNMP daemon
    We use the typical service call:
    [root@<hostname> ~]# service snmpd restart
    Stopping snmpd:                                            [FAILED]
    Starting snmpd:                                            [  OK  ]
    [root@<hostname> ~]#
    

If this is all done, don't forget to save and configure SCOM to detect the new network device. Don't discover it as a Linux or Unix computer!

Before performing the discovery I started a port scan with nmap and found that only port 161 is opened on the SCOM 2012 installation. This is the reason why SCOM 2012 isn't able to act as a SNMP trap receiver out of the box unless the device discovery is successful done!

[root@<hostname> ~]# nmap -sU -p U:161 <FQDN-SCOM>

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2014-06-20 12:28 CEST
Interesting ports on <FQDN-SCOM> (<SCOM-IP>):
PORT    STATE         SERVICE
161/udp open|filtered snmp
MAC Address: 00:50:56:A3:xx:yy (VMWare)

Nmap finished: 1 IP address (1 host up) scanned in 0.325 seconds
[root@<hostname> ~]#
[root@<hostname> ~]# nmap -sU -p U:162 <FQDN-SCOM>

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2014-06-20 12:28 CEST
Interesting ports on <FQDN-SCOM> (<SCOM-IP>):
PORT    STATE  SERVICE
162/udp closed snmptrap
MAC Address: 00:50:56:A3:xx:yy (VMWare)

Nmap finished: 1 IP address (1 host up) scanned in 0.118 seconds
[root@<hostname> ~]# 

But now let us configure the MS Windows Server 2012 and the SCOM 2012 R2 update rollup 2 in several separate steps.

  1. First we have to do some prerequisites on the Windows operating system
    1. Click on Server Manager -> Dashboard -> Add roles and features

      Add roles and features - 1

    2. Read and click next

      Add roles and features - 2

    3. Select role-based of feature-based installation

      Add roles and features - 3

    4. Select a server from the server pool

      Add roles and features - 4

    5. Just click next

      Add roles and features - 5

    6. Check SNMP service (should be installed)

      Add roles and features - 6

    7. Click on Server Manager -> Tools -> Services

      Configure Windows services - 1

    8. Click on SNMP Service -> Properties

      Configure Windows services - 2

    9. Set SNMP community

      Configure Windows services - 3

    10. Select accept SNMP packets from any host

      Configure Windows services - 4

    11. SNMP Service startup type has to be automatic and Service status has to be running

      Configure Windows services - 5

    12. SNMP Trap startup type has to be disabled and Service status has to be stopped

      Configure Windows services - 6

  2. Second we have to create a Management Pack rule in SCOM
    1. Click on Authoring -> Management Pack Objects -> Rules -> Create a new rule...

      Create MP rule - 1

    2. Select SNMP Trap (Event) rule type and click on New... near Management pack

      Create MP rule - 2

    3. Create the Management pack

      Create MP rule - 3
      Enter a useful name like "Override for SNMP Trap". Click on Next.

    4. Enter some knowledge (or not) and click on Create

      Create MP rule - 4

    5. Back on rule type selection click on Next

      Create MP rule - 5

    6. Select a rule target first

      Create MP rule - 6

    7. Select Node as target

      Create MP rule - 7

    8. Select rule name

      Create MP rule - 8
      Enter a useful rule name like "Collect all SNMP traps". Check if rule is enabled and click on Next.

    9. Configure the trap OIDs to collect

      Create MP rule - 9
      In SCOM 2012 R2 update rollup 2 you can leave the Object Identifier field empty. Click on Create.

    10. Enable the scope view

      Create MP rule - 10

    11. Select the target "Node"

      Create MP rule - 11
      Enable the target "Node". Click on OK.

    12. Review the settings of the created rule of type node

      Create MP rule - 12

    13. "Collect all SNMP traps" properties

      Create MP rule - 13
      Click on Edit... near "Data sources".

    14. SNMP Trap Event trap provider properties

      Create MP rule - 14
      The Object Identifier Properties are empty. Click on Cancel.

  3. Third we have to create a trap monitor in SCOM
    1. Click on Monitoring -> New -> Folder...

      Create a trap monitor - 1

    2. Review the settings of the created rule of type node

      Create a trap monitor - 2
      Enter a useful folder name like "SNMP Management" and select the formerly created MP "Override for SNMP Trap". Click on OK.

    3. Create an Event View

      Create a trap monitor - 3
      Click on SNMP Management -> New -> Event View

    4. Edit the properties of the Event View

      Create a trap monitor - 4
      In tab "Criteria" choose "Node", select "generated by specific rules" and choose the "Collect all SNMP traps" rule. Click on OK.

  4. Fourth we have to create a Run As Account
    1. Click on Administration -> Run As Configuration -> Create Run As Account...

      Create a run as account - 1

    2. Specify general account properties

      Create a run as account - 2
      Choose "Community String" as account type and enter a useful display name like "SNMP Proxy". Click on Next.

    3. Set credentials

      Create a run as account - 3
      Enter the community string and click on Next.

    4. Select a distribution security option

      Create a run as account - 4
      Choose "More secure" and click on Create.

    5. Prompt to create a profile

      Create a run as account - 5

  5. Fifth we have to create a Run As Profile
    1. Click on Administration -> Run As Configuration -> Profiles -> Create Run As Profile...

      Create a run as profile - 1

    2. Specify general profile properties

      Create a run as profile - 2
      Enter the same display name "SNMP Proxy" like we used for the "Run As Account". Click on Next.

    3. Add Run As Account

      Create a run as profile - 3
      Select "Add", choose "SNMP Proxy" and select "All targeted objects". Click on OK.

    4. Create profile

      Create a run as profile - 4
      Click on Create.

    5. Warning about account distribution

      Create a run as profile - 5
      First click on "SNMP Proxy" (green hook appeares) and then click on Close.

Start of the network device discovery

  1. Click on Administration -> Network Management -> Discovery Wizard...

    Discover network device - 1
    Select "Network devices" and click on Next.

  2. Select a management or gateway server

    Discover network device - 2
    If there is just the message that no management servers are available, please check the discovery rules!

  3. Check discovery rules

    Discover network device - 3
    Go to Administration -> Network Management -> Discovery Rules. If there are all your SCOM servers listet, it is not allowed to configure any further discovery rule. You have to delete at least one rule to go further! Again I can't understand this behavior. Why isn't it possible to define various rules and select one of them for a specific discovery? Once more a lack of functionality into SCOM.

  4. Set general properties

    Discover network device - 4
    Enter the name (FQDN) of your network device, select a management server and pool. Click on Next.

  5. Set general properties

    Discover network device - 5
    Enter the name (FQDN) of your network device, select a management server and pool. Click on Next.

  6. Select a discovery type

    Discover network device - 6
    Select "Explicit discovery" and click on Next.

  7. Specify the default Run As accounts for discovery

    Discover network device - 7
    Select "SNMP Proxy" as account name and click on Next.

  8. Specify devices

    Discover network device - 8
    Click on "Add", enter name or IP of the device, choose "ICMP and SNMP" as access mode, select "v1 or v2" as snmp version and "SNMP Proxy" as account. Click on OK and then on Next.

  9. Schedule the network discovery

    Discover network device - 9
    Choose "Run the discovery rule manually". Click on Next.

  10. Confirm the settings

    Discover network device - 10
    Click on Create.

  11. Wait for completion

    Discover network device - 11
    Select "Run the network discovery rule after the wizard is closed". Click on Close.

And voila there is a new network SNMP version 1 device discovered on port 161!

Network device discovered
The information showed are mainly the content of snmpd.conf of the Linux server and not from snmptrapd.conf!
Furthermore the net-snmp agent version seems to be hardcoded in SCOM, because the version displayed is older than the installed version.

A nmap scan again shows that port 162 is now listening on the SCOM 2012 system.

[root@<hostname> ~]# nmap -sU -p U:162 <FQDN-SCOM>

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2014-06-20 13:31 CEST
Interesting ports on <FQDN-SCOM> (<SCOM-IP>):
PORT    STATE         SERVICE
162/udp open|filtered snmptrap
MAC Address: 00:50:56:A3:xx:yy (VMWare)

Nmap finished: 1 IP address (1 host up) scanned in 0.321 seconds
[root@<hostname> ~]#

Now it's time to disable (stop) the SNMP daemon on the Linux server and try to send some SNMP traps. There is a command line utility which can be used for sending SNMP traps. Don't confound things, this is not snmptrapd!

[root@<hostname> ~]# service snmpd stop
Stopping snmpd:                                            [  OK  ]
[root@<hostname> ~]# 
[root@<hostname> ~]# snmptrap -v 1 -c public <FQDN-SCOM> .1.3.6.1.4.1.8072.2.3.1 "" 6 17 "" .1.3.6.1.2.1.1.6.0 s "Just there"
[root@<hostname> ~]# 

or

[root@<hostname> ~]# snmptrap -v 1 -c public <FQDN-SCOM> NET-SNMP-EXAMPLES-MIB::netSnmpExampleNotification "" 6 17 "" SNMPv2-MIB::sysLocation.0 s "Just here"
[root@<hostname> ~]# 

shows following result in SCOM. Please note that SCOM doesn't translate the OIDs to the appropriate entity names by itself!

SCOM shows v1 SNMP trap - 1

For the interested reader now a small collection of SNMP v2c and v1 traps and how SCOM detected them.

Now we see a bunch on non-speaking object identifiers. There is a tool called snmptranslate into the net-snmp package which translates the OID into the entity name and vice versa:

ObjectIdentifier snmptranslate -m ALL Syntax Value snmptranslate -m ALL
.1.3.6.1.2.1.1.3.0 DISMAN-EVENT-MIB::sysUpTimeInstance Timeticks 26266607 -
.1.3.6.1.6.3.1.1.4.1.0 SNMPv2-MIB::snmpTrapOID.0 Oid .1.3.6.1.4.1.8072.2.3.0.1 NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification
.1.3.6.1.4.1.8072.2.3.2.1 NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatRate Integer 60 -
.1.3.6.1.6.3.1.1.4.3.0 SNMPv2-MIB::snmpTrapEnterprise.0 Oid .1.3.6.1.4.1.8072.2.3 NET-SNMP-EXAMPLES-MIB::netSnmpExampleNotifications

If you use the option -m ALL snmptranslate will scan the MIB directory, RHEL default is /usr/share/snmp/mib, parse all MIBs and build the entity names out of the OIDs.

I've bundled the needed MIBs into a download file [vmware-mibs.tar.gz;82 kByte], which you can request here. More information about the latest MIBs are available from VMware: http://kb.vmware.com/kb/1013445.

This was a short trip into the world of object identifiers and entity names. Let us now switch back to real world. We are now able to discover a network device, which sends SNMP traps to SCOM 2012. But what about network devices which are unable to answer SNMP requests but able to send SNMP traps? Is it possible to monitor them with System Center Operations Manager?

The answer is: yes, but not the smart way. First you have to deploy a server like the RedHat Enterprise Linux I've mentioned above and configure it identical (hostname and ip address and SNMP version) to your network device. If this Linux server was successful discovered by SCOM just remove it and substitute it with the "real" SNMP trap sending network device. This should work pretty good! But beware the circumstance that SCOM 2012 just shows the object identifier and is not able to import MIBs! This is a severe shortage.

I guess some guys know that the snmptrap daemon is able to forward SNMP traps. Now, maybe that it is possible to forward SNMP traps from a MS Windows server, let's say VMware vCenter Server, using this RHEL server? Well I've extend my scenario to verify this:

Scenario trap forwarder - 1

Linux A sends a snmptrap to Linux B. Linux B is configured as a forwarder for SNMP traps and sends the trap immediatly back to Linux A which is a snmptrap receiver, too. I do this without including SCOM to verify that the setup is full functional. We can monitor the behavior of the two systems by observing the content of /var/log/messages on both servers.

send trap from Linux A to Linux B:

[root@<hostnameA> ~]# snmptrap -v 1 -c public <hostnameB> .1.3.6.1 localhost 6 17 "" .1.3.6.1 s "Just a test"
[root@<hostnameA> ~]# 

Logfile Linux B:

Jun 18 14:46:04 <hostnameB> snmptrapd[4215]: 2014-06-18 14:46:04 localhost.localdomain [127.0.0.1] (via UDP: [<hostnameA-IP>]:38153) TRAP, SNMP v1, community public SNMPv2-SMI::internet Enterprise Specific Trap (17) Uptime: 265 days, 5:59:58.77 SNMPv2-SMI::internet = STRING: "Just a test"

Logfile Linux A:

Jun 18 14:46:04 <hostnameA> snmptrapd[32387]: 2014-06-18 14:46:04 localhost.localdomain [127.0.0.1] (via UDP: [<hostnameB-IP>]:37360) TRAP, SNMP v1, community public SNMPv2-SMI::internet Enterprise Specific Trap (17) Uptime: 265 days, 5:59:58.77 SNMPv2-SMI::internet = STRING: "Just a test"

Hey, that looks good.

/var/log/messages from Logfile Linux A:

Jun 18 14:47:00 <hostnameA> snmptrapd[32387]: 2014-06-18 14:47:00 <vCenter-FQDN> [<vCenter-IP>] (via UDP: [<vCenter-IP>]:56912) TRAP, SNMP v1, community public  VMWARE-PRODUCTS-MIB::vmwVC Enterprise Specific Trap (VMWARE-VC-EVENT-MIB::vpxdDiagnostic) Uptime: 34 days, 21:34:31.53

Isn't it cool? It will work pretty good from the Linux point of view. Let's transfer the scenario using SCOM 2012 as SNMP trap receiver, which means that there is an additional trap receiver to configure:

Scenario trap forwarder - 2

As you might guess, the result is: nothing! I can't see any new trap into SCOM 2012 by using the last scenario, just the old ones sending direct from Linux B. Oh, what a mess!

These are the configuration changes on Linux B:

  1. /etc/snmp/snmptrapd.conf
    authCommunity net,log public
    forward default <hostnameA-IP> public
    forward default <SCOM-IP> public
    traphandle VMWARE-VC-EVENT-MIB::vpxdDiagnostic /root/scom/bin/traps vpxdiag
    traphandle VMWARE-PRODUCTS-MIB::vmwVC          /root/scom/bin/traps vcenter
    traphandle .1.3.6.1.4.1.6876.4.3               /root/scom/bin/traps vcenteroid
    traphandle .1.3.6.1.4.1.6876.4.3.0.202         /root/scom/bin/traps vpxdiagoid
    
    disableAuthorization yes
    

    You see the forward rules, they are self-explaining. The next lines are trap handlers, I will tell a bit later about them. The last line (disableAuthorization yes) is needed if you use net-snmp v5.3 or greater.

  2. Create the trap handler script /root/scom/bin/traps

    The trap handlers are able to execute a script if a specific trap is received. Because the forwarding doesn't work, what about triggering a new trap with the received values? Let's see what happens if we create the script.

    #!/bin/bash
    
    BASE='/usr/bin/snmptrap -v 1 -c public <SCOM-FQDN> .1.3.6.1.4.1.6876.4.3 "" 6'
    VPXDIAG='202 ""'
    VPXDALARMINFO='203 "" '
    vars=
    
    while read oid val
    do
      OBJECT="$VPXDALARMINFO"
      if [ "$oid" = "<vCenter-FQDN>" ]
      then
        continue
      fi
      if [ "$oid" = "<UNKNOWN>" ]
      then
        continue
      fi
      if [ "$oid" = "UDP:" ]
      then
        continue
      fi
      if [ "$oid" = "SNMP-COMMUNITY-MIB::snmpTrapAddress.0" ]
      then
        continue
      fi
      if [ "$oid" = "SNMP-COMMUNITY-MIB::snmpTrapCommunity.0" ]
      then
        continue
      fi
      if [ "$oid" = "DISMAN-EVENT-MIB::sysUpTimeInstance" ]
      then
        continue
      fi
      if [ "$oid" = "SNMPv2-MIB::snmpTrapOID.0" ]
      then
        continue
      fi
      if [ "$oid" = "SNMPv2-MIB::snmpTrapEnterprise.0" ]
      then
        continue
      fi
      if [ "$oid" = "VMWARE-VC-EVENT-MIB::vmwVpxdTargetObjType.0" ]
      then
        if [ "$val" = "unknown" ] ; then
         val=1
        elif [ "$val" = "host" ] ; then
         val=2
        elif [ "$val" = "vm" ] ; then
         val=3
        elif [ "$val" = "other" ] ; then
         val=4
        fi
      fi
      if [ "$val" = "VMWARE-VC-EVENT-MIB::vpxdDiagnostic" ]
      then
        OBJECT="$VPXDIAG"
        break
      fi
      if [ "$vars" = "" ]
      then
        if [ "$oid" = "VMWARE-VC-EVENT-MIB::vmwVpxdTargetObjType.0" ]
        then
          vars="$oid i \"$val\""
        else
          vars="$oid s \"$val\""
        fi
      else
        if [ "$oid" = "VMWARE-VC-EVENT-MIB::vmwVpxdTargetObjType.0" ]
        then
          vars="$vars $oid i \"$val\""
        else
          vars="$vars $oid s \"$val\""
        fi
      fi
    done
    
    echo "trap: $1 # $BASE $OBJECT $vars" >> /root/scom/bin/snmptraphandle.txt
    clean_vars=$(echo $vars | sed -e 's/Ä/Ae/g' -e 's/Ö/Oe/g' -e 's/Ü/Ue/g' -e 's/ä/ae/g' -e 's/ö/oe/g' -e 's/ü/ue/g' -e 's/ß/ss/g')
    eval "$BASE" "$OBJECT" "$clean_vars"
    
  3. Changes of /etc/init.d/snmptrapd
    ...
    . /etc/init.d/functions
    
    OPTIONS="-m ALL -Ls 3 -p /var/run/snmptrapd.pid"
    #OPTIONS="-Lsd -p /var/run/snmptrapd.pid"
    if [ -e /etc/sysconfig/snmptrapd.options ]; then
    ...
    
  4. Restart of snmptrapd
    [root@<hostnameA> ~]# service snmptrapd restart
    Stopping snmptrapd:                                        [  OK  ]
    Starting snmptrapd:                                        [  OK  ]
    [root@<hostnameA> ~]#
    
  5. The vpxd diagnostic trap

    To receive the vpxd diagnostic trap from vCenter every minute follow the kb article about how to change the default sending period of the diagnostic SNMP trap on vCenter: http://kb.vmware.com/kb/2020271. If you don't know how to send SNMP traps from vCenter there is a kb article, too: http://kb.vmware.com/kb/1006438.

    vCenter vpxd diagnostic trap

    Yeah, we've got it! It works as expected. Now we can receive SNMP traps send by a vCenter Server running Microsoft Windows using a Linux server as a kind of trap rewriter.

Now let me write some words about the configuration. You may have recognized in snmptrapd.conf the duplicate lines, e.g.
traphandle VMWARE-VC-EVENT-MIB::vpxdDiagnostic /root/scom/bin/traps vpxdiag
and
traphandle .1.3.6.1.4.1.6876.4.3.0.202 /root/scom/bin/traps vpxdiagoid
In your final configuration you could delete one of the lines. It doesn't matter if you use OIDs or entity names. In my opinion it's better to use the entity names, because they are more "human readable" instead of the OIDs. Additionally you can delete the forward lines, they are obsolete.

I was inspired from the handler script example from the net-snmp team for writing the traps shell script. The script is just a kind of filter which throws away unwanted information. It converts the ASCII string of the vmwVpxdTargetObjType to integer again (the MIB files provide information about the needed data type) and at least it coverts the german umlauts (some guys say "funny or dotted characters") to the valid ASCII range. If you don't do that SCOM interpretes this as the binary representation of octet string as you can see in the following picture:

SNMP Event including Umlaut

Just by replacing the umlaut ä to "ae" the octet string get readable again:

SNMP Event including converted Umlaut

Now my dear reader is it up to you to build a free piece of software which displays the vCenter SNMP traps in a meaningful way into SCOM 2012. I've provided you with all information and tips I've currently discovered.

You can download this page as pdf file [2000 kB].


Frank Ickstadt - Am Dattenbach 9-11 - 65817 Eppstein - Germany Frank Ickstadt
Am Dattenbach 9-11
65817 Eppstein
Germany
  Phone: not available Phone: not available

frank [dot] ickstadt [at] removethis gmail [dot] com

frank [dot] ickstadt [at] removethis gmail [dot] com   Fax: currently out of order Fax: currently out of order

jEdit Programmer's Text Editor button