Using PowerShell Command Notifications in SCOM 2012

When SCOM produces alerts we have a few ways of notifying people that an alert has occurred. We can simply look in the console and see alerts there. We can send an Email, Instant Message, we can send an SMS or even forward alerts to other applications like a ticketing system. We can also use a Command Notification to run a command or a script. So in this post, I’ll show you how you can use a Command Notification to run a Windows PowerShell script which will allow us to perform some sort of task after an alert has been generated.

As with any new Notifications, we’ll need to create a Channel, a Subscriber and a Subscription.

  • Channels are the type of notification – eMail, SMS, IM or run a command/script.
  • Subscribers are who we will notify – an email address, a Distribution List, a script to perform some task.
  • Subscriptions are the what we will notify people of. Specific alerts, critical alerts, new alerts, alerts from specific classes and so forth.

 

Creating a Notification Channel

So in the SCOM Console, we’ll go to the Administration Window

Administration Window

 

 

 

 

 

 

and under the Notifications Folder we’ll choose “Channels”.

Notification Channels

 

 

 

 

Now we’ll right click and choose “New –> Command” or head over to the Actions Pane and choose “New –> Command”.

New Command

 

 

 

 

 

 

The good news is that this will be a very short wizard…we just need to provide a name for our Command Channel and a description advising other administrators of what this Command Channel will be used for.

 

Command Channel Description

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Now we’ll click “Next” and we’ll need to enter in where our script can be found and any parameters we might like to pass it.

 

Command Channel Settings

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

So here’s the command line options I’ve chosen for this example.

Full path of the command line:

c:\windows\system32\windowspowershell\v1.0\powershell.exe

 

Command line parameters:

“C:\Scripts\Notifications\Culham-TestNotifications.ps1” ‘$Data/Context/DataItem/AlertName$’ ‘$MPElement$’ ‘$Data/Context/DataItem/AlertId$’

 

Startup folder for the command line:

c:\windows\system32\windowspowershell\v1.0\

 

So we’re providing the path to Windows PowerShell and passing it the name of our script in the first text box and we’ll use the same path of the PowerShell folder for the startup folder field. But in the Command line parameters field, we’re going to pass our script some parameters.

In my example I’m passing in the AlertName, the MPElement ID and the AlertID. If we click the little arrow icon to the right of this field we can see some of the other parameters we can pass to our script.

Command Channel Flyout

 

You could use any of these:

AlertID GUID
$Data/Context/DataItem/AlertId$

Alert Name
$Data/Context/DataItem/AlertName$

Alert Category
$Data/Context/DataItem/Category$

UTC Date/Time of DataItem Created
$Data/Context/DataItem/DataItemCreateTime$

LocalTime Date/Time of DataItem Created
$Data/Context/DataItem/DataItemCreateTimeLocal$

UTC Date/Time DataItem was Modified
$Data/Context/DataItem/LastModified$

Local Date/Time DataItem was Modified
$Data/Context/DataItem/LastModifiedLocal$

Custom Field 1
$Data/Context/DataItem/Custom1$

Custom Field 2
$Data/Context/DataItem/Custom2$

Custom Field 3
$Data/Context/DataItem/Custom3$

Custom Field 4
$Data/Context/DataItem/Custom4$

Custom Field 5
$Data/Context/DataItem/Custom5$

Custom Field 6
$Data/Context/DataItem/Custom6$

Custom Field 7
$Data/Context/DataItem/Custom7$

Custom Field 8
$Data/Context/DataItem/Custom8$

Custom Field 9
$Data/Context/DataItem/Custom9$

Custom Field 10
$Data/Context/DataItem/Custom10$

Created by monitor? (true/false)
$Data/Context/DataItem/CreatedByMonitor$

Recipient Name
$Data/Recipients/To/Address/Address$

Management Server Principal Name
$Target/Property[Type=”Notification!Microsoft.SystemCenter.AlertNotificationSubscriptionServer”]/PrincipalName$

Web Console URL
$Target/Property[Type=”Notification!Microsoft.SystemCenter.AlertNotificationSubscriptionServer”]/WebConsoleUrl$

Workflow ID (GUID)
$Data/Context/DataItem/WorkflowId$

UTC Date/Time the Alert was Resolved
$Data/Context/DataItem/TimeResolved$

Time Raised (Local)
$Data/Context/DataItem/TimeRaisedLocal$

Time Raised (UTC)
$Data/Context/DataItem/TimeRaised$

Time Added (Local)
$Data/Context/DataItem/TimeAddedLocal$

UTC Time Added
$Data/Context/DataItem/TimeAdded$

TicketID
$Data/Context/DataItem/TicketId$

Alert Severity ID
$Data/Context/DataItem/Severity$

Person Resolving the Alert
$Data/Context/DataItem/ResolvedBy$

The Resolution State Name (New, Closed etc.)
$Data/Context/DataItem/ResolutionStateName$

Date/Time Resolution State was Last Modified (Local)
$Data/Context/DataItem/ResolutionStateLastModifiedLocal$

Date/Time Resolution State was Last Modified (UTC)
$Data/Context/DataItem/ResolutionStateLastModified$

Resolution State ID (0=New 255=Closed etc.)
$Data/Context/DataItem/ResolutionState$

Alert Repeat Count
$Data/Context/DataItem/RepeatCount$

Alert Owner
$Data/Context/DataItem/Owner$

Managed Entity Path
$Data/Context/DataItem/ManagedEntityPath$

ManagedEntity Full Name
$Data/Context/DataItem/ManagedEntityFullName$

ManagedEntity Display Name
$Data/Context/DataItem/ManagedEntityDisplayName$

ManagedEntity GUID
$Data/Context/DataItem/ManagedEntity$

 

Or you can find a complete list of variables here.

And speaking of our script, let’s take a look at the example I’m using for this blog post.

Param (
[String]$AlertName,
[String]$SubscriptionID,
[String]$AlertID
)

# Load the OpsMgr Provider
Import-Module OperationsManager

# Locate the Specific Alert
$alert = Get-SCOMAlert -Id $alertID
$strResolutionState = ($alert.resolutionstate).ToString()
$localtimeRaised = ($alert.timeraised).tolocaltime()
$severity = $alert.Severity.ToString()

# Write to a log file
$Logstring = $AlertName + ” ” + $SubscriptionID + ” ” + $AlertID + ” ” + $severity
$Logfile = “c:\Scripts\Notifications\LogAlerts.log”
$DateTime = Get-Date -Uformat “%y-%m-%d %H:%M:%S”
$Logstring = $DateTime + ” ” + $Logstring
Add-content $Logfile -value $Logstring

# Set the Alert in SCOM Console to a New Resolution State
$alert | Set-SCOMAlert -ResolutionState 2

 

So basically what this script aims to do is to take the 3 input parameters of the AlertName, the Subscription ID and the AlertID and then write that information out to a logfile and then updates the resolution state in SCOM itself. And it’s not a particularly useful example for the real world, but it does highlight how we can take those input parameters, run a script, output that data to a file which will let us know that we’ve found the right alert and successfully processed it and also update SCOM to complete the round trip.

Now since this script does call for updating a SCOM resolution state, let’s set up a new one. In the script I’ve used the Resolution State of 2 which doesn’t exist in a new SCOM environment. So in our SCOM console we’ll go to the Administration view and under Settings we’ll choose “Alerts”.

We’ll add a New Resolution State and provide a Name and a numeric ID.

Gobal Alert Settings - Add New Notification

 

 

 

 

 

 

 

 

 

We’ll add a New Resolution State and provide a Name and a numeric ID.

I’ll use the values of “Notification Sent” and give it a state of 2 which will match the value of my script.

 

Gobal Alert Settings - Notification Sent

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Now that we’ve created the new Resolution State and obviously we’ve created our Channel, we need to create both a Subscriber and a Subscription.

 

Creating the Subscriber

So for our subscriber, we’ll right click and choose “New” or use the Actions Pane, choosing “New”.

New Subscriber

 

 

 

First we’ll enter in a name for our Subscriber.

Subscriber - Description

 

 

 

 

 

 

 

 

 

 

 

Now we’ll choose a schedule for this subscriber to adhere to. I’ve left the defaults to ‘Always send notifications’.

Subscriber - Schedule

 

 

 

 

 

 

 

 

 

 

 

 

Finally we’ll choose the addresses. We don’t have any here so we’ll click “Add”.

Subscriber - Address Add

 

 

 

 

 

 

 

 

 

 

 

 

We’ll provide this with a name and click “Next”.

Subscriber Address - General

 

 

 

 

 

 

 

 

 

 

 

We’ll choose a Command Channel Type and select our Command Channel from the drop down box. Then we’ll click “Next”.

Subscriber Address - Channel

 

 

 

 

 

 

 

 

 

 

 

 

Finally we’ll choose the schedule. Again, I’ll leave the defaults and click “Finish”.

Subscriber Address - Schedule

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

And we’ll click “Finish” once more.

Subscriber - Finish

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Our Subscriber has been created.

 

Creating the Subscription

The last thing we need to do is create a subscription, or in other words, set what criteria we’ll choose to run our PowerShell script. Will it be critical alerts? Any alert? Alerts from a specific class? Basically it comes down to what you wish to use. For the purposes of this post I’ll just use any new alert.

So we’ll create a “New” Subscription.

Subscription - New

 

 

 

We’ll give it a Name and a Description then click “Next”.

Subscription - Description

 

 

 

 

 

 

 

 

 

 

We’ll choose the Criteria that must be met in order to trigger this Command Notification. We have so many options to choose from here, and most likely you’ll be choosing specific types of alerts or alerts from certain classes – maybe you want to assign them to different teams or something else entirely. But for the purposes of this post, I’m going to choose nothing and that’ll leave this Criteria at “All Alerts”.

Subscription - Criteria

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Next we’ll choose the “Subscribers” that will subscribe to this subscription…in other words…who will we notify. We’ll click “Add”.

Subscription - Subscribers Add

 

 

 

 

 

 

We’ll type in the name of our Subscriber and click “Search” then we’ll select it from the “Available subscribers” list and choose “Add”, then click “OK”.

Subscription - Subscriber Search

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Next we’ll need to choose the Channel by which Notifications will be delivered. We’ll click “Add”.

Subscription - Channels Add

 

 

 

 

 

 

 

 

 

 

We’ll type in the name of our Channel and click “Search” then we’ll select it from the “Available channels” list and choose “Add”, then click “OK”.

Subscription - Channel Search

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Now we’ll click “Finish”. But you can uncheck the box as highlighted below if you don’t want to turn on this Subscription just yet which is a good idea if you don’t quite want notifications being sent until you’re ready.

Subscription - Summary Enabled

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Right we’re done. Our PowerShell Script is ready, our Notification Channel, Subscriber and Subscriptions are all complete. We just need an alert to be generated.

 

Triggering an Alert

Obviously since I chose to trigger my command notification on ANY new alert, I could just wait and SCOM will trigger this command notification when an alert is raised, but I’ll use my trusty old Event Log script from other posts. This PowerShell Script simply sends an alert to the Application Log with the ID of 56 and I have a Rule configured that will alert on it.

So when we fire off our alert a few times, we can see our PowerShell Command Notification spring into action and update the alerts Resolution State to our new value of 2…which is “Notification Sent”

Alerts in Console

 

 

 

 

 

 

 

And remember we also wrote some information to our log file, so let’s open that up here you can see those entries being written.

Logfile

 

Everything is working well.

 

Wait, we have a problem

Yes we do. Whilst this solution works fine as you’ve just seen, it wont work when we have a lot of events come in at the same time. And by a lot, I mean more than 5…although that’s not really a lot as such.

When we do have more than 5 alerts try and use our Command Notification at the same time we’ll get a Warning Event that puts a halt on things. So here I’ve triggered my alert 30 times… and SCOM has chipped in with the following Warning Event: Operations Manager failed to start a process due to lack of resources.

In the Alert Description it clearly says that the number of asynchronous responses (5) has been reached and our command notification is clearly identified as the culprit. So what this means in the real world is that the first 5 alerts all were processed fine and the script ran perfectly. But the other 25 alerts never got processed by our script. They still have a Resolution State of New.

Operations Manager failed to start a process

 

Ouch. So can we fix this?

Sure we can but SCOM does have a fixed limit of 100 Asynchronous Responses, even though the default is set to 5. So here’s how you can change this value if you wish.

Open up “Regedit” and navigate to the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Modules\Global

Regedit - Global

 

 

 

 

 

 

 

 

 

If you have a key called “Command Executer” then select it. If not, right click the “Global” key and choose “New –> Key”. Call the key “Command Executer”.

Regedit - Global - New Key

 

 

 

 

 

 

 

 

Now right click on the new “Command Executer” key and choose “New –> DWORD (32bit) Value” and give it a name of “AsyncProcessLimit”. Switch to Decimal and enter in a new value anywhere up to 100. I wouldn’t recommend going straight to 100, maybe try a lower value like 10 or 20 and increase as you require. For the purposes of this lab I’ll set mine to 50.

Click “OK”.

Regedit - Global - DWORD

 

 

 

 

 

 

 

 

 

 

You’ll now have to restart the HealthService on your Management Server for this to take effect. At a Windows PowerShell console type in:

Restart-Service HealthService

 

And that’s it. This change should now be live and in effect.

So I’m going to fire off my Event Log script again, this time I’ll set it to 75 entries to try and trigger an error.

PowerShell Events

 

 

 

 

 

 

And this time we’re well past the default limit of 5 command notification executions and once we hit 75, our Warning Event is triggered but this time it clearly shows the maximum number has been increased to 50.

Operations Manager failed to start a process 50

 

 

So as you’ve seen working with Command Notifications isn’t difficult but you will need to give some thought as to how you want to work with them and if you have a really busy environment you might like to be selective in what alerts you’re passing to your script to avoid these Asynchronous warnings.

 

 
Comments

No comments yet.

Leave a Reply