When SCOM produces alerts we have a few ways of notifying people that an alert has occurred. We can simply look in the console and see alerts there. We can send an Email, Instant Message, we can send an SMS or even forward alerts to other applications like a ticketing system. We can also use a Command Notification to run a command or a script. So in this post, I’ll show you how you can use a Command Notification to run a Windows PowerShell script which will allow us to perform some sort of task after an alert has been generated.
As with any new Notifications, we’ll need to create a Channel, a Subscriber and a Subscription.
- Channels are the type of notification – eMail, SMS, IM or run a command/script.
- Subscribers are who we will notify – an email address, a Distribution List, a script to perform some task.
- Subscriptions are the what we will notify people of. Specific alerts, critical alerts, new alerts, alerts from specific classes and so forth.
Creating a Notification Channel
So in the SCOM Console, we’ll go to the Administration Window
and under the Notifications Folder we’ll choose “Channels”.
Now we’ll right click and choose “New –> Command” or head over to the Actions Pane and choose “New –> Command”.
The good news is that this will be a very short wizard…we just need to provide a name for our Command Channel and a description advising other administrators of what this Command Channel will be used for.
Now we’ll click “Next” and we’ll need to enter in where our script can be found and any parameters we might like to pass it.
So here’s the command line options I’ve chosen for this example.
Full path of the command line:
Command line parameters:
“C:\Scripts\Notifications\Culham-TestNotifications.ps1” ‘$Data/Context/DataItem/AlertName$’ ‘$MPElement$’ ‘$Data/Context/DataItem/AlertId$’
Startup folder for the command line:
So we’re providing the path to Windows PowerShell and passing it the name of our script in the first text box and we’ll use the same path of the PowerShell folder for the startup folder field. But in the Command line parameters field, we’re going to pass our script some parameters.
In my example I’m passing in the AlertName, the MPElement ID and the AlertID. If we click the little arrow icon to the right of this field we can see some of the other parameters we can pass to our script.
You could use any of these:
UTC Date/Time of DataItem Created
LocalTime Date/Time of DataItem Created
UTC Date/Time DataItem was Modified
Local Date/Time DataItem was Modified
Custom Field 1
Custom Field 2
Custom Field 3
Custom Field 4
Custom Field 5
Custom Field 6
Custom Field 7
Custom Field 8
Custom Field 9
Custom Field 10
Created by monitor? (true/false)
Management Server Principal Name
Web Console URL
Workflow ID (GUID)
UTC Date/Time the Alert was Resolved
Time Raised (Local)
Time Raised (UTC)
Time Added (Local)
UTC Time Added
Alert Severity ID
Person Resolving the Alert
The Resolution State Name (New, Closed etc.)
Date/Time Resolution State was Last Modified (Local)
Date/Time Resolution State was Last Modified (UTC)
Resolution State ID (0=New 255=Closed etc.)
Alert Repeat Count
Managed Entity Path
ManagedEntity Full Name
ManagedEntity Display Name
Or you can find a complete list of variables here.
And speaking of our script, let’s take a look at the example I’m using for this blog post.
# Load the OpsMgr Provider
# Locate the Specific Alert
$alert = Get-SCOMAlert -Id $alertID
$strResolutionState = ($alert.resolutionstate).ToString()
$localtimeRaised = ($alert.timeraised).tolocaltime()
$severity = $alert.Severity.ToString()
# Write to a log file
$Logstring = $AlertName + ” ” + $SubscriptionID + ” ” + $AlertID + ” ” + $severity
$Logfile = “c:\Scripts\Notifications\LogAlerts.log”
$DateTime = Get-Date -Uformat “%y-%m-%d %H:%M:%S”
$Logstring = $DateTime + ” ” + $Logstring
Add-content $Logfile -value $Logstring
# Set the Alert in SCOM Console to a New Resolution State
$alert | Set-SCOMAlert -ResolutionState 2
So basically what this script aims to do is to take the 3 input parameters of the AlertName, the Subscription ID and the AlertID and then write that information out to a logfile and then updates the resolution state in SCOM itself. And it’s not a particularly useful example for the real world, but it does highlight how we can take those input parameters, run a script, output that data to a file which will let us know that we’ve found the right alert and successfully processed it and also update SCOM to complete the round trip.
Now since this script does call for updating a SCOM resolution state, let’s set up a new one. In the script I’ve used the Resolution State of 2 which doesn’t exist in a new SCOM environment. So in our SCOM console we’ll go to the Administration view and under Settings we’ll choose “Alerts”.
We’ll add a New Resolution State and provide a Name and a numeric ID.
We’ll add a New Resolution State and provide a Name and a numeric ID.
I’ll use the values of “Notification Sent” and give it a state of 2 which will match the value of my script.
Now that we’ve created the new Resolution State and obviously we’ve created our Channel, we need to create both a Subscriber and a Subscription.
Creating the Subscriber
So for our subscriber, we’ll right click and choose “New” or use the Actions Pane, choosing “New”.
First we’ll enter in a name for our Subscriber.
Now we’ll choose a schedule for this subscriber to adhere to. I’ve left the defaults to ‘Always send notifications’.
Finally we’ll choose the addresses. We don’t have any here so we’ll click “Add”.
We’ll provide this with a name and click “Next”.
We’ll choose a Command Channel Type and select our Command Channel from the drop down box. Then we’ll click “Next”.
Finally we’ll choose the schedule. Again, I’ll leave the defaults and click “Finish”.
And we’ll click “Finish” once more.
Our Subscriber has been created.
Creating the Subscription
The last thing we need to do is create a subscription, or in other words, set what criteria we’ll choose to run our PowerShell script. Will it be critical alerts? Any alert? Alerts from a specific class? Basically it comes down to what you wish to use. For the purposes of this post I’ll just use any new alert.
So we’ll create a “New” Subscription.
We’ll give it a Name and a Description then click “Next”.
We’ll choose the Criteria that must be met in order to trigger this Command Notification. We have so many options to choose from here, and most likely you’ll be choosing specific types of alerts or alerts from certain classes – maybe you want to assign them to different teams or something else entirely. But for the purposes of this post, I’m going to choose nothing and that’ll leave this Criteria at “All Alerts”.
Next we’ll choose the “Subscribers” that will subscribe to this subscription…in other words…who will we notify. We’ll click “Add”.
We’ll type in the name of our Subscriber and click “Search” then we’ll select it from the “Available subscribers” list and choose “Add”, then click “OK”.
Next we’ll need to choose the Channel by which Notifications will be delivered. We’ll click “Add”.
We’ll type in the name of our Channel and click “Search” then we’ll select it from the “Available channels” list and choose “Add”, then click “OK”.
Now we’ll click “Finish”. But you can uncheck the box as highlighted below if you don’t want to turn on this Subscription just yet which is a good idea if you don’t quite want notifications being sent until you’re ready.
Right we’re done. Our PowerShell Script is ready, our Notification Channel, Subscriber and Subscriptions are all complete. We just need an alert to be generated.
Triggering an Alert
Obviously since I chose to trigger my command notification on ANY new alert, I could just wait and SCOM will trigger this command notification when an alert is raised, but I’ll use my trusty old Event Log script from other posts. This PowerShell Script simply sends an alert to the Application Log with the ID of 56 and I have a Rule configured that will alert on it.
So when we fire off our alert a few times, we can see our PowerShell Command Notification spring into action and update the alerts Resolution State to our new value of 2…which is “Notification Sent”
And remember we also wrote some information to our log file, so let’s open that up here you can see those entries being written.
Everything is working well.
Wait, we have a problem
Yes we do. Whilst this solution works fine as you’ve just seen, it wont work when we have a lot of events come in at the same time. And by a lot, I mean more than 5…although that’s not really a lot as such.
When we do have more than 5 alerts try and use our Command Notification at the same time we’ll get a Warning Event that puts a halt on things. So here I’ve triggered my alert 30 times… and SCOM has chipped in with the following Warning Event: Operations Manager failed to start a process due to lack of resources.
In the Alert Description it clearly says that the number of asynchronous responses (5) has been reached and our command notification is clearly identified as the culprit. So what this means in the real world is that the first 5 alerts all were processed fine and the script ran perfectly. But the other 25 alerts never got processed by our script. They still have a Resolution State of New.
Ouch. So can we fix this?
Sure we can but SCOM does have a fixed limit of 100 Asynchronous Responses, even though the default is set to 5. So here’s how you can change this value if you wish.
Open up “Regedit” and navigate to the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Modules\Global
If you have a key called “Command Executer” then select it. If not, right click the “Global” key and choose “New –> Key”. Call the key “Command Executer”.
Now right click on the new “Command Executer” key and choose “New –> DWORD (32bit) Value” and give it a name of “AsyncProcessLimit”. Switch to Decimal and enter in a new value anywhere up to 100. I wouldn’t recommend going straight to 100, maybe try a lower value like 10 or 20 and increase as you require. For the purposes of this lab I’ll set mine to 50.
You’ll now have to restart the HealthService on your Management Server for this to take effect. At a Windows PowerShell console type in:
And that’s it. This change should now be live and in effect.
So I’m going to fire off my Event Log script again, this time I’ll set it to 75 entries to try and trigger an error.
And this time we’re well past the default limit of 5 command notification executions and once we hit 75, our Warning Event is triggered but this time it clearly shows the maximum number has been increased to 50.
So as you’ve seen working with Command Notifications isn’t difficult but you will need to give some thought as to how you want to work with them and if you have a really busy environment you might like to be selective in what alerts you’re passing to your script to avoid these Asynchronous warnings.