Installing SCOM AIX (Unix or Linux) Agents behind Chained Gateway Servers

Today I was called to help configure SCOM monitoring for some AIX 7 Servers. Ok, not a hard task as such but during the meeting I found out that I was going to have quite a few hurdles to contend with. These AIX Servers were located in a Customers Network behind some chained gateways and the Management Servers were in one network and the only port opened between the two networks was TCP 5723 inbound from the Customer Networks Gateway Server to an Internal Gateway Server on the Management Network where all of the SCOM Servers were located. I also didn’t have an account in the customer network so I can’t push out the agent…oh and I need to get it working today! Sounds like it’s going to be a fun day…

So the network in question looks like this:

AIX Network

 

So from our Management Servers we cannot connect to the AIX Servers directly.

AIX MS to AIX No

 

From the Internal Gateway we don’t have any access either. We simply have no access from the Management Network.

AIX Internal Gateway to AIX No

 

The only server that does have any direct connection to the AIX Servers is the Customer Gateway Server.

AIX Customer Gateway to AIX

 

 

So we’ll use this Customer Gateway to manage the AIX Agents. Makes sense anyway, it’s on the same network as the AIX Server.

My second issue is that I do not have the appropriate system rights to push the agent to the AIX Servers given the fact that the required Ports for monitoring Unix/Linux (TCP Ports 22 and 1270) are not open from the Management Network and they will not be allowed to be open. I also don’t have an account on the AIX Servers so someone else will be installing the agent. So, the agent installation in this case will be manual and we’ll discover the agent from the Management Consoles back in the Management Network.

So there’s a few steps required, let’s summarize those steps first.

  1. Download and install the Appropriate Management Packs for AIX (or any other Unix/Linux flavor).
  2. Restart the HealthService on the Management Server.
  3. Copy the AIX Agent file over to the AIX Server and install it.
  4. Copy the scx-host-<hostname>.pem file to the appropriate server and sign it.
  5. Copy the signed scx_new.pem file back to the AIX Server then restart the Agent Service.
  6. Create a Resource Pool and assign the appropriate servers to it.
  7. Create a standard user account on the AIX Server.
  8. Create a Run As Account in SCOM, use the details of the AIX User account created in the previous step.
  9. Add this Run As Account to our Unix/Linux Run As Profiles.
  10. Discover the AIX Agent.

 

Download and Install the Agents and Management Packs

So let’s begin by locating the appropriate Agent Software and Management Packs. You can find those here: https://www.microsoft.com/en-au/download/details.aspx?id=29696

Run the MSI file to extract the compacted files and then on your Management Server import the required Management Packs. To do this go to Administration –> Management Packs and choose “Import Management Packs”.

Then we’ll choose “Add –> Add from disk …”

AIX Add MP

 

 

 

 

 

 

 

 

 

In my case I didn’t need HPUX, Solaris or any of the Linux Flavors, so I just installed the generic Unix Files and of course the AIX 7 Management Pack.

AIX MP

 

 

 

 

 

 

 

 

 

Restart the HealthService on the Management Server

Now let’s restart the Microsoft Monitoring Agent Service on our Management Server. Open up the Services MMC and restart the service shown below.

AIX Restart Microsoft Monitoring Agent Service

 

 

 

 

Copy the Agent to the AIX Server and Install it

Now lets navigate to the location where we have SCOM installed, the default being “C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\AgentManagement” and we should have a “UnixAgents” folder. Let’s open that up…

AIX Files

 

 

 

 

 

 

 

and inside the “DownloadedKits” folder we will find our Agent files. In my case I only have the AIX 7 agent as that’s all I need but you might have more depending on what Operating Systems you are going to be monitoring.

AIX Agent Files

 

 

 

 

 

 

So we’ll need to copy this Agent file to our AIX Server. How you do that is up to you.

Because of the situation I found myself in, I was not in a position to automatically push the agent out to the AIX Servers and I also wasn’t able to install it myself due to the fact that I didn’t have access to the AIX Servers. But here are the steps required to do it.

Transfer the agent (scx-<version>-<os>-<arch>.gz) to the AIX server, type:
cp scx-<version>-<os>-<arch>.gz

To unzip the package, type:
gzip -d scx-<version>-<os>-<arch>.gz

To install the package, type:
/usr/sbin/installp -a -d scx-<version>-<os>-<arch> scx

To verify that the package is installed, type:
swlist scx

To verify that the Microsoft SCX CIM Server is running, type:
ps -ef|grep omi

Look for the following process in the list:
omiserver

 

You can find the steps for the other flavors of Unix and Linux here: https://technet.microsoft.com/en-us/system-center-docs/om/manage/install-agent-and-certificate-on-unix-and-linux-computers-using-the-command-line

 

When we deploy a Unix Agent from the Management Server using the discovery wizard, it will handle the agent installation and certificate signing. In our case though since we installed our agent manually, it created a self signed certificate. That’s no good to us because our SCOM environment won’t trust it as SCOM is located in a totally different network. We do have a certificate already installed on the Customer Gateway server so that it can be trusted. So the next step is to sign that AIX Certificate from a source that we do trust – The Customer Gateway Server!

Now in cases where you might have your Unix Agents in the same network as your SCOM Management Servers, then these certificates would probably be signed from the SCOM Management Servers themselves (or at least those that are part of the appropriate Resource Group, but more on that step later). But in my case, I’ll need to get the client side Gateway Server to sign that certificate. My Management Servers trust that Gateway and it can trust the AIX Server.

So the first step is to navigate following directory on our AIX Server:
/etc/opt/microsoft/scx/ssl/ and locate the file scx-host-<hostname>.pem.

Copy this file over to our Gateway Server and for the purposes of this post, I’m going to place the file in C:\Temp.

Now from a command prompt we’re going to run the scxcertconfig command which we can find in “C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server”.

scxcertconfig.exe -sign C:\temp\scx-host-AIXServer01.pem C:\temp\scx_new.pem

This will output a new file in C:\temp\ called scx_new.pem.

AIX Certificate Signing on Gateway

 

 

 

 

 

 

 

 

 

 

 

We now need to take the resulting signed certificate file “scx_new.pem” and copy it back to our AIX Server and overwrite the original scx-host-<hostname>.pem file.

Finally we will need to restart the AIX agent by typing
scxadmin -restart

 

Create an AIX User Account

You might already have this step performed ahead of time but in order for SCOM to be able to do its job and monitor our server we’ll need to have a username/password.

Since my installation here was for AIX, here’s a link for creating a user account but you can Google/Bing for your own flavor of Unix/Linux. https://www.ibm.com/developerworks/aix/library/au-aixuseradmin/

 

Create a Resource Pool

A SCOM Resource Pool by definition is a group of Management Servers or Gateway Server (not a mix of each) that can distribute work amongst each other to manage objects. In our case we’re talking about Unix Servers so we could have one or more servers in a resource pool to help performance and if we have more than one server in that resource pool and one server goes down, the other can take over giving us some form of fault tolerance.

In my case I only have a single Gateway Server in the Customers Network and we’re going to add this to our Resource Pool. But first, we need to create the Resource Pool.

So in the SCOM Console we’ll navigate to “Administration –> Resource Pools” and as you can see in a default SCOM installation there are 3 Resource Pools.

AIX Default Resource Pools

 

 

 

 

We’re not going to use any of these, we’ll create a new Resource Pool specifically for our AIX Servers.

We’ll right click and choose “Create Resource Pool”.

AIX Create Resource Pool

 

 

 

 

 

 

We’ll need to give our Resource Pool a name…just bear in mind this post I wrote a while back regarding the use of special characters in a Resource Pool name. Call it something simple 🙂

AIX Resource Pools - 01 Name

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Now we need to add the Management Servers or Gateway Servers that will be members of this resource pool.

AIX Resource Pools - 02 Add

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

So we’ll click “Add”.

And let’s find the server or servers that are going to be part of this Resource Pool. In my case there is only a single Gateway Server out at the Customers Site, I’ll choose that and Add it to my Resource Pool by selecting it, clicking “Add” and then “OK”.

AIX Resource Pools - 03 Add Gateway

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Finally we’ll click “Next” and then click “Create”.

Now in my case because I only have a single Gateway Server added to my Resource Pool, this step is complete. But if you have more than one Gateway Server or Management Server in your pool then you will need to exchange cross platform certificates. This is very easy to do and is a required step only if you have more than one server.

So in my diagram at the top of this post you can see I have 2 Management Servers. So if hypothetically I was using those servers to monitor these AIX Servers, then in I would need to add both servers to my Resource Pool, then exchange certificates. So here’s how we exchange those certificates.

 

On Management Server 1:

Open a command prompt and type in the following:
cd C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server
scxcertconfig.exe -export C:\Temp\MS1.cer

AIX Export MS1 Cer

 

 

 

This will export the certificate to our C:\Temp folder. We could copy it to a share if it makes it easier as we’ll need to import this certificate to our second Management Server. I’ll just copy the file to the C:\Temp folder on Management Server 2.

So on Management Server 2:

Open a command prompt and type in the following:
cd C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server
scxcertconfig.exe -import C:\Temp\MS1.cer

AIX Import MS1 Cer

 

 

 

Now we need to do the same but in reverse.

On Management Server 2:

Open a command prompt and type in the following:
cd C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server
scxcertconfig.exe -export C:\Temp\MS2.cer

AIX Export MS2 Cer

 

 

 

 

Then we’ll copy the file to the C:\Temp folder on Management Server 1.

So on Management Server 1:

Open a command prompt and type in the following:
cd C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server
scxcertconfig.exe -import C:\Temp\MS2.cer

AIX Import MS2 Cer

 

 

 

Now we’ve successfully exchanged certificates. If you have more than 2 servers, just do the same again but make sure that every management server has the certificate for all other servers in the Resource Pool.

 

If you open up the Certificates MMC and go to Trusted Root Certification Authorities you should then see both certificates imported. But again, in a single server scenario like I have here, this step is not required.

AIX Cross Platform Certificates

 

 

Create Unix/Linux Run As Account

So in the SCOM Console we’ll navigate to “Administration –> Run As Configuration –> Unix/Linux Accounts” and we’ll right click and choose “Create Run As Account”.

AIX Create Run As Account

 

 

 

 

We’ll choose a “Monitoring account” and click “Next”.

AIX Unix Monitoring Account

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

We’ll give it a name. I’m calling mine “Unix Run As Account”.

AIX Unix Run As Account

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

We’ll then need to enter in the name and the password for the account we created on the AIX Server.

AIX Your Unix Account

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

We’re going to choose a “More Secure” Distribution and we’ll come back and configure this in a moment once our Run As Account is created.

AIX Distribution Security

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

And we’re done so let’s click “Create”.

Now we’ll double click on our newly created Unix Run As Account

AIX Unix Run As Account Created

 

 

 

 

 

and go to “Distribution Security” and click “Add”.

AIX Distribution Security Add

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

We’ll locate our Resource Pool, not the individual Management Servers or Gateways. Remember it’s our Resource Pool members that we want managing these Unix Servers, so from the drop down we’ll choose “search by resource pool name”

AIX Search Resource Pool

 

 

 

 

 

 

 

 

and we’ll locate our Unix Resource Pool and click “Add” then click “OK”.

AIX Locate Unix Resource Pool

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

And finally we’ll click “Save”.

AIX Unix Resource Pool Save

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The next step is to associate our new Run As Account with our Unix Run As Profiles.

AIX Unix Run As Profiles

 

 

 

 

So let’s open up the first one, the Action Account and we’ll click “Next” a couple of times, then “Add”.

AIX Run As Profile Add

 

From the drop down window we’ll choose our “Unix Run As Account” and leave the default at “All targeted objects” and click “OK”.

 

AIX Add a Run As Account

 

 

 

 

 

 

 

 

 

And then we’ll click “Save”.

Now if you’re planning on using the same account for all Run As Profiles as I will be doing since I’ve only been given a single account, we’ll just repeat this process for the remaining two Run As Profiles.

 

Discover the Agent

The final step is to discover the agent and have it managed by SCOM. So from “Administration” we’ll click the “Discovery Wizard…” link.

AIX Discovery Wizard Link

 

 

 

 

And choose “UNIX/Linux computers” in the middle of the window and click “Next”.

AIX Discovery Wizard

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Now we’ll need to define the discovery criteria for our AIX Agent or Agents. We’ll click “Add”.

AIX Discovery Criteria Add

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Then we will enter in the details for the agents we wish to discover. We can of course enter in multiple servers by adding more rows. I only have the one for this example.

In the drop down box, we’ll choose “Only computers with an installed agent and signed certificate” as we’ve already installed the agent manually and applied a certificate.

AIX Discovery Criteria Servers with Agent

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Finally we’ll click “Set credentials…”.

AIX Discovery Your User Account

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

We’ll enter in our Username and Password and click “OK”.

AIX Discovery Your User Account Save

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Now we’ll click “Save” and we can start the Discovery Process…

AIX Discovery Discover in Progress

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

When the agent/s have been discovered we can click “Manage” and the agent will be monitored by SCOM.

AIX Discovery Successul

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Click “Done” and now we’ll have to wait for the agent to become healthy. This might take a little while so be patient.

And if all goes well we can now see our agent is healthy.

AIX Agent Healthy

 

 

 

 

 

 

Just a final piece of advice, sometimes I’ve found some agents immediately go grey after being discovered and don’t go healthy, so after a successful discovery you could restart the AIX Agent service and the agent should go green.

Happy AIX Monitoring 🙂

 

 
Comments

No comments yet.

Leave a Reply