Page History

\nContents

...

...

Overview

...

This article is part of a series:

...

Part 1 - Creating Custom Service Monitors in

...

Uptime Infrastructure Monitor
Part 2 - Creating

...

Custom Service Monitors with Retained Data Collection

...

Part 3 - Creating

...

Plugin Service Monitors in Uptime Infrastructure Monitor

Uptime Infrastructure Monitor

...

...

Article \nContents \n

...

\n \n
• How \nCustom Service Monitors Work
\n \n
• Step \n1 - Creating the Agent Side Script
\n \n
• Step \n2 - Creating the Monitoring Station Script
\n \n
• Step \n3 - Adding your Custom Service Monitor to up.time
\n \n

...

Overview
\n \n

...

up.time provides many built in service monitors that allow \nyou you to monitor and alert on a wide array of services and applications \nwithin within your enterprise. However, the standard up.time Uptime Infrastructure Monitor service monitors \nmay may not be adequate to monitor all of the systems, applications and \nproprietary proprietary devices in your environment. In some cases, you may want to \ncapture capture metrics that are unique to your environment. To do this, you \ncan can define custom service monitors to monitor custom services and \napplications applications to provide the level of in depth performance monitoring \nthat that you require. Using custom service monitors, you can monitor and \ngraph graph historical performance data \nfrom from just about any device in your environment.

...

This article series will focus on taking a simple command line \nscript script and incorporating it into up.timeUptime Infrastructure Monitor. It will then demonstrate how \nto to expand that script for use within a Custom with Retained Data \nservice service monitor that will enable you to not only register service \noutages outages in the event of a failure but also graph the historical \nperformance performance trends. The final article in this \nseries series will guide you through the steps to further expand your custom \nservice service monitor so that it can be directly plugged in to the up.time \ninterface Uptime Infrastructure Monitor interface for use by your end users.

...

...

How Custom Service Monitors Work

...

In Uptime Infrastructure Monitor

...

In up.time, you can configure custom service monitors to run \nany any application or script. The application or script can then take an \nactionaction, including launching commands on one of your agent systems using \na a built-in agent authentication protocol. Or, your script can use a \ndifferent different transport method to contact other servers and applications, \nor or simple run commands on the monitoring station itself. The only \nlimitation limitation is that the application or script must return status \ninformation that up.time information that Uptime Infrastructure Monitor will then use to determine the custom service \nmonitor monitor status.

...

The built-in agent authentication protocol provides a quick \nand and easy way for you to have the up.time Uptime Infrastructure Monitor agent run commands (or any \napplicationapplication) on your agent-side system without the fear of unknown \nusers users being able to execute the same commands. The output from the \nagentagent-side commands will be returned to the monitoring station custom \nscript script for parsing. This enables you to create a script that can \nmonitor monitor many different agent systems without duplicating the script \nmany many times. The following diagram illustrates the custom \nservice service monitor process:

...

 

\n \n

...

 

...

Step 1 - Creating the Agent Side Script

...

...

When designing a custom service monitor, you should consider \nwhat what information that you want the monitor to collect. For example:

...

• If you want to monitor the status of an application, you \nshould should first create a command line script or utility (or \nusing using an existing one) that returns the general status of a \nservice service that is on the agent system. 
\n \n
• . 
• If you want to gather performance metrics, ensure that the script \nto to return a series of performance indicators to the console, otherwise \nsimple simple output a message that can later be used to validate the status \nof of the service or application. 
\n \n

...

Once you have written your agent-side script, you will need to \nconfigure your up.time configure your Uptime Infrastructure Monitor agent to execute it. As a minimum, you must configure the following settings for each agent-side command:

...

• For agents running on non-Windows systems, define a password and command pair in the agent-side password file.
\n \n
• For agents running on Windows, set a password via the Agent Console.
\n \n
• Verify that the agent-side \ncommand command permissions allow the execution of the command by the user that the \nagent agent is run as. If the commands within your script require additional \nsystem system permissions, you may have to alter the security settings for the execution of the \nagentagent.
\n \n

...

For the purpose of this example, lets assume that we have \ndesigned designed an agent side script that returns information about the \ntemperature temperature and humidity levels around a given agent system. If we were \nto to run this agent side script from the command line it would be \nexecuted executed as follows:

...

Note
title Note
These example commands are formatted for non-Windows systems. The

...

command format is very similar on Windows systems, but require minor syntax and

...

path changes.

...

Code Block
language powershell

...

# su -

...

uptime $ /opt/uptime-agent/my-scripts/show_temp.sh

...

temp 74 rh 30

...

To define your password and command pair on a non-Windows \nagent agent system, do the following:

...

\n \n
1. Ensure that a file named .uptmpasswd is created in the \nup.time Uptime Infrastructure Monitor agent bin directory -- either /opt/SPYNuptm/bin/ or \n /opt/uptime-agent/bin, depending the version of your agent. This file must \nbe be owned and readable by the user that the agent is run as (ex. uptimeagent or uptime users).
\n \n
3. Open the file .uptmpasswd in a text editor.
\n
5. Enter a password and command pair for each command that you want to run on the agent \nsystemsystem. The following is the example format \nand and an example of a agent-side commands: \n

   Format: [password] [command path, no arguments]

   \n

   ]

   Example file contents:

   \n

   Code Block
   language powershell
   secretpassword /opt/uptime-agent/my-scripts/show_temp.sh

   
   

   $%^& /usr/local/bin/appstatus.sh

   \n
\n

...

Windows agents do not require a password file. However, you must enter the equivalent settings into the Agent Console with the following steps:

...

1. Launch the Agent Console on the system where the script will run. By default this is located in the Start Menu under “All Programs -> up.time Uptime agent -> up.time Uptime Agent Console.”
\n
3. The first step after launching the Agent console is to set a password in the ‘Agent Options’ section and click save. This password will be used to authenticate with the agent before running the script.
\n
5. After setting the password, tell the agent where the custom script is located on the server by clicking on the ‘Advanced’ option in the dropdown list and selecting ‘Custom Scripts.’
\n
7. On the Custom Scripts screen, you should see an example similar to:
   \n
   • Custom Command: dir
   \n
   • Path to Script: cmd.exe /c "dir c:"
\n
9. To continue this show-temp example, set the parameters to something similar to: \n
   • Custom Command: temp
   \n
   • Path to Script: cmd.exe /c “C:scriptsshow_temp.pl” \n (note: make sure to click Add/Edit before closing the window, otherwise your command will not be saved).
\n
10. If you haven’t done so yet, click Save, and Yes when asked to restart the agent. This will ensure that your new settings take effect.
\n

...

The following is a sample configuration. Note the additional folder named rexec_commands, which contains a key/value pair of the command key.

...

Note
title Note
If any changes are made to the Windows agent registry please restart the

...

"up.time agent" service to see these changes take effect.

...

...

Code Block
language text

[HKEY_LOCAL_MACHINE\SOFTWARE\uptime software\up.time agent]

...

"CmdsTimeout"=dword:00000014

...

"Port"=dword:0000270e

...

"MaxClients"=dword:00000005

...

"Debug"=dword:00000000

...

"CmdsPassword"="secretpassword"

...

"LogFile"="log.txt"

...

[HKEY_LOCAL_MACHINE\SOFTWARE\uptime software\up.time agent\rexec_commands]

...

"dir"= cmd.exe /c "dir c:"

...

"show-temp"= cmd.exe /c "C:\scripts\show_temp.pl"

...

...

Step 2 - Creating the Monitoring

...

Station Script

The next step is to write the monitoring station script that \nwill will attempt to execute your agent side script, validate the output, \nand and then return status or performance information to up.time.

...

Uptime Infrastructure Monitor.

The monitoring station script can be written in any language \nand and format, provided that it adheres to these general rules:

...

\n \n
1. The script must return at least one line of output to the \nsystem system console. up.time Uptime Infrastructure Monitor will use this output to determine the status of \nthe the custom service monitor.
\n \n
3. The script must exit with a success status, unless \nthere there has been a problem during the execution of the script or you want to \nforce force a status for the service monitor. Acceptable return codes are: \n
   \n \n
   • exit 0 - The \nscript script executed without any errors, all output produced from the script \nwill will be parsed by up.time Uptime Infrastructure Monitor to determine the service monitor status.
   \n \n
   • exit 1 - The \nservice service monitor status will be set to WNG (warning) and an outage will be \nrecordedrecorded. The console output from the script will be placed in the \ncustom custom service monitor status message.
   \n \n
   • exit 2 - The \nservice service monitor status will be set to CRIT (critical) and an outage will be \nrecordedrecorded. The console output from the script will be placed in the \ncustom custom service monitor status message.
   \n \n
   \n \n
\n \n
4. The script must accept the host name of the agent system as \nthe the first argument. up.time Uptime Infrastructure Monitor will automatically add this argument to the \narguments arguments that are passed to the script. \n
\n \n

...

In most cases, the monitoring station script will contact an agent system and attempt to run a predefined agent-side script like the one that was defined in step 1 above. You can use any transport tool -- such as rexec, rsh, or ssh -- to contact the agent system.

...

However, uptime software recommends using the bundled agentcmd utility the netcat utility to contact your agent systems from your monitoring station script:

...

:

• To get started with custom scripts, or for implementations with only a handful of custom monitor instances, use agentcmd. The agentcmd utility is commonly called in the format listed below to execute a command on the agent side system. Please note that this utility is a helper tool, if you are using many custom monitor instances we recommend using an alternate tool.
  \nThis This example uses the settings that were configured above to execute the agent side script: \n

  Format:

  Code Block
  language text
  /usr/local/uptime/scripts/agentcmd [-s/+s] -p [agent port] [agent hostname] rexec [password] [path]

  \n \n

  Example:

  Code Block
  language text
  /usr/local/uptime/scripts/agentcmd -p 9998 my-agent rexec secretpassword /opt/uptime-agent/my-scripts/show_temp.sh my-arguments

  \n \n

  For more information click on Using the syntax used with agentcmd, see this Knowledge Base article.

  \n

  agentcmd Utility.

• If you plan on implementing a number of custom monitor instances, use netcat. netcat must be downloaded and installed on the monitoring station if it isn't already installed as part of your operating system. The netcat utility is commonly called in the format listed below to execute a command on the agent side system. \nThis This example uses the settings that were configured above to execute the agent side script: \n

  Format:

  Code Block
  language text
  echo -n rexec [password] [path] | /usr/local/uptime/bin/netcat [agent

  \nhostname

  hostname] [agent port]

  \n \n

  Example:

  Code Block
  language text
  echo -n rexec secretpassword /opt/uptime-agent/my-scripts/show_temp.sh my-arguments | /usr/local/uptime/bin/netcat my-agent 9998

  \n
\n \n \n

...

...

You will normally use netcat or agentcmd in the monitoring station script to return the results of an agent-side script, validate the status of those results, and return the status to up.timeUptime Infrastructure Monitor. The following is an example of a monitoring station script:

...

#!/bin/sh

...

# This script takes the following arguments:
# check_temp.sh hostname port [temp|rh] wng

...

 crit
# Example execution:
# ./check_temp.sh my-agent 9998 temp 60

...

 80
# ./check_temp.sh my-agent 9998 rh 20

...

 30

# This script can be placed anywhere on the monitoring station system as long as it is
# executable by the uptime user.

#First, collect our arguments
AGENT=$1
PORT=$2
TYPE=$3
WNG=$4
CRIT=$5

TMPFILE=/tmp/$$.temp

#
\n
\n \n# now use the info above to contact our agent, store the output in a file for parsing
\n \n`echo

`echo -n rexec secretpassword /opt/uptime-agent/my-scripts/show_temp.sh

...

 my-arguments | /usr/local/uptime4/bin/netcat $AGENT $PORT > $TMPFILE`

...

...

...

...

'/usr/local/uptime/scripts/agentcmd my-agent -p 9998 rexec secretpassword /opt/uptime-agent/my-scripts/show_temp.sh my-arguments > $TMPFILE'

...

# we have the output from the agent. If it is ERR that means there was a problem running the script on the agent
`grep ERR $TMPFILE`
if [ $? -eq 0 ]
then
echo "Could not execute agent side script!" 
# by exiting with a 2 we are forcing a CRIT service outage
exit 2
fi

# given our parameters we can now extract the correct value from the agent output
if [ $TYPE -eq "temp" ]
then
VALUE=`head -1 $TMPFILE | awk '{print $2}'`
MSG="temperature"
else
VALUE=`tail -1 $TMPFILE | awk '{print $2}'`
MSG="humidity"
fi

# now lets check our values to see if they are over the thresholds and set our status message
RET="OK - $msg is $VALUE on $AGENT"

if [ $VALUE -ge "$WNG" ]
then
# this is our warning message
# include WARNING for use in the web interface thresholds
RET="WARNING - $MSG is $VALUE on $AGENT"
fi
if [ $VALUE -ge "$CRIT" ]
then
# this is our critical message
# include CRITICAL for use in the web interface thresholds
echo "CRITICAL - $MSG is $VALUE on $AGENT"
fi

# here we simply print our status message to the console and exit with a 0, 
# the thresholds provided in the up.time web interface will be used to set the monitor status
rm $TMPFILE
echo "$RET"
exit 0

...

Step 3 - Adding your Custom Service Monitor to Uptime Infrastructure Monitor

...

\n# we have the output from the agent. If it is ERR that means there was \na problem running the script on the agent
\n \n`grep ERR $TMPFILE`
\n \nif [ $? -eq 0 ]
\n \nthen
\n \necho "Could not execute agent side script!"
\n \n# by exiting with a 2 we are forcing a CRIT service outage
\n \nexit 2
\n \nfi
\n \n
\n \n# given our parameters we can now extract the correct value from the \nagent output
\n \nif [ $TYPE -eq "temp" ]
\n \nthen
\n \nVALUE=`head -1 $TMPFILE | awk '{print $2}'`
\n \nMSG="temperature"
\n \nelse
\n \nVALUE=`tail -1 $TMPFILE | awk '{print $2}'`
\n \nMSG="humidity"
\n \nfi
\n \n
\n \n# now lets check our values to see if they are over the thresholds and \nset our status message
\n \nRET="OK - $msg is $VALUE on $AGENT"
\n \n
\n \nif [ $VALUE -ge "$WNG" ]
\n \nthen
\n \n# this is our warning message
\n \n# include WARNING for use in the web interface thresholds
\n \nRET="WARNING - $MSG is $VALUE on $AGENT"
\n \nfi
\n \nif [ $VALUE -ge "$CRIT" ]
\n \nthen
\n \n# this is our critical message
\n \n# include CRITICAL for use in the web interface thresholds
\n \necho "CRITICAL - $MSG is $VALUE on $AGENT"
\n \nfi
\n \n
\n \n# here we simply print our status message to the console and exit with \na 0,
\n \n# the thresholds provided in the up.time web interface will be used to \nset the monitor status
\n \nrm $TMPFILE
\n \necho "$RET"
\n \nexit 0
\n \n

...

Step 3 - Adding your Custom \nService Monitor to up.time
\n \n

...

The next step is to add a custom service monitor to the up.time Uptime Infrastructure Monitor Web interface using the same process \nthat that you would use to add a standard custom service monitor to up.time. \nThe Uptime Infrastructure Monitor. The Custom monitor option is found in the List Other Monitors \nsection section of the Add New \nService Service Instance page..

...

The Custom service monitor template has the following \nmonitor monitor specific settings available:
\n \n
\n \n 

...

Based on the settings used in the example monitoring station script, configure the monitor with the following setting: \n

...

\n \n
• Enter a name \nand and description for the monitor.
\n
• Select a host from \nthe the dropdown menu. Be sure to select the same host that your agent side \nscript script is on
\n
• In the Script Name field, \nenter enter the path to the custom script on your monitoring station. On \nWindows Windows systems be sure to use UNIX style / instead of \ \nand put and put quotation marks around your path. For example: "C:/my files/check_temp.bat" \n
\n
• In the Arguments field, \nenter enter the arguments for the script. up.time Uptime Infrastructure Monitor adds the agent name as the \nfirst first argument automatically so do not include it.
\n
• Select contains from the Warning dropdown and \nenter enter WARNING \nas as the search text.
\n
• Select contains from the Critical drop down \nand and enter CRITICAL \nas as the search text.
\n
• Complete the \nremainder remainder of the monitor template as you would for a normal service \nmonitor.
\n

...

• monitor.

Example \nmonitor configuration

...

monitor configuration

The image below illustrates a sample monitor configuration. \nThis This service monitor will indicate a WARN or CRIT whenever \nthe the monitoring station \ncustom custom script returns WARNING \nor  or CRITICAL \nin in its output.

...

Image Removed

...

Image Added