Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Web Application Transactions

A Web transaction is a series of Web pages that together fulfill a specific function for end users. A common Web transaction example is the checkout process on an e-commerce site, during which end users select a shipping option, pay for their items, and have their credit card verified. During this transaction, many calls are made to the application and data layers as the end-user provides, and the servers process, information.

Although the type of Web application that is monitored by Uptime Infrastructure Monitor users is typically different (e.g., intranet applications), the structure of the transaction is the same: an end user steps through a sequence of Web pages that take inputted information and initiate appropriate actions with application or database servers.

The Uptime Infrastructure Monitor Web Application Transaction monitor tests the speed and availability of an end-user Web transaction. Specifically, the Web Application Transaction monitor performs two roles:

  • it confirms the general availability of an end-user Web transaction by executing a previously recorded script then reporting whether all pages that make up the web transaction were successfully processed
  • it reports on the speed of the Web transaction both as a whole, and broken down by previously defined stages

Both the availability and speed of Web transactions can be used in reports and as triggers for alerts.

Using the Web Application Transaction Monitor

Use the Web Application Transaction monitor to record a series of URLs that together make up a transaction. This recording should be of a transaction that acts as a suitable test of your Web application delivery infrastructure.

During the recording process, declare checkpoints that demarcate significant stages in the Web transaction. Isolating the different stages in an end-user transaction allows you to view stage-specific speed tests in reports, which ultimately helps you identify where problem areas exist.

For example, if a transaction relies on processing on the application layer, makes multiple calls to the data layer, and is accessible worldwide, creating checkpoints during the recording phase helps you ascertain whether the application server, database management server, or network may be the reason behind a poorly performing transaction.

The following sample checkpoints could be created for an e-commerce transaction:

  • Browse Catalog
  • Add to Shopping Cart
  • Checkout
  • Credit Card Validation

The following sample checkpoints could be created for an internal office transaction:

  • Login
  • Browse Orders
  • View Order Details

Configuring Web Application Transaction Monitors

You can define Web application transactions by manually stepping through one and declaring checkpoints at key stages:

  1. Open a Web browser, and configure its proxy settings so that you can record a transaction:
    • Open the dialog where network and connection settings are made.
    • Configure the browser’s proxy to “localhost” on port 8001.
    • Ensure these settings also are applied to SSL or secure communications.
    • Set the proxy to bypass the Monitoring Station.
      This step requires you to select an option such as “no proxy” or “bypass proxy server,” and may also require you to manually enter your Monitoring Station URL or IP address.
    Using the monitor as a proxy allows it to intercept Web traffic as you generate it.
  2. In the browser, navigate to the starting point of the Web application whose performance you are monitoring.
  3. In the Uptime Infrastructure Monitor Add Service window, select the Web Application Transaction monitor, then click Continue.
    The Web Application Transaction Recorder is displayed, and the monitor is now listening on port 8001 for traffic.

    Info

    Ensure your browser's Java plugin is updated to the latest, most secure version.

  4. Begin stepping through the Web transaction as an end user, providing the required data or actions.
    Every URL visited during the transaction is logged and displayed in the recorder.

    Info
    The Web Application Transaction monitor records all data inputted during recording: this includes any login information. It is recommended that you use a test account for the Web application, otherwise any user data is visible in the recorded script.
  5. At each major step in the Web transaction that signals a new analysis point, enter a checkpoint name in the text box at the top of the window, then click Mark Checkpoint.
    For example, create a checkpoint at a transaction step where the application takes user-inputted data and makes database calls.

    Info
    You can later set Warning and Critical thresholds that apply to every segment declared in your recording. It is recommended that the divisions between your checkpoint intervals are reasonably consistent.
  6. Continue to repeat steps 4 and 5 until you have completed enough of the Web transaction to test it, then click Next.
  7. Complete the monitor information fields.
    To learn how to configure monitor information fields, see Monitor Identification.

    Info
    Even though the Web application performance is not directly tied to an Element’s performance, making this selection is still required: the service based on this monitor needs to be associated with an Element in order to be viewed in areas such as Global Scan or Infrastructure.
  8. Configure the Web Application Transaction Settings:
    • Script to play back
      If desired, optimize the playback script (e.g., remove extraneous URLs such as image downloads).
    • Text that must appear
      Enter a text string that can be used to confirm the script playback was successful (e.g., a phrase that appears on the final page of the application). If the monitor does not find this text, its status changes to Critical. By providing mandatory text, you can ensure an alert is triggered in cases where a Web application is malfunctioning, but checkpoint-to-checkpoint times are fast enough to fulfill response time requirements.
    • Text that must not appear
      Enter a text string that should not appear at any point during the script playback (e.g., a client- or server-error HTTP status code). If the monitor finds this text, its status changes to Critical. Use this feature, as you would use mandatory text, to ensure a malfunctioning application triggers an alert.
    • User Agent String
      Select the Web browser and version used to record the script. This selection determines the user agent string used in the HTTP requests to the application server, and should be provided in case the application blocks access by scripts.
    • Checkpoint Times
      Enter the Warning and Critical Checkpoint Time thresholds. An alert is generated with these thresholds if any of the recorded Web transaction’s checkpoint times exceeds the supplied values.
    • Response Time
      Enter the Warning and Critical Response Time thresholds. An alert is generated with this threshold if the entire transaction playback time exceeds the supplied values. For more information, see See Configuring Warning and Critical Thresholds .
  9. Enter Warning- and Critical-level thresholds for the overall response time of the monitor.
    Most of the monitor’s Response Time is comprised of the Delivery Time and the Retrieve Time. Ensure the values provided for the Response Time thresholds roughly correspond with those provided for the other thresholds. For more information, see Configuring Warning and Critical Thresholds.
  10. Complete the following settings:
  11. Click Finish.

Viewing and Diagnosing Web Transaction Performance

To view Web transaction performance via playback, create a Service Metrics graph for the Web Application Transaction monitor’s system. To generate a Service Metrics graph, either select the system to which the Web Application Transaction monitor is associated in Infrastructure, or the monitor itself in the main Services panel. Click the Graphics tab, then click Service Metrics.

The Service Metrics graph shows how long each transaction segment took to complete during playback, and in doing so, provides an end-to-end performance snapshot of the components of your infrastructure that deliver applications to users. For example, the following metrics graph shows that the execution of the comments found in checkpoint3 took excessively long to complete:

Image RemovedImage Added

Because other checkpoints performed well, the poor performance of a single checkpoint indicates possible issues with a particular server, and not the network infrastructure. This theory can be further investigated by looking at the performance metrics for the server in question.

Use the Web Application Transaction monitor’s playback script to verify which servers are used during a problem checkpoint. In the Service Instances panel, click the monitor to view the script, then locate the system that is accessed (e.g., with GET and POST commands). Use this as an investigative starting point: although an application or Web server is often referenced in the script, the problem may be found deeper in the application stack (e.g., a database server to which the referenced Web server makes calls during the checkpoint).

Using Web Transaction Performance in SLA Reports

Your Web applications typically call on systems on the application and database tiers, as well as make use of internal- and external-facing network devices. Because the Web Application Transaction monitor directly reports on the performance of a Web transaction, it in effect indirectly reports on the health of your IT infrastructure as a whole.

This broad reporting coverage makes the Web Application Transaction monitor an ideal monitor to include in service level agreement reports.

For more information on SLA reports, see Reports for Service Level Agreements.

Email Delivery Monitor

Although specific Uptime Infrastructure Monitor monitors are available for your POP, IMAP, and SMTP servers, their monitoring duties focus on availability and response time. To test your IT infrastructure’s ability to send or receive emails within a reasonable amount of time, use the Email Delivery monitor.

Typically, email delivery tests include a server that is part of your IT infrastructure and monitored by Uptime Infrastructure Monitor. In these cases, you must test either incoming mail delivery times by supplying information about a monitored POP3 or IMAP server, or test outgoing mail delivery times by supplying information about a monitored SMTP server.

The Email Delivery executes several steps in order to calculate mail delivery and retrieval time:

  • the monitor requests an internal or external SMTP server to send a generated test mail (when the monitor asks the SMTP server to send the mail, the monitor records the delivery time)
  • the monitor waits for five seconds, then logs in to and checks an internal or external POP3 or IMAP mail server to verify the mail was received
  • if the test mail is not found, the monitor waits another five seconds and checks again (and continues to check until the process has either timed out or the mail is found)
  • the monitor confirms the mail was received and reports both the delivery and retrieval times

Configuring Email Delivery Monitors

Define the Email Delivery monitor by providing information about the outgoing and incoming mail servers:

  1. Complete the monitor information fields.
    To learn how to configure monitor information fields, see Monitor Identification.

    Info

    Once created, the Email Delivery monitor service can be included with status reports for the system or group you select. If this monitor is reporting outgoing mail delivery times, the system should be a monitored SMTP server; if incoming mail delivery times are measured, the system should be a monitored POP3/IMAP mail server.

  2. Complete the Outgoing Email Settings:
    • SMTP Hostname
      Provide the name or IP address of the SMTP server.
    • SMTP Port
      Provide the port used to communicate with the SMTP server. Leave this field blank to use the default SMTP port (25).
    • SMTP Username
      Provide the authenticated SMTP user name.
    • SMTP Password
      Provide the authenticated SMTP user password.
    • SMTP Uses SSL
      Specify whether the SMTP server sends and receives encrypted communication using SSL.
    • Destination Email Address
      Enter the test email address used by the monitor. The monitor sends an email to this address, and this address is checked for receipt of the test email.

      Info

      Although the Email Delivery monitor attempts to promptly find and delete test emails, network issues may prevent timely cleanups. To avoid potential Inbox clutter, it is recommended that you create a dedicated test email account as the destination address.

    • Delivery Time
      Enter the Warning and Critical Delivery Time thresholds. The smallest unit of time used for these thresholds is seconds. Given the speed at which SMTP servers should finish processing an outgoing email, is it recommended that you set the Warning threshold to one second.
  3. Complete the Incoming Email Settings:
    • POP3/IMAP Hostname
      Provide the name or IP address of the mail server.
    • POP3/IMAP Port
      Provide the port used to communicate with the mail server. Leave this field blank to use the default POP3 or IMAP port (110 and 143, respectively).
    • POP3/IMAP Username
      Provide the login name for the destination email account.
    • POP3/IMAP Password
      Provide the password for the destination email account.
    • POP3/IMAP Uses SSL
      Specify whether the mail server sends and receives encrypted communication using SSL.
    • Retrieve Time
      Enter the Warning and Critical retrieval time thresholds. The smallest unit of time used for these thresholds is seconds, and the monitor checks for receipt of the test mail in five-second intervals. Enter values in multiples of five.
  4. Enter Warning- and Critical-level thresholds for the overall response time of the monitor.
    Enter the Warning and Critical Response Time thresholds. An alert is generated with this threshold if the combined email delivery and response time exceeds the supplied values. For more information, see Configuring Warning and Critical Thresholds.
  5. Complete the following settings:
  6. Click Finish.

Diagnosing and Reporting Email Delivery Problems

If the Email Delivery monitor reaches a Critical state, the first investigation step is to review the message produced by Uptime Infrastructure Monitor. In the System Status panel, view the message belonging to the system to which the monitor is attached, which should point you in the right direction. For example, the status message below indicates the monitor reached a critical state because the retrieval time from an external POP3 server exceeded the defined threshold; your SMTP server is most likely not responsible for the delay:

 

Image Removed

Image Added

Speculation based on the status message can be confirmed using a Service Metrics graph for the Email Delivery monitor’s system.  This graph indicates whether the delivery and retrieval time are within acceptable limits (below left), or if one or both are unusually long (below right):

 

Image Removed

Image Added

To generate a Service Metrics graph, either select the system to which the Email Delivery monitors are associated in Infrastructure, or the monitor itself in the main Services panel. Click the Graphics tab, then click Service Metrics.

Even if the Service Metrics graph indicates delivery and retrieval times are not exceeding defined thresholds (and Uptime Infrastructure Monitor is not sending out critical alerts), it is still an ideal investigative starting point if you are getting critical feedback from your users about email delivery times.

If the Email Delivery monitor’s Service Metrics graph confirms that there are delays somewhere within your network infrastructure, you can investigate further by using the service monitor you created for your mail server. Co-ordinate your Email Delivery monitor’s metrics graphs or reports with those from a service monitor you have assigned to your mail server (e.g. Exchange) while focusing on metrics that may be related outgoing or incoming mail time delays.  For example, in the Exchange service monitor metrics graph below, the mail server experienced a high SMTP Local Queue Length that did not always coincide with the SMTP Messages Per Second count:

Image RemovedImage Added

Splunk Query

Splunk is a third-party search engine that indexes log files and data from the devices, servers, and applications in your network. Using Splunk, you can quickly analyze your logs to pinpoint problems on a server or in a network, or ensure that you are in compliance with a regulatory mandate, or service level agreements. You install Splunk on a server in your datacenter.

When you integrate Uptime Infrastructure Monitor and Splunk, a Splunk icon appears beside any service that is in a WARN or CRIT state, when viewing the service in a monitoring or diagnostic view (e.g., the My Alerts section of My Portal, the Outages list or Status page for an Element):

Image RemovedImage Added

Clicking this icon takes you to your Splunk search page.

You can use the Splunk Query monitor to perform Splunk queries on log files to pinpoint an error condition.

Info

Before you can use a Splunk Query monitor, you must add settings to your Uptime Infrastructure Monitor Configuration panel that allow Uptime Infrastructure Monitor to interface with your Splunk installation. See Splunk Settings for more information.

Configuring Splunk Query Monitors

To configure a Splunk Query monitor, do the following:

  1. Complete the monitor information fields.
    See Monitor Identification for more information.
  2. Complete the Splunk Query Settings:
    • Splunk query
      The Splunk query string that is used to search the log file for an error condition. You can enter any Splunk query string in this field. For example, the following searches the log files for any instances of sendmail and error in relation to the mailServer host within the last two hours:
      host::mailServer sendmail error hoursago::2
      For more information on the syntax of Splunk queries, refer to search information found in the Splunk User Manual.

      Info

      To minimize the risk of the monitor timing out, avoid using open-ended queries; instead use relative time ranges for a block of time, or with the snap-to-time modifier such as -1d@d (yesterday starting from 12:00:00 AM).

    • Result count of splunk query
      Enables Uptime Infrastructure Monitor to alert you when the number of results that match your Splunk query exceeds the defined warning and critical thresholds.
      For example, you can configure the monitor to issue a Warning alert when five or more Splunk results matching your query are returned, and a Critical alert when 10 or more results for your query are returned.
    • Response Time
      Enter the Warning and Critical Response Time thresholds. For more information, see Configuring Warning and Critical Thresholds.
  3. To save the result-count or response-time data for graphing or reporting, click the Save for Graphing checkbox beside the appropriate checkbox(es).
  4. Complete the following settings:
  5. Click Finish.

Live Splunk Listener

Live Splunks are scheduled searches of Splunk queries that are saved on the Splunk server. A Live Splunk automatically runs a search, can initiate an alert, and can perform actions based on that alert. You can, for example, set up a Live Splunk to search for all critical error conditions.

The Live Splunk Listener monitor enables you to capture the information generated by a Live Splunk (from Splunk 4.x only). This monitor is very similar to the External Check monitor, and uses scripts that are bundled with Uptime Infrastructure Monitor (found in the /scripts subdirectory) to return Live Splunk information to the Monitoring Station.

To use this monitor, you must first modify the two Splunk scripts that are included with Uptime Infrastructure Monitor:

  • alertUptimeStatusHandler.sh
  • alertUptime.py

This pair of scripts take the following options:

  • --message
    A message that is returned to the Uptime Infrastructure Monitor Monitoring Station. For example, if the Live Splunk is configured to search for warning conditions, you can enter the message “Changed to WARN” .
  •  --status
    The script can return the following status codes:
    •  0 - OK
      The services are functioning properly.
    •  1 - Warning
      There is a potential problem with one of more of the monitored services.
    •  2 - Critical
      There is a critical problem with one or more of the monitored services.
    •  3 - Unknown
      There is an error in the configuration of the monitor itself, or Uptime Infrastructure Monitor cannot execute the service check.
  •  --monitor
    The name of the Uptime Infrastructure Monitor monitor to which the information from the Live Splunk is directed.

The following is an example of the script with all of its options specified:

No Format
nopaneltrue
alertUptimeStatusHandler.sh --message="sendmail has some traffic going through new command!"
                            --status=2 --monitorName="Live Splunk"

Uptime Infrastructure Monitor captures the output from the script, which appears in the service status section of the Global Scan dashboard (see Understanding the Status of Services). The Uptime Infrastructure Monitor monitoring framework picks up any error codes and triggers the appropriate monitoring action.

Before You Begin

Before you can monitor Live Splunks generated on a Splunk server, you must do the following:

  1. Edit the alertUptime.py script to point to the Uptime Infrastructure Monitor Monitoring Station:
    • Navigate to the /scripts directory on the Monitoring Station.
    • Open the file alertUptime.py in a text editor.
    • Find the following entry in the file:

      No Format
      nopaneltrue
      host = "uptime-host"
      port = "9996"
    • Change the values for host and port to the host name and port of the Monitoring Station.
    • Save and close the file.
  2. Edit the alertUptimeStatusHandler.sh script to configure how the Live Splunk is reported on the Monitoring Station:
    • Open alertUptimeStatusHandler.sh in a text editor (found in the /scripts directory on the Monitoring Station).
    • For the message option, enter a diagnostic message that accompanies a Live Splunk captured by the Uptime Infrastructure Monitor service monitor.
    • For the status option, enter the status of the monitored service.
    • For the monitorName option, enter the name of the service monitor that is listening to the Live Splunk.
    • Save and close the file.
  3. Copy the alertUptimeStatusHandler.sh and alertUptime.py scripts from the Monitoring Station’s /scripts directory to the
    /data/splunk/bin/scripts directory on the Splunk server.
  4. Configure a Live Splunk. For information on configuring Live Splunks, see the Splunk user manual.
    When setting up your Live Splunk, select the Run the shell script option on the configuration page. Then, enter the path to alertUptimeStatusHandler.sh in the field.

Configuring the Live Splunk Listener Monitor

To configure a Live Splunk Listener monitor, do the following:

  1. Complete the monitor information fields.
    To learn how to configure monitor information fields, see Monitor Identification.
  2. Complete the following settings:
  3. Click Finish.

Save

Save