Webpage Data Collection

You can use Webpage (HTTP/HTTPS) data collection to retrieve data from web endpoints by sending HTTP or HTTPS requests. This collection method enables you to monitor availability, measure response time, and extract values from web page content or API responses.

Webpage data collection is commonly used to monitor web applications, APIs, or services that expose data over HTTP or HTTPS, validate endpoint availability and response behavior, and extract values from structured or semi-structured responses.

When you select Webpage (HTTP/HTTPS) as the collection method in a DataSource, you configure how the Collector connects to the endpoint, sends a request, and interprets the response.

Requirements for Collecting Data using Webpage

To configure Webpage (HTTP/HTTPS) data collection, do the following:

• The Collector reaches the target endpoint
  For more information, see Configuring Collector to Communicate with HTTP Proxies.
• Requests must be properly formatted
  • Use valid HTTP/1.0 or HTTP/1.1 requests. HTTP/1.1 requires a Host header.
  • Use \n for line breaks.

• Authentication credentials are defined as properties (for example, http.user, http.pass, and ntlm.domain )
• HTTPS endpoints present a valid SSL certificate

If you are using NTLM authentication, you need the following:

• Use HTTP/1.1 requests
• Do not include the Connection: close header
• Ensure the Collector runs with sufficient privileges

Configuring Webpage Data Collection

1. In LogicMonitor, navigate to Modules > and either create a new DataSource or select an existing one you want to configure Webpage data collection for.
   For more information, see Custom Module Creation or Modules Management.
2. Configure or modify your DataSource as needed, and select “WEBPAGE” from the Collection Method dropdown menu.
   For more information, see DataSource Configuration.
   
3. To define how frequently data is collected, select a time in minutes from the Collection Schedule dropdown menu.
4. In the Access Groups section, add access groups as needed to control who can view or edit the module.
   For more information, see Access Groups for Modules. 
5. In the AppliesTo section, based on your requirements, add any AppliesTo scripting method. 
   This determines which resources the module applies to. For more information, see AppliesTo Function Scripting Overview.
6. In the Collection section, do the following:
   • (Optional) To configure requests to use HTTPS, enable the SSL switch.
     When the switch is disabled, requests use HTTP.
     
     Note: When enabled, the default port changes to 443.
   • To specify the TCP port used for the request, enter the port exposed by the endpoint into the Port field.
     For example, 80 for HTTP or 443 for HTTPS.
   • (Optional) To configure how long the Collector waits for a connection to be established, enter the time in milliseconds into the Connect Timeout (milliseconds) field.
   • (Optional) To configure how long the Collector waits for a response after a request is sent, enter a time in millisecond into the Read Timeout field (milliseconds) field.
   • (Optional) To define the HTTP request being sent to the endpoint, enter a valid HTTP/1.1 or HTTP/1.1 request into the Request field.
     
     To make the DataSource reusable across resources or instances, you can input LogicMonitor tokens ##HOSTNAME## or ##WILDVALUE##.
     For more information, see Available Tokens for Data Collection.
     The following code displays an example:
     

     GET /status HTTP/1.1\n
     Host: ##HOSTNAME##\n
     \n

     Copy
   • (Optional) To determine whether the Collector follows HTTP redirect, enable the Follow Redirect? switch.
     The redirect returns up to three redirects by the endpoint.
     Keeping this switch disabled ensures the endpoint returns only the initial response.
     
7. In the Datapoint section, add or edit a datapoint as needed.
   For more information, see Datapoint overview.
8. During configuration, do the following:
   • From the Datapoint Source dropdown, select one of the following:
     • Status code—Select to collect the HTTP response status.
       Status values map to the following structure: 1 (HTTP 200), 2 (HTTP 3xx), 3 (HTTP 4xx), 4 (HTTP 5xx), 5 (connection refused), and 6 (invalid SSL certificate).
     • RTT to load the last byte of the page—Select to measure how long the request takes to complete. The value is returned in milliseconds.
     • Entire HTTP response—Select to collect the full HTTP response, including headers and body.
     • HTTP response body—Select to collect only the response body. Typically used when extracting values from returned content.
   • (Optional) To define how Entire HTTP response or HTTP response body responds, select an option from the Interpret output with dropdown menu.
     For more information, see Output Interpretation Methods.
     
   • To save your configuration, select Save.
   • (Optional) If the endpoint requires authentication, navigate to the Resource or Resource group and define credentials as properties such as http.user and http.pass. Add ntlm.domain if NTLM authentication is required.
     For more information, see Resource and Instance Properties.

After completing this configuration, the DataSource begins collecting data from the specified endpoint, and datapoint values are available for monitoring and alerting.