Sending Logs to Ingestion API

Last updated on 12 June, 2023

If a log integration isn’t available or you have custom logs that you want to analyze, you can send the logs directly to your LogicMonitor account through the logs ingestion API. The received events are mapped to resources in LogicMonitor before they are further processed or stored.

Log Ingestion Endpoint

Endpointhttps://<account>.logicmonitor.com/rest/log/ingest
ProtocolHTTPS:443
HeadersAuthorization: <LMv1 token>
MethodPOST
Content Typeapplication/json
Content Encodinggzip, deflate

Resource Path

For the logs ingestion endpoint, https://<account>.logicmonitor.com/rest/log/ingest:

  • The base URL is https://<account>.logicmonitor.com/rest
  • The resource path is /log/ingest

Where <account> is the company name or account name for your LogicMonitor portal.

Note: The base URL for the logs ingestion endpoint is /rest (not /santaba/rest, which is the URL for the LogicMonitor REST API).

Authentication

All requests to the log ingestion API use LogicMonitor’s LMv1 API tokens for authentication. You can create the API tokens in your LogicMonitor account Settings under Users & Roles.

Every request must include an HTTP authorization header in the format:

Authorization: LMv1 AccessId:Signature:Timestamp

The Signature is a base64 and HMAC encoded hash based on your Access Key, a timestamp in epoch milliseconds, and the resource path of the endpoint:

signature = base64(
  HMAC-SHA256(
    Access Key, 
    HTTP VERB 
    + TIMESTAMP
    + POST DATA
    + RESOURCE PATH
  ) 
)

Note: An API-only user that submits logs through the Log Ingestion API must have “Manage” permissions for Logs. For more information, see Roles.

Limitations

The log ingestion API expects an array of entries with the following constraints on the content size:

DescriptionSize
Maximum content size per payload8 MB
Maximum array size for a single log message32 KB
Maximum array size for sending multiple logs/events in an array
Rate limit for calls to the LM Logs ingestion endpoint (per account/API key)240K requests/min

Example Request

Sending a log message for an Amazon EC2 instance to the ingestion endpoint using curl:

curl --location \
--request POST 'https://<account>.logicmonitor.com/rest/log/ingest' \
--header 'Authorization: <LMv1 token>' \
--header 'Content-Type: application/json' \
--data-raw '[{
    "msg": "Generating example log message 31687",
    "_lm.resourceId":{"<property>": "<value>"}
}]'

Parameters

The log ingestion endpoint expects an array of JSON objects as events with any JSON attribute. Reserved attributes are described in the following.

ParameterDescription
message | msg | Msg(Required) The log message. See Limitations.

If an event does not contain this attribute, the event is not forwarded to the log processing pipeline.
timestamp | date | _timestamp | Timestamp | eventTime | published_date(Optional) The timestamp when the event occurred. Supported date formats are ISO8601 and Unix Epoch (in secs, ms, ns).

If events don’t contain any of the default date attributes and you haven’t defined your own timestamp, the event will be assigned the current date as the timestamp.

The timestamp can be either int or string. For more accurate nanosecond times, use string (to avoid rounding errors of int values in JSON).
_lm.resourceId: {<property>: <value>}_lm.resource.id is not required for log ingestion, only for mapping to a LogicMonitor resource. To map events to a unique LM resource, any resource property can be used, for example: deviceId, AWS ARN, Kubernetes pod Id.

<property> is the LogicMonitor resource property to use for device mapping.
<value> is the value of the resource property.

If more than one property exists, only the first property will be mapped.

Example 1 – deviceId known:
_lm.resourceId: { system.deviceId: "<deviceId>" }

Example 2 – in the AWS Lambda function using unique ARN: _lm.resourceId: { system.aws.arn: <arn> }

To send additional information along with the log message, you can add more JSON items in the message. The information displays as “metadata” in the Logs table when reviewing logs. For more information, see Viewing Logs and Log Anomalies.

Note: Logs can have a severity associated with each event if it matches the criteria for an anomaly or matches an alert condition. This severity is not set during ingestion. For more information, see Log Alert Conditions.

Response Codes

The log ingestion API uses conventional HTTP response codes to indicate the success or failure of an API request.

Status CodeDescription
202 - AcceptedThe request has been accepted for processing, but the processing has not been completed.
207 - Multi-statusSome events in the batch have not been accepted for processing. See the “errors” property within the response for details.
400 - Bad RequestThe request is invalid. For example, it may be missing headers or the request body is incorrectly formatted.
401 - UnauthorizedAuthentication failed. The API key provided is not valid.
402 - Payment RequiredThe LM Logs feature has not been enabled. Please contact Support.
403 - ForbiddenThe API key doesn’t have permission to perform the request.
413 - Payload Too LargeThe payload exceeds the maximum content size of 8 MB.
429 - Too Many RequestsThe number of requests exceeds the rate limit.
500 - Server ErrorSomething went wrong on LogicMonitor’s end.
502 - Bad GatewayA dependency failed to respond within a reasonable time.

Example: 202 – Accepted

Header:
     X-Request-ID: 38b78dd6-3bc0-4cd9-8a15-6af552d49c3e
Body:
{
    "success": true,
    "message": "Accepted"
}

Example: 207 – Multi-status

Header:
     X-Request-ID: 38b78dd6-3bc0-4cd9-8a15-6af552d49c3e
Body:
{
    "success": false,
    "message": "Some events were not accepted. See the 'errors' property for additional information.",
    "errors": [
        {
            "code": 4001,
            "error": "Resource not found",
            "event": {
                "_lm.resourceId": {
                    "system.deviceId": "kish"
                },
                "message": "test"
            }
        },
    ]
}

Custom Error Codes

Custom error codes provide insights into what went wrong when log ingestion fails. If events do not meet the log ingestion constraints, such as failed mapping to a LogicMonitor resource or field requirements, one of the error messages described in the following is returned.

Error CodeDescription
4001 - Resource not foundThe query for the resource was attempted but the device was not found.
4002 - More than one resource has been foundThe event source must be mapped to only one LogicMonitor resource.
4003 - Insufficient information for device lookupThe information is not enough for resource lookup. For example, the _lm.resourceId is not correctly specified and device mapping cannot be performed.
4004 - Missing message fieldLogicMonitor will not accept logs without a message field.
4005 - Event too largeA single event may not be larger than 1 MB.
4006 - Event too old or futureThe event falls outside the configured ingestion time window, ±3 hours.

Example: Failures

Header:
    X-Request-ID: 38b78dd6-3bc0-4cd9-8a15-6af552d49c3e
Body:
{
    "success": false,
    "message": "Some events were not accepted. See the 'errors' property for additional information.",
    "errors": [
        {
            "code": 4003,
            "error": "Insufficient information for device lookup",
            "event": {
                "message": "test"
            }
        },
        {
            "code": 4004,
            "error": "Missing message field",
            "event": {
                "_lm.resourceId": {
                    "system.deviceId": "kish"
                },
            }
        }
 
    ]
}
In This Article