Support Center Home


Setting up AWS Logs Ingestion

The Amazon Web Services (AWS) integration for LM Logs sends Amazon CloudWatch logs to LogicMonitor using a Lambda function configured to forward the log events. LogicMonitor provides two methods to automate this process: an AWS CloudFormation Stack template and a Terraform configuration.

The AWS integration for LM Logs can be found at the following link: https://github.com/logicmonitor/lm-logs-aws

Prerequisites

Deploy using AWS CloudFormation

To deploy the Lambda function using a CloudFormation stack template for LM Logs:

1. On the AWS integration for LM Logs, click “Launch Stack“.

2. Configure the stack options in the template.

Once you create the stack, a Lambda function will be deployed and subscribed to the specific CloudWatch Logs group to forward logs to LogicMonitor.

3. Refer to Forwarding AWS Logs for service-specific instructions for sending logs to your CloudWatch logs group if it doesn’t already include the logs you want to forward (if it does, you can skip the information below).

Once logs are sent to the right CloudWatch Logs group, the Lambda function will automatically forward them to the log ingestion API. You should see logs and log anomalies show up in the UI (on both the Logs page and Alerts pages) shortly thereafter.

CloudFormation Stack options

Parameter Description
FunctionName (Required) The name for the log forwarding Lambda function. Defaults to LMLogsForwarder.
LMIngestEndpoint (Required) Your LogicMonitor account URL: https://<account>.logicmonitor.com
Where <account> is your LogicMonitor sandbox account or company name.
LMAccessId (Required) The LogicMonitor API tokens access ID. We recommend creating an API-only user.
LMAccessKey (Required) The LogicMonitor API tokens access key.
LMRegexScrub (Optional) Regular expression pattern to remove matching text from the log messages.

We recommend using this parameter to filter out any logs that contain sensitive information so that those logs are not sent to LogicMonitor.
LogGroupName (Required) The name of an existing CloudWatch Logs group. The logs that you want to forward to LogicMonitor should be collected into this log group.
FunctionMemorySize (Optional) The memory size for the log forwarding lambda function.
FunctionTimeoutInSeconds (Optional) The timeout for the log forwarding lambda function.
LogRetentionInDays (Optional) The CloudWatch log retention for logs sent to the specified log group

CloudFormation permissions

To deploy the CloudFormation Stack with the default options, you need to have the permissions below to save your LM Credential as a secret and create an S3 bucket to store the Forwarder’s code (zip file), and create Lambda functions (including execution roles and log groups).

{
           "Effect": "Allow",
           "Action": [
               "cloudformation:*",
               "secretsmanager:CreateSecret",
               "secretsmanager:TagResource",
               "secretsmanager:DeleteSecret",
               "s3:CreateBucket",
               "s3:GetObject",
               "s3:PutEncryptionConfiguration",
               "s3:PutBucketPublicAccessBlock",
               "s3:DeleteBucket",
               "iam:CreateRole",
               "iam:GetRole",
               "iam:PassRole",
               "iam:PutRolePolicy",
               "iam:AttachRolePolicy",
               "iam:DetachRolePolicy",
               "iam:DeleteRolePolicy",
               "iam:DeleteRole",
               "lambda:CreateFunction",
               "lambda:GetFunction",
               "lambda:GetFunctionConfiguration",
               "lambda:GetLayerVersion",
               "lambda:InvokeFunction",
               "lambda:PutFunctionConcurrency",
               "lambda:AddPermission",
               "lambda:RemovePermission",
               "logs:CreateLogGroup",
               "logs:DescribeLogGroups",
               "logs:PutRetentionPolicy",
               "logs:PutSubscriptionFilter",
               "logs:DeleteSubscriptionFilter"
           ],
           "Resource": "*"
}

The following capabilities are required when creating a CloudFormation stack:

  • CAPABILITY_AUTO_EXPAND, because the forwarder template uses macros.
  • CAPABILTY_IAM, CAPABILITY_NAMED_IAM, because the Forwarder creates IAM roles.

Deploy using Terraform

Run the following terraform command to deploy the Lambda function (filling in the necessary variables):

# terraform apply --var 'lm_access_id=<lm_access_id>' --var 'lm_access_key=<lm_access_key>' --var 'lm_company_name=<lm_company_name>'

For more information, see the Sample Configuration for the LM Logs Forwarder.

Forwarding AWS Logs

After deploying the Lambda function, sending logs from individual AWS services should be configured separately.

This integration currently supports the following AWS services: Amazon Elastic Cloud Compute (EC2), Simple Storage Service (S3), Relational Database (RDS), and Elastic Load Balancing (ELB).

Send EC2 Instance Logs

Before the EC2 instance logs can be forwarded to LM Logs, they need to be collected into CloudWatch Logs. For more information, see Installing the CloudWatch Agent.

Note: When sending EC2 logs to LogicMonitor, the logstream name must be the instance ID (typically this is the default).

After you start receiving the EC2 logs in the CloudWatch log group: 

1. In CloudWatch, select the log group (where the EC2 logs will be forwarded from).

2. Under Actions > Create Lambda subscription filter, select “Lambda function” and choose “LMLogsForwarder” (or whatever you named the Lambda function during stack creation).

3. Click Start streaming.

Send S3 Bucket Access Logs

To send Amazon access logs from an S3 bucket to LM Logs:

1. Under the source bucket’s Properties, enable Server access logging.

You will need to select a Target bucket where the access logs will be stored. If this target bucket doesn’t exist, you need to create it. (This is different from the source bucket.)

2. Go to the target bucket, and under Advanced settings > Events, add a notification for “All object create events“.

3. Send to “Lambda function” and choose “LMLogsFowarder” (or whatever you named the Lambda function during stack creation).

4. Click Start streaming.

Send ELB Access Logs

To send Amazon ELB access logs to LM Logs:

1. In the EC2 navigation page, choose Load Balancers and select your load balancer.

2. Under Attributes > Access logs, click “Configure access logs“.

3. Select “Enable access logs” and specify the S3 bucket to store the logs. (You can create a bucket if it doesn’t exist.)

4. Go to the S3 bucket (from Step 3) and under Advanced settings > Events, add a notification for “All object create events“.

5. Send to “Lambda function” and choose “LMLogsFowarder” (or whatever you named the Lambda function during stack creation).

6. Click Start streaming.

Send RDS Logs

To send Amazon RDS logs to LM Logs:

1. Configure the RDS instance to send the logs to CloudWatch.

2. In CloudWatch, select the log group (where the RDS logs will be forwarded from).

3. Under Actions > Create Lambda subscription filter, select “Lambda function” and choose “LMLogsForwarder” (or whatever you named the Lambda function during stack creation).

4. Click Start streaming.

Send Lambda Logs

To send Lambda logs to LM Logs:

1. In CloudWatch, select the Lambda’s log group (where the logs will be forwarded from).

2. Under Actions > Create Lambda subscription filter, select “Lambda function” and choose “LMLogsForwarder” (or whatever you named the Lambda function during stack creation).

3. Click Start streaming.

The Lambda logs should be forwarded from the log group to LogicMonitor.

Troubleshooting

To help troubleshoot logs forwarded from Amazon CloudWatch, enable debug logging in your Lambda logs:

1. In the AWS console, go to AWS Lambda > Functions and select “LMLogsForwarder” (or whatever you named the log forwarding Lambda function during setup).

2. Add an environment variable with the key DEBUG and value true.

In This Article