Support Center Home


Setting up AWS Logs Ingestion

The Amazon Web Services (AWS) integration for LM Logs sends Amazon CloudWatch logs to LogicMonitor using a Lambda function configured to forward the log events. LogicMonitor provides two methods to automate this process: an AWS CloudFormation Stack template and a Terraform configuration.

The AWS integration for LM Logs can be found at the following link: https://github.com/logicmonitor/lm-logs-aws

Prerequisites

Deploy using AWS CloudFormation

To deploy the Lambda function using a CloudFormation stack template for LM Logs:

1. On the AWS integration for LM Logs, click “Launch Stack“.

2. Configure the stack options in the template.

Once you create the stack, a Lambda function will be deployed and subscribed to the specific CloudWatch Logs group to forward logs to LogicMonitor.

NOTE: The FunctionName has a default value of LMLogsForwarder. When a new Function is created, a CloudWatch log group is created with the same name (LMLogsForwarder in this case) with /aws/lambda/ prefix. If you specify a different FunctionName when creating the function, the log group will be created with that same name (aws/lambda/myfunctionname)

3. Refer to Forwarding AWS Logs for service-specific instructions for sending logs to your CloudWatch logs group if it doesn’t already include the logs you want to forward (if it does, you can skip the information below).

Once logs are sent to the right CloudWatch Logs group, the Lambda function will automatically forward them to the log ingestion API. You should see logs and log anomalies show up in the UI (on both the Logs page and Alerts pages) shortly thereafter.

CloudFormation Stack options

Parameter Description
FunctionName (Required) The name for the log forwarding Lambda function. Defaults to LMLogsForwarder.
LMIngestEndpoint (Required) Your LogicMonitor account URL: https://<account>.logicmonitor.com
Where <account> is your LogicMonitor sandbox account or company name.
LMAccessId (Required) The LogicMonitor API tokens access ID. We recommend creating an API-only user.
LMAccessKey (Required) The LogicMonitor API tokens access key.
LMRegexScrub (Optional) Regular expression pattern to remove matching text from the log messages.

We recommend using this parameter to filter out any logs that contain sensitive information so that those logs are not sent to LogicMonitor.
FunctionMemorySize (Optional) The memory size for the log forwarding lambda function.
FunctionTimeoutInSeconds (Optional) The timeout for the log forwarding lambda function.
LogRetentionInDays (Optional) The CloudWatch log retention for logs sent to the specified log group

CloudFormation permissions

To deploy the CloudFormation Stack with the default options, you need to have the permissions below to save your LogicMonitor Credential as a secret and create an S3 bucket to store the Forwarder’s code (zip file), and create Lambda functions (including execution roles and log groups).

{
           "Effect": "Allow",
           "Action": [
               "cloudformation:*",
               "secretsmanager:CreateSecret",
               "secretsmanager:TagResource",
               "secretsmanager:DeleteSecret",
               "s3:CreateBucket",
               "s3:GetObject",
               "s3:PutEncryptionConfiguration",
               "s3:PutBucketPublicAccessBlock",
               "s3:DeleteBucket",
               "iam:CreateRole",
               "iam:GetRole",
               "iam:PassRole",
               "iam:PutRolePolicy",
               "iam:AttachRolePolicy",
               "iam:DetachRolePolicy",
               "iam:DeleteRolePolicy",
               "iam:DeleteRole",
               "lambda:CreateFunction",
               "lambda:GetFunction",
               "lambda:GetFunctionConfiguration",
               "lambda:GetLayerVersion",
               "lambda:InvokeFunction",
               "lambda:PutFunctionConcurrency",
               "lambda:AddPermission",
               "lambda:RemovePermission",
               "logs:CreateLogGroup",
               "logs:DescribeLogGroups",
               "logs:PutRetentionPolicy",
               "logs:PutSubscriptionFilter",
               "logs:DeleteSubscriptionFilter"
           ],
           "Resource": "*"
}

The following capabilities are required when creating a CloudFormation stack:

  • CAPABILITY_AUTO_EXPAND, because the forwarder template uses macros.
  • CAPABILTY_IAM, CAPABILITY_NAMED_IAM, because the Forwarder creates IAM roles.

Deploy using Terraform

Run the following terraform command to deploy the Lambda function (filling in the necessary variables):

# terraform apply --var 'lm_access_id=<lm_access_id>' --var 'lm_access_key=<lm_access_key>' --var 'lm_company_name=<lm_company_name>'

For more information, see the Sample Configuration for the LM Logs Forwarder.

Forwarding AWS Logs

After deploying the Lambda function, you should configure the individual AWS services to send their logs to the Lambda function. You can find instructions for supported AWS services below.

Send EC2 instance logs

Before the EC2 instance logs can be forwarded to LM Logs, they need to be collected into CloudWatch Logs. For more information, see Installing the CloudWatch Agent.

Note: When sending EC2 logs to LogicMonitor, the logstream name must be the instance ID (typically this is the default).

After you start receiving the EC2 logs in the CloudWatch log group: 

1. In CloudWatch, select the log group (where the EC2 logs will be forwarded from).

2. Under Actions > Create Lambda subscription filter, select “Lambda function” and choose “LMLogsForwarder” (or whatever you named the Lambda function during stack creation).

3. Click Start Streaming.

Send ELB access logs

To send Amazon ELB access logs to LM Logs:

1. In the EC2 navigation page, choose Load Balancers and select your load balancer.

2. Under Attributes > Access logs, click “Configure access logs“.

3. Select “Enable access logs” and specify the S3 bucket to store the logs. (You can create a bucket if it doesn’t exist.)

4. Go to the S3 bucket (from Step 3) and under Advanced settings > Events, add a notification for “All object create events“.

5. Send to “Lambda function” and choose “LMLogsFowarder” (or whatever you named the Lambda function during stack creation).

6. Click Start streaming.

Send S3 Bucket access logs

To send Amazon access logs from an S3 bucket to LM Logs:

1. Under the source bucket’s Properties, enable Server access logging.

You will need to select a Target bucket where the access logs will be stored. If this target bucket doesn’t exist, you need to create it. (This is different from the source bucket.)

2. Go to the target bucket, and under Advanced settings > Events, add a notification for “All object create events“.

3. Send to “Lambda function” and choose “LMLogsFowarder” (or whatever you named the Lambda function during stack creation).

4. Click Save changes.

Send logs from RDS

To send Amazon RDS logs to LM Logs:

1. Configure the RDS instance to send the logs to CloudWatch.

2. In CloudWatch, select the log group (where the RDS logs will be forwarded from).

3. Under Actions > Create Lambda subscription filter, select “Lambda function” and choose “LMLogsForwarder” (or whatever you named the Lambda function during stack creation).

4. Click Save changes.

Send Lambda Logs

To send Lambda logs to LM Logs:

1. In CloudWatch, select the Lambda’s log group (where the logs will be forwarded from).

2. Under Actions > Create Lambda subscription filter, select “Lambda function” and choose “LMLogsForwarder” (or whatever you named the Lambda function during stack creation).

3. Click Save changes.

The Lambda logs should be forwarded from the log group to LogicMonitor.

Send flow logs from EC2

To send EC2 flow logs to LM Logs:

1. Add the following lines to the Permissions of the Lambda’s Role policy:

"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"

2. Add the following line to Service tag Role, under Trust Relationship:

"vpc-flow-logs.amazonaws.com"

3. A Log group in CloudWatch should be created with the name /aws/ec2/networkInterface

4. Search the Network Interfaces page for your EC2 instance ID. Select that Network Interface row and create a flow log with the following settings:

  • Destination Log Group: /aws/ec2/networkInterface
  • IAM Role: the role you created in Steps 1 and 2.

5. In the Log record format, select Custom Format. The first value of the Log Format should be instance-id. Set other values depending on your requirements. For more information refer to the Available Fields for Flow Logs.

6. Go to the /aws/ec2/networkInterface Log Group. In Actions > Subscription filters > Create Lambda subscription filter, select “LMLogsForwarder” (or whatever you named the Lambda function during stack creation) and provide Subscription filter name. Click Start Streaming.

The logs will start to propagate through the Lambda to the Log Ingestion API.

Send flow logs from NAT Gateway

To send NAT Gateway flow logs to LM Logs:

1. Add the following lines to the Permissions of the Lambda’s Role policy:

"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"

2. Add the following line to Service tag Role, under Trust Relationship:

"vpc-flow-logs.amazonaws.com"

3. A Log group in CloudWatch should be created with the name /aws/natGateway/networkInterface

4. Search the Network Interfaces page for your NAT Gateway ID. Select that Network Interface row and create a flow log with the following settings:

  • Destination Log Group: /aws/natGateway/networkInterface
  • IAM Role: the role you created in Steps 1 and 2.

5. Go to the /aws/ec2/networkInterface Log Group. In Actions > Subscription filters > Create Lambda subscription filter, select “LMLogsForwarder” (or whatever you named the Lambda function during stack creation) and provide Subscription filter name. Click Start Streaming.

The logs will start to propagate through the Lambda to the Log Ingestion API.

Send logs from CloudTrail

To send logs from AWS CloudTrail to LM Logs:

1. On the CloudTrail page of your AWS portal, click Create Trail.

2. Provide Trail name.

3. Uncheck “Log file SSE-KMS encryption” if you do not want to SSE-KMS encrypt your log files.

4. Check “CloudWatch Logs Enabled” and provide log group name as: /aws/cloudtrail

5. If you have existing IAM role CloudTrail permissions, provide it as input in IAM role box. Else a new role can also be created, make sure to provide a name for the new role.

6. In the next page choose the type of logs that you would like to be collected. Supported logs include Kinesis Data Stream logs, Kinesis Firehose API logs, and ECS logs.

7. In the next page review the provided configuration and click Create Trail.

8. Go to CloudWatch’s log group page and select the /aws/cloudtrail log group.

9. In Actions > Subscription filters > Create lambda subscription filter. In lambda function select “LMLogsForwarder” (or whatever you named the Lambda function during stack creation) and provide Subscription filter name. Click Start Streaming.

Logs will start to propagate to LM Logs. You will be able to see logs in the AWS account name resource.

Send logs from CloudFront

To send logs from AWS CloudFront to LM Logs:

1. In the CloudFront page of your AWS portal, select the distribution for which you would like to collect logs.

2. Select “On” for Standard Logging.

3. In S3 bucket for logs, select the bucket in which you want to store the logs.

4. Click Create Distribution.

5. Go to S3 bucket that you had selected in 3rd step.

6. Go to Properties > Event notifications and click Create event notification.

7. Provide an Event name.

8. In Destination’s Lambda function tab, select “LMLogsForwarder” (or whatever you named the Lambda function during stack creation).

9. Click Save changes.

You will be able to see logs from your S3 bucket in LM Logs.

Send logs from Kinesis Data Streams

Since logs from Amazon Kinesis Data Streams are filtered from AWS CloudTrail, you can follow the CloudTrail instructions to ingest these logs.

Send logs from Kinesis Data Firehose

Amazon Kinesis Data Firehose consists of two kinds of logs: API Logs and Error Logs. API Logs are collected from CloudTrail, and you can follow the CloudTrail instructions to ingest these logs.

To ingest Error logs:

1. In Create delivery system > Configure system, select “Enabled” for Error Logging.

This creates a log group in CloudWatch with the delivery system’s name in the format: /aws/kinesisfirehose/<Delivery system name>

2. In Actions > Subscription filters > Create lambda subscription filter. In lambda function select “LMLogsForwarder” (or whatever you named the Lambda function during stack creation) and provide Subscription filter name. Click Start Streaming.

Logs will start to propagate through lambda to LogIngest. You will be able to see logs with the Kinesis Firehose delivery system’s name.

Send logs from ECS

Since logs from Amazon ECS are filtered from AWS CloudTrail, you can follow the CloudTrail instructions to ingest these logs.

Troubleshooting

To help troubleshoot logs forwarded from Amazon CloudWatch, enable debug logging in your Lambda logs:

1. In the AWS console, go to AWS Lambda > Functions and select “LMLogsForwarder” (or whatever you named the log forwarding Lambda function during setup).

2. Add an environment variable with the key DEBUG and value true.

In This Article