AWS Service Configuration for Log Ingestion

LogicMonitor needs Amazon Web Services (AWS) service configuration for log ingestion because it enables the platform to collect and analyze log data generated by your AWS services. LogicMonitor provides a centralized view of your entire cloud environment, enabling you to monitor performance, troubleshoot issues, and identify potential problems within your AWS applications by analyzing the detailed logs generated by different AWS services. 

For logs that are ingested using CloudFormation stacks, you need to create a CloudFormation template that defines stacks. Stacks are AWS resources for LogicMonitor to access and monitor your infrastructure, including IAM roles with appropriate permissions. After you deploy the CloudFormation stack to provision the resources within your AWS environment, LogicMonitor will begin to collect data from your AWS services. For more information about captured logs using AWS CloudWatch, see Lambda Function Deployment for AWS Logs Ingestion.

Metadata for AWS Logs

The following table lists metadata fields for the AWS Logs integration with LM Logs. The integration looks for this data in the log records and adds the data to the logs along with the raw message string.

Property	Description	LM Mapping	Default
arn	Amazon Resource Name, unique identifier for AWS resources.	arn	No
awsRegion	Region for the AWS resource.	region	No
eventsource	The source service sending the event.	_type	Yes
ResourceType	Indicates from where the AWS logs are coming from. It also indicates the location of the deployed service.	ResourceType	Yes

Supported AWS Services for Log Ingestion

For AWS logs that do not use CloudFormation stacks, configurations must occur on the AWS side. The following lists each supported AWS service:

• AWS EC2 Logs
• AWS ELB Access Logs
• AWS RDS Logs
• AWS Lambda Logs
• AWS EC2 Flow Logs
• AWS NAT Gateway Flow Logs
• AWS CloudTrail Logs
• AWS CloudFront Logs
• AWS Kinesis Data Streams Logs
• AWS Kinesis Data Firehose Logs
• AWS ECS Logs
• AWS EKS Logs
• AWS Bedrock Logs
• AWS EventBridge CloudWatch Events Logs

Configure Log Forwarding for AWS

After deploying the Lambda function, configure the individual AWS services to forward logs to the Lambda function. You can find instructions for forwarding logs from the supported AWS services below.

Forwarding EC2 logs

1. Open the CloudWatch console using the AWS CloudWatch Logs Agent. For more information, see Quick Start: Install and configure the CloudWatch Logs agent on a running EC2 Linux instance from Amazon.
2. Select Actions > Create Lambda subscriptions filter. 
3. In the Create Lambda subscription filter window, select Lambda Function > LMLogsForwarder (or a different name you used for the Lambda function during stack creation). 
4. Select Start streaming.

Forwarding ELB Access logs

1. In the EC2 navigation page, select Load Balancers and select your load balancer.
2. Navigate to Attributes > Access logs andselect Configure access logs.
3. Select Enable access logs and select the S3 bucket used to store the logs. You can create a bucket if it doesn’t exist.
4. In the S3 bucket, select Advanced settings > Events andadd a notification to All object create events.
5. Select Lambda function.
6. Select LMLogsFowarder (or a different name you used for the Lambda function during stack creation).
7. Select Start streaming.

Forwarding RDS logs

1. Configure your RDS instance to send the logs to CloudWatch. For more information, see Monitor Amazon Aurora MySQL, Amazon RDS for MySQL and MariaDB logs with Amazon CloudWatch from Amazon.
2. In the CloudWatch console, select the log group that is forwarding the RDS logs.
3. Navigate to Actions > Create Lambda subscription filter andselect Lambda function.
4. Select LMLogsFowarder (or a different name you used for the Lambda function during stack creation).
5. Select Save changes.

Forwarding Lambda logs

1. In the CloudWatch console, select log group that is forwarding the Lambda logs.
2. Navigate to Actions > Create Lambda subscription filter andselect Lambda function.
3. Select LMLogsFowarder (or a different name you used for the Lambda function during stack creation).
4. Select Save changes.

Forwarding EC2 Flow logs

1. In the Lambda role policy, add the following lines to the permissions:

   "logs:CreateLogGroup", 
   "logs:CreateLogStream", 
   "logs:PutLogEvents"

   Copy
2. Navigate to Trust Relationship and add the following line to Service Tag Role:

   "vpc-flow-logs.amazonaws.com"

   Copy
3. Create a log group in CloudWatch with the name /aws/ec2/networkInterface.
4. Navigate to the Network Interfaces pageto find your EC2 instance ID. Select the Network Interface row and create a flow log with the following settings:
   • Destination Log Group: /aws/ec2/networkInterface
   • IAM Role: the service tag role you previously created.
5. In the Log record format, select Custom Format. The first value under Log Format should be instance-id. Set other values depending on your requirements. For more information, see Logging IP traffic using VPC Flow Logs from Amazon.
6. Navigate to the /aws/ec2/networkInterface Log Group.
7. Navigate to Actions > Subscription filters > Create Lambda subscription filter.
8. Select LMLogsFowarder (or a different name you used for the Lambda function during stack creation) and provide the Subscription filter name.
9. Select Start Streaming.

Forwarding NAT Gateway Flow logs

1. In the Lambda role policy, add the following lines to the permissions:

   "logs:CreateLogGroup", 
   "logs:CreateLogStream", 
   "logs:PutLogEvents"

   Copy
2. Navigate to Trust Relationship and add the following line to Service Tag Role:

   "vpc-flow-logs.amazonaws.com"

   Copy
3. Create a log group in CloudWatch with the name /aws/natGateway/networkInterface.
4. Navigate to the Network Interfaces page for your NAT Gateway ID. Select the Network Interface row and create a flow log with the following settings:
   • Destination Log Group: /aws/natGateway/networkInterface
   • IAM Role: the service tag role you previously created.
5. Navigate to the /aws/ec2/networkInterface Log Group.
6. Navigate to Actions > Subscription filters > Create Lambda subscription filter.
7. Select LMLogsFowarder (or a different name you used for the Lambda function during stack creation) and provide the Subscription filter name.
8. Select Start Streaming.

Forwarding CloudTrail logs

1. On the CloudTrail page of your AWS portal, select Create Trail.
2. Provide the Trail name.
3. Uncheck Log file SSE-KMS encryption if you do not want to SSE-KMS encrypt your log files.
4. Check CloudWatch Logs Enabled and provide the following log group name: /aws/cloudtrail
5. If you have existing IAM role CloudTrail permissions, you can provide them in the IAM role field. Otherwise, you can create a new role.
6. On the next screen, select the Event type for the logs that you would like to collect. For more information, see CloudTrail concepts from Amazon.
7. On the next screen, review the provided configuration and select Create Trail.
8. Navigate to the CloudWatch log group page and select the /aws/cloudtrail log group.
9. Navigate to Actions > Subscription filters > Create Lambda subscription filter.
10. Select LMLogsFowarder (or a different name you used for the Lambda function during stack creation) and provide the Subscription filter name.
11. Select Start Streaming.

Forwarding CloudFront logs

1. On the CloudFront page of your AWS portal, select the distribution to collect logs.
2. Select On for Standard Logging.
3. In S3 bucket for logs, select the bucket to store the logs.
4. Select Create Distribution.
5. Navigate to the S3 bucket you selected to store logs.
6. Navigate to Properties > Event notifications and select Create event notification.
7. Provide an Event name.
8. In the Destination’s Lambda function tab, select LMLogsForwarder (or a different name you used for the Lambda function during stack creation).
9. Select Save changes.

Forwarding Kinesis Data Streams logs

Since logs from Amazon Kinesis Data Streams are filtered from AWS CloudTrail, you can follow the CloudTrail instructions to ingest these logs. For more information, see Forwarding CloudTrail logs.

Forwarding Kinesis Data Firehose logs

Amazon Kinesis Data Firehose consists of two kinds of logs: API Logs and Error Logs. API Logs are collected from CloudTrail, so you can follow the CloudTrail instructions to ingest these logs. For more information, see Forwarding CloudTrail logs.

To ingest Error logs:

1. In Create delivery system > Configure system, select Enabled for Error Logging.
2. This creates a log group in CloudWatch with the delivery system’s name in the format: /aws/kinesisfirehose/<Delivery system name>
3. Navigate to Actions > Subscription filters > Create Lambda subscription filter.
4. Select LMLogsFowarder (or a different name you used for the Lambda function during stack creation) and provide the Subscription filter name.
5. Select Start Streaming.

Forwarding ECS logs

Since logs from Amazon ECS are filtered from AWS CloudTrail, you can follow the CloudTrail instructions to ingest these logs. For more information, see Forwarding CloudTrail logs.

Forwarding EKS logs

Pre-requisites:

On EKS Cluster, do the following:

1. Create a nodegroup on cluster.
2. Add Amazon CloudWatch Observability plugin (add-on).

– Or –

Before the EC2 instance logs can be forwarded to LM Logs, they need to be collected into CloudWatch Logs. Forward different EKS logs to cloudwatch.

1. To forward application metrics to cloudwatch, see Setting up the CloudWatch agent to collect cluster metrics from Amazon.
2. To forward application logs to cloudwatch, see Send logs to CloudWatch Logs from Amazon.  
3. To forward logs using Fluenbit, see Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs from Amazon.  

After adding the plugin, the following five different log groups are created into CloudWatch using your cluster-name:

• /aws/containerInsights/<cluster-name>/application
• /aws/containerInsights/<cluster-name>/host
• /aws/containerInsights/<cluster-name>/performance
• aws/containerInsights/<cluster-name>/dataplane
• /aws/eks/<cluster-name>/cluster

1. In the EKS Cluster, create a nodegroup in the EKS cluster.
2. Add the Amazon CloudWatch Observability plugin (add-on). For more information, see Quick start with the Amazon CloudWatch Observability EKS add-on from Amazon.

Sending EKS logs

1. Go to CloudWatch and select the log group that you want to forward logs. 
2. Navigate to Actions > Subscription filters > Create Lambda subscription filter.
3. Select LMLogsFowarder (or a different name you used for the Lambda function during stack creation) and provide the Subscription filter name.
4. Select Start Streaming.

Forwarding Bedrock logs

1. The following two types of logs are supported by AWS Bedrock:
   • Model invocation logging. For setting up the Model Invocation Logging, see Monitor model invocation using CloudWatch Logs from Amazon.
   • Knowledge Base Logging. For sending logs from the knowledge base to Cloudwatch, see Monitor knowledge bases using CloudWatch Logs from Amazon.
2. Create a log group in CloudWatch with name that contains “bedrock”.

1. Navigate to the Log Group you created with the name that contains “bedrock”.
2. Navigate to Actions > Subscription filters > Create lambda subscription filter.
3. Select Lambda Function and select LMLogsForwarder (or a different name you used for the Lambda function during stack creation).
4. Select Start streaming.

The Model Invocation logs will be mapped to the Bedrock model resource created in LogicMonitor. The knowledge-base logs will be mapped to the AWS account resource created in LogicMonitor.

Forwarding EventBridge (CloudWatch) logs

Amazon EventBridge (CloudWatch) Events provide real-time event streams that describe changes in AWS resources. You can use simple rules to setup, match, and route events to functions or streams through AWS EventBridge. For more information, see CloudWatch Events from Amazon.

Requirements

Set up your AWS and LogicMonitor integration. For more information, see AWS Logs Ingestion.

Creating Amazon EventBridge Rule

Create one rule for each service. The event pattern in the rule specifies which service events will be sent to the target. For the target, specify the Lambda function as LMLogsForwarder or the Lambda that sends logs to LM. The currently supported services for CloudWatch events are S3, Lambda, ECS, Kinesis, SQS, EC2, and Account.

To create a rule to setup CloudWatch events, do the following:

1. Go to Amazon EventBridge web page and select Rules.
2. On the Define rule detail page, provide the name and description for the rule. 
3. To choose a Rule type, select Rule with an event pattern. This ensures that whenever an event is generated, it sends the events to the target.
4. For Method, select the Use Pattern Form method to get readymade pattern for a specific service.
5. In the Event source field, select AWS services.
6. In the AWS service field, select a service. LogicMonitor currently supports S3, Lambda, ECE, Kinesis, SQS, EC2, and Account services for CloudWatch events.
7. In the Event type field, select AWS API Call via Cloudtrail. As per the current implementation, LogicMonitor supports only AWS API calls via Cloudtrail. The events coming from the other types are ignored.
8. In the Select target(s) tab, do the following:

• For Target types, select AWS service.
• For Select a target, select Lambda function.
• For Function, select LMLogsForwarder or the Lambda that sends logs to LM.

9. Select Next to review the rule and then select Create.

When the rule is successfully created, you can view the logs in the LM Logs page. You can also view the message in JSON format in the Overview panel.

For more information, see Creating Amazon EventBridge rules that react to events from Amazon.