Support Center Home


Setting up Azure Logs Ingestion

The Microsoft Azure integration for LM Logs is implemented as an Azure function that consumes logs from an Event Hub and forwards the logs to the LogicMonitor logs ingestion API.

LogicMonitor provides a Terraform script that automates the setup and configuration of the integration. Alternatively, you can deploy the integration using Gradle.

The Azure integration for LM Logs can be found at the following link: https://github.com/logicmonitor/lm-logs-azure

Prerequisites

  • LogicMonitor API tokens to authenticate all requests to the log ingestion API.
  • Install the Azure CLI (command line interface)
  • Azure devices can only send logs to the Event Hubs within the same region. Each Azure region requires a separate Azure Function deployment. See Forwarding Azure logs to Event Hub.

Deploy using Terraform

Terraform is included in the Azure CLI. To deploy the Azure Function using Terraform:

1. Sign in to the Azure CLI: az login

2. Download the file: deploy.tf

This script creates the Event Hub, resource group, and storage account needed in your Azure environment to run the Azure Function. If there are no errors, it will deploy and start the Azure Function.

3. (Optional) Update app_settings in deploy.tf to set the optional parameters. See Configure Application Settings.

4. Start Terraform: terraform init

5. Run the following command, filling in the necessary variables:

# terraform apply --var 'lm_company_name=<ACCOUNT>' --var 'lm_access_id=<ID>' --var 'lm_access_key=<KEY>' --var 'azure_region=<REGION>’

(Optional) you can have these variables configured in a .tfvars file. If so, run the following sequence of commands to apply it:

# terraform plan --var-file terraform.tfvars -out tf.plan
# terraform apply tf.plan

5. In your Azure portal, configure you devices to redirect logs to the new Event Hub. For more information, see Forwarding Azure logs to Event Hub.

Once setup is complete, you should start seeing the forwarded logs appear in the LogicMonitor Logs page.

Deploy using Gradle

Use the Gradle plugin to build the function app and deploy it to Azure.

1. Before you can use the plugin: Create an Event Hub and Function app.

The runtime stack should be set to Java version 11.

2. Update the Application settings for the function. See Configure Application Settings.

3. Run the following command:

# ./gradlew -DazureResourceGroup=<your Azure Function's Resource Group name> -DazureFunction=<your Azure Function name> azureFunctionsDeploy

If your account has multiple subscriptions, you need to add the following flag to the previous command:

-DazureSubscription=<subscription ID>

Once setup is complete, you should start seeing the forwarded logs appear in the LogicMonitor Logs page.

Configure Application Settings

You can configure the application settings used by the log forwarder Azure Function with the following information:

Parameter Description
LogsEventHubConnectionString (Required) The Event Hub connection string. The Event Hub settings are located in the file: host.json.
LogicMonitorCompanyName (Required) Your LogicMonitor company or account name in the target URL: https://<account>.logicmonitor.com
LogicMonitorAccessId (Required) The LM API tokens access ID. We recommend an API only user to be used with this integration.
LogicMonitorAccessKey (Required) The LM API tokens access key.
LogApiClientConnectTimeout (Optional) Connection timeout in milliseconds. Default is 10000.
LogApiClientReadTimeout (Optional) Read timeout in milliseconds. Default is 10000.
LogApiClientDebugging (Optional) HTTP client debugging: true|false. Default is false
LogRegexScrub (Optional) Regular expression pattern to remove matching text from the log messages.

We recommend using this parameter to filter out any logs that contain sensitive information so that those logs are not sent to LogicMonitor.

Forwarding Azure logs to Event Hub

After the deployment is complete, the Azure Function listens for logs from the Event Hub. For most Azure resources, you can direct their diagnostic settings to the Event Hub.

If the function was deployed using Terraform, the logs should be sent to an Event Hub named log-hub in the namespace lm-logs-<LM company name>-<Azure region>.

Forwarding logs from virtual machines will require some more configuration.

Forward Virtual Machine (VM) logs

To configure VMs to forward their system and application logs:

1. Install a diagnostic extension on the VM.

2. Sign in to the Azure CLI: az login

3. Install wget: sudo apt-get install wget

4. Download the configuration script:

# wget https://raw.githubusercontent.com/logicmonitor/lm-logs-azure/master/vm-config/configure-lad.sh

5. Run the script to create the storage account and configuration files needed by the diagnostic extension:

# ./configure-lad.sh <LM company name>

6. Update lad_public_settings.json to configure types of system logs and their levels (syslogEvents) and application logs (filelogs) to collect.

7. Run the following command to configure the extension:

# az vm extension set --publisher Microsoft.Azure.Diagnostics --name LinuxDiagnostic --version 3.0 --resource-group azure-logs-us --vm-name vm-logs-us --protected-settings lad_protected_settings.json --settings lad_public_settings.json

Troubleshooting

Enable debug logging

For logs forwarded from Microsoft Azure, you can enable application logging for the Azure Function in the Azure CLI using the commands: webapp log config and webapp log tail.

To enable logging and configure the type and level of logging:

# az webapp log config --resource-group <Azure Function's Resource Group name> --name <Azure Function name> --application-logging true --level verbose --detailed-error-messages true

To review the logs:

# az webapp log tail --resource-group <Azure Function's Resource Group name> --name <Azure Function name>

Finding Azure logs

You can check that logs are being sent from the Azure Event Hub to LogicMonitor by looking in the logs folder for the specific agent sending the logs to the Event Hub.

For example, if you are running a Windows VM with a IaaSDiagnostics extension, agent logs will be in the Azure directory:

C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.Diagnostics.IaaSDiagnostics##VERSION####WADID##\Configuration

Debugging with Gradle

You can use Gradle to run the Azure Function locally for debugging purposes.

1. Install Azure Functions Core Tools 2.0 and above.

2. Copy the application settings to the file: local.settings.json

3. Execute the following command: ./gradlew azureFunctionsRun

4. Use the remote debugging port 5005. It can be modified in the build.gradle file, setting localDebug.

In This Article