Log Monitoring

Log monitoring 

Logs are strings of text that record events occurring on a system. They come in many different formats and can be written locally to a log file or sent over the network when an event happens.

There are two basic types of logs:

• System logs provide information about events happening at the OS (operating system) level. Examples of events in system logs include system-level authentication, connection attempts, service and process starts and stops, configuration changes, errors, and point-in-time usage and performance metrics.
• Application logs provide information about events happening at the software level. “Software level” includes specialized server software, such as dedicated proxies and firewalls, as well as other software applications. These logs include events like application-level authentication, CRUD operations, software configuration changes, errors, and application-specific functions. Examples of application-specific logs include proxy logs, firewall logs, and developer-inserted log statements.

Logging strategies

Log monitoring is the practice of reviewing logs to determine what events are occurring on your systems. Logs contain valuable information and granular detail. There are different ways you can monitor logs:

• Local log monitoring is done by directly accessing a system and reviewing local log files. It’s the fastest way to monitor logs for a single system and can be helpful for developers building and testing code locally. However, it isn’t easily scalable, and log retention policies can leave your systems’ disk space clogged with logs. Local monitoring also isn’t feasible for infrastructure you don’t have direct access to, such as Functions-as-a-Service (FaaS) and Software-as-a-Service (SaaS).
• Vendor-provided log consoles are interfaces that allow you to review logs without direct server access. These are necessary as SaaS solutions don’t expose the underlying server, but users are still responsible for monitoring application-level behavior. The flexibility in searching, filtering, and exporting these logs varies. These consoles can be helpful for admins working specifically in the tool, but scaling and log retention are issues here too. An organization with multiple SaaS vendors will find it cumbersome to validate even basic activity across different SaaS platforms. Administrators can’t easily manage all the different logins, UI navigation, and search languages. Additionally, vendors generally have log retention policies that don’t align with the time you may be legally required to retain logs.
• Centralized log management moves your logs from their origin to a central repository that can have its own data retention, storage, and access policies. You can normalize your data, search across your entire environment, and schedule alerts in this centralized location. Log management platforms can be on-prem, self-hosted, or SaaS deployments.

Log monitoring use cases

Local log monitoring may be sufficient for developers, and vendor-provided log consoles may be adequate for specialized admin teams. Still, there are many operational and security reasons to centralize your log management. For example, centralized logging shines when you need to standardize monitoring across servers or services, correlate disparate data sources, or monitor ephemeral environments. Common use cases for log monitoring include:

• Monitor all authentication systems for password spray attacks and alert the Security Operations Center (SOC) for investigation.
• Correlate Intrusion Detection System (IDS) logs with vulnerability scan data and asset management data to determine when an attack occurs against a vulnerable system. When an attack is detected, the system can alert both the SOC and the asset owner.
• Scope a project to upgrade all servers to use TLS 1.2 for communication by identifying active usage trends and high-volume systems.
• Monitor server health and escalate tickets to the operations team when servers show troubling KPIs.
• Notify the applications team when a spike in errors occurs within the application logs of a critical system.
• Troubleshoot dropped traffic between two endpoints.
• Discover system relationships by injecting test data into a workflow and tracing its progression.
• Investigate user activity across all systems to assist with a help desk call about persistent lockout issues.

Log monitoring features

These are the key capabilities to evaluate when selecting a log management solution:

Ingestion types

A log management solution should support different kinds of log ingestion. Syslog-ng and other collectors provide flexible, generalized ingestion points. Agents installed on endpoints can simplify configuration for endpoint log collection. API or webhook integrations, both push and pull methodologies, are useful for SaaS platforms and microservice infrastructures.

Log filtering and masking

During log ingestion, you want mechanisms to filter out unneeded logs and mask sensitive data. Verbose debug logging is helpful for application developers. However, it’s generally not useful for centralized log management. Dropping these logs can help you save on ingest cost and downstream compute costs. It can also simplify user search results.

As a result, filtering can help improve performance and protect against accidental data exposure, such as the disclosure of customer PII in overly descriptive application logs.

PII concerns are also why you want to make sure your log management solution has masking capabilities. Suppose your logs include sensitive information such as customer names, email addresses, IDs, or plaintext passwords. In that case, you want to put safeguards in place to prevent this information from being ingested. Otherwise, you risk exposing PII.

Secure transport and storage

Logs should be encrypted both in transit and at rest. Encryption prevents tampering, data exposure, and other security concerns. It also simplifies log onboarding for sensitive log sources that contain data subject to regulatory minimum encryption standards. The transport mechanism itself should be robust and mitigate the risk of dropping logs. Although UDP was previously the standard for log transport, RFC 5424 now recommends TLS-based transport. You may still see UDP logging in legacy equipment, but it isn’t best practice due to the potential of log loss.

Parsing

A log management solution should provide up-to-date and easily configurable parsing for common data structures and popular vendor tools. The power of a log management tool lies in its ability to parse logs, allowing users to search parsed fields efficiently.

Unfortunately, there’s no global standard for log formatting. Commonly used structures include JSON, XML, and key-value pairs, but vendors and application teams can and will use custom formats. Choosing a log management solution with an existing parsing library and auto-parsing on standard formats will save a lot of time during log onboarding and product upgrades that impact the log format.

Parsing occurs at different points in different log management solutions. Some are more flexible than others, but that can come at a processing cost. When choosing a solution, it’s important to understand when parsing happens, and the process to re-parse logs should the parsing not work correctly or the log formatting change.

Searching

A log management solution should offer a flexible, robust searching language for querying your logs efficiently. UI-based filtering can be helpful for new users, but text-based queries make creating and sharing queries much more efficient in the long term.

In addition to simple keyword searching against parsed data, you should also be able to:

• Search using wildcards
• Query the raw text of the logs
• Manipulate and store information in variables
• Use regex to extract and match text
• Perform basic mathematical and statistical functions
• Manipulate strings
• Perform basic comparisons and aggregations across a set of logs
• Combine disparate datasets

Supplemental data

A log management solution should provide easy upload mechanisms to add supplemental data to the platform. Logs record events that occur in the environment, but they don’t capture the environment’s state. Correlating logs with infrastructure metrics, application traces, and end-user experience data gives teams the full picture, from user actions to root causes.

A business can have a lot of supplemental information that correlates against logs and can significantly minimize manual triage. This information can help answer questions like:

• Who owns this application?
• What office does this server live in?
• Who is the manager of this employee?
• What error text does this error number map to?

IT ticket info, asset data, employee data, frameworks, and manual uploads are all common supplemental data sources.

Saved searches and alerting

Users should be able to save searches for reuse and schedule saved searches for proactive alerting. Saved searches should be shareable with the option to run on demand or be scheduled for periodic execution to generate alerts and reports in a common data format. Alerts are typically configured using a cron schedule.

Resource allocation and scaling

Understand platform limitations regarding user, ingest, and compute. Where does the capacity to grow exist, and where would usage patterns cause a problem? If you’ve never built a log management solution before, accurately scoping is a challenge. Safeguards should exist to limit the impact users on the system have on each other. For example, a user who kicks off a compute-intensive search shouldn’t monopolize the platform’s resources.

Documentation

Centralized logging is intended to be a company-wide log management solution. This means the documentation must be excellent to keep the barrier to entry low. You don’t want to buy something so complicated that your team needs formal training before getting any value from it. Make sure your vendor has clear, concise, easily accessible documentation on how to perform key functions. Bonus if they have free self-led training.

Retention and recovery

From both a billing and an efficiency standpoint, implementing a hot/cold storage strategy can be advantageous. Most users don’t need more than the past 30 days of logs, but legal storage requirements can mandate years-long retention. Storing unused logs can reduce processing overhead and storage costs. Ensure all log storage has recovery capabilities in the event of log destruction or system failure.

Residency

Make sure your log management solution can accommodate any data residency laws in play in your countries of business. Some data may be legally mandated to remain on in-country servers.

Pricing

Pricing strategies for log management solutions are diverse, and many companies use a mix of pricing components that add up to your final bill. Different pricing strategies work better depending on your ingest, storage, compute, search, user, and functionality requirements.

Pricing strategies may include the following components:

• Flat rate pricing charges a single fee for usage of the tool, regardless of component costs.
• Ingestion pricing charges a fee per amount of data ingested.
• Storage pricing charges a fee per amount of data stored.
• Compute pricing charges a fee per amount of compute resources consumed, i.e., per the deployment’s search load.
• Seat pricing charges the customer per number of users or “seats” that are in use.
• Features pricing charges based on specific components of the solution, often described as “features,” “modules,” “functionality,” etc.

Best practices

These log management best practices help teams get more value from centralized logging.

Maintain a current asset inventory

Asset management is key to good log hygiene. You can only capture logs from what you know is out there.

Logs document events, not state

Logs are useful for detecting events in the environment, but they can’t provide on-demand state values.

Take configuration, for example. A log management solution can tell you which servers have recently initiated a session using TLS version below 1.2. However, it can’t spot machines with TLS 1.0 enabled that haven’t used the protocol in any transactions.

Know your legal landscape

Nothing grinds a technical working session to a halt faster than uncertainty about whether onboarding a particular set of logs is legally allowed. Have your legal counsel survey relevant data residency laws.

Within your company, clearly identify and document which logs have in-country restrictions, which countries require paperwork to be submitted and the status of that paperwork, and which countries have no data residency restrictions. Maintain a similar document regarding data retention policies for different log sources.

Plan for sensitive data

Identify high-risk logs that may expose sensitive information. Even if you mask the data, create a response plan for how you’ll react if that sensitive data is exposed.

Log formats can change, temporarily invalidating masking. Developers may turn on more verbose logging to help debug an issue, only to discover that the new logs contain sensitive information. Make sure to have well-defined answers for whether you’ll restrict access to these logs, delete logs containing sensitive data, or let them roll off on schedule.

Take a balanced approach to logging

Deciding which logs to ingest is a balancing act. A good equilibrium has to take into account ingestion fees, value from potential alerts generated, convenience, maintenance/administrative overhead, vendor tools that can offer similar functionality for a slice of the environment, and legal or compliance requirements.

Sometimes an alert is enough

Some log sources can be prohibitively large to ingest. In those cases, ingesting alerts and other action-item events can be a good compromise. Users who need more information will need to go elsewhere to retrieve the full logs (such as a vendor-provided log console or directly onto a server). However, you can still take advantage of a log management platform’s centralized correlation and alerting functionalities by ingesting alerts.

Configure health monitoring for log ingestion

Ingesting health monitoring logs into the log management platform can be tricky. Consistent, high-volume log types are easy enough to spot when they go down. If you have infrastructure that’s only used occasionally, such as a failover device or a small application with infrequent user activity, you’ll find setting the right threshold for when you consider the logging “broken” is more an art than a science.

Normalize timestamps

Regardless of ingestion type, the log management solution should create a universal timestamp of when the log was ingested. The log itself will contain a timestamp of when the event occurred.

To avoid timezone issues, you should normalize to a universal timestamp. By comparing the event timestamp with the ingestion timestamp, you can identify ingestion delay issues.

Normalization can also be helpful when log timestamps don’t clearly reflect the timeline of events. For example, a popular IT vendor will timestamp security alerts with the time the suspicious event occurred, rather than when the activity was determined malicious and the security alert was generated. The log timestamp can make it look like a security alert was ignored for days, when the reality is that the alert just came in.

Normalize your data

When parsing your logs, use vendor-agnostic field names. This simplifies the environment for exploring users and reduces administrative overhead when technology of the same kind is added or swapped out.