The idea of paying money to store logs nobody is looking at may seem like a waste.
Well, that is until you need those logs.
At that point, you see how valuable log retention is, especially if there’s a security or compliance issue. When you prioritize log retention, you can look back to investigate an incident or provide data for an audit — especially when you centralize log and metric data in one platform.
Here is what you need to know about log retention, the importance of a log retention policy, best practices, and more.
Contents
- What Is Log Retention?
- What Is a Log Retention Policy?
- What Is a Log Retention Period?
- How Long Should You Store Log Data?
- What Are Log Retention Best Practices?
What Is Log Retention?
A log doesn’t need to be a lengthy, complicated text file. Logs are temporal data events and can come from anywhere — client update events, network flow data, web server logs, you name it. All digital actions create logs. However, even if all logs are collected, not all are stored.
Following the log collection and centralized log aggregation process comes log retention. Retaining logs is a critical component of any successful cybersecurity strategy.
Log retention refers to the archiving of event logs, particularly those related to security, concerning the duration for which you store these log entries. These entries typically refer to all cybersecurity, allowing companies to hold information on security-related activities. By keeping records, IT teams can gain critical insight into the activities happening on their network.
For example, what activities were performed on the system and when? What user account or software was involved in the activity and what was the outcome?
What Is a Log Retention Policy?
While following best practices, one of the steps you need to consider is creating and managing a log retention policy. Within this policy, you will outline log storage requirements.
Other considerations will include:
- What type of information you’ll store and for how long
- Data confidentiality
- Online vs. offline data
A critical starting point is to store compressed copies of your firewall logs (host or network), intrusion detection system (IDS) logs, and audit logs.
Here is an example of the University at Buffalo’s log retention policy. For example, log file retention times are specified in the Retention Guidelines for Log Files for this policy. The default log retention time is 30 days unless the log file type falls under one of three categories, such as an IDM audit or UNIX auth/login. In these cases, the retention time is six months. Instructions are outlined in the policy concerning how log files are handled. Depending on your organization and infrastructure, your policy may be more complex.
You may also create several retention policies for each of your systems, which are then prioritized based on their importance. Anything concerning security should be part of an extended log retention policy. For example, data related to user authentication or credit card authorization. Accounting and tax compliance software may also require longer retention policies since corporate auditors and government authorities can request retroactive analysis.
How often an application is used is also something to consider. If there is an application you do not often use, such as once a month, you’ll want to consider an extended retention policy just in case developers need to verify the source of an issue. Lower frequency of use would require developers to go back further to debug effectively.
Classifying Business Records
Within your log retention policy, you must classify logs based on varying categories. These include the following.
- ‘Must keep’ logs — These are the files and events you are legally required to store. This category should also prioritize logs that may relate to potential legal issues. Store these logs based on the length of time required by regulations and laws relevant to your industry or until they are no longer relevant for potential legal disputes.
- ‘Want to keep’ logs — These logs will hold value to your business but are not required to be stored by law. Your policy should address an appropriate log retention period for these records. You should also state how often these logs are reviewed to determine whether they still hold any importance.
- ‘Destroy’ logs — Just as you would shred any hard copy documents that are no longer needed, it’s important to delete logs that are either obsolete or logs that are not being used by your company yet act as a potential security threat.
Although the most important logs for your organization will vary depending on the scope of your business and industry as a whole, most organizations prioritize the following log types.
- User IDs and credentials
- Data and time of access to critical assets (e.g., activation and deactivation of firewalls, important security events, etc.)
- Failed and successful login attempts
- Changes made to system configurations
- Details of incidents
What Is a Log Retention Period?
A log retention period is the amount of time you keep logs. For example, you may keep audit logs and firewall logs for two months. However, if your organization must follow strict laws and regulations, you may keep the most critical logs anywhere between six months and seven years. This timeframe is the log retention period.
Retaining logs for extended periods is optimal for security and compliance measures. However, this can become costly. That is why you need to create log retention policies that outline set log retention periods.
Think about a typical breach discovery timeline to justify a log retention period. The average timeline to discover an incident is 100 to 200 days depending on the source. In that sense, keeping logs for a year is sensible.
How Long Should You Store Log Data?
The answer to this question is it depends.
While some companies store log data for a year or less, some organizations must comply with regulatory mandates. In this case, you may need to retain a security log for several years. Every organization operates in varying business environments. While one organization may retain logs for six months, another may keep logs for 18+ months.
The key is knowing the requirements your organizations need to comply with, based on the nature of your business.
Most organizations find that a minimum of one year meets most regulatory requirements. However, you must be mindful of your organization’s industry standards, regulations, and laws, as well as your company’s cybersecurity concerns.
For example, according to the Basel II Accord, international banks must keep their activity log for 3 to 7 years. In comparison, an eCommerce corporation must keep audit logs for at least six months, as outlined by the VISA Cardholder Information Security Program (CISP). If you’re in the healthcare industry, you’ll need to consider the Health Insurance Portability and Accountability Act (HIPAA), protecting patients’ data and retaining logs for up to six years.
Here are some of the primary considerations:
- Compliance — What regulatory reasons are there to keep log messages and for how long? GDPR, PCI, NISPOM, and corporate policies all come into play here. For example, should you keep specific logs containing personally identifiable information for shorter periods? Should some of the fields of these logs be encrypted?
- Operational needs — Are there use cases where logs would be beneficial after the fact? For example, concerning a third-party copyright claim or other potential legality issues?
- Costs — Logs take up space and space costs money. There comes a time when storing certain logs for longer outweighs the benefits they bring.
What Are Log Retention Best Practices?
The key to better security log retention is to follow best practices, implementing proactive strategies from day one. When it comes to your company’s security, you always want to remain proactive, not reactive. If something happens and you do not have the systems in place, your efforts moving forward may be too little, too late.
While organizing a large IT environment, log management can be challenging, which is even more reason to follow best practices. Successfully maintaining critical logs is not just a good security posture — taking the following steps will improve peace of mind and ongoing business operations if something happens. You need to protect your overall IT infrastructure while gaining valuable insight.
Archive Data Centrally
If something happens, your security logs act as the evidence you need to investigate. Without these logs, you cannot conduct an analysis. It is recommended that you archive your logs centrally to ensure their integrity and compliance.
When archiving these logs, you’ll want to encrypt and time-stamp them. Other techniques, such as hashing, are also optimal to enhance security. You should also monitor your logs, implementing a tool to identify issues and send alerts.
Read more about log alerts and the power of log analysis.
Maximize Security Log Size
Depending on your organization’s requirements, you should set the maximum-security log size. That way, you can scale efficiently by adding more data to the network. The last thing you want is to lose data because of insufficient space, especially when you think you’re remaining compliant.
Create a Log Retention Policy
Security logs should be retained for longer periods compared to application log data. If you experience a breach or attack, your logs are your evidence. However, it’s not beneficial to hold logs forever, which is where a log retention policy comes into play, as discussed earlier. Creating a log retention policy sets critical guidelines for your team to follow.
When creating your policy, focus on both log security and log retention. For example, are you storing clients’ data or internal access keys for API? These examples are the type of data you want to encrypt.
Log retention guidelines are critical, but so is log monitoring. Implement alerts and automate log monitoring to improve security. You can achieve this by leveraging a tool or platform that simplifies this process — do not underestimate the value of log analysis. Tools that offer AI and machine learning abilities sift through logs to determine a baseline and detect anomalies.
Don’t Overcomplicate the Logging Process
The most critical logs can become challenging to find if you log too many events. Log the most critical events, so that essential information isn’t lost or overlooked. These events include account lockouts, login failures, and file access. Again, taking this step will ensure network security and optimal regulatory compliance. Also, ensure that all system clocks are synchronized for more accurate time stamps.
The best way to ensure the logging process isn’t overly complicated is to invest in a platform that makes this step seamless. Also, the more humans you involve in the process, the more likely errors will result. A unified platform will allow you to take advantage of greater automation. When you can easily access all the information you need to investigate an issue across your technology ecosystem, this will significantly speed up the process.