Recently, I had the pleasure of speaking with Anton Chuvakin, Director of Product Management at LogLogic. We had an interesting discussion about log management and the open source project he's involved in that collects Windows event logs. Here's an overview of our chat.

Drowning in logs is all too common nowadays when organizations are struggling with a combination of operational, security, and compliance requirements. A typical organization will have logs from a wide array of log sources such as server operating systems (Unix and Windows), desktops, mainframes, network gear such as routers and switches, web proxies, security gear such as network IDS, IPS or anti-virus tools, Web, e-mail, and DNS server software as well as enterprise applications.

Large organizations typically have tens of thousands of servers generating log files. The challenge for IT is how an enterprise can efficiently collect logs from all these servers without losing any data. In fact, almost 30% of all enterprise data is log data. Owing to compliance requirements from regulations like Sarbanes-Oxley and PCI, the archived log data must be stored. A single organization can easily be required to store hundreds of terabytes of log data. How IT manages this large set of data continues to be a challenge for enterprises, regardless of size.

Log Management and Intelligence is an approach to dealing with large volumes of computer-generated log messages (also known as audit records, audit trails, event logs, etc.), which consists of log collection, centralized aggregation, long-term retention, log analysis (in real-time and in bulk after storage) as well as sharing the information with the relevant parties across the organization. Such analysis is usually done for security, operational (such as system or network administration), or regulatory compliance.

Effectively analyzing large volumes of diverse logs is a challenge. From huge log volumes - often reaching hundreds of gigabytes of data a day for a large organization - to log format diversity, obstacles in dealing with log data confounds IT daily. Couple that with undocumented proprietary log formats that often hinder analysis and the presence of false log records found in certain logs, such as intrusion detection logs, and the situation becomes more complex.

To unravel the complexity, tools to handle log collection and analysis are sometimes built by users, assembled from various open source components or acquired from commercial vendors in the form of LMI or Log Management and Intelligence solutions. So far, the open source community hasn't been able to come up with a single tool to deal with most log challenges that confront IT. But there are some promising contenders.

Moreover, the open source community has been pretty effective in building pieces of log management infrastructure. Syslog-NG enables log collection from Unix servers and network devices, serving as a better replacement for standard syslog daemons than is typically provided by operating system vendors as a primary example of open source excellence., There are also a huge number of simple scripts and small programs such as logwatch, logsentry, and fwanalog that were written by the open source community over the years to handle specific logs or a particular slice of a log puzzle. At times it seems that it was easier for some people to create their own script instead of looking for one online. However, most of these tools focused on Unix and Linux platforms and largely ignored Windows-based systems.

One of the recent open source solutions that enables a critical part of log management is Project LASSO, a Windows-based open source software designed to collect Windows event logs, including custom application logs, and provide for the central collection and transport of Windows log data via TCP syslog to any syslog-NG compatible log receivers. Before Project LASSO incorporating Windows server and workstation logs in an overall log management process was extremely onerous. One had to use agents installed on every single Windows system to collect the logs or be stuck with super-expensive proprietary solutions. And deploying agents on every system is one of the most dreaded tasks in all of enterprise IT.

Open source tools such as syslog-ng existed for years to simplify log management for Unix and Linux as well as network devices that support syslog (such as Cisco routers and firewalls), but the Windows part of the world was largely excluded because binary Windows event logs aren't syslog. Project LASSO bridges this gap, allowing remote Windows log collection without agents, as well being deployed as an agent on each server, if needed. LASSO enables the inclusion of logs in log management systems, such as the one by LASSO's sponsor LogLogic or other companies.

Overall, Project LASSO enables users to connect the dots by allowing central collection and analysis of Windows event logs with the same ease that they are used to with Unix and Linux. After the data is collected by LASSO, users can use report and search features to review and analyze logs across all the systems in an enterprise: Windows, Unix, network systems, applications, etc. Moreover, LASSO greatly reduces the impact on monitored servers in terms of storage and processing, as well being able to capture application-specific and custom Windows event logs.

Using LASSO, IT can gain invaluable insight into its network. For example, a query for an account holder can be run across all the systems in an enterprise, identifying the files or applications that he or she touched. Such capability is critical for compliance, as well as for incident response and forensics.

Log Management is increasingly making its way onto the IT agenda. Today, a simple Google search of "log management" drives this home with over 240 million hits - and it's growing daily. As more organizations move toward implementing policies for compliance, log management systems have taken on a vital role. LMI's greatest value lies not only in the improvements it creates in automating compliance and providing forensics, but there are great benefits to be found in ensuring operational efficiency by giving IT visibility into the details of what has happened on every system in its network. As Log Management and Intelligence matures, the open source tools that intersect with log files will surely continue to evolve and mature.