Logging network activity is one of the most powerful tools in your security arsenal; it lets you confirm or deny suspicious behavior and it lets you reconstruct events long after they happen. According to many security analyses, organizations often learn about intrusions long after the initial compromise, and having a history of events allows you to go back to see what happened and figure out what can be done to prevent similar attacks in the future.
Log data is invaluable not only to investigate specific events, but also to detect the existence of events in the first place. Comprehensive log data allows staff to investigate hunches or theories that a SIEM is unable to assist with. For example, an analyst may notice a PC acting strangely, and suspect that it has some sort of yet-undetected malware. An investigation into the PCs history can help determine if the suspicion is warranted or misplaced: what systems has this PC connected to? Has it attempted other suspicious activity such as scanning other PCs? Has it attempted to connect to unusual or unauthorized devices? If the PC has connected to suspicious websites, who else in the organization has also connected to these websites? Another example is where an analyst notices an abnormality such as a user logging in at odd hours. Does that user do it often, or is this the only time? Is the source address from a foreign country?
With comprehensive network logging, an analyst can answer all these questions quickly.
It is important to gather logs from as many sources as possible. Most organizations simply collect logs from routers and firewalls. If you’ve compartmentalized your network as explained above, you should also be logging attempted policy violations. That is, whenever an access list blocks traffic, that event should be logged.
You should not stop at logging just your routers and firewalls. You should be logging all authentication events (logins) from your Active Directory, Linux and VPN devices. Be sure you log both successful logins as well as failures. You should also include proxies and IDS sensors. If other enterprise applications, such as order processing, CRM systems or accounting systems generate logs, you should be collecting those too. In short, collect all the logs you can from all your systems.
Of course, its one thing to collect logs. Searching for the right log entry out of gigabytes of log data is another matter. Many organizations collect lots of log data and dutifully archive it, never to be seen again. But the ability to find relevant information quickly can make or break a your security monitoring. Fortunately there are a number of log analysis systems, both commercial and open source, that allow you to search through months of log data based on any number of criteria. Having a detailed log history can help you answer questions such as “When did jsmith log into the system last month?” “Has the PC at IP address 18.104.22.168 ever been attempted to access a restricted server?” “Who logged in to the database at 2:00 am last Saturday night?”
A search system needs to combine structured and unstructured (i.e. full text) searches. Searches need to be fast enough so that analysts can perform ad-hoc queries, follow up on hunches, etc. Ideally search data can be presented in several forms, including tables and graphs. In some cases, it is necessary to control access to certain users, so they have access to only a subset of log data.
Should you build an open source system, or buy a commercial product? This is the perennial question, as there are both commercial products and open-source utilities that fit the bill for logging and retrieving data. Often, commercial logging systems are often built on top of or duplicate existing open-source products, and give them user-friendly interfaces. Many open-source logging utilities are part of Unix/Linux utilities and are fully functional systems in their own right.
Open-source applications do need more system administrative effort than commercial products, but they usually offer more flexibility than their commercial counterparts. Commercial products, on the other hand, are almost “plug-and-play,” but you are often locked into managing your data the way the products developers think it ought to be done.
In the end, the build or buy decision often comes down to the available system administrative skills. If you have staff that is comfortable in the Linux environment, open-source software may be a cost-effective option. If not, then commercial products may be a better choice. But whatever you end up using, having this log data available will be one of your most powerful tools to detect and defend against network attacks.