As there’s been a lot of SIEM hype over the last few years, I’d thought I’d go over a few common problems I’ve come across whilst deploying new, and fixing existing SIEM deployments. This won’t be a complete list, but just some of the most common. I may update this post from time to time.
SIEM Deployment Never Ends
This title might sound like a negative, but it’s meant to highlight the reality of SIEM. I started this blog by saying I would discuss problems I’ve come across whilst fixing existing deployments – perhaps a misnomer there as SIEM deployments are never ending in the sense that they require attention and modification on an ongoing basis.
A good IT team will look for continuous improvement, and with that improvement comes changes to your infrastructure. Those changes are important to a SIEM as SIEMs are heavily reliant on context and sufficient log sources to provide benefit. Due to this very fact, SIEM deployments are never finished.
Lack of Visibility
In order for a SIEM to provide you useful information, you need to ensure it is receiving appropriate log sources. For example, if you want a SIEM to tell you a machine has been infected with a virus, you need to make sure you are sending the appropriate logs to SIEM. Seems like a simple point, but it’s something IT teams often fall down on.
Another point is making sure sufficient logs are being collected. Windows can be a bit of a maze trying to ensure everything you need is being logged, WMI generally isn’t enough to discover a lot of common attack methods and Sysmon isn’t always the most straightforward to configure logging for. Further, it’s very common for devices such as firewalls to have rule logging not enabled across the board (in particular the implicit deny), logging is a common area for mis-configuration as it’s not seen as service impacting.
Lack of Context
I mentioned context previously – and what what I mean by this is that SIEMs will come pre-installed with a large number of rules. No every rule applies to every device type, and so some context is needed to allow the SIEM to more intelligently apply these rules. This will help to reduce false positives and general noise from alerts.
Poor Tuning Practices
Another common problem is poor tuning. This usually takes the form of either tuning too much or not at all – and what I mean by too much is that your rule modification matches too much traffic – so much so that you might miss important information in the future.
A good way to prevent this is through the use of thresholds – assuming your solution supports this. Instead of say white listing all DNS traffic for a particular host, try and establish a base line of usage and set the rule to alert after a higher threshold is exceeded.
People are usually dazzled during the sales stage. They are told the SIEM will show them all sorts of mysterious information that was previously unavailable to them. The reality is that SIEMs can show you a lot of information – but not all that information is interesting. Generally, you can probably get 80% of information you want from a SIEM direct from the log source itself – what the SIEM adds (and what sales guys often bang on about) is correlation. The ability to take logs from to distinct sources and alert should a malicious link between logs be found.
Reality is, if you’re getting something that is considered “interesting” to a secuirty analyst – it’s likely to be a breach, in which case you don’t want that. Other useful information that could come out of a SIEM is a breach of acceptable use policy or perhaps mis-configuration of your devices might become apparent.
This is where your getting soo many notifications, alarms or incidents, that you’re just overwhelmed with information and become desensitised to it – ultimately resulting in all incidents getting ignored. The reality is three actions should come out of every incident, and if they aren’t completed then you’re going to be stuck with incident fatigue. The three most common actions will be:
- Tuning required on SIEM – this is a false positive and requires tuning on the SIEM to prevent the incident reoccurring.
- Tuning required on the log source – this is where tuning is required on the log source instead of the SIEM. This could be due to a false positive which could be caused by mis-configuraiton on the device.
- The incident is valid and requires further investigation and/or remediation.
Quite often, after further investigation you’ll find step 3 is actually a false positive and requires tuning in line with steps 1 or 2.
I didn’t want this post to become critical of SIEM. SIEMs are valid and useful tools and they usually fall down due to expectations of the solution and not having a full understanding of how much work they require, let a lone the resources to do that work. These issues are common between self managed and outsourced solutions. When work loads are high, the first things to get neglected are often the security/visibility solutions.
I’ll try to refine this post as I go along. I initially wrote this a while ago but hadn’t had the time to complete or refine it. I was just conscious we haven’t posted a great deal of content recently. Nonetheless, I hope you find this post useful and please feel free to post your SIEM horror stories in the comments.