Edition 14: To WAF or not to WAF
Effectiveness of WAFs are a hotly debated subject in AppSec circles. This editions tries to bring a structure to that discussion.
First, a confession: For the longest time, I have been biased against WAFs. Most WAF tool sales pitches are too good to be true and often feels like NetSec folks trying to solve AppSec issues, without understanding AppSec well enough. Unsurprisingly, there’s more nuance than that.
A few days ago, On a Twitter space with Chris Folini and others, his answer to a question of mine (34m mark) got me thinking. It’s clear that WAFs work well against some kind of attacks and can only respond in a limited set of ways (block or slow down traffic), but it’s unclear what those attacks are and what types of responses they work well against. It maybe useful to develop a hypothesis which helps us determine when WAFs can be useful.
The hypothesis
WAFs can be used as a part of the defense strategy only when two conditions are met: (1) The cost of fixing the defect in the target software is prohibitively more expensive than blocking attack traffic and (2) the organization can tolerate a percentage of legitimate traffic being blocked.
Testing the hypothesis
Let’s take a few examples to understand the hypothesis better
L7 (application layer) Denial of Service attacks
Traditionally DoS attacks involved flooding the victim with a lot of L3/L4 (network/transport layer) traffic (think SYN flood attacks), which overwhelm servers and cause them to go down. Over the last few years though, the most successful [sic] DoS attacks have been on the application layer (L7). The attacker floods an application with a large number of HTTP packets. The number isn’t large enough to bring down the infrastructure, but the cost of processing these HTTP packets is so high (given the volume), that it brings down down stream systems like the database, message queues and so on.
Is the cost of fixing a defect is prohibitively more expensive than blocking attack traffic for that defect?
L7 DoS attacks can be stopped without WAFs. If the application is capable of processing all requests and block attack traffic (as opposed to processing them), the attack can be thwarted without blocking any traffic at the WAF layer. However, making these systems resilient comes at a cost. If a system is built to handle 500 RPS (requests per second) of legitimate traffic, making it resilient to 100x or 1000x the traffic is very expensive, especially given the additional cost does not add any value to legitimate users.
Is there is tolerance in the orgranization to some legitimate traffic being blocked?
Properly configured WAFs block traffic based on well defined rate limits set for particular endpoints (especially true for APIs) . This means, in the event of a DoS attack, there is still a chance that some legitimate traffic is blocked (e.g.: If some of the attack traffic is coming up from a VPN endpoint with a large number of users, it’s possible that some of the users from that IP are legitimate and the others aren’t). In the middle of a L7 DoS attack which could compromise your key applications, I would guess that most companies are OK with blocking some legitimate traffic (the value of “some legitimate traffic” is key here, but every org and every app within an org has a different threshold for that).
In summary, WAFs are a good tool to respond to L7 DoS attacks. Of course, the usual caveats of using the WAF intelligently is important, but it’s probably worth investing sufficient time and money to configure your WAF properly to respond to L7 DoS attacks.
SQL injection
We often see a “OWASP Top 10” rulepack in WAFs, claiming to block attacks against software with common defects like SQL injection. Are these rulepacks worth it?
Is the cost of fixing a defect is prohibitively more expensive than blocking attack traffic for that defect?
This one is easy. The cost of fixing SQL injection(e.g.: Using Parameterized queries) is dramatically lower in a majority of the cases than finding every possible attack vector against SQL injection and blocking them at the WAF layer (which is a needle in a haystack).
Is there is tolerance in the orgranization to some legitimate traffic being blocked?
Answer to the first question is sufficient to rule out WAF as an effective strategy against SQL injection, but let’s answer the second question anyway.
While the cost of fixing SQL injection is low, one could argue that the cost of finding them itself is high (even before you fix something, you need to find it). Given how crafty injection payloads can get, you will need to block a lot of payload patterns to stop injection attacks. In other words, applying broad WAF rules to all input can stop SQL injection attacks. For instance, If you drop all requests with a special characters (slight exaggeration), it will block most SQL injection attacks, but it will also block a lot of legitimate traffic (poor Ms. O’connor will never have her name stored in a database again).
In summary, WAFs are a blunt tool which will cause a lot of collateral damage when used to block attacks like SQL injection (or most common injection attacks). This maybe acceptable is if there is no way to fix known defects (think maintaining legacy apps, where all the developers have left the building). But if that’s the case, configuring WAFs well are least of your problems.
Log4J RCE response
It’s December 10th 2021. You are prepping for a nice weekend. Maybe some beer or just catching up on lost sleep or maybe there are a couple of good cricket matches you want to watch. But Log4J had other plans for you . All InfoSec professionals know what happened next. In the first few hours/days after the CVE was released, a key problem all teams faced was how to identify all instances of Log4J in their portfolio. It was clear that for large organizations, this process would take weeks (if not months). In the meantime, WAF came to the rescue.
Is the cost of fixing a defect is prohibitively more expensive than blocking attack traffic for that defect?
For the first few days after the CVE dropped, the cost of upgrading Log4J was “unknown” for most orgs. Like we discussed before, how do you secure something when you don’t know if and where it exists? WAF rules turned out to be extremely useful this time. Most SRE teams will tell you that within 24 hours of the CVE dropping, most servers were hit with many attacks containing the payload attempting to exploit vulnerable versions of Log4J. While implementing the right WAF rules wasn’t easy, it turned out to be way cheaper than upgrading Log4J everywhere, over a December weekend.
Is there is tolerance in the orgranization to some legitimate traffic being blocked?
The WAF community ( open source tools and commercial vendors) quickly came up with rules that could block attack traffic. The nature of the attack also meant the obscure payload needed to make the attack successful would probably not block a lot of legitimate traffic (it was still non-zero). This made the call easy. You rather block a small chunk of obscure traffic for a few days than risk RCE.
But as Chris mentioned, this only bought us time (which was precious at the time). In the medium to long run, the effectiveness of WAF response fo the Log4J RCE mirrors that of SQL injection. The cost of performing the upgrade is much lower than maintaining a WAF block list with constantly changing attack vectors.
A few scenarios where the hypothesis may fail:
WAFs are a blunt tool whose primary function is to block bad traffic. In cases where it’s difficult to distinguish bad traffic from good traffic, the hypothesis fails (even if the answer to both questions are “yes”)
WAF configuration is non-trivial. This is not a “set it and forget it” tool. It needs regular monitoring. You want to have a cross functional team (SRE, CloudSec & app teams) to meet regularly and make sure the WAF rules still make sense. If this isn’t done, the effectiveness of the WAF goes down over time.
It isn’t practical to go through each attack type and determine if a WAF is a good response option. A better idea maybe to assume WAFs don’t work for all types, and then consider WAFs only in areas where cost of fixing is high and the tolerance for dropped traffic is non-zero.
That’s it for today! Are there other questions we should answer before considering WAFs as a viable solution? Am I grossly underestimating the usefulness of WAFs? Hit me up! You can drop me a line on twitter, LinkedIn or email. If you find this newsletter useful, do share it with a friend, colleague or on your social media feed.