The (Great) Web Application Firewall

Introduction

Web application firewalling (WAF) isn't anything particularly new to technology, but it is relatively new to me and my organisation. In this post I talk about our experimentation with it, subsequent implementation and what we might do in the future.

WTF?

What is a WAF? Well, in simple terms it's a protection layer between nasty people on the internet and your runtime web applications. It actively strives to prevent attackers from using common vectors to break into your systems and ultimately databases, where all the golden eggs lie. It's not a total solution, but rather another mechanism you should consider deploying in your many-layered approach to good web application and therefore data security.

Imagine this scenario:

aitchteeteepee://mywebsite.com/page.aspx?variable=' update users set isadmin=1 where emailaddress='mike@appsecbloke.com'-- (not a valid email address BTW :))

It's a basic (yet potentially devastating) SQL injection attack. Injection is and has been the OWASP supremo for years. Other variations of this attack are available.

On a vulnerable website, invoking the above HTTP request would elevate my permissions as a user to those of an administrator, based on a few prerequisites, including creating an account using an email address, finding an error on a page that calls into a database, but very little else.

Once you're an admin, you can knock yourself out, poking around, altering stuff and stealing personal data, should the application process it.

A WAF can be configured to identify requests of this nature and carry out a responding action, for example return an error, drop a connection, or redirect the requestor to somewhere else, like a honeypot.

If you're a fan of automation, tools such as Havij and SQLMap will carry out injection attacks while you watch in awe, but as they make HTTP requests to do it, those requests can be snared and dealt with using a WAF.

Cross-site scripting (XSS), or in other words, forcing an often innocent user to do something on a website they don't intend (or want) to do, is a major problem. It's in the OWASP TOP THREE and has been for ages.

Imagine this scenario:

aitchteeteepee://mywebsite.com/search?searchterm="'+alert("XSS")+'"'

On a vulnerable website, this request would run a XSS attack and again, a WAF could be configured to block it. Again, XSS comes in many different flavours, behaving in different ways, but generally with a pretty bad effect.

Properly configured security headers would also provide some protection against XSS on modern browsers (another layer of protection), but not everyone uses a modern browser.

OK, so hopefully you get it.

Pre-WAF Life

Back when we weren't entirely certain of the security posture of our web applications, we suffered a major distributed denial of service (DDoS) attack. It was pretty devastating and effectively took our business offline for around three days. Not a great look for an ISP. It was a varied attack, composed of TCP Syn flooding, UDP randomness and HTTP pingback, using pwned Wordpress implementations. A small army of folks in the business pulled levers and changed advertising and routing to get us back online, but it was tough and felt extremely laboured and unsophisticated. It didn't feel like something we could repeat quickly and easily next time.

Let's RnD

After the abovementioned event, a few of us (managers and engineers) got together and came to the obvious conclusion that come what may, we needed a resilient web presence. A presence that showed we were still in business, even if other systems were unavailable. How did we go about doing this?

So, we picked our primary marketing website as the sensible candidate and started to flesh out a plan.

As a business, we'd been using (and selling) Amazon Web Services (AWS) for a while and had a few experts or keen learners dotted around the business.

We all got in a room and this is what happened:

How about we front the site in AWS? We can't immediately move it there, as there are far too many dependencies, but we could route access to it that way.

This sounded good, so we did a proof of concept project (time boxed to two weeks, with Adrian, Carl, (another) Carl and I helming the piece). This was a success and also enlighted a few people as to the technology available to us; ELB for your load balancing, CloudFront for your caching, S3 for your storage, Route53 for your DNS and so on.

Importantly, there was no actual WAF involved at this point, but we had figured out that we could route traffic (good and bad) via AWS to our primary marketing website and that we had far more control over it. So, we made that all live.

This was late 2015 / early 2016...

New Problems

Or in fact old ones, we hadn't dealt with.

Zero DDoS attacks later, our attention turned to something entirely unrelated - a secure gateway for our published APIs. We'd probably been complacent to some extent, as the fact we hadn't had an attack on our marketing website didn't mean it was invulnerable, nor the many other applications that hadn't had the AWS treatment. That said, we had something new to deal with.

Many attackers these days are bypassing the traditional surface of the front-end website being the door into everything, in favour of hitting up APIs. This makes sense, as APIs are often seen as 'plumbing' that no one pays too much attention to. This is obviously risky. We realised this and by taking the decision to allow controlled, yet public access to our APIs, we knew needed to protect them.

Ok, so we revisited our approach to protecting our marketing website, for signs of opportunities to reuse it or indeed update it. That's precisely what we did.

What Happened Next?

We didn't change much from an AWS point of view, but we added some new things.

We decided that continuing to use proxies were a great idea, as this gave us flexibility around routing, based on the GeoIP location of source traffic. So, we built a selection of stacks; UK, Rest of World and Failover, all sat behind Amazon's elastic load balancing technology.

If you are UK based and hit a site, you get the UK stack, if you're from outside the UK, you get the RoW (OK, you get it), but if for whatever reason a site is unavailable, you'll land at a friendly page telling you this and providing information about other ways to get in touch with us. All of this is handled by Route53 DNS.

Right, let's assume you're a bad guy / gal. Certain things happen:

• We look at your client IP address and if it exists in a public black list, we drop your connection, period. The same thing happens if you hit us through TOR, because you'll only use TOR if you're A. a researcher (in which case you'll understand) or B. a hacker (in which case we don't care)
• You'll likely find that private resources you're probing are already IP restricted down to ranges we control
• We inspect your user agent string and if it stinks, we HTTP 400 you
• We inspect your HTTP request and if it contains an injection or script based attack variable, NAXSI HTTP 418s you (for LOLs)
• If your request is for resources that contain things like /admin or anything similar, we route you to a Honeypot server and analyse you
• If your IP repeats any of the above more than a determined number of times, Fail2Ban, well bans you

Additionally and because of our use of AWS Route53, we're able to control requests to our web applications at country level, which might sound a little insular and weird, but in a DDoS scenario, this could prove vital; access to our systems is predominantly UK-based, so in a difficult situation, we can make the rest of the World effectively disappear. Certain web applications we operate are already only accessible if you're in the UK.

Other benefits include our ability to utilise Amazon's certificate manager for our SSL certs. We no longer have to worry about expiry, or renewal for our public facing web applications.

As we route all traffic through our WAF, we get the bonus of consolidated HTTP traffic logging, which we then pipe into our monitoring platforms, giving us visibility of who's doing what and when.

As we reverse proxy requests, we are able to control access to resources in clever ways, transparent to clients. It's all good!

Our WAF is in itself a variety of different technologies, each serving their own purpose. The configuration build repeatability of this technology is all orchestrated using Puppet.

OK, so this doesn't protect from every attack, including where an attacker is spoofing all their things (IP, user agent etc), but then I did make it clear to begin with that WAF is but a layer of protection, not absolute protection in itself.

You still need to get your web application security and everything else right, which is something I talk about in another post.

What Next?

Well, we're in a reasonably good place at the moment, as far as attacks via the internet are concerned, but in my role, it pays to be on edge and assume you're getting something wrong, somewhere.

I'd like us to use Mod_Security, as it offers a higher level of sophistication in terms of protection against web application attacks. It also plays nicely with our dynamic application scanning tool of choice, as Netsparker will write rules to it.

It's also a far better understood technology amongst my colleagues across the business, as it's already in use there.

Also on the look into list is Amazon's own WAF offering, which looks powerful and would obviously sit well alongside their other technologies we use.

Summary

WAF isn't a solution, it's part of a broader or deeper solution. It does however sit in the front row, alongside hardening of infrastructure, secure network design, well implemented transport security, strong access control and good old / new fashioned secure coding practices.

Where TOR operates by the concept of layered anonimity, WAF sits towards the outer layer of the web application security onion. It isn't a panacea and shouldn't be thought of as such, but it is important and it has its place.

As our use of technologies such as those described above develops, I'll write about it some more.

Update

In August 2018, I had the privilege of delivering a talk around this subject at BSides Manchester. The content of the talk was pretty much the content of this post, but with obligatory supporting slides. I've embedded both a recording of my talk at BSides Manchester and a video of the slide deck below, for anyone interested.

Recording

Slide Deck

Update 2019

We're now in the implementation stage of upgrading our WAF to (as mentioned above) replace NAXSI with ModSecurity. I think we now know more than enough about how our technology has behaved over the last couple of years to make that jump.

It's no small task, as there'll be OS upgrades, rule migrations, all new config changes and other tweaks, but as we get into a stable state, I'll write a fresh post.