Fuzz Testing for Online Services

Want to skip straight to the action? Check out this GitHub repo for a lightweight dotnet core implementation.

Years ago, I was an engineer on a large-scaled web service. At the time, I managed a service to do traffic forking - sending a sanitized copy of production traffic into the test environments to drive out bugs in new code. It was pretty effective at finding bugs, but after attending an internal security conference, I thought I had a way to make it better.

Enter, then, the fuzzing service. This service took normal HTTP requests of a vein we might normally receive in production, and applied targeted model-based fuzzing to them. Using attacks provided by an internal security service, we combined with an accurate model of our particular HTTP request usage and a smart strategy to emphasize testing only the parts of our data model that was important. We were able to find an order of magnitude of new bugs - some of which had been living in production for years as intermittent crashes.

We didn't just find crashes, though. We also found memory leaks, stray un- and mis-handled exceptions, and even a couple of actual security vulnerabilities. All without any engineering effort beyond the time to write the fuzzer. Once it was on, it required very little maintenance and was able to provide very broad assurance that code under fuzz was stable enough to run in production.

What does fuzzing do? It takes something like this:

Sample HTTP Fuzzing Strategy JSON

{
   "Name": "My HTTP Fuzzing Strategy",
   "QueryParam" : {
     "MaxManipulations": 2",
      "Probability": 25"
   },
   "Path" : {
      "MaxManipulations": 1,
      "Probability": 5
   },
   "Headers": {
      "Key": { Probability = 5 },
      "Value": { "MaxItemsToFuzz": 3, "Probability": 10 },
      "HeaderWhitelist": [ "Cookie", "Authorization" ]
   },
   "UseAllRelevantManipulations": true
}

GET https://queenofcode.net/search?q=ponies&qs=n&form=QBRE HTTP/1.1
User-Agent: Fiddler/4.6.20171.9220 (.NET 4.6.2)
Host: queenofcode.net
Accept-Language: en-US
Accept-Encoding: gzip, deflate, br
Accept: text/html,application/xhtml+xml
Pragma: no-cache
Cookie: DUP=Q=N51ltffjRFaNUXQ2&T=69727&A=1&IG=140419BB6CF3209258263; SRCHS=PC=U316; SRCHD=AF=CHROMN; SRCHUID=V=2&GUID=2222333DDDD3333C0C0C0C0F00D33227 &dmnchg=1; _EDGE_V=1; MUID=2222333DDDD3333C0C0C0C0F00D33227; _ITAB=STAB=REC;

And turns it into this:

GET https://queenofcode.net/search?q=AAAAAA%20&qs=n&form=QBRE HTTP/1.1
User-Agent: Fiddler/4.6.20171.9220 (.NET 4.6.2)
Host: queenofcode.net
Accept-Language: en-US
Accept-Encoding: gzip, deflate, br
Accept: text/html,application/xhtml+xml
Pragma: no-cache
Cookie: DUP=Q=N51ltffjRFaNUXQ2&T=69727&A=1&IG=140419BB6CF3209258263; SRCHS=PC=U316; SRCHD=AF=CHROMN; SRCHUID=V=2&GUID= 2222333DDDD3333C0C0C0C0F00D33227 &dmnchg=1; _EDGE_V=1; MUID=(oNcliCk=alert()); _ITAB=STAB=REC;

The keys to successfully fuzzing a service are relatively simple: an accurate data model, a trusted set of attacks, and a plan to apply those attacks sanely to your data model. Nobody should be re-inventing the wheel for attack set - those are well-documented in the security industry. Not sure where to start? Try FuzzDB for a great overview of attack patterns and payloads. Still, never underestimate the power of a string of 'A' of indeterminate length. I've gotten as much value out of [AAAAA]{10,1000} as I have out of the most intricately crafted and applied XSS exploit.

Keys to successful fuzz testing

Why does this matter? If you're working on a traditional application - one that works on standard or file IO - then you probably have tools solved for you. AFL (American Fuzzy Lop) is a great solution if you are working in C or C++ on an application with data input/output that it models easily. It uses code coverage to guide generational fuzzing to explore as many branches as possible through your application.

But I, personally, work in dotnet core. And I don't work on applications that ship to someone's desktop. I've found that, over the years, the tools I used to use to test web services have migrated into all-in-one offerings that are expensive or require engagement with a specific company. Open-source base libraries appear to have evaporated. But that's no reason to fret! As shown with the sample code in this repo it does't take a security expert and decades of experience to spin up a fuzzing solution to meet your problem set. Any software engineer - tester, dev, security expert, etc - can spin up their own fuzzing solution by following these basic principles.

This is part of a larger talk - interested in hearing about it in person? Drop me a line! I love giving this talk.