Testing ModSecurity rules – Spartan Website

Developing effective ModSecurity rule sets is often an iterative process, there may be many ways to get the exact same result. This article discusses several alternatives to test ModSecurity rules.
Let’s suppose the CRS is already up and running and the false positives are already under control (see this Handling ModSecurity false positives tutorial if not the case).

HTTP requests and ModSecurity variables

By far the most two common HTTP methods used in most web applications are GET and POST, there are many others, but the CRS allows only GET, HEAD, POST, and OPTIONS methods. Usually, GET has no request body only the REQUEST_LINE and the request headers. It is not forbidden by the RFC to use http message bodies with GET, however, CRS rule 920170 will complain if GET and HEAD ever use it.

ModSecurity will load the parameter names contained into the request line (query string) to the variables ARGS_NAMES and ARGS_GET_NAMES, the respective values to the variables ARGS and ARGS_GET, the parameter names contained in the POST bodies into the ARGS_NAMES and ARGS_POST_NAMES and the values to ARGS and ARGS_POST. It is possible to have the same parameter names duplicated in both the query string and the message body, or have multiple parameters with the same name in either the query string or the message body but when paranoia level 3 (PL3) or higher is in use the CRS rule 921170 will block such anomalies (that normal for some applications).

There are no restrictions on how many times a request header can be used in the CRS so far, even if the RFC (RFC7230 – HTTP header order) states it must not be used except for headers with known comma separated values or known exceptions, such as Set-Cookie.

Figure 1: Graphical interpretation of the parsing into variables for a GET request

Figure 2: Graphical interpretation of the parsing into variables for a POST request

Body processors

ModSecurity uses the request body processors to parse into variables the request http message body. The use of the wrong processor may result in a lot of false positives or false negatives. Only urlencoded and multipart processors are selected automatically depending on the Content-Type header. The other two body processors available are XML and JSON. This is highlighted in the ModSecurity documentation at the end of the ctl action section (ModSecurity Reference Manual – ctl).

“The requestBodyProcessor option allows you to configure the request body processor. By default, ModSecurity will use the URLENCODED and MULTIPART processors to process an application/x-www-form-urlencoded and a multipart/form-data body, respectively. Other two processors are also supported: JSON and XML, but they are never used implicitly. Instead, you must tell ModSecurity to use it by placing a few rules in the REQUEST_HEADERS processing phase. After the request body is processed as XML, you will be able to use the XML-related features to inspect it.”

(P.S. JSON is missing as possible value under ModSecurity Reference Manual – Request Body Processor).

The body processor is important when inspecting contents of the message body is required, it makes life easier. The JSON processor allows to use variables like “ARGS:user.name” to match a username property instead of parsing using regular expressions the request “@rx \{.*user:[^\}]+\{name:(.+)\}”. The XML processor do something similar but it uses XPath to address the fields and use “XML:/user/name” instead of struggling around all the layers of nested elements in XML “@rx .+(.+).+”. The body processors introduce another important change to the variables by removing the wrapping delimiters; XML contents cause especially high levels of content injection false positives, and JSON contents cause issues with non-word characters.

Testing a ModSecurity rule requires forging a payload that simulates the real application requests and matches the rule conditions. In general, testing will require to write/capture a payload, send the payload and check the response and or logs for the desired behavior.

Manually writing payloads

Manually writing a payload can be the faster method for small tests, the minimum usable HTTP/1.1 payload will have at least:

REQUEST_LINE containing separated by blank spaces
Host header, this should match the server name or server alias or the targeted virtual host

Ideally, the request also should include some headers to identify the type of content, encoding and the acceptable content responses such as:

Accept header specifies the type of content the client want (text/HTML, */*)
Content-Type header specifies the type of content the http message body contains (application/x-www-form-urlencoded)
Content-Length header specifies the length in bytes the http message body contains (Used with content-type)
User-Agent header specifies the client software used to send the request (Mozilla/5.)
Connection header specifies if the connection will be reused or if it should be closed immediately (keep-alive, close)

According to the RFC, every line should be terminated by a (\r\n) CRLF (RFC7230 – HTTP message format), but in practice, many web servers will accept an LF (\n) without issues. This is not important for Windows users, as CRLF is the line delimiter, but it is for Linux users as LF is the line delimiter. This causes some issues while scripting or when typing requests directly to the terminal using netcat or any other telnet like client as the web server will respond with a 400 error. In Apache 2.4, the “HttpProtocolOptions Unsafe” directive will allow the server to accept LF terminations.

Capturing payloads

A very simple way to have a payload with all the headers our client software use is by capturing it using an application proxy such as Burp, Zap or Fiddler, most browsers and client software can be configured to use an HTTP proxy to send all the requests. Outdated clients not compatible with proxies a transparent proxy can be implemented to redirect all the traffic to the proxy as if it was the target system using additional software like iptables, proxychains, burpsuite, etc.

Once the application is configured to use the proxy, it will intercept all traffic, and it is capable of modifying the responses and requests as required and simplify scripting and automation tasks.

Figure 3: Capturing requests and responses with an application proxy

The application proxies keep track of every request and response and can modify manually or automatically the transactions if desired to adjust values or include or remove elements.

Burp extensions can speed up the payload capture and replay process by simply right clicking on the requests to reproduce and use any of the following options (may require installing from the BApp store in the Extender tab):

Copy as requests (Copy As Python-Requests)
Copy as curl command
Generate script (Reissue Request Scripter)

Figure 5: Burp – Generate script using python requests

The ModSecurity audit log can be used to capture requests and responses, and even as a source to replay the requests over again with tools as modsec-replay (modsec-replay). The tool supports auditlog using native or JSON format, but it will not perform changes to the payloads on the fly to adjust to the testing needs, any modifications required must be performed directly on the source files that will be used for replaying the transactions.

Let’s assume that we want to test posting comments to a blog and we generate the full request with payload below and store it in /tmp/post.
POST /index.html HTTP/1.1 Host: localhost User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Referer: https://localhost Cookie: __cfduid=d0d6dblahblahblahblah5; _pk_id.6.5894=db4311174blahblahblahblah18.; comment_author_f895blahblah06=Manuel+Spartan; comment_author_email_f895blahblah06=spartantri%40gmail.com; comment_author_url_f895blahblah06=https%3A%2F%2Fspartantri.com%2FModSecurity%2F Connection: close Upgrade-Insecure-Requests: 1 Content-Type: application/x-www-form-urlencoded Content-Length: 309

comment=This+is+the+link+to+the+YAML+format+details+github.com%2fCRS-support%2fftw%2fblob%2fmaster%2fdocs%2fYAMLFormat.md&author=Manuel+Spartan&[email protected]&url=spartantri.com%2fModSecurity%2f&submit=Post+Comment+%c2%bb&comment_post_ID=616&comment_parent=0&akismet_comment_nonce=264da13699

Replaying the payloads

After designing all the possible payloads, there are several options to send them to the server, the obvious manual copy/paste into a telnet, netcat, openssl s_client console is fine for single quick tests, for repeated requests it may be easier to script the process (e.g. python, bash, ab) or use the application proxy (e.g. Burp Repeater) to repeat the request over and over or when required.

Netcat manual testing

Netcat can be used to do manual requests and for scripting, however it may be problematic to send requests to some servers due to the line termination, this can be solved by some minor adjustments. Add the netcat upper case C switch to send CRLF as line-ending if the web server use strict HTTP compliance settings and CRLF is not the line terminator it will respond with a 400 Bad Request status code.

Sample manual request:
$ nc -C 127.0.0.1 80 POST /index.html HTTP/1.1 Host: localhost User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Referer: https://localhost Cookie: __cfduid=d0d6dblahblahblahblah5; _pk_id.6.5894=db4311174blahblahblahblah18.; comment_author_f895blahblah06=Manuel+Spartan; comment_author_email_f895blahblah06=spartantri%40gmail.com; comment_author_url_f895blahblah06=https%3A%2F%2Fspartantri.com%2FModSecurity%2F Connection: close Upgrade-Insecure-Requests: 1 Content-Type: application/x-www-form-urlencoded Content-Length: 309

Sample server response:
HTTP/1.1 200 OK Date: Tue, 23 Jan 2018 06:32:45 GMT Server: Apache/2.4.29 (Debian) Last-Modified: Mon, 07 Aug 2017 20:28:42 GMT ETag: "13-5562fb16155f7" Accept-Ranges: bytes Content-Length: 19 Connection: close Content-Type: text/html

Done!

Netcat scriptable testing

Repeating requests or sending many different requests is not something that should be done manually, for that kind of testing it is better to start scripting or use a tool to do the job, the equivalent single liner command using netcat also have to take the line termination into account. Netcat is great for clear text HTTP testing but not for HTTPS.
* The content-length should be adjusted to fit the http message body to send.

Payload use CRLF for line termination
cat /tmp/post | nc -C 127.0.0.1 80

Payload use LF for line termination, using sed
cat /tmp/post |sed 's,$,\r,' | nc -C 127.0.0.1 80

Payload use LF for line termination, using awk
cat /tmp/post |awk 'sub("$", "\r")' | nc 127.0.0.1 80

Payload use LF for line termination, using unix2dos
cat /tmp/post |unix2dos | nc 127.0.0.1 80

Curl testing

Curl is an excellent tool for sending payloads to either HTTP or HTTPS servers, in curl headers can be specified one by one or from a file containing all headers, same as the payload contents, it supports proxy and TLS Server Name Indication(SNI). Burp have the “Copy as curl command” option when right clicking a request that will put into the memory the curl command to use directly from the command line.

Sample curl command generated by burp using the exact same request payload we have stored in /tmp/post
curl -i -s -k -X $'POST' \ -H $'Host: localhost' -H $'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0' -H $'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H $'Accept-Language: en-US,en;q=0.5' -H $'Accept-Encoding: gzip, deflate' -H $'Referer: https://localhost' -H $'Cookie: __cfduid=d0d6dblahblahblahblah5; _pk_id.6.5894=db4311174blahblahblahblah18.; comment_author_f895blahblah06=Manuel+Spartan; comment_author_email_f895blahblah06=spartantri%40gmail.com; comment_author_url_f895blahblah06=https%3A%2F%2Fspartantri.com%2FModSecurity%2F' -H $'Connection: close' -H $'Upgrade-Insecure-Requests: 1' -H $'Content-Type: application/x-www-form-urlencoded' -H $'Content-Length: 309' \ -b $'__cfduid=d0d6dblahblahblahblah5; _pk_id.6.5894=db4311174blahblahblahblah18.; comment_author_f895blahblah06=Manuel+Spartan; comment_author_email_f895blahblah06=spartantri%40gmail.com; comment_author_url_f895blahblah06=https%3A%2F%2Fspartantri.com%2FModSecurity%2F' \ --data-binary $'comment=This+is+the+link+to+the+YAML+format+details+github.com%2fCRS-support%2fftw%2fblob%2fmaster%2fdocs%2fYAMLFormat.md&author=Manuel+Spartan&[email protected]&url=spartantri.com%2fModSecurity%2f&submit=Post+Comment+%c2%bb&comment_post_ID=616&comment_parent=0&akismet_comment_nonce=264da13699' \ $'https://coreruleset.org/index.html'

The same request as above can be divided into files by headers and post data to make a shorter command and possibly send binary data (using the --data-binary switch) or very long payloads, curl will automatically compute the content-length so that header is not required in the headers file.
curl -XPOST http://localhost/index.html -H @/tmp/headers --data @/tmp/payload
Use the “-i” switch to display the response headers.

OpensSSL manual testing

For TLS/SSL enabled webservers openssl s_client can be used to send manual or scripted requests from the command line, there are a couple switches that are required to connect to strict servers, the switch -crlf to use CRLF line termination, and the switch -servername FQDN to specify the SNI, if the connect target does not match the SNI.

$ openssl s_client -connect 127.0.0.1:443 -crlf -servername spartantri.com $ (echo –ne "GET / HTTP/1.1\r\nhost:spartantri.com\r\n\r\n"; sleep 1) | openssl s_client -connect 127.0.0.1:443 -crlf -servername spartantri.com

OpenSSL script testing

Piping the standard output of a command to openssl is possible but s_client shutdown immediately as standard input closes, so a small delay added on the command will suffice to make it work.

$ (cat /tmp/post |sed 's,$,\r, ; sleep 1) |openssl s_client -connect 127.0.0.1:443 -crlf -servername spartantri.com
* Usually the host header, the SNI, and the server name or server alias should all match to the FQDN.

FTW testing

The Framework for Testing WAFs (FTW) was released in late 2017 (FTW), this framework can be used to do positive and negative testing of the CRS rules, FTW uses YAML for describing the tests including all headers and http message body contents and the expected responses from server as well as what string to match or not to match on the error log for a successful test.

The framework is quite configurable but not very fast. FTW can send just a few requests per second. Usually, that is not a problem unless sending several thousand or millions of requests per campaign is desired.

The FTW documentation describes the YAML format (FTW – YAML format) listing all the configuration parameters and also examples of the usage. Writing FTW tests is not complex and can be done in any text editor. A good test coverage of what the ModSecurity rule set should do or not do, should include tests for every use case of desirable and undesirable behavior.

Positive tests will send requests to the WAF that should trigger a rule, for example, writing a test that will trigger a specific keyword in an argument value. The tests 944350-1 and 944350-2 rules below attempt to trigger the PL3 rule 944350 from the CRS 3.1 by including a combination java, runtime, and processbuilder. When the tests run, FTW will send the payload to the server and will tail the log looking for the pattern described by the log_contains parameter.
--- meta: author: "spartantri" enabled: true name: "944350.yaml" description: "Description" tests: - test_title: 944350-1 desc: Argument test includes keywords java and Runtime stages: - stage: input: dest_addr: "127.0.0.1" port: 80 headers: Host: "localhost" User-Agent: "ModSecurity CRS 3 Tests" Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Accept-Encoding: gzip,deflate Accept-Language: en-us,en;q=0.5 Content-Type: "application/x-www-form-urlencoded" method: POST version: HTTP/1.0 data: "test=java.blablabla.Runtime" output: log_contains: "id \"944350\""

–
test_title: 944350-2
desc: Argument test includes keywords java and ProcessBuilder
stages:
–
stage:
input:
dest_addr: “127.0.0.1”
port: 80
headers:
Host: “localhost”
User-Agent: “ModSecurity CRS 3 Tests”
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Content-Type: “text/plain”
method: POST
version: HTTP/1.0
data: “test=java.blablabla.ProcessBuilder”
output:
log_contains: “id \”944350\””

FTW can also allow crafting of requests with non-ascii characters or non-standard requests using raw_requests or base64 encoded_requests which will override all other settings, these type of tests are useful to perform tests that otherwise will be hard or not possible to describe with the current YAML parameters.
--- meta: author: "spartantri" enabled: true name: "944300.yaml" description: "Description" tests:

–
test_title: 944300-0
desc: Argument test includes java serialization magic bytes, raw request
stages:
–
stage:
input:
dest_addr: “127.0.0.1”
port: 80
raw_request: “POST / HTTP/1.0\r\nHost: localhost\r\nUser-Agent: ModSecurity CRS 3 Tests\r\nAccept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: gzip,deflate\r\nAccept-Language: en-us,en;q=0.5\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: 9\r\n\r\ntest=\xac\xed\x00\x05\r\n\r\n”
output:
log_contains: “id \”944300\””

–
test_title: 944300-1
desc: Argument test includes java serialization magic bytes, base64 encoded request
stages:
–
stage:
input:
dest_addr: “127.0.0.1”
port: 80
encoded_request: “UE9TVCAvIEhUVFAvMS4wDQpIb3N0OiBsb2NhbGhvc3QNClVzZXItQWdlbnQ6IE1vZFNlY3VyaXR5IENSUyAzIFRlc3RzDQpBY2NlcHQ6IHRleHQveG1sLGFwcGxpY2F0aW9uL3htbCxhcHBsaWNhdGlvbi94aHRtbCt4bWwsdGV4dC9odG1sO3E9MC45LHRleHQvcGxhaW47cT0wLjgsaW1hZ2UvcG5nLCovKjtxPTAuNQ0KQWNjZXB0LUNoYXJzZXQ6IElTTy04ODU5LTEsdXRmLTg7cT0wLjcsKjtxPTAuNw0KQWNjZXB0LUVuY29kaW5nOiBnemlwLGRlZmxhdGUNCkFjY2VwdC1MYW5ndWFnZTogZW4tdXMsZW47cT0wLjUNCkNvbnRlbnQtVHlwZTogYXBwbGljYXRpb24veC13d3ctZm9ybS11cmxlbmNvZGVkDQpDb250ZW50LUxlbmd0aDogOQ0KDQp0ZXN0PaztAAUNCg0K”
output:
log_contains: “id \”944300\””

The purpose of negative tests is sending a payload and checking that such test do not trigger a given rule, this is especially useful for identifying regressions caused due to rule changes. A pool of known good requests that should be accepted by the WAF is a good source for FTW negative tests.
--- meta: author: "spartantri" enabled: true name: "944350.yaml" description: "Description" tests: - test_title: 944350-3 desc: Argument test264da13699 should not trigger any rule stages: - stage: input: dest_addr: "127.0.0.1" port: 80 headers: Host: "localhost" User-Agent: "ModSecurity CRS 3 Tests" Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Accept-Encoding: gzip,deflate Accept-Language: en-us,en;q=0.5 Content-Type: "application/x-www-form-urlencoded" method: POST version: HTTP/1.0 data: "test264da13699=This+payload+should+cause+no+errors" output: no_log_contains: "test264da13699"

FTW no doubts is the way to go for complex testing, also when testing and comparing changes between rule sets is important or when tests must be repeated over and over. The generation of positive rules (FTW – base_positive_rules script) is not a difficult task and can be scripted.
The base_positive_rules python script will generate rules for every provided payload to test different elements of a request such as header names, header values, cookie names, cookie values, argument names, argument values, JSON argument names, JSON argument values, XML element names, XML element values, XML element property names, XML element property values and some body processor and content type bypasses.

The script will generate tests associated for the items marked in red, the items in green require customization with production like data.

Final words

I found very useful to detect rule errors and regression issues to have FTW running before and after every change and write a set of positive tests for every rule and a set for negative tests to ensure the WAF configuration changes will not break my applications.