Web application firewalls (WAF) are being adopted by many as the first line of defense for their applications and platforms, that is good no doubt but there are several questions that need to be made first before jumping into the hype, what is it and what can it do to protect my website and what are some side effects which are discussed in this article.

WAFs

A WAF is either a hardware appliance or software that can interpret and inspect the web application requests and responses looking for patterns. Generally, WAF’s are used to perform negative filtering, so it usually will search for anomalies or known bad patterns and perform actions that can manipulate the response or the request to reduce the risk of having the application affected adversely because of that specific content.

The effectiveness in most cases depends on the configuration, some vendors have nice and easy to use graphical interfaces while others use text files to get their configuration, that means that the protection level on a given web application protected by different WAF can be different depending on the configuration even if the application and the WAF use the same version.

ModSecurity and the OWASP CRS

ModSecurity is an open source WAF version that is used in this project, by itself ModSecruity without any rule will not protect against web attacks. There are multiple free or commercial rule sets available for ModSecurity, in many Linux distributions the OWASP core ruleset (CRS) is installed by default.

The CRS is a generic attack detection open source rule for ModSecurity and compatible WAF. Some of the common attack categories that the CRS protect against are:

• Malicious automation
  • security scanners, robots
• Leakage
  • source code
    • php, java
  • metadata, error
• Web attacks
  • Injection
    • command, file, code, SQL
• Web application defects
  • session fixation, session hijacking

False positives, false negatives

Every WAF have false positives and false negatives, the first refers to events generated often due to a string that matches one of the patterns in the ruleset considered to be malicious and the second refers to events that are malicious or a true match but that the ruleset configuration or the WAF engine missed to raise an alert or action. Both can be problematic and cause service disruptions or hide attacks.

The balance between false positives and false negatives depends on the type and content of web application. Some WAF will lean towards easier implementation and fewer disruptions which often translates to many false negatives that would allow an attacker to hack a website and download entire databases. Some others would lean to security and will cause many false positives which often translates to broken content and lots of forbidden error pages.

The OWASP CRS may be configured to different sensitivities depending on the user risk appetite, there settings as anomaly scores, anomaly score thresholds and paranoia levels that can affect the behaviour and turn the WAF from a very relaxed setting with low anomaly scores, high thresholds and paranoia level 1 that will ignore many known and unknown attacks or have the opposite behaviour by using high anomaly scores, low anomaly score thresholds and paranoia level 4 that will block a lot of the known and many unknown web attacks but will also block innocent requests.

Negative filtering

Negative filtering can be very powerful to prevent known attacks and to some extent unknown attacks, but it is generic so it is expected to have false positives and that several generic rules may not be compatible with the application it is protecting or with the language the website uses.

One common false positive is triggered due to the use of quotation marks on forms, it is normal in English, French, and other lenguages to have them in last names as “O’Hara”, or phrases as “je t’aime”.

Other common false positives come from sentences people write like “I’m at home between 9 and 12”, sometimes we can avoid those if the application properly escapes and encode the user inputs but WAF’s also decode and parse the encodings which will trigger the alert.

Pseudo random strings as session identifiers or base64 encoded contents can trigger some alerts from time to time if a string like xOr or mSdB matches a rule related to SQL injection attacks.

Positive filtering

Positive filtering on the other side is meant to validate the possible type of fields, data types, value ranges, length, etc of the elements known to the application to allow those requests or responses that match those conditions and reject everything else. It is a very strong protection and very hard to bypass depending on the application. If the application is very closed and have defined element values it can be easy to write a rule set to accept only valid requests but if the is a lot of user inputs as in chats or forums it will be a challenge to protect all of the elements so using a combination of both positive and negative filtering will produce the best result.

When using the both filtering methods a rule from the CRS can be whitelisted for the elements we are already validating with little risk of accepting malicious requests and at the same time relax the validation of some input fields as the CRS will check for evil requests.

Having a clean setup free of false positives and with low false negatives will take some effort, some small applications and common CMS that have some rules built in the CRS will be easy to setup while others may take a huge work.

WAF logs monitoring

Once the WAF rule set is tuned and ready for production it should be continuously monitored to check for new false positives and especially for attacks, even the most strict configurations may be vulnerable to bypass or application logic issues, also logs can give an early warning when someone is trying to hack into the application so corrective and preventive actions can be taken.

A WAF may generate millions of events a day depending of the number on requests a website receives and the configuration the amount of log can be much more than the total amount of traffic so storage and system capacity must be taken into consideration, the logs can be stored locally or shipped remotely to a central collection facility.

Modsecurity does not have a log management application to aggregate the events from multiple systems but supports shipping the logs with mlogc to other applications like AuditConsole (unmaintained, outdated, not prod level) or pipe the logs to syslog and ingest later in splunk (free up to 500mb, very small projects, modsec content in splunk focus on monitoring availability not security) or some other siem or pipe as discussed by Trustwave (not very well documented) and my personal favorite is using filebeats to ship the logs to ELK (do it yourself, scalable, cool) logstash use java meaning more RAM and not so safe to be facing the internet but filebeats is very fast and can do TLS mutual authentication to connect to a secure logstash, and finally it is also possible to use the wild west style using the CLI (does not scale well).

Modsecurity have two major log files, the regular log which in the case of Apache is the error log and the audit log which can use native or JSON formats, the JSON format is much better for computers and programmatic manipulation but uses more space due to the added tags, while the native format is better for humans but a pain to parse.

Parsers for auditlog (native format):

• logstash https://github.com/bitsofinfo/logstash-modsecurity
• fluent https://github.com/bitsofinfo/fluentd-modsecurity

The logs are there to be monitored and act on the events, while doing that many application and server errors can be identified and diagnosed even before the client complain or even notice there is something wrong and logs must be configured so that a transaction can be traced back through the different log files and servers (adding the unique_id at the perimeter to a header and into all logs can do the trick). I will write a future post on this.

Sample logs

Here is a sample of a basic request that will trigger an SQL injection alert and the corresponding logs.

• Request:

curl "127.0.0.1/a?b='%20or%201=1;--"

Apache error log (log):

[Tue Aug 29 16:20:52.183189 2017] [:error] [pid 10567] [client 127.0.0.1:48620] [client 127.0.0.1] ModSecurity: Warning. detected SQLi using libinjection with fingerprint 's&1;c' [file "/usr/share/modsecurity-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "68"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: s&1;c found within ARGS:b: ' or 1=1;--"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "127.0.0.1"] [uri "/a"] [unique_id "WaV4RH8AAQEAAClH1OgAAAAB"]

• Audit log (Native):

--23e5884d-A--
[29/Aug/2017:16:20:52 +0200] WaV4RH8AAQEAAClH1OgAAAAB 127.0.0.1 48620 127.0.0.1 80
--23e5884d-B--
GET /a?b='%20or%201=1;-- HTTP/1.1
Host: 127.0.0.1
User-Agent: curl/7.52.1
Accept: */*

--23e5884d-F--
HTTP/1.1 403 Forbidden
Content-Length: 285
Content-Type: text/html; charset=iso-8859-1

--23e5884d-E--
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /a
on this server.<br />
</p>
<hr>
<address>Apache/2.4.27 (Debian) Server at 127.0.0.1 Port 80</address>
</body></html>

--23e5884d-H--
Message: Warning. detected SQLi using libinjection with fingerprint 's&1;c' [file "/usr/share/modsecurity-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "68"] [id "942100"] [rev "1"] [msg "SQL Injection Attack Detected via libinjection"] [data "Matched Data: s&1;c found within ARGS:b: ' or 1=1;--"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"]

[.... REMOVED MULTIPLE EVENTS RELATED TO SAME ATTACK TO SAVE SPACE ....]

....
Action: Intercepted (phase 2)
Stopwatch: 1504016452181543 2026 (- - -)
Stopwatch2: 1504016452181543 2026; combined=1390, p1=424, p2=888, p3=0, p4=0, p5=78, sr=35, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.1 (http://www.modsecurity.org/); OWASP_CRS/3.0.0.
Server: Apache/2.4.27 (Debian)
Engine-Mode: "ENABLED"

--23e5884d-Z--

• Audit log (JSON):

{
    "audit_data": {
        "action": {
            "intercepted": true,
            "message": "Operator GE matched 5 at TX:anomaly_score.",
            "phase": 2
        },
        "engine_mode": "ENABLED",
        "error_messages": [
            "[file \"apache2_util.c\"] [line 273] [level 3] [client 127.0.0.1] ModSecurity: Warning. detected SQLi using libinjection with fingerprint 's&1;c' [file \"/usr/share/modsecurity-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf\"] [line \"68\"] [id \"942100\"] [rev \"1\"] [msg \"SQL Injection Attack Detected via libinjection\"] [data \"Matched Data: s&1;c found within ARGS:b: ' or 1=1;--\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"1\"] [accuracy \"8\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-sqli\"] [tag \"OWASP_CRS/WEB_ATTACK/SQL_INJECTION\"] [tag \"WASCTC/WASC-19\"] [tag \"OWASP_TOP_10/A1\"] [tag \"OWASP_AppSensor/CIE1\"] [tag \"PCI/6.5.2\"] [hostname \"127.0.0.1\"] [uri \"/a\"] [unique_id \"WaWSHX8AAQEAACp1Iz8AAAAD\"]",
            [.... REMOVED MULTIPLE EVENTS RELATED TO SAME ATTACK TO SAVE SPACE ....]
        ],
        "messages": [
            "Warning. detected SQLi using libinjection with fingerprint 's&1;c' [file \"/usr/share/modsecurity-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf\"] [line \"68\"] [id \"942100\"] [rev \"1\"] [msg \"SQL Injection Attack Detected via libinjection\"] [data \"Matched Data: s&1;c found within ARGS:b: ' or 1=1;--\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"1\"] [accuracy \"8\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-sqli\"] [tag \"OWASP_CRS/WEB_ATTACK/SQL_INJECTION\"] [tag \"WASCTC/WASC-19\"] [tag \"OWASP_TOP_10/A1\"] [tag \"OWASP_AppSensor/CIE1\"] [tag \"PCI/6.5.2\"]",
            [.... REMOVED MULTIPLE EVENTS RELATED TO SAME ATTACK TO SAVE SPACE ....]
        ],
        "producer": [
            "ModSecurity for Apache/2.9.1 (http://www.modsecurity.org/)",
            "OWASP_CRS/3.0.0"
        ],
        "response_body_dechunked": true,
        "server": "Apache/2.4.27 (Debian)",
        "stopwatch": {
            "gc": 0,
            "l": 0,
            "p1": 202,
            "p2": 2146,
            "p3": 0,
            "p4": 0,
            "p5": 98,
            "sr": 11,
            "sw": 0
        }
    },
    "request": {
        "headers": {
            "Accept": "*/*",
            "Host": "127.0.0.1",
            "User-Agent": "curl/7.52.1"
        },
        "request_line": "GET /a?b='%20or%201=1;-- HTTP/1.1"
    },
    "response": {
        "body": "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p>You don't have permission to access /a\non this server.<br />\n</p>\n<hr>\n<address>Apache/2.4.27 (Debian) Server at 127.0.0.1 Port 80</address>\n</body></html>\n",
        "headers": {
            "Content-Length": "285",
            "Content-Type": "text/html; charset=iso-8859-1"
        },
        "protocol": "HTTP/1.1",
        "status": 403
    },
    "transaction": {
        "local_address": "127.0.0.1",
        "local_port": 80,
        "remote_address": "127.0.0.1",
        "remote_port": 50082,
        "time": "29/Aug/2017:18:11:09 +0200",
        "transaction_id": "WaWSHX8AAQEAACp1Iz8AAAAD"
    }
}

Conclusion

A WAF is a great addition to security but as any other control needs attention and resources (human, time, services, etc), failing to monitor and tune a WAF will cause service disruptions, bad user experience and a false sense of security but also it is a valuable place to identify intrusion attempts and behavior anomalies.