DOS filter rules

The denial of service filter works on a basic premise of two rules

1. Dynamic requests
2. Static requests

MageStack classifies the type of request using a few rules; based on file extension, URI and URL. The rule is applied before fetching content, so mime-type is not considered.

Thresholds

A threshold exists for each type of request, and if this amount is exceeded in a given window of time - then a soft block is given with a (customisable) notice page, if the requests continue on the block page - then a hard block is issued. The block is lifted almost immediately after the requests subside.

The default thresholds are:

Type	Warning	Soft Limit	Hard Limit	Measurement Window	Action
Static	750	950	3000	5 seconds	503 Header
Dynamic	25	35	100	5 seconds	TCP Block

These can be fine tuned by a support member to suit your traffic load.

Identifying requests

You can evaluate how MageStack interprets the request by checking the Debug Headers, specifically MageStack-Tag.

It is very important that the requests are accurately tagged, as it could be possible that static requests are incorrectly being marked as dynamic - which could cause a false positive.

If you have a URI which appears to be dynamic content, but is actually static, eg. A resized 'on-the-fly' image - then a rule can be put in place to mark that URI as a static content. At present, only our support team can make adjustments to these rules.

Search Engines

Some search engines have quite aggressive crawl rates that can trigger the DOS filter. So to protect you against any SEO penalty, the major search engines (Google|Yahoo|Bing|Yandex) are all automatically white-listed based on their reverse and forward DNS values when they reach the warning level.

Eg.

Sep 16 15:16:18 lb1 dos-filter: dynamic - warn (30) - 66.249.73.221 - Bot: googlebot (added to bot-whitelist)

False positives

The default thresholds are set very conservatively - and are considered extremely safe both for normal visitors and search engines alike.

Wrongly tagged

If you have requests that are being incorrectly tagged, then a false positive can occur. Ensuring that static content is marked as static is very important given the much higher potential for number of requests. On a single page, there are usually <5 dynamic requests, but >30 static requests; so if any static request is incorrectly marked as dynamic, it can quickly cause a ban.

NAT

If you also have multiple people behind a single IP accessing your website, eg. a warehouse/office location, it can also be possible to trigger a false positive.

Whitelisting

It is possible to whitelist via a few means,

1. Source IP address
2. Request URI
3. User agent

If you have a static IP address, a permanent addition to the whitelist can be added.

This can be done via the DOS Filter management page, we have a Guide available.

If you have a specific URI that needs to either be marked as static, or bypassed from the filter completely, you should contact the support team to make this adjustment.

If you are using a crawl utility that has its own user-agent, you should contact the support team to add the user-agent to the whitelist.

You can verify if an entry is whitelisted by looking at the PHP $_SERVER variable for any of the following headers

MageStack-Whitelisted-Http-Host
MageStack-Whitelisted-Uri
MageStack-Whitelisted-Ip
MageStack-Whitelisted-User-Agent

Viewing DOS Hits in Kibana

You can identify what IPs or requests are triggering DOS warnings or blocks by utilising Kibana. View the load balancer (lb) log panel and use the following search string

 app:dos-filter AND ( dos_block_type:hard-block or dos_block_type:soft-block )

This will filter the results to only display soft and hard blocks (ie. those that display the error message or block the user entirely). You can then filter further by source IP, or by request to narrow down your results.