The denial of service filter works on a basic premise of two rules
- Dynamic requests
- Static requests
MageStack classifies the type of request using a few rules; based on file extension, URI and URL. The rule is applied before fetching content, so mime-type is not considered.
A threshold exists for each type of request, and if this amount is exceeded in a given window of time - then a soft block is given with a (customisable) notice page, if the requests continue on the block page - then a hard block is issued. The block is lifted almost immediately after the requests subside.
The default thresholds are:
|Type||Warning||Soft Limit||Hard Limit||Measurement Window||Action|
|Static||750||950||3000||5 seconds||503 Header|
|Dynamic||25||35||100||5 seconds||TCP Block|
These can be fine tuned by a support member to suit your traffic load.
You can evaluate how MageStack interprets the request by checking the Debug Headers, specifically
It is very important that the requests are accurately tagged, as it could be possible that static requests are incorrectly being marked as dynamic - which could cause a false positive.
If you have a URI which appears to be dynamic content, but is actually static, eg. A resized 'on-the-fly' image - then a rule can be put in place to mark that URI as a static content. At present, only our support team can make adjustments to these rules.
Some search engines have quite aggressive crawl rates that can trigger the DOS filter. So to protect you against any SEO penalty, the major search engines (Google|Yahoo|Bing|Yandex) are all automatically white-listed based on their reverse and forward DNS values when they reach the warning level.
Sep 16 15:16:18 lb1 dos-filter: dynamic - warn (30) - 126.96.36.199 - Bot: googlebot (added to bot-whitelist)
The default thresholds are set very conservatively - and are considered extremely safe both for normal visitors and search engines alike.
If you have requests that are being incorrectly tagged, then a false positive can occur. Ensuring that static content is marked as static is very important given the much higher potential for number of requests. On a single page, there are usually <5 dynamic requests, but >30 static requests; so if any static request is incorrectly marked as dynamic, it can quickly cause a ban.
If you also have multiple people behind a single IP accessing your website, eg. a warehouse/office location, it can also be possible to trigger a false positive.
It is possible to whitelist via a few means,
- Source IP address
- Request URI
- User agent
If you have a static IP address, a permanent addition to the whitelist can be added. Contact the support team to make this adjustment for you.
If you have a specific URI that needs to either be marked as static, or bypassed from the filter completely, you should contact the support team to make this adjustment.
If you are using a crawl utility that has its own user-agent, you should contact the support team to add the user-agent to the whitelist.
You can verify if an entry is whitelisted by looking at the PHP
$_SERVER variable for any of the following headers
MageStack-Whitelisted-Http-Host MageStack-Whitelisted-Uri MageStack-Whitelisted-Ip MageStack-Whitelisted-User-Agent
Viewing DOS Hits in Kibana
You can identify what IPs or requests are triggering DOS warnings or blocks by utilising Kibana. View the load balancer (
lb) log panel and use the following search string
app:dos-filter AND ( dos_block_type:hard-block or dos_block_type:soft-block )
This will filter the results to only display soft and hard blocks (ie. those that display the error message or block the user entirely). You can then filter further by source IP, or by request to narrow down your results.