Bots targeting your web server have become more advanced, bypassing traditional security methods such as rate-limiting and static IP blocking. These malicious actors can overload your server by using distributed IP addresses and sophisticated traffic patterns, making detection and mitigation a challenge. In this article, we will demonstrate how to use Fail2ban with CrowdSec to dynamically detect and block abusive IPs without relying on a massive static blocklist. This approach ensures your server remains secure and operational without overloading its resources.

Why Standard Mitigation Techniques Are Insufficient

Traditional measures such as rate-limiting, static blocklists, or regex-based detection often fall short when dealing with modern bots. Here’s why:

  • Dynamic IPs: Bots often rotate their IP addresses, making static blocklists ineffective.
  • Low-Rate Attacks: By limiting requests to a few per second and using distributed IPs, bots evade rate-limiting measures.
  • Pattern Evasion: Attackers use encoded or randomized requests to bypass basic regex filters.
  • Large Blocklists: Loading millions of IPs into your firewall is inefficient and can degrade server performance.

By combining Fail2ban with CrowdSec, we can create a lightweight, dynamic solution that checks incoming IPs against a global threat intelligence network in real-time and blocks only those actually hitting your server.

Solution Overview

We’ll configure Fail2ban to monitor your server logs and use a custom action to query CrowdSec’s database for suspicious IPs. If an IP is deemed malicious, it will be dynamically blocked using a lightweight approach. This ensures only relevant IPs are blocked, minimizing unnecessary resource usage.

What You’ll Need

  • A server running Debian/Ubuntu or RHEL/CentOS.
  • Web server logs (e.g., Apache or Nginx).
  • Fail2ban installed.
  • CrowdSec installed and configured.
  • Basic familiarity with Linux commands and configuration files.

Step 1: Install Fail2ban and CrowdSec

Fail2ban and CrowdSec are the core components of this solution. First, let’s install them on your server.

Installing Fail2ban

On Debian/Ubuntu:


sudo apt update
sudo apt install fail2ban -y

On RHEL/CentOS:


sudo dnf install epel-release -y
sudo dnf install fail2ban -y

Installing CrowdSec

CrowdSec is a collaborative threat intelligence platform that provides updated lists of malicious IPs.

On Debian/Ubuntu:


curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | sudo bash
sudo apt install crowdsec -y

On RHEL/CentOS:


curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.rpm.sh | sudo bash
sudo dnf install crowdsec -y

After installation, start and enable the CrowdSec service:


sudo systemctl enable --now crowdsec

Step 2: Configure CrowdSec for Threat Intelligence

CrowdSec uses collections to manage threat detection. Install the necessary collection for detecting web-based attacks:


sudo cscli hub update
sudo cscli collections install crowdsecurity/http-bad-behaviors
sudo systemctl restart crowdsec

This collection ensures that CrowdSec can identify IPs associated with malicious web activity. You can add more collections as needed.

Setting Up the Bouncer

The CrowdSec bouncer integrates with your firewall or Fail2ban to block malicious IPs.


sudo cscli bouncers add fail2ban-bouncer

Step 3: Create a Custom Fail2ban Action

To integrate Fail2ban with CrowdSec, we’ll create a custom action that queries CrowdSec’s API for each detected IP.

Create the Action Script

Save the following configuration as /etc/fail2ban/action.d/crowdsec-check.conf:


[Definition]
actionstart =
actionstop =
actionban = curl -s -G "http://127.0.0.1:8080/v1/decisions?ip=<ip>" | jq '.new'
actionunban =

This configuration sends the detected IP to CrowdSec’s local API for analysis and retrieves a response indicating whether the IP is malicious.

Step 4: Define a Fail2ban Filter

Fail2ban filters specify which log patterns to monitor. For example, to detect malicious input in Apache logs, create the following filter file:

Example Apache Filter

Save the filter as /etc/fail2ban/filter.d/apache-malicious.conf:


[Definition]
failregex = ^<HOST> - .* "(GET|POST).*%3Cscript.*" .*$
ignoreregex =

This filter detects requests with encoded malicious input (e.g., %3Cscript).

Step 5: Configure a Fail2ban Jail

Define a Fail2ban jail that uses your custom filter and action. Create the jail configuration file:

Add the Jail Configuration

Save the following as /etc/fail2ban/jail.d/apache-malicious.local:

[apache-malicious]

enabled = true port = http,https filter = apache-malicious logpath = /var/log/apache2/access.log bantime = 3600 findtime = 300 maxretry = 1 action = crowdsec-check[name=apache, ip=<ip>]

This configuration enables the jail, monitors Apache logs, and uses the custom CrowdSec action to block malicious IPs dynamically.

Step 6: Restart Services

Restart Fail2ban and CrowdSec to apply your configurations:


sudo systemctl restart fail2ban
sudo systemctl restart crowdsec

Step 7: Test and Monitor

Monitor Fail2ban Logs

Check the status of the apache-malicious jail to ensure it is working correctly:


sudo fail2ban-client status apache-malicious

Simulate a Malicious Request

To verify that the system blocks malicious requests, simulate one using curl:


curl -A "malicious-bot" "http://yourserver.com/%3Cscript%3Ealert(1)%3C/script%3E"

The IP should be flagged and blocked after detection.

Optional: GeoIP Filtering

If you want to block requests from specific countries, you can integrate GeoIP filtering with Fail2ban.

Install GeoIP Dependencies

Install the necessary packages:


sudo apt install xtables-addons-common geoip-database -y

Example GeoIP Rule

Add the following to the jail configuration to block non-US traffic:


actionban = iptables -A INPUT -m geoip ! --src-cc US -j DROP

Conclusion

By integrating Fail2ban with CrowdSec, you can dynamically detect and block malicious bot traffic without overloading your server with massive blocklists. This lightweight solution is highly effective against sophisticated attacks and ensures your server remains secure and responsive.

For additional assistance, feel free to contact us.