Bots generate roughly half of all Internet traffic. Some are clearly malicious (password crackers, vulnerability scanners, application-level/L7 DDoS), and others are merely unwanted (web scrappers, carting, appointment etc) bots. Traditional challenges (CAPTCHAs, JavaScript checks) degrade user experience, and some vendors are deprecating them. An alternative is traffic and behavior analytics, which is much more sophisticated, but can be far more effective.

Complicating matters, there are cloud services not only helping to bypass challenges, but also mimic browsers and human behavior. It’s tough to build a solid protection system withstand such proxy services.

In this talk, we present WebShield, a small open-source Python daemon that analyzes Tempesta FW, an open-source web accelerator, access logs and dynamically classifies and blocks bad bots.

You’ll learn:

• Which bots are easy to detect (e.g., L7 DDoS, password crackers) and which are harder (e.g., scrapers, carting/checkout abuse).
• Why your secret weapon is your users’ access patterns and traffic statistics—and how to use them.
• How to efficiently deliver web-server access logs to an analytics database (e.g., ClickHouse).
• Traffic fingerprints (JA3, JA4, p0f): how they’re computed and their applicability for machine learning.
• Tempesta fingerprints: lightweight fingerprints designed for automatic web clients clustering.
• How to correlate multiple traffic characteristics and catch lazy bot developers.
• Baseline models for access-log analytics and how to validate them.
• How to block large botnets without blocking half the Internet.
• Scoring, behavioral analysis, and other advanced techniques are not yet implemented.