Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extend user agent filter by "python", "wget" and "monitor" #151

Merged
merged 2 commits into from
May 17, 2020

Conversation

stklcode
Copy link
Contributor

Fixes #149

Back in #125 the user agent filter for bot detection has been reworked.
As @KittMedia already mentioned in #125 (comment) the wget identifier has been forgotten while curl is present. Never taken care of that...

Also commonly used monitoring tools like Nagios/Icinga or PRTG are not excluded. They are not "bots" in terms of crawlers/spiders, but also non-human requests which are not meant to be counted. Both contain the word "monitor(ing)" while afaik no client webbrowser contains this.

I've also added python, because it's quite common for analysis scripts and bots as well.

Example UA strings that pass the 1.7 filter, but shouldn't:

Wget/1.20.3 (linux-gnu)
Python-urllib/3.8
python-requests/2.22.0
check_http/v2.2 (monitoring-plugins 2.2)
Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)

The latter also passed the <=1.6 filter, because it contains "Windows"...

@stklcode stklcode added this to the 1.7.1 milestone May 12, 2020
@stklcode stklcode force-pushed the fix/extend-ua-filter branch from 01fe7e0 to dd8318f Compare May 16, 2020 08:07
Copy link
Member

@patrickrobrecht patrickrobrecht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@patrickrobrecht patrickrobrecht merged commit d0adbe7 into master May 17, 2020
@patrickrobrecht patrickrobrecht deleted the fix/extend-ua-filter branch May 17, 2020 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Heavily more visitors since v1.7.0
2 participants