Mailing List ecs-isp@2rosenthals.com Archived Message #314

Fra: "Massimo S." <ecs-isp@2rosenthals.com> Full Headers
Undecoded message
Emne: Re: [eCS-ISP] unwanted bots
Dato: Tue, 20 Apr 2021 10:33:19 +0200
Til: eCS ISP Mailing List <ecs-isp@2rosenthals.com>



Il 19/04/2021 16:58, Steven Levine ha scritto:
In <list-1821020@2rosenthals.com>, on 04/19/21
    at 11:18 AM, "Massimo S." <ecs-isp@2rosenthals.com> said:

Hi Massimo,

thanks, but it seems is not working

Why would you expect it to?

Ian, FWIW, Massimo is trying to do URL matching, whic does allow
wildcards.

Massimo, that said, you are trying to match a URL to a User-Agent name
which is almost never going to work.

What you need to do is match on the bot name in the User-Agent header.

I recommend you start with a logging rule so you can see what the packet
actually contains.  Baring typos, you want:

Log-mj12_2-URLs
    Comment = "Log HTTP requests for MJ bot",
    Destination-Port = "80 443",
    Log-Control = Enabled,
    Log-Mask = "date time severity message rule source resolved_source prot
packet_data",
    Log-Severity = Major,
    Hex-String = "<nocase>12bot",
    Depth = 200
    Rule-Action = Log,
    Direction = Incoming

Once you understand the packet you want to match on, you can convert this
into a Deny Rule-Action, tuning the Depth and Log-Mask to optimized the
filter performance.

FWIW, if this were my problem, I would use an observe rule to blacklist
the source.

Massimo, is the really the logged URL:

   /referenze.html/.../referenze.html/

It does not look valid to me.  If you edited it, don't do that in the
future.  I am only willing to try so hard to help you.

Steven

MJ12bot is over with the string and depth you suggested
thanks a lot

i'me trying to use log to understand that stuff

i now have this (similar) issue:

[2021/04/19][00:07:07][05:Major][MSG:][URL-Filter-bot_petalsearch23][SRC:114.119.147.152][SRC:petalbot-114-119-147-152.petalsearch.com][tcp]
 0000:  474554202F746573 - 692E7068703F7465     GET /tesi.php?te
 0010:  73694F726465723D - 536F727465725F4C     siOrder=Sorter_L
 0020:  6976656C6C6F4C61 - 7572656126746573     ivelloLaurea&tes
 0030:  694469723D415343 - 2674657369506167     iDir=ASC&tesiPag
 0040:  653D382048545450 - 2F312E310D0A4163     e=8 HTTP/1.1..Ac
 0050:  636570743A207465 - 78742F68746D6C2C     cept: text/html,
 0060:  6170706C69636174 - 696F6E2F7868746D     application/xhtm
 0070:  6C2B786D6C2C6170 - 706C69636174696F     l+xml,applicatio
 0080:  6E2F786D6C3B713D - 302E392C2A2F2A3B     n/xml;q=0.9,*/*;
 0090:  713D302E380D0A55 - 7365722D4167656E     q=0.8..User-Agen
 00A0:  743A204D6F7A696C - 6C612F352E302028     t: Mozilla/5.0 (
 00B0:  4C696E75783B2041 - 6E64726F69642037     Linux; Android 7
 00C0:  2E303B2920417070 - 6C655765624B6974     .0;) AppleWebKit
 00D0:  2F3533372E333620 - 284B48544D4C2C20     /537.36 (KHTML,
 00E0:  6C696B6520476563 - 6B6F29204D6F6269     like Gecko) Mobi
 00F0:  6C65205361666172 - 692F3533372E3336     le Safari/537.36
 0100:  2028636F6D706174 - 69626C653B205065      (compatible; Pe
 0110:  74616C426F743B2B - 68747470733A2F2F     talBot;+https://
 0120:  7765626D61737465 - 722E706574616C73     webmaster.petals
 0130:  65617263682E636F - 6D2F736974652F70     earch.com/site/p
 0140:  6574616C626F7429 - 0D0A416363657074     etalbot)..Accept
 0150:  2D4C616E67756167 - 653A20656E2C7A68     -Language: en,zh
 0160:  3B713D302E310D0A - 4163636570742D45     ;q=0.1..Accept-E
 0170:  6E636F64696E673A - 20677A69702C6465     ncoding: gzip,de
 0180:  666C6174650D0A48 - 6F73743A20777777     flate..Host: www
 0190:  2E65636F6469636F - 6D2E6E65740D0A43     .tecnicom.com..C
 01A0:  616368652D436F6E - 74726F6C3A206D61     ache-Control: ma
 01B0:  782D6167653D300D - 0A436F6E6E656374     x-age=0..Connect
 01C0:  696F6E3A206B6565 - 702D616C6976650D     ion: keep-alive.
 01D0:  0A0D0A                                  ...


this rule filter something:

URL-Filter-bot-petal-string1
Comment = "Deny HTTP requests for petalsearch-bot 19-4-2021",
Destination-Port = "80 443",
Protocol = TCP,
Log-Control = Enabled,
Log-Mask = "date time severity message rule source resolved_source prot",
Log-Size = 9990,
Log-Severity = Major,
Depth = 200,
Hex-String = "<nocase>petalbot",
                Rule-Action = Deny,
Direction = Incoming

these 2 nothing at all:

URL-Filter-bot-petal-string2
Comment = "Deny HTTP requests for petalsearch-bot2 19-4-2021",
Destination-Port = "80 443",
Protocol = TCP,
Log-Control = Enabled,
Log-Mask = "date time severity message rule source resolved_source prot",
Log-Size = 9990,
Log-Severity = Major,
Depth = 200,
Hex-String = "<nocase>petalsearch",
                Rule-Action = Deny,
Direction = Incoming

URL-Filter-bot-petal-string3
Comment = "Deny HTTP requests for petalsearch-bot2 19-4-2021",
Destination-Port = "80 443",
Protocol = TCP,
Log-Control = Enabled,
Log-Mask = "date time severity message rule source resolved_source prot",
Log-Size = 9990,
Log-Severity = Major,
Depth = 200,
Offset = URL,
Hex-String = "*PetalBot*",
                Rule-Action = Deny,
Direction = Incoming

thanks

massimo

Abboner: Feed, Digest, Index.
Stopp abbonement
E-post til ListMaster