From: "Massimo S." <ecs-isp@2rosenthals.com>
Reply-To: ml@ecomstation.it
Subject: Re: [eCS-ISP] unwanted bots
To: eCS ISP Mailing List <ecs-isp@2rosenthals.com>
References: <list-1821467@2rosenthals.com>
Organization: eComStation dot it
Message-ID: <00b0f36c-c0c1-aa53-a5ad-4a3b0311800c@ecomstation.it>
Date: Tue, 20 Apr 2021 10:33:19 +0200
User-Agent: Mozilla/5.0 (OS/2; U; Warp 4.5; it-IT; rv:1.7.13) Gecko/20060424
 Thunderbird/1.0.8 Mnenhy/0.7.4.0
MIME-Version: 1.0
In-Reply-To: <list-1821467@2rosenthals.com>
Content-Type: text/plain; charset=iso-8859-15; format=flowed
Content-Language: it-IT
Content-Transfer-Encoding: 7bit



Il 19/04/2021 16:58, Steven Levine ha scritto:
> In <list-1821020@2rosenthals.com>, on 04/19/21
>     at 11:18 AM, "Massimo S." <ecs-isp@2rosenthals.com> said:
> 
> Hi Massimo,
> 
>> thanks, but it seems is not working
> 
> Why would you expect it to?
> 
> Ian, FWIW, Massimo is trying to do URL matching, whic does allow
> wildcards.
> 
> Massimo, that said, you are trying to match a URL to a User-Agent name
> which is almost never going to work.
> 
> What you need to do is match on the bot name in the User-Agent header.
> 
> I recommend you start with a logging rule so you can see what the packet
> actually contains.  Baring typos, you want:
> 
> Log-mj12_2-URLs
>     Comment = "Log HTTP requests for MJ bot",
>     Destination-Port = "80 443",
>     Log-Control = Enabled,
>     Log-Mask = "date time severity message rule source resolved_source prot
> packet_data",
>     Log-Severity = Major,
>     Hex-String = "<nocase>12bot",
>     Depth = 200
>     Rule-Action = Log,
>     Direction = Incoming
> 
> Once you understand the packet you want to match on, you can convert this
> into a Deny Rule-Action, tuning the Depth and Log-Mask to optimized the
> filter performance.
> 
> FWIW, if this were my problem, I would use an observe rule to blacklist
> the source.
> 
> Massimo, is the really the logged URL:
> 
>    /referenze.html/.../referenze.html/
> 
> It does not look valid to me.  If you edited it, don't do that in the
> future.  I am only willing to try so hard to help you.
> 
> Steven

MJ12bot is over with the string and depth you suggested
thanks a lot

i'me trying to use log to understand that stuff

i now have this (similar) issue:

[2021/04/19][00:07:07][05:Major][MSG:][URL-Filter-bot_petalsearch23][SRC:114.119.147.152][SRC:petalbot-114-119-147-152.petalsearch.com][tcp]
  0000:  474554202F746573 - 692E7068703F7465     GET /tesi.php?te
  0010:  73694F726465723D - 536F727465725F4C     siOrder=Sorter_L
  0020:  6976656C6C6F4C61 - 7572656126746573     ivelloLaurea&tes
  0030:  694469723D415343 - 2674657369506167     iDir=ASC&tesiPag
  0040:  653D382048545450 - 2F312E310D0A4163     e=8 HTTP/1.1..Ac
  0050:  636570743A207465 - 78742F68746D6C2C     cept: text/html,
  0060:  6170706C69636174 - 696F6E2F7868746D     application/xhtm
  0070:  6C2B786D6C2C6170 - 706C69636174696F     l+xml,applicatio
  0080:  6E2F786D6C3B713D - 302E392C2A2F2A3B     n/xml;q=0.9,*/*;
  0090:  713D302E380D0A55 - 7365722D4167656E     q=0.8..User-Agen
  00A0:  743A204D6F7A696C - 6C612F352E302028     t: Mozilla/5.0 (
  00B0:  4C696E75783B2041 - 6E64726F69642037     Linux; Android 7
  00C0:  2E303B2920417070 - 6C655765624B6974     .0;) AppleWebKit
  00D0:  2F3533372E333620 - 284B48544D4C2C20     /537.36 (KHTML,
  00E0:  6C696B6520476563 - 6B6F29204D6F6269     like Gecko) Mobi
  00F0:  6C65205361666172 - 692F3533372E3336     le Safari/537.36
  0100:  2028636F6D706174 - 69626C653B205065      (compatible; Pe
  0110:  74616C426F743B2B - 68747470733A2F2F     talBot;+https://
  0120:  7765626D61737465 - 722E706574616C73     webmaster.petals
  0130:  65617263682E636F - 6D2F736974652F70     earch.com/site/p
  0140:  6574616C626F7429 - 0D0A416363657074     etalbot)..Accept
  0150:  2D4C616E67756167 - 653A20656E2C7A68     -Language: en,zh
  0160:  3B713D302E310D0A - 4163636570742D45     ;q=0.1..Accept-E
  0170:  6E636F64696E673A - 20677A69702C6465     ncoding: gzip,de
  0180:  666C6174650D0A48 - 6F73743A20777777     flate..Host: www
  0190:  2E65636F6469636F - 6D2E6E65740D0A43     .tecnicom.com..C
  01A0:  616368652D436F6E - 74726F6C3A206D61     ache-Control: ma
  01B0:  782D6167653D300D - 0A436F6E6E656374     x-age=0..Connect
  01C0:  696F6E3A206B6565 - 702D616C6976650D     ion: keep-alive.
  01D0:  0A0D0A                                  ...


this rule filter something:

URL-Filter-bot-petal-string1
		Comment = "Deny HTTP requests for petalsearch-bot 19-4-2021",
		Destination-Port = "80 443",
		Protocol = TCP,
		Log-Control = Enabled,
		Log-Mask = "date time severity message rule source resolved_source prot",
		Log-Size = 9990,
		Log-Severity = Major,
		Depth = 200,
		Hex-String = "<nocase>petalbot",
                 Rule-Action = Deny,
		Direction = Incoming

these 2 nothing at all:

URL-Filter-bot-petal-string2
		Comment = "Deny HTTP requests for petalsearch-bot2 19-4-2021",
		Destination-Port = "80 443",
		Protocol = TCP,
		Log-Control = Enabled,
		Log-Mask = "date time severity message rule source resolved_source prot",
		Log-Size = 9990,
		Log-Severity = Major,
		Depth = 200,
		Hex-String = "<nocase>petalsearch",
                 Rule-Action = Deny,
		Direction = Incoming

URL-Filter-bot-petal-string3
		Comment = "Deny HTTP requests for petalsearch-bot2 19-4-2021",
		Destination-Port = "80 443",
		Protocol = TCP,
		Log-Control = Enabled,
		Log-Mask = "date time severity message rule source resolved_source prot",
		Log-Size = 9990,
		Log-Severity = Major,
		Depth = 200,
		Offset = URL,
		Hex-String = "*PetalBot*",
                 Rule-Action = Deny,
		Direction = Incoming

thanks

massimo