From: "Massimo S." <ecs-isp@2rosenthals.com>
Reply-To: ml@ecomstation.it
Subject: Re: [eCS-ISP] clamscan issue - directories with a lot of files
To: eCS ISP Mailing List <ecs-isp@2rosenthals.com>
References: <list-11203715@2rosenthals.com>
Organization: Massimo S.
Message-ID: <49887cfd-36e3-c68e-6785-8e814e17261a@ecomstation.it>
Date: Mon, 4 Nov 2024 09:43:20 +0100
User-Agent: Mozilla/5.0 (OS/2; U; Warp 4.5; it-IT; rv:1.7.13) Gecko/20060424
 Thunderbird/1.0.8 Mnenhy/0.7.4.0
MIME-Version: 1.0
In-Reply-To: <list-11203715@2rosenthals.com>
Content-Type: text/plain; charset=iso-8859-15; format=flowed
Content-Language: it-IT
Content-Transfer-Encoding: 8bit



Il 02/11/2024 16:37, Steven Levine ha scritto:
> In <list-11202333@2rosenthals.com>, on 11/01/24
>     at 12:05 PM, "Massimo S." <ecs-isp@2rosenthals.com> said:
> 
> HI Massimo,
> 
>>>> damn, i just deleted a debug.txt output for mistake.. :-(
> 
> Oops.
> 
>>>> anyway it was a 4GB file, i don't have it in the bkups,
>>>> maybe ZIP is not able to add a so large file
> 
> Depends on the version of zip.
> 
>>>> and i guess that the issue of "Can't read file ERROR"
>>>> is not related to a directory with a lot of files inside
>>>> since it appears allmost the times in "big dirs", but
>>>> it may happens that clamscan keep on working again in the same dir
>>>> e.g. in this case:
> 
>>>> X:/weasel/MailRoot/mydomain.com/myemail/OLAYI6.MSG: Can't read file ERROR
>>>> X:/weasel/MailRoot/mydomain.com/myemail/OLAZFS.MSG: Can't read file ERROR
>>>> X:/weasel/MailRoot/mydomain.com/myemail/OLB25S.MSG: Can't read file ERROR
>>>> X:/weasel/MailRoot/mydomain.com/myemail/OLB3TA.MSG: Can't read file ERROR
>>>> X:/weasel/MailRoot/mydomain.com/myemail/OLB5K9.MSG: Can't read file ERROR
> 
>>> i'm able to reproduce the debug output
>>> uncompressed it's 5,6GB..  doh  :-)
>>> compressed is 2,6GB
> 
> For a file this large, I recommed you use 7z.
> 
>>> i can put it on an ftp
>>> let me know
> 
> Not yet.  I busy with some other projects.
> 
>> when it show the errors i see a lot of these in the output on the screen:
>> LibClamAV Error: fmap_readpage: pread error: Not enough memory
> 
> That message pretty much explains itself.
> 
>> the VM has 4GB ram assigned (i know /2 see only about 3,3GB) and
>> vaddresslimit is 3072
> 
> That's as good as it's going to get.
> 
> Clamscan is a memory hog.  There are two issues - overall memory usage and
> fragmentation.

hi,

could that be improved?

> I don't seem to have a copy of your clamscan command line.

clamscan --database=X:\usr\local\clamav\share\clamav --tempdir=Y:\temp --scan-archive=no --exclude=MSGLIST.DAT 
--exclude=DOMAIN.*   --move=X:\quarantine -lZ:\clamav_day.log X:\weasel\MailRoot\mydomain.com

> I suspect you are going have to break the clamscan run into smaller chunks
> to avoid the memory issues.

there are about 25 mail domains and more than 350 subdirectories..
with the small ones with just one or two emails i'm already doing this

> Steven

massimo