From: "Steven Levine" <ecs-isp@2rosenthals.com>
Message-ID: <652ceda8.8.mr2ice.fgrirsq@earthlink.net>
Date: Mon, 16 Oct 2023 00:00:40 -0800
To: "eCS ISP Mailing List" <ecs-isp@2rosenthals.com>
In-Reply-To: <list-8113084@2rosenthals.com>
Subject: Re: [eCS-ISP] ClamAV

In <list-8113084@2rosenthals.com>, on 10/15/23
   at 05:45 PM, "Paul Smedley" <ecs-isp@2rosenthals.com> said:

Hi Paul,

>IIRC - it creates a text file somewhere locally for this - remove the 
>text file and you're un-banned :)

Thanks.  I know what to look for if it happens again.  Based on how the
ban appeared to work, it really looked like the Cloudflare CDN was keeping
the access data on the server.

>your wish is my command - https://github.com/psmedley/clamav-os2 & 
>https://smedley.id.au/tmp/clamav-0.103.10-os2-20231015-debug.zip

These are working pretty well so far.  I've been running freshclam under
idebug and have been able to break at

others_common.c:237
    if (!alloc) {
logg("cli_realloc(): Can't re-allocate memory to %lu bytes.\n", (unsigned
long int)size);
        perror("realloc_problem");
        cli_errmsg("cli_realloc(): Can't re-allocate memory to %lu
bytes.\n", (unsigned long int)size);
        return NULL;
    } else
        return alloc;
}

when the heap runs out of memory.  So far it's been in the code that adds
signatures to the internal list.

matcher-ac.c:2483
cl_error_t cli_ac_addsig(struct cli_matcher *root, const char *virname,
const char *hexsig, uint8_t sigopts, uint32_t 

If you look at how ClamAV handles memory, it does an large number of
reallocs with small size increments.  This is good way to fragment the
heap.

When we run out of memory, the realloc size is not all that large. 
Something like 16 million or so, IIRC.

Here's the thing.  As you probably know .cvd files are basicall tgz files
with a ClamAV specific header.  To update the database, freshclam unpacks
the local .cvd files and any files it downloads to a temp directory and
does some processing, probably to prune stale data.  Then it packs the
file set into a temp .clb, which is just a .cvd with a different
extension.  This is given to fc_test_database to test and that is where we
run out of memory.

The file freshscan runs out of memory on is:

10-15-23  23:31     196,264,448    124 
clamav-169820d0846ffa6b2d6f3ff06dea27ce.tmp-daily.cld

Unlike main.tgz and daily.tgz which are compressed, the .cld file is not
compressed.  This means that even assuming 100% overhead for links and
hash tables and such, the memory requirements should be 400MB or so.  This
is why I suspect fragmentation is the problem.  I'll know more when I look
what Theseus has to say about address space usage.

Take a look at umalloc.h.  We might want to use _ustats to get an idea of
how much fragmentation there is.

Steven

-- 
----------------------------------------------------------------------
"Steven Levine" <steve53@earthlink.net>  Warp/DIY/BlueLion etc.
www.scoug.com www.arcanoae.com www.warpcave.com
----------------------------------------------------------------------