If you look at how ClamAV handles memory, it does an large number of
reallocs with small size increments. This is good way to fragment the
When we run out of memory, the realloc size is not all that large.
Something like 16 million or so, IIRC.
Here's the thing. As you probably know .cvd files are basicall tgz files
with a ClamAV specific header. To update the database, freshclam unpacks
the local .cvd files and any files it downloads to a temp directory and
does some processing, probably to prune stale data. Then it packs the
file set into a temp .clb, which is just a .cvd with a different
extension. This is given to fc_test_database to test and that is where we
run out of memory.
Unlike main.tgz and daily.tgz which are compressed, the .cld file is not
compressed. This means that even assuming 100% overhead for links and
hash tables and such, the memory requirements should be 400MB or so. This
is why I suspect fragmentation is the problem. I'll know more when I look
what Theseus has to say about address space usage.
Take a look at umalloc.h. We might want to use _ustats to get an idea of
how much fragmentation there is.