From: "Massimo S." <ecs-isp@2rosenthals.com>
Received: from [192.168.100.201] (HELO mail.2rosenthals.com)
  by 2rosenthals.com (CommuniGate Pro SMTP 5.4.10)
  with ESMTPS id 511484 for ecs-isp@2rosenthals.com; Wed, 26 Feb 2020 04:02:33 -0500
Received: from [192.168.200.201] (port=48236 helo=mail2.2rosenthals.com)
	by mail.2rosenthals.com with esmtp (Exim 4.82_1-5b7a7c0-XX)
	(envelope-from <ml@ecomstation.it>)
	id 1j6saU-0002xD-2p
	for ecs-isp@2rosenthals.com; Wed, 26 Feb 2020 04:02:38 -0500
Received: from mail2.quasarbbs.net ([80.86.52.115]:10195)
	by mail2.2rosenthals.com with esmtp (Exim 4.82_1-5b7a7c0-XX)
	(envelope-from <ml@ecomstation.it>)
	id 1j6sVc-0001jK-34
	for ecs-isp@2rosenthals.com; Wed, 26 Feb 2020 03:57:37 -0500
Received: from [192.168.10.199] (dtp [192.168.10.199]) by srv2 (Weasel v2.71); 
 26 Feb 2020 09:57:32 
X-CTCH-RefID: str=0001.0A020202.5E5632FF.0008,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0
Reply-To: ml@ecomstation.it
Subject: Re: [Apache/2] apache 2.2.34 php 5.4.45 mod_ssl/2.2.34 openssl/1.0.2k
 hangs - was a faulty RAM! :) - NO it's not... :(
To: apache2@googlegroups.com
References: <3c8a5043-c957-e49a-03f9-e25ddd3c1528@ecomstation.it>
 <5e67d368-bb19-45ad-a9f6-4b5d3abbad18@ecomstation.it>
 <e004bf0f-8004-589b-268d-5079420adb1c@ecomstation.it>
 <5E28C5A8.3040303@2rosenthals.com>
 <7c8eb079-84bb-5402-4c1c-f2ddbfedc73d@ecomstation.it>
 <5E292FF8.9050108@2rosenthals.com>
 <c1c19863-0a7d-0ce5-fa7b-b87f783b8707@ecomstation.it>
 <5E29CEC7.2020703@2rosenthals.com>
 <fcd21a40-c761-198b-8a47-a2fd47ef9c7f@ecomstation.it>
Cc: Paul Smedley <paul@smedley.id.au>, Steven Levine <steve53@earthlink.net>,
 eCS ISP Mailing List <ecs-isp@2rosenthals.com>
Organization: eComStation dot it
Message-ID: <eca69d2c-1294-f4e8-ef9e-fba883ec0421@ecomstation.it>
Date: Wed, 26 Feb 2020 09:57:30 +0100
User-Agent: Mozilla/5.0 (OS/2; U; Warp 4.5; it-IT; rv:1.7.13) Gecko/20060424
 Thunderbird/1.0.8 Mnenhy/0.7.4.0
MIME-Version: 1.0
In-Reply-To: <fcd21a40-c761-198b-8a47-a2fd47ef9c7f@ecomstation.it>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: it-IT
Content-Transfer-Encoding: 8bit

Il 06/02/2020 10:09, Massimo S. ha scritto:
> 
> 
> Il 23/01/2020 17:50, Lewis G Rosenthal ha scritto:
>> Hi, Max...
>>
>> (Keeping most of this for context; apologies to all for the absurdly long quoting.)
>>
>> On 01/23/20 07:09 am, Massimo S. wrote:
>>>
>>>
>>> Il 23/01/2020 06:32, Lewis G Rosenthal ha scritto:
>>>> On 01/22/20 05:46 pm, Massimo S. wrote:
>>>>>
>>>>>
>>>>> Il 22/01/2020 22:59, Lewis G Rosenthal ha scritto:
>>>>>> On 01/21/20 04:52 pm, Massimo S. wrote:
>>>>>>>
>>>>>>>
>>>>>>> Il 21/01/2020 22:46, Massimo S. ha scritto:
>>>>>>>>
>>>>>>>>
>>>>>>>> Il 20/01/2020 10:54, Massimo S. ha scritto:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> in the last months (since the end of october 2019) i've a number of hangs
>>>>>>>>> of the web server due to attacks (ddos or other) that comes on apache.
>>>>>>>>>
>>>>>>
>>>>
>>>> <snip>
>>>>
>>>>>>
>>>>>>>>> Often, sometimes 2 times per day i receive some kind of attacks on http or https, some 
>>>>>>>>> kind of them i can filter with Injoy Firewall, some of them
>>>>>>>>> i still have to understand how they act.
>>>>>>>>>
>>>>>>>>> Attacks like the slow loris, syn rotating packets attack etc..
>>>>>>>>>
>>>>>>>>> I also put a limit to 29 max tcp/ip connections from the same ip on http and https.
>>>>>>>>>
>>>>>>>>> All theese protections seems not to be sufficient.
>>>>>>>>>
>>>>>>>>> Any idea?
>>>>>>>>> Any help?
>>>>>>>>>
>>>>>>
>>>>>> Not enough info. No idea what modules you have loaded, 
>>>>>
>>>>>
>>>>> loadModule authn_file_module modules/authn_fi.dll
>>>>> loadmodule authn_default_module modules/authn_de.dll
>>>>> loadmodule authz_host_module modules/authz_ho.dll
>>>>> loadmodule authz_groupfile_module modules/authz_gr.dll
>>>>> loadmodule authz_user_module modules/authz_us.dll
>>>>> loadmodule authz_default_module modules/authz_de.dll
>>>>> loadmodule auth_basic_module modules/auth_bas.dll
>>>>> loadmodule auth_digest_module modules/auth_dig.dll
>>>>> loadmodule dumpio_module modules/dumpio.dll
>>>>> loadmodule echo_module modules/echo.dll
>>>>> loadmodule ext_filter_module modules/ext_filt.dll
>>>>> loadmodule include_module modules/include.dll
>>>>> loadmodule filter_module modules/filter.dll
>>>>> loadmodule deflate_module modules/deflate.dll
>>>>> loadmodule log_config_module modules/log_conf.dll
>>>>> loadmodule log_forensic_module modules/log_fore.dll
>>>>> loadmodule env_module modules/env.dll
>>>>> loadmodule mime_magic_module modules/mime_mag.dll
>>>>> loadmodule expires_module modules/expires.dll
>>>>> loadmodule headers_module modules/headers.dll
>>>>> loadmodule ident_module modules/ident.dll
>>>>> loadmodule usertrack_module modules/usertrac.dll
>>>>> loadmodule setenvif_module modules/setenvif.dll
>>>>> loadmodule version_module modules/version.dll
>>>>> loadmodule ssl_module modules/ssl.dll
>>>>> loadmodule mime_module modules/mime.dll
>>>>> loadmodule status_module modules/status.dll
>>>>> loadmodule autoindex_module modules/autoinde.dll
>>>>> loadmodule asis_module modules/asis.dll
>>>>> loadmodule info_module modules/info.dll
>>>>> loadmodule vhost_alias_module modules/vhost_al.dll
>>>>> loadmodule negotiation_module modules/negotiat.dll
>>>>> loadmodule dir_module modules/dir.dll
>>>>> loadmodule imagemap_module modules/imagemap.dll
>>>>> loadmodule alias_module modules/alias.dll
>>>>> loadmodule rewrite_module modules/rewrite.dll
>>>>> loadModule php5_module modules/modphp5.dll
>>>>>
>>>>
>>>> Suggestion #1: comment any modules you don't need. I doubt that you need *all* those auth 
>>>> modules (authz_groupfile? really?). Do you really need echo? 
>>> removed echo_module and authz_groupfile
>>> less modules -> less resources used, less potential security flaws
>>>
>>> >usertrack consumes considerable
>>> > overhead. Do you really track your users via cookies? If not, you don't >need this module.
>>>
>>> dunno exactly, but i know that some websites use cookies
>>> i've here also CMS joomla and wp CMSs (so cookie's banners etc.)
>>>
>>
>> You don't need to have the *server* instance track cookies just becuase various applications 
>> running on the server use cookies. These are different issues.
>>
>> I see tha you have determined that you really do need authz_groupfile. That's an interesting 
>> one. I haven't seen that on a rooted Apache installation on a single-user OS before. The idea 
>> is that you should be aware of what modules you actually use and try to avoid loading the 
>> ones you don't, as they are just so much extra baggage to take along with you.
>>
>>>>>> or anything else, let alone what the server is serving. PHP is likely the biggest drag on 
>>>>>> the system,
>>>>>
>>>>> i know this i've used apache with only html websites without php
>>>>> and it's rock solid stable
>>>>>
>>>>
>>>> All well and good. When using php as module, there is a considerable amount of interaction 
>>>> between httpd and php resources (php is loaded *all the time*). You need to balance http 
>>>> performance against php performance (sometimes), and you have to remember that you are 
>>>> essentially sharing resources between the two environments (shared memory resources, that is).
>>>>
>>>>>> and that's a whole other set of configurations (php.ini and the settings for the PHP apps 
>>>>>> themselves).
>>>>>
>>>>> php.ini (resource limit part)
>>>>>
>>>>> max_execution_time = 180
>>>>> max_input_time = 600
>>>>> max_input_time = 180
>>>>> memory_limit = 256M
>>>
>>> max_input_time = 600 this is a typo (since it's commented in php.ini)
>>>
>>
>> Okay.
>>
>>> others now reduced to theese:
>>>
>>> max_execution_time = 60
>>> max_input_time = 120
>>>
>>> i raised up them in the past years for Prestashop CMS, but now i've no prestashop websites, 
>>> so....
>>>
>>
>> The higher they are, the more the chance the PHP instance will get stuck for no good reason.
>>
>> <snip>
>>
>>>>>>>> about MPM_MPMT
>>>>>>>>
>>>>>>>> i have:
>>>>>>>>
>>>>>>>> MinSpareThreads       29
>>>>>>>> MaxSpareThreads       33
>>>>>>>>
>>>>>>>> but i see this:
>>>>>>>>
>>>>>>>> P-ID PPID Session Thr Prio    CPU Time Name
>>>>>>>
>>>>>>>> after some time..
>>>>>>>>
>>>>>>>> 3790 3785 020 VIO  36 0200    0:00:04.65 HTTPD.EXE
>>>>>>>>
>>>>>>>> where 3, 35, 33 and 34 are threads, is this normal?
>>>>>>>>
>>>>>>>> thanks
>>>>>>>>
>>>>>>>> massimo
>>>>>>>>
>>>>>>>
>>>>>>> where 36, 35, 33 and 34 are threads
>>>>>>> sorry the typo and the bad copy&paste
>>>>>>>
>>>>>>
>>>>>> No idea. How would I know, I wonder?
>>>>>
>>>>> it sound to me strange as MaxSpareThreads were 33
>>>>> and the threads of the childs show 36..
>>>>>
>>>>
>>>> 3 threads in use and 33 spares were available. The point of setting min/max values is that 
>>>> these are created and idle (the min value) at daemon start and will be added (put to use 
>>>> and the number of spares possibly increased) as demanded. The numbers should bounce between 
>>>> the in-use threads + MinSpare and in-use + MaxSpare.
>>>>
>>>>>> As posted a gazillion times on this list already:
>>>>>>
>>>>>> <IfModule mpm_mpmt_os2_module>
>>>>>>      ThreadStackSize    262144
>>>>>>      StartServers            3
>>>>>>      MinSpareThreads        50
>>>>>>      MaxSpareThreads        60
>>>>>>      MaxRequestsPerChild  1000
>>>>>> </IfModule>
>>>>>>
>>>>>> Been that way for ages. If your min/max values are too close, the server will labor to 
>>>>>> constantly adjust the amount of spare threads. Note I said *spare* threads, as in, idle 
>>>>>> threads available for work but which are *not* in use. Without knowing what your overall 
>>>>>> thread count 
>>>>>
>>>>> There are 32 Processes with 305 Threads.
>>>>> This machine's uptime is 0d 2h 40m 25s 949ms.
>>>>>
>>>>
>>>> You need to monitor your threads over time to see what happens to this value. 
>>>
>>> i use MRTG
>>>
>>
>> Okay. It doesn't really matter what you use. The point is that you need to watch the value 
>> over time and under different conditions to get a good feel for how many threads you're using.
>>
>>>> Also, 2 hours is a vry short time to get an impression. Check it at random intervals or 
>>>> when the system seems to be loaded and over a few days of uptime. That should give you a 
>>>> better picture.
>>>
>>> this server do 2 reboots (setboot/b) per days in the night
>>> and since 18 jen 2020 i've 2 hang/freeze per day
>>> i've moved here from another webserver a terrible WP website
>>> but now i've moved back to the external webserver
>>>
>>
>> You should not have to bounce the box twice per day. I was down to every four days, and now 
>> every 6 (I think; I need to check crontabs). Of course, if your memory is becoming fragmented 
>> sooner, you'll need to reboot sooner, and all of these values play into that fragmentation 
>> problem, as we have seen over the years.
>>
>>>
>>>>> apache:
>>>>>
>>>>> P-ID PPID Session Thr Prio    CPU Time                    Name
>>>>>
>>>>> 4440   16 020 VIO   1 0200    0:00:05.12      HTTPD.EXE
>>>>> 4974 4440 020 VIO  36 0200    0:00:02.31        HTTPD.EXE
>>>>> 4973 4440 020 VIO  36 0200    0:00:07.87        HTTPD.EXE
>>>>> 4972 4440 020 VIO  34 0200    0:00:06.90        HTTPD.EXE
>>>>> 4971 4440 020 VIO  33 0200    0:00:06.75        HTTPD.EXE
>>>>>
>>>>>
>>>>>> is in the box, your THREADS= in CONFIG.SYS, how could anyone possibly know whether your 
>>>>>> thread settings for the MPM are sane?
>>>>>
>>>>> threads in config.sys are
>>>>>
>>>>> THREADS=1024
>>>>>
>>>>> i've the same value on my 2 server, but also on the new AOS 503 on the
>>>>> test virtual machine that in the next monts will handle only apache+php
>>>>> i've seen that AOS put this value as default
>>>>>
>>>>
>>>> Yes, it's a sane default figure. For a server, I wouldn't recommend anything lower.
>>>>
>>>>> other values in httpd.conf that may be of interested are set to:
>>>>>
>>>>> Timeout 60
>>>>> KeepAlive On
>>>>> MaxKeepAliveRequests 45
>>>>> KeepAliveTimeout 5
>>>>>
>>>>
>>>> I would cut timeout to 30 (again, the longer you sit around waiting, the more resources 
>>>> you're going to waste in the process). In the days of 56Kbps connections, longer timeouts 
>>>> were necessary; today, not so much.
>>>
>>> will try 45 then 30 in the next days today i've moved too much parameters :)
>>>
>>
>> Agreed, and a point worth repeating. As with any tuning changes, making too many at one time 
>> is just bad practice. You need to do this gradually and methodically to watch for unexpected 
>> changes, and document behavior as you go (and not rely on sheer brain memory alone).
>>
>>>> MaxKeepAliveRequests should be a higher value (Apache docs recommend 500 or so; I use 500). 
>>>> The server will labor when this is set lower, as connections will have to be reestablished. 
>>>> Reestablishing a connection is an expensive proposition. There are conflicting user reports 
>>>> about this, with some recommending very low values. I've *never* had good luck on OS/2 with 
>>>> this set low (on NetWare, I've not found it to make much difference; again, it all depends 
>>>> upon the memory model and how expensive these operations are on a given platform).
>>>
>>> ok raised from 45 to 100
>>> i've to do all theese tests slowly to understand the behaviour
>>>
>>
>> The mantis links I provided should give you some background as to the tests we conducted and 
>> what we found at the time.
>>
>>>> KeepAliveTimeout 5 is a reasonable number. I would *not* recommend setting this higher.
>>>>
>>>>>
>>>>> about MinSpareTd and MaxSpareTd i've right now modified the apache configuration and now i 
>>>>> have:
>>>>>
>>>>> <IfModule mpm_mpmt_os2_module>
>>>>>     StartServers           4
>>>>>     MinSpareThreads       29
>>>>>     MaxSpareThreads       39
>>>>>     MaxRequestsPerChild   95
>>>>> </IfModule>
>>>>>
>>>>>
>>>>> ..ThreadStackSize    262144
>>>>> what this value?
>>>>> i don't know much about this function is this value the one to use on mpmt_os2 or is a 
>>>>> generic one?
>>>>>
>>>
>>> i will try also this option
>>> since the default i read i 65536 and it seems that's too low
>>>
>>
>> It's only too low if you are seeing crashes. If Apache isn't crashing with a blown stack, you 
>> don't need to bump this value; you're just wasting resources. That said, these are rather 
>> cheap by today's standards, so feel free to experiment. I should probably lower this figure, 
>> I guess. My notes aren't exceptionally clear, though I did note in one of the Mantis tickets 
>> that values *below* 65536 seemed to cause consistent problems (crashes). Again, YMMV.
>>
>>>> If you are *not* getting crashes with Apache blowing the stack, you probably don't need to 
>>>> adjust this. Various MPMs use this directive to set the stack size allocated for each 
>>>> worker thread. The default on OS/2 is (still, IIRC) 65536 (see: 
>>>> https://mantis.smedley.id.au/view.php?id=558 and 
>>>> https://mantis.smedley.id.au/view.php?id=637 for more).
>>>>
>>>> (That second link contains a lot of research and testing we did back in 2014/2015.)
>>>>
>>
>> <snip>
>>
>>>>> slowly i'm moving websites to https
>>>>>
>>>>
>>>> Best recommendation:
>>>>
>>>> https://www.ssllabs.com/ssltest/
>>>>
>>>> Test your server at the above. While you're there, try testing www.arcanoae.com or 
>>>> www.2rosenthals.com (A+ on both of those). Don't discount the server load for poor SSL 
>>>> configuration.
>>>
>>> tried and i get B due to this:
>>>
>>> This server's certificate chain is incomplete. Grade capped to B.
>>
>> You probably need to install the intermediate cert. I don't use LE certs, and I 
>> *specifically* remove the intermediates as they are unnecessary for my CA (this is for 
>> performance reasons, as every cert fetch is another transaction). I have no idea for LE what 
>> may be required.
>>
>>> since i don't have this entry in the dns zone:
>>>
>>> mydomain.org. CAA 128 issue "letsencrypt.org"
>>>
>>> i'm running bind 9.6.0pl1 and it don't support CAA function
>>> i've to upgrade at least to 9.9.6
>>>
>>
>> Unfortunately (or fortunately, depending on one's POV), CAA is a whole new record type and 
>> not something which can be faked with a TXT record.
>>
>>> i've tried newer binds in the last months, but it seems to me that all were
>>> giving issues, so i sticket to 9.6.0
>>>
>>
>> Not having a CAA record should only limit you to A, not B. B cap is due to the cert chain 
>> being incomplete, not the lack of CAA.
>>
>>> will see what to do thanks
>>>
>>> for now i've moved too much parameters so i will wait 1 or 2 days to see how the things goes on
>>>
>>
>> Good idea.
> 
> 
> sorry for the big quota
> 
> i realized since some months that the real source of the problem was a faulty ram (Kingston 
> KTH-XW4300E/2G)
> 
> in the last month i've changed: UPS, power supply, motherboard, CPU and all the cables...
> 
> ...but, i only kept ram... sigh
> 
> now i'm running with only 2GB of ram sometimes the machine (due for apache)
> go at 512KB of ram (no ram free at all:D) so that rarely something trap
> at ring0 and the machine reboot since i have REIPL=ON
> 
> i've ordered 1GB ram, i guess that with 3GB will surely not run out of memory
> 
> 
> massimo

after the change of the RAM anyway the server has had a number of freezes

so that i changed completely another mother board (see $$$) and freezes continues...  about 1-3 
per days

it' clear that since i've modified the apache mmpt configuration
the situation has worstned not improved

i suspect the high number of threads

what i did...

i bought a new server now i'm using Vbox 6.1.2 VMs
the server web has an AOS 5.0.3 machine and *it only run:
apache+php, ftpd, cron, injoy fw and nothing more

i'm moving all the websites into the new AOS503 web server only guest VMs

the guest VMs still has reboot scheduled, 2 days ago come into production
yesterday at 00 o'clock at a scheduled reboot (setboot /b) the server freezed/hang due to 
apache running out of "who knows system resources"

this is no good

i've *isolated apache+php in a 4GB VM only* to have it running completely
away by itself and also with this dedicated configuration there are issues

now i'm asking to Paul and Steven a bit of help, we have to understand what goes wrong *in 
apache and php* and i'm not sufficient to undrestand what cause this i've seen in the past 
maybe an HTTPD child that become unkillable
so bad that even a setboot /b can't reboot the server

setboot /b is the last resource if this freeze the server we are fuc$ed

only Steven and Paul can help here

Server Version: Apache/2.2.34 (OS/2) PHP/5.4.45 mod_ssl/2.2.34 OpenSSL/1.0.2k
Server Built: Feb 3 2019 17:11:06

i'm at full disposal to run diagnostics and such

this is the apache setup:

Timeout 60
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
UseCanonicalName On
ServerTokens Full
HostnameLookups On

<IfModule mpm_mpmt_os2_module>
     StartServers           4
     MinSpareThreads       40
     MaxSpareThreads       60
     MaxRequestsPerChild   99
</IfModule>



please help me



massimo