Il 23/01/2020 17:50, Lewis G Rosenthal ha scritto:

Hi, Max...

(Keeping most of this for context; apologies to all for the absurdly long quoting.)

On 01/23/20 07:09 am, Massimo S. wrote:

Il 23/01/2020 06:32, Lewis G Rosenthal ha scritto:

On 01/22/20 05:46 pm, Massimo S. wrote:

Il 22/01/2020 22:59, Lewis G Rosenthal ha scritto:

On 01/21/20 04:52 pm, Massimo S. wrote:

Il 21/01/2020 22:46, Massimo S. ha scritto:

Il 20/01/2020 10:54, Massimo S. ha scritto:

Hi all,

in the last months (since the end of october 2019) i've a number of hangs
of the web server due to attacks (ddos or other) that comes on apache.

<snip>

Often, sometimes 2 times per day i receive some kind of attacks on http or https, some kind of them i can filter with Injoy Firewall, some of them
i still have to understand how they act.

Attacks like the slow loris, syn rotating packets attack etc..

I also put a limit to 29 max tcp/ip connections from the same ip on http and https.

All theese protections seems not to be sufficient.

Any idea?
Any help?

Not enough info. No idea what modules you have loaded,

loadModule authn_file_module modules/authn_fi.dll
loadmodule authn_default_module modules/authn_de.dll
loadmodule authz_host_module modules/authz_ho.dll
loadmodule authz_groupfile_module modules/authz_gr.dll
loadmodule authz_user_module modules/authz_us.dll
loadmodule authz_default_module modules/authz_de.dll
loadmodule auth_basic_module modules/auth_bas.dll
loadmodule auth_digest_module modules/auth_dig.dll
loadmodule dumpio_module modules/dumpio.dll
loadmodule echo_module modules/echo.dll
loadmodule ext_filter_module modules/ext_filt.dll
loadmodule include_module modules/include.dll
loadmodule filter_module modules/filter.dll
loadmodule deflate_module modules/deflate.dll
loadmodule log_config_module modules/log_conf.dll
loadmodule log_forensic_module modules/log_fore.dll
loadmodule env_module modules/env.dll
loadmodule mime_magic_module modules/mime_mag.dll
loadmodule expires_module modules/expires.dll
loadmodule headers_module modules/headers.dll
loadmodule ident_module modules/ident.dll
loadmodule usertrack_module modules/usertrac.dll
loadmodule setenvif_module modules/setenvif.dll
loadmodule version_module modules/version.dll
loadmodule ssl_module modules/ssl.dll
loadmodule mime_module modules/mime.dll
loadmodule status_module modules/status.dll
loadmodule autoindex_module modules/autoinde.dll
loadmodule asis_module modules/asis.dll
loadmodule info_module modules/info.dll
loadmodule vhost_alias_module modules/vhost_al.dll
loadmodule negotiation_module modules/negotiat.dll
loadmodule dir_module modules/dir.dll
loadmodule imagemap_module modules/imagemap.dll
loadmodule alias_module modules/alias.dll
loadmodule rewrite_module modules/rewrite.dll
loadModule php5_module modules/modphp5.dll

Suggestion #1: comment any modules you don't need. I doubt that you need *all* those auth modules (authz_groupfile? really?). Do you really need echo?

removed echo_module and authz_groupfile
less modules -> less resources used, less potential security flaws

>usertrack consumes considerable
> overhead. Do you really track your users via cookies? If not, you don't >need this module.

dunno exactly, but i know that some websites use cookies
i've here also CMS joomla and wp CMSs (so cookie's banners etc.)

You don't need to have the *server* instance track cookies just becuase various applications running on the server use cookies. These are different issues.

I see tha you have determined that you really do need authz_groupfile. That's an interesting one. I haven't seen that on a rooted Apache installation on a single-user OS before. The idea is that you should be aware of what modules you actually use and try to avoid loading the ones you don't, as they are just so much extra baggage to take along with you.

or anything else, let alone what the server is serving. PHP is likely the biggest drag on the system,

i know this i've used apache with only html websites without php
and it's rock solid stable

All well and good. When using php as module, there is a considerable amount of interaction between httpd and php resources (php is loaded *all the time*). You need to balance http performance against php performance (sometimes), and you have to remember that you are essentially sharing resources between the two environments (shared memory resources, that is).

and that's a whole other set of configurations (php.ini and the settings for the PHP apps themselves).

php.ini (resource limit part)

max_execution_time = 180
max_input_time = 600
max_input_time = 180
memory_limit = 256M

max_input_time = 600 this is a typo (since it's commented in php.ini)

Okay.

others now reduced to theese:

max_execution_time = 60
max_input_time = 120

i raised up them in the past years for Prestashop CMS, but now i've no prestashop websites, so....

The higher they are, the more the chance the PHP instance will get stuck for no good reason.

<snip>

about MPM_MPMT

i have:

MinSpareThreads       29
MaxSpareThreads       33

but i see this:

P-ID PPID Session Thr Prio    CPU Time Name

after some time..

3790 3785 020 VIO  36 0200    0:00:04.65 HTTPD.EXE

where 3, 35, 33 and 34 are threads, is this normal?

thanks

massimo

where 36, 35, 33 and 34 are threads
sorry the typo and the bad copy&paste

No idea. How would I know, I wonder?

it sound to me strange as MaxSpareThreads were 33
and the threads of the childs show 36..

3 threads in use and 33 spares were available. The point of setting min/max values is that these are created and idle (the min value) at daemon start and will be added (put to use and the number of spares possibly increased) as demanded. The numbers should bounce between the in-use threads + MinSpare and in-use + MaxSpare.

As posted a gazillion times on this list already:

<IfModule mpm_mpmt_os2_module>
     ThreadStackSize    262144
     StartServers            3
     MinSpareThreads        50
     MaxSpareThreads        60
     MaxRequestsPerChild  1000
</IfModule>

Been that way for ages. If your min/max values are too close, the server will labor to constantly adjust the amount of spare threads. Note I said *spare* threads, as in, idle threads available for work but which are *not* in use. Without knowing what your overall thread count

There are 32 Processes with 305 Threads.
This machine's uptime is 0d 2h 40m 25s 949ms.

You need to monitor your threads over time to see what happens to this value.

i use MRTG

Okay. It doesn't really matter what you use. The point is that you need to watch the value over time and under different conditions to get a good feel for how many threads you're using.

Also, 2 hours is a vry short time to get an impression. Check it at random intervals or when the system seems to be loaded and over a few days of uptime. That should give you a better picture.

this server do 2 reboots (setboot/b) per days in the night
and since 18 jen 2020 i've 2 hang/freeze per day
i've moved here from another webserver a terrible WP website
but now i've moved back to the external webserver

You should not have to bounce the box twice per day. I was down to every four days, and now every 6 (I think; I need to check crontabs). Of course, if your memory is becoming fragmented sooner, you'll need to reboot sooner, and all of these values play into that fragmentation problem, as we have seen over the years.

apache:

P-ID PPID Session Thr Prio    CPU Time                    Name

4440   16 020 VIO   1 0200    0:00:05.12      HTTPD.EXE
4974 4440 020 VIO  36 0200    0:00:02.31        HTTPD.EXE
4973 4440 020 VIO  36 0200    0:00:07.87        HTTPD.EXE
4972 4440 020 VIO  34 0200    0:00:06.90        HTTPD.EXE
4971 4440 020 VIO  33 0200    0:00:06.75        HTTPD.EXE

is in the box, your THREADS= in CONFIG.SYS, how could anyone possibly know whether your thread settings for the MPM are sane?

threads in config.sys are

THREADS=1024

i've the same value on my 2 server, but also on the new AOS 503 on the
test virtual machine that in the next monts will handle only apache+php
i've seen that AOS put this value as default

Yes, it's a sane default figure. For a server, I wouldn't recommend anything lower.

other values in httpd.conf that may be of interested are set to:

Timeout 60
KeepAlive On
MaxKeepAliveRequests 45
KeepAliveTimeout 5

I would cut timeout to 30 (again, the longer you sit around waiting, the more resources you're going to waste in the process). In the days of 56Kbps connections, longer timeouts were necessary; today, not so much.

will try 45 then 30 in the next days today i've moved too much parameters :)

Agreed, and a point worth repeating. As with any tuning changes, making too many at one time is just bad practice. You need to do this gradually and methodically to watch for unexpected changes, and document behavior as you go (and not rely on sheer brain memory alone).

MaxKeepAliveRequests should be a higher value (Apache docs recommend 500 or so; I use 500). The server will labor when this is set lower, as connections will have to be reestablished. Reestablishing a connection is an expensive proposition. There are conflicting user reports about this, with some recommending very low values. I've *never* had good luck on OS/2 with this set low (on NetWare, I've not found it to make much difference; again, it all depends upon the memory model and how expensive these operations are on a given platform).

ok raised from 45 to 100
i've to do all theese tests slowly to understand the behaviour

The mantis links I provided should give you some background as to the tests we conducted and what we found at the time.

KeepAliveTimeout 5 is a reasonable number. I would *not* recommend setting this higher.

about MinSpareTd and MaxSpareTd i've right now modified the apache configuration and now i have:

<IfModule mpm_mpmt_os2_module>
    StartServers           4
    MinSpareThreads       29
    MaxSpareThreads       39
    MaxRequestsPerChild   95
</IfModule>


..ThreadStackSize    262144
what this value?
i don't know much about this function is this value the one to use on mpmt_os2 or is a generic one?

i will try also this option
since the default i read i 65536 and it seems that's too low

It's only too low if you are seeing crashes. If Apache isn't crashing with a blown stack, you don't need to bump this value; you're just wasting resources. That said, these are rather cheap by today's standards, so feel free to experiment. I should probably lower this figure, I guess. My notes aren't exceptionally clear, though I did note in one of the Mantis tickets that values *below* 65536 seemed to cause consistent problems (crashes). Again, YMMV.

If you are *not* getting crashes with Apache blowing the stack, you probably don't need to adjust this. Various MPMs use this directive to set the stack size allocated for each worker thread. The default on OS/2 is (still, IIRC) 65536 (see: https://mantis.smedley.id.au/view.php?id=558 and https://mantis.smedley.id.au/view.php?id=637 for more).

(That second link contains a lot of research and testing we did back in 2014/2015.)

<snip>

slowly i'm moving websites to https

Best recommendation:

https://www.ssllabs.com/ssltest/

Test your server at the above. While you're there, try testing www.arcanoae.com or www.2rosenthals.com (A+ on both of those). Don't discount the server load for poor SSL configuration.

tried and i get B due to this:

This server's certificate chain is incomplete. Grade capped to B.

You probably need to install the intermediate cert. I don't use LE certs, and I *specifically* remove the intermediates as they are unnecessary for my CA (this is for performance reasons, as every cert fetch is another transaction). I have no idea for LE what may be required.

since i don't have this entry in the dns zone:

mydomain.org. CAA 128 issue "letsencrypt.org"

i'm running bind 9.6.0pl1 and it don't support CAA function
i've to upgrade at least to 9.9.6

Unfortunately (or fortunately, depending on one's POV), CAA is a whole new record type and not something which can be faked with a TXT record.

i've tried newer binds in the last months, but it seems to me that all were
giving issues, so i sticket to 9.6.0

Not having a CAA record should only limit you to A, not B. B cap is due to the cert chain being incomplete, not the lack of CAA.

will see what to do thanks

for now i've moved too much parameters so i will wait 1 or 2 days to see how the things goes on

Good idea.

sorry for the big quota

i realized since some months that the real source of the problem was a faulty ram (Kingston KTH-XW4300E/2G)

in the last month i've changed: UPS, power supply, motherboard, CPU and all the cables...

...but, i only kept ram... sigh

now i'm running with only 2GB of ram sometimes the machine (due for apache)
go at 512KB of ram (no ram free at all:D) so that rarely something trap
at ring0 and the machine reboot since i have REIPL=ON

i've ordered 1GB ram, i guess that with 3GB will surely not run out of memory


massimo