[mdlug] [Apache] High Load Issues... ?

[Wojtak, Greg]

Thanks for the ideas, I'll check that.

Another thing I noticed (that I forgot to mention) is that cpu time is indeed being split pretty much 50/50 between user and sys - not a good sign.  I'll have to check with our Network guy, because people out on the intertubes aren't connecting directly to the apache servers - there are a couple of F5's in between them.  Perhaps they are doing some connection clean up or something.  One of my co-workers noticed that when it gets in this state there are an awful lot of connections in a CLOSE_WAIT or FIN_WAIT (can't remember which he said, I'll ask him tomorrow).

Thanks again!

Greg

-----Original Message-----
From: mdlug-bounces at mdlug.org [mailto:mdlug-bounces at mdlug.org] On Behalf Of Jonathan Billings
Sent: Monday, April 13, 2009 6:21 PM
To: MDLUG's Main discussion list
Subject: Re: [mdlug] [Apache] High Load Issues... ?

On Mon, Apr 13, 2009 at 04:07:10PM -0400, Wojtak, Greg wrote:
>
> I recently updated from RHEL 4 to CentOS 5, taking Apache along for
> the ride (2.0 -> 2.2).  Recently (only in our production
> environment), we've been seeing load randomly shoot up to around 130
> or so.  One of the things I've noticed is that during these
> occurrences, apache is always on top when running top, though it is
> always different PID's, and (this is the one that gets me) memory
> utilization drops tremendously.  I assume there is some kind of
> garbage cleanup going on.   Does anyone know what apache might be
> doing that frees up approximately 200 MB or more of physical RAM?
> Any other insights? 

Often when you see a high load like that is when you have a lot of
processes waiting on a kernel call, like disk wait or a TCP socket.
Check your apache configuration, I suspect the maximum number of
httpds that are to be running is close to 130.

Keep in mind:  high load != lots of CPU operations.  "Load" is defined
by a complex algorithm, basically giving the lenght of the "queue" of
operations to run.  If there are a lot of processes waiting on a
kernel call to complete, each one of those processes increments the
load by one.

Perhaps what you're seeing is a scan on your web server where the
client doesn't close the connection correctly.  Next time you see it,
check out lsof or netstat.  Or maybe you're serving data out with each
connection, and they're waiting on IO?

-- 
Jonathan Billings <billings at negate.org>
_______________________________________________
mdlug mailing list
mdlug at mdlug.org
http://mdlug.org/mailman/listinfo/mdlug