[mdlug] [Apache] High Load Issues... ?

[Michael S. Mikowski]

Can you watch top when you start up apache?  Maybe all the procs are having 
this problem from the start.   I would expect you should see some notification 
in the error logs.

On Monday 13 April 2009 20:23:17 Wojtak, Greg wrote:
> Thanks for the ideas, I'll check that.
>
> Another thing I noticed (that I forgot to mention) is that cpu time is
> indeed being split pretty much 50/50 between user and sys - not a good
> sign.  I'll have to check with our Network guy, because people out on the
> intertubes aren't connecting directly to the apache servers - there are a
> couple of F5's in between them.  Perhaps they are doing some connection
> clean up or something.  One of my co-workers noticed that when it gets in
> this state there are an awful lot of connections in a CLOSE_WAIT or
> FIN_WAIT (can't remember which he said, I'll ask him tomorrow).
>
> Thanks again!
>
> Greg
>
> -----Original Message-----
> From: mdlug-bounces at mdlug.org [mailto:mdlug-bounces at mdlug.org] On Behalf Of
> Jonathan Billings Sent: Monday, April 13, 2009 6:21 PM
> To: MDLUG's Main discussion list
> Subject: Re: [mdlug] [Apache] High Load Issues... ?
>
> On Mon, Apr 13, 2009 at 04:07:10PM -0400, Wojtak, Greg wrote:
> > I recently updated from RHEL 4 to CentOS 5, taking Apache along for
> > the ride (2.0 -> 2.2).  Recently (only in our production
> > environment), we've been seeing load randomly shoot up to around 130
> > or so.  One of the things I've noticed is that during these
> > occurrences, apache is always on top when running top, though it is
> > always different PID's, and (this is the one that gets me) memory
> > utilization drops tremendously.  I assume there is some kind of
> > garbage cleanup going on.   Does anyone know what apache might be
> > doing that frees up approximately 200 MB or more of physical RAM?
> > Any other insights?
>
> Often when you see a high load like that is when you have a lot of
> processes waiting on a kernel call, like disk wait or a TCP socket.
> Check your apache configuration, I suspect the maximum number of
> httpds that are to be running is close to 130.
>
> Keep in mind:  high load != lots of CPU operations.  "Load" is defined
> by a complex algorithm, basically giving the lenght of the "queue" of
> operations to run.  If there are a lot of processes waiting on a
> kernel call to complete, each one of those processes increments the
> load by one.
>
> Perhaps what you're seeing is a scan on your web server where the
> client doesn't close the connection correctly.  Next time you see it,
> check out lsof or netstat.  Or maybe you're serving data out with each
> connection, and they're waiting on IO?