[mdlug] [Apache] High Load Issues... ?

[Jonathan Billings]

On Mon, Apr 13, 2009 at 11:23:17PM -0400, Wojtak, Greg wrote:
> Thanks for the ideas, I'll check that.
> 
> Another thing I noticed (that I forgot to mention) is that cpu time
> is indeed being split pretty much 50/50 between user and sys - not a
> good sign.  I'll have to check with our Network guy, because people
> out on the intertubes aren't connecting directly to the apache
> servers - there are a couple of F5's in between them.  Perhaps they
> are doing some connection clean up or something.  One of my
> co-workers noticed that when it gets in this state there are an
> awful lot of connections in a CLOSE_WAIT or FIN_WAIT (can't remember
> which he said, I'll ask him tomorrow). 

All those CLOSE_WAIT or FIN_WAITs are likely the cause of the high
load.  That's the thing I suspected you'd see in the output of
netstat. 

Perhaps the load balancers aren't properly closing sockets, or there
is some sort of firewall rule that's interferring?  Personally, I've
seen some odd behavior on web servers with hand-crafted firewall rules
that were meant to block SYN-related attacks.

-- 
Jonathan Billings <billings at negate.org>