Slow for anyone else?

[haha]

Unbelievably slow this evening- 7pm pst.

Ha

[cute fuzzy bunny]

Consistently good ping times to both. Within 1-2ms of each other.

Shouldnt matter if its one bad one and one good one. Whichever one is bad will have web server logs showing the 500 errors and the root cause.

For the benefit of the users who dont know what a "server 500" error is, that occurs when something goes wrong at the server end, its unable to respond to your http request, so it sends you a "500" error to indicate that its had a problem and cant comply with your request.

[Nords]

Originally Posted by Andy R

Thanks for the feedback, it's very helpful. I will pass it on. If possible, can you try adding a 1 or 2 in the URL when this happens. It might lead to some more info.

Examples:
www1.early-retirement.org
www2.early-retirement.org

The server admins are investigating things right now.

Yeah, that'll keep us all busy while the server admins are getting their stories straight...

[Andy R]

The server admins were able to find some scripts that are crashing fastcgi (php). I now have access to the logs and am keeping an eye on the errors. I think we are going to need to change the web servers from running lighttp to another web server like Apache 2 or LightSpeed.

Please post back here if the servers slow to a crawl again.

[cute fuzzy bunny]

A whole flurry of 500 errors again today and they were on www2. www1 responded just fine. I got about 30 500 errors in a row on www2.

[Rich_in_Tampa]

Uh oh. Slow and a few time-outs this evening for me, too.

[HFWR]

Recent performance has often been slow, with timeouts and 500 errors. Might be my imagination, but the slow loads are at least partly caused waiting for Goodle ads to load...

It's a bit frustrating to be surfing along, and then nothing...

I guess you get what you pay for...

[Andy R]

I am investigating. Also, just as an FYI, after the last round of issues reported earlier in this thread I found some 3rd party monitoring services and have them setup to monitor the site. Here are the public reports for January.

E-R.org WWW1, January 2008
E-R.org WWW2, January 2008

I will speak to the server admins now about the issues and put together a game plan to ditch lighttpd and move to Apache next month. Lighttpd (link) is the web server and it runs in a way that causes the 500 server errors. Apache will not have those types of errors so it should be easier to monitor and restart using automated tools.

[Rich_in_Tampa]

Good to hear, Andy. I have never seen a more stable and robust combination than PHP, mySQL and Apache. That combination is as good as it gets in my experience.

[Andy R]

I went with lighttpd because a guy I was working with was telling me about it's scaling capability. You can set it up to have seperate php processing machines effectively making an application layer. That way when you need more power you can just add a machine optimized to process php. I thought I was going to utilize that feature but it just gets too complex. The switch to Apache is going to happen around the middle of February. I will keep monitoring the servers and see if we can keep things running as well as possible until then.

Do we have any php developers, database admin or other technical members? I am always looking for good people, it's seems that every programmer I know is gainfully employed and do not have any free time for moonlighting.

[BigMoneyJim]

Originally Posted by Andy R

I went with lighttpd because a guy I was working with was telling me about it's scaling capability. You can set it up to have seperate php processing machines effectively making an application layer. That way when you need more power you can just add a machine optimized to process php. I thought I was going to utilize that feature but it just gets too complex. The switch to Apache is going to happen around the middle of February. I will keep monitoring the servers and see if we can keep things running as well as possible until then.

Do we have any php developers, database admin or other technical members? I am always looking for good people, it's seems that every programmer I know is gainfully employed and do not have any free time for moonlighting.

Funny you should mention this. I've been running lighttpd since October 2006 and just switched back to Apache in the past 24 hours for similar reasons. I didn't have the 500 error problems like you've had, but everything is designed for Apache (rewrites, premade .htaccess files) and I got tired of trying to hack lighttpd to work with my php sites.

Also patches and upgrades are easier with Apache since it updates with your OS updates.

I just set up Squid as a web accelerator. It has the great advantage of taking the static file load off of the big mod_php Apache processes so they can deal mainly with dynamic requests. The downside is that logging is different, and some php apps may not play well with it, but vBulletin seems to be set up well for it. In my case I'm scaling down and this helps me run with fewer Apache processes in my limited RAM. It scales up and allows for redundant backends, but then we get into complexity again. Anyway, it's just a thought. You may not need it if you have enough server and enough RAM, though.

Anyway, maybe I'm ready to do some moonlighting. Whatcha think?

EDIT: From a distance, the 500 errors are clearly something in the backend PHP processes. Probably not script errors, so either there are some timeout issues between servers, or processes are segfaulting or something is amiss so that the fastcgi channels are failing.

[Andy R]

I think the 500 errors are happening when a php fastcgi process uses up all the RAM that's allocated to it, it then locks up and stops processing the php script. The web server stays running but throws out the error. Anyway, I look forward to switching back to Apache for all the reason you stated Jim. Everything is written for apache and the extra work needed to run lighttpd is just not worth it. I think Apache 2.x makes big improvements also.

[BigMoneyJim]

Originally Posted by Andy R

I think the 500 errors are happening when a php fastcgi process uses up all the RAM that's allocated to it, it then locks up and stops processing the php script. The web server stays running but throws out the error.

That would certainly do it. If the data stream isn't finished the frontend server has no choice but to reply with a 500 error.

You can increase the PHP memory per process in the php.ini file. On my server it's in /etc/php4/cgi/php.ini and was set to 8MB by default. I had to up it to 16MB for my software.

If that's where the problem is then Apache will have problems, too, but in that case the file (for my server, anyway) is /etc/php4/apache2/php.ini.

memory_limit = 8M      ; Maximum amount of memory a script may consume (8MB)

A quick search tells me vBulletin people are setting this at 16MB, 32MB or even higher, although I'd be surprised if 32MB weren't too much.