Dr. Conspiracy
Unless you’re interested in the technical aspects of WordPress blogging, stop here.
I tried to get on the blog late yesterday afternoon and I kept getting Internal Server Errors and 404 Page Not Found errors. What this means is that the script processes that feed this web site’s content were running out of memory and getting canceled. This can happen during times of very heavy traffic and (I believe) when the shared server where the site is located has very heavy traffic on other sites.
This site uses WordPress caching technology (WP Super Cache, Widget Cache and Plug-in Cache) so that when the pages don’t change, the software will feed you a copy it created earlier, requiring hardly any computer resources. That’s great in theory. However, this is a heavily-commented blog meaning that every time someone leaves a comment on an article, that page has to be built again to include the new comment.
Back to yesterday. Analysis of yesterday’s site statistics shows strong activity, but not remarkably strong, and not the heaviest of the month. The peak load yesterday was about 11.5 pages a minute (which doesn’t seem all that much) from 5 pm to 6 pm. Still, generating a page with a couple hundred comments on it can take a while, and if a few requests collide they can start a jam just like cars on the highway.
This brings me, finally (!) to what this article is about. Last night at 8:20 pm I made a significant change in how the site works. It’s called FastCGI and it’s a trade off between memory and speed. In theory, it should smooth out the memory usage (albeit at a high level) and make the scripts run faster, making it less likely to jam up.
Normally FastCGI isn’t very helpful when caching is enabled, but on this blog with lots of comments, I think it might help, and perhaps help a lot. FastCGI is technically more complex leading the down side of more things potentially to go wrong.
So if you notice performance today different from yesterday (or not), leave a comment.
I was having those same errors over the past 2 days. I haven’t had any problems today. It seems faster, but I don’t know if that is just due to lesser traffic at the moment or even my own perception bias, by having read this article before really thinking about it.
I’m getting errors when I click on your links from Twitter.
Sorry for the inconvenience. The URL shortening service used to make short article links suitable for Twitter is broken.
Read more at: http://thenextweb.com/2009/06/16/popular-url-shortener-cligs-hacked/
This comment came in email:
Thanks for your comment. The Obama Conspiracy Theories blog runs on a shared server that runs many other web sites. I don’t know how many, but I suppose it is as many as the hosting company can put on it while allowing them to run reasonably well.
In a peak hour, the site processes about one page every 4 seconds (on average). There are over 46,000 comments in the database, though.
If you view page source on one of the pages from this site, you will see at the very bottom something like this:
Showing that the page generation takes about .4 seconds (now that FastCGI is on). One hopes that most of the time, however, you receive a cached page, one already generated, which takes very little server resources.
So I agree with you that site traffic shouldn’t be a problem at the levels this site experiences.
By the way, I found that one of the caching plug-ins I installed was not turned on. This has been fixed.
Actually, HuffPo is running off of multiple servers. I would suspect the most of them are dedicated serves on top of that. But one would expect that of a large online news company with a technical staff. It is not an apples to apples comparison.
Doc, I presume you are aware of this, but it appears that the recent caching changes at your server are preventing updates of the “Recent Comments” section. Previously a page refresh would get the changes, but that’s not working now.
I was not aware of this. The Recent Comments widget is supposed to be refreshed whenever a comment is posted. I’ll look into it.
I noticed the same problem late last night, so it is not just Sef experiencing this.
It seemed that the problem was not consistent either. For instance, I could post a comment on an article and then see my comment appear as the most recent comment while still on that article. However, if I then went to just the main home page and refreshed, I somehow saw an earlier recent comments view, prior to *any* comments I had made.
So yeah, definitely a confusing problem, as Recent Comments are all I have to go by to try to ascertain where there is new info that I haven’t seen yet.
Please let us know when you’ve found/fixed the problem.
@ G and Sef re: Caching issues
Late this morning I installed WordPress 3.0 and I reset some parameters in the caching. Please let me know if this is still happening. In any case, all the pages should get refreshed at least every 30 minutes.