We are Stable Now! Last 14 hours have been Hell! Thanks for Being Patient :)

May 21, 2009 by Abbas

(No Ratings Yet)

Loading ...

TNerd It started around 11 PM last night when I started approving comments for the day. I started approving comments and realized that the site crashed.

TNerd is hosted on Media Temple’s DV-Extreme with added RAM the server mostly holds up against the traffic it experiences, thanks to WP-Supercache. Once the server crashed I immediately rebooted the machine which usually solves the issue but it did not.

Logged into my SHELL (SSH) to see which process is eating away all the resources and I noticed MySQL shooting up to 600% of the CPU power. Called the Media Temple (MT) guys and they initially suggested me to upgrade RAM which should resolve the issue, I did add more RAM but no luck!

Called MT guys again and started working with Robert to solve the issue, we together zeroed down on the issue which was causing MySQL to consume all the resources.

This query “SELECT c . * , p . * FROM wp_comments c INNER JOIN wp_posts p ON p.” with query status “Sending Data” was jamming the database by queuing up, up to 50 instances running for over 1500 seconds per instance.

So we figured out what the problem was but I simply had no idea what exactly was causing it or how to resolve it. I tried “Flush Tables”, etc in MySQL but no help.

I put up a new database and installed the latest backup and pointed the domain to the new database. So far things seem to be normal MySQL is consuming between 0-12% of CPU resources.

I still don’t know what really went wrong but one thing I realized is that it is extremely important to have a hosting service with good technical support and what’s even more important is backing up the database.

TNerd database is backedup every single day as new posts and comments are published on the site every day. I use WordPress Database Backup plugin to take schedule backups.

I would like to thank our readers for being patient, sending e-mails and try to reach me over to phone to let me know that the site is down or to know what is going on. Let’s hope this does not happen again!

UPDATE:

It happened again and continued to happen everytime I approved comments. The next logical thing to do was to check on the plugins and disable the ones causing trouble, and I did every other thing but that.

Eventually I got to the plugins and figured that a plugin in use, called "Comment Relish" is causing the problem. Removed the plugin and problem solved!!