Further problems with the DB server (updated)

13 Jan 2007 14:19 (Edited: 13 Jan 2007 14:19)

Had to take the secondary DB server "out of action" today, because the MySQL relay log had corrupted during an unclean reboot, and there wasn't any feasible way of fixing it quickly enough.

After the server was taken offline a database dump was taken from a development database server (which contains data identical to the master server, but cannot be used for production use), and transferred to the secondary production server. Unfortunately the problems with the HDD caused the server to lock up when trying to restore data from the dump, so the HDD replacement that was scheduled for later, had to be done immediately.

As I'm typing the HDD is being replaced by the techs at our ISP. After the HDD swap the RAID array needs to be re-built on the server, so it'll be a while before the server is online, and even a longer while before the database dump is restored, and the server is ready to handle queries again.

Unfortunately a single DB server cannot cope with the demands of all our sites that well. Even though the queries are mostly SELECTs, an odd UPDATE or INSERT can lock a table for long enough to cause, for example, forum thread views to take long to show up or timeout altogether. The amount of queries is reasonably low thanks to heavy caching (both query caching and file caching). The database server currently handles around 200 queries per second.

Update
The server is now back online, and uncompressing the dump. The cable and the PERC 4 RAID Adapter on the server were swapped instead of a single drive in the array. That's because the errors given by the server related to the entire array -- not a single disk.

Update 2
The database restore is now complete, and the secondary server should be back in action shortly.

Tags: afterdawn  database  hdd  maintenance  problems  server 

List of my blog entries

• Power consumption of common home electronics devices (19 Nov 2007)
• Downloads more accessible to users with firewalls (25 Jun 2007)
• More download capacity added (28 Feb 2007)
• Fixed problems with blog image scaling functions (20 Jan 2007)
• Yesterday's problems with the site (12 Jan 2007)
• Bypassing lock code on Nokia phones (27 Dec 2006)
• iptables and vsftpd with ssl (25 Sep 2006)
• Replying to feedback mail (24 Jan 2006)
• Windows XP HDD stuck in PIO mode instead of DMA (13 Jan 2006)
• Problems with Windows Update (07 Jan 2006)
• auditd causing crond (cron) and sshd (ssh) to lock up (06 Jan 2006)
• Squeezebox under the microscope (25 Nov 2005)
• And so it begins... (10 Nov 2005)

User comments

(No comments made)

Post your comment

In order to post your comments here, you need be logged in to our system. Simply follow this link in order to login and to post your comments here.