homecomputerlab

I’ve recently been experimenting with Amazon AWS services having spun up a few EC2 micro instances and registered for Amazon Route 53 DNS services. The idea is to have my home RaspberryPi serve up my website in the first instance and if for whatever reason my RaspberryPi is not available Amazon Route 53 DNS failover will kick in and show my website from an EC2/S3 instance. Whilst playing around I noticed that my EC2 mirco instance would often fail at the first hurdle with MySQL frequently crashing after ~2days uptime.

After examining my log files I found the following entries:

cat /var/log/mysql/error.log

InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: Completed initialization of buffer pool
InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...

Not very telling about what has gone wrong other than mysql seems to be recovering from a crash of some sort.

cat /var/log/dmesg

Contained the following which seems to give a little more insight into the problem:

[1296220.616387] Out of memory: Kill process 12685 (mysqld) score 33 or sacrifice child
[1296220.616416] Killed process 12685 (mysqld) total-vm:1448560kB, anon-rss:256800kB, file-rss:0kB
[1296221.075058] init: mysql main process (12685) killed by KILL signal
[1296221.075159] init: mysql main process ended, respawning
[1296252.243501] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0,

Indeed after a quick google search it seems to be a relatively common issue for mysql instances to fall over running on Amazon EC2 micro instances, the chief reason for this being that EC2 instances are provisioned without any swap space configured and out of memory errors then occur.

The recommended and also the quickest fix is simply to provision some swap storage space on your server, which can be accomplished with the following commands:

Provision some space for your swap file:
dd if=/dev/zero of=/swapfile bs=1M count=1024

Setup the swap file:
mkswap /swapfile

To enable the swap file immediately but not automatically at boot time:
swapon /swapfile

To enable it at the boot time, add the following entry into /etc/fstab:
/swapfile swap swap defaults 0 0