Skip to main content

Computer Clocks Cause More Issues

Two nights ago, a leap second was added to system clocks running Linux, causing much-undesired havoc.

On July 1st at 12:00 AM UTC, both of my Amazon EC2 instances fired an alarm indicating high CPU usage. I investigated to find that it was MySQL that was eating all of the CPU. I logged in and ran SHOW PROCESSLIST to find that no queries were running (these servers don't get hit much after business hours). I stopped MySQL, CPU utilization dropped back down to 1-3% (as normal). I restarted MySQL, and it started eating a lot of CPU again. Then, I restarted the server (shutdown -r now), and the problem went away. Both servers had the exact same problem (running Ubuntu 12.04 LTS).

In my particular case, MySQL began eating CPU, even after being restarted.  It was a livelock.

The only relevant item I saw in the syslog was:

Jun 30 23:59:59 hostname kernel: [14152976.187987] Clock: inserting leap second 23:59:60 UTC

Oh yeah... leap seconds.  Those are super important... so important that we need our servers to freeze up.

http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
http://it.slashdot.org/story/12/07/01/1920217/leap-second-bug-causes-crashes
https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=6b43ae8a619d17c4935c3320d2ef9e92bdeed05d
http://stackoverflow.com/questions/11294582/mysql-eating-80-100-cpu-starting-july-1-midnight-a-day-or-so-following-a-ebs-c
Labels:

Comments

Post a Comment

Popular posts from this blog

Developing a lightweight WebSocket library

Late in 2016, I began development on a lightweight, isomorphic WebSocket library for Node.js called ws-wrapper .  Today, this library is stable and has been successfully used in many production apps. Why?  What about socket.io ?  In my opinion, socket.io and its dependencies are way too heavy .  Now that the year is 2018, this couldn't be more true.  Modern browsers have native WebSocket support meaning that all of the transports built into the socket.io project are just dead weight.  On the other hand, ws-wrapper and its dependencies weigh about 3 KB when minified and gzipped.  Similarly, ws-wrapper consists of about 500 lines of code; whereas, socket.io consists of thousands of lines of code.  As Dijkstra once famously said: "Simplicity is prerequisite for reliability." ws-wrapper also provides a few more features out of the box.  The API exposes a two-way, Promise-based request/response interface.  That is, clients can request dat...
Post a Comment
Read more

Wedding Prediction - October, 2013

Carla and I are planning on getting married sometime in October next year.  We need to pick a date, and that decision may  involve some science and mathematics.  :) For example, we want the weather to be nice.  To be more precise, we'd like the high temperature for the wedding day to be between 60 and 80 degrees Fahrenheit.  Obviously, we have both lived in Ohio our entire lives, and we have a pretty good idea of what the weather will be like.  We both hypothesised that October was a "hit or miss" sort of month; it could be cold, or it could be nice. But, for me, a simple hypothesis was not enough; I really wanted to know the probabilities of decent weather based on historical weather data.  Many websites on the Internet (i.e. almanac.com) charge you to review historical weather data, but Carla and I discovered a cool page on cleveland.com that provided exactly what we wanted.  I loaded the historical temperature data from 1903 to 2011 f...
2 comments
Read more

Node.JS + MySQL + Transactions

If you're like me, then you are probably building web applications using Node.JS and MySQL (and maybe Redis, too).  If so, you're probably going to need transactions, and you've probably already noticed that the current version of node-mysql doesn't support transactions yet.  :( But that's OK because I have a solution for you.  Check out node-mysql-queues on github .  This project provides pretty good support for MySQL transactions with a fairly simple API.  There are a couple of things to remember, though.  For one, Node.JS is very "callback-centric," so when executing a series of queries, you would normally chain the queries together with a series of callbacks.  node-mysql sort of changes this model, by allowing you to place queries on a queue to be executed in order.  If you only care about doing something when all of your queries are done, you can simply put your callback in the final query.  node-mysql-queues allows you to do the same...
Post a Comment
Read more