Skip to main content

Computer Clocks Cause More Issues

Two nights ago, a leap second was added to system clocks running Linux, causing much-undesired havoc.

On July 1st at 12:00 AM UTC, both of my Amazon EC2 instances fired an alarm indicating high CPU usage. I investigated to find that it was MySQL that was eating all of the CPU. I logged in and ran SHOW PROCESSLIST to find that no queries were running (these servers don't get hit much after business hours). I stopped MySQL, CPU utilization dropped back down to 1-3% (as normal). I restarted MySQL, and it started eating a lot of CPU again. Then, I restarted the server (shutdown -r now), and the problem went away. Both servers had the exact same problem (running Ubuntu 12.04 LTS).

In my particular case, MySQL began eating CPU, even after being restarted.  It was a livelock.

The only relevant item I saw in the syslog was:

Jun 30 23:59:59 hostname kernel: [14152976.187987] Clock: inserting leap second 23:59:60 UTC

Oh yeah... leap seconds.  Those are super important... so important that we need our servers to freeze up.

http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/
http://it.slashdot.org/story/12/07/01/1920217/leap-second-bug-causes-crashes
https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=6b43ae8a619d17c4935c3320d2ef9e92bdeed05d
http://stackoverflow.com/questions/11294582/mysql-eating-80-100-cpu-starting-july-1-midnight-a-day-or-so-following-a-ebs-c
Labels:

Comments

Post a Comment

Popular posts from this blog

Beware the Ides of March...in 9 days

Stupid heading for this blog, but whatever.  I was amused.   So, a lot has happened since my last entry, which I believe was sometime in January.  I have officially started a new business -- OnlineFixShop, LLC.  The web address is http://www.onlinefixshop.com/ .  Check it out!  For the next few months, my business will be focusing on home PC repair.     I am offering services that can help you:   Rid your computer of spyware and viruses Retrieve lost information and data Gain access to the Internet Increase your computer's performance and speed Learn your way around various types of software Setup a secure wireless or wired network Back-up personal and valuable data Secure your computer(s) and protect your data Eventually, I am planning to focus on repairing computers over the Internet using remote administration technology, which I have yet to design.   Right now, I am working to setup an onlin...
1 comment
Read more

JavaScript Sticky Footer and Scroll Effect

This post talks about two different HTML/JavaScript effects: How to keep a page footer stuck at the bottom of the browser window. How to create a scrolling <div> without using a scroll bar OK. So... you have a website. You want a header stuck at the top of your page and the footer stuck at the bottom of your page. The stuff in the middle, you want to be able to scrollable. But, you don't want those ugly scrollbars to the right of your scrollable text. Maybe, instead, you'll have up arrows and down arrows above and below your <div>. When you mouseover the arrows, the text in the <div> will move up or down and create a scrolling effect. Suppose your page looks like this... <html> <head> <title>Test</title> </head> <body> <div style="position: relative; width: 700px; margin-left: auto; margin-right: auto;"> <div id="header">Header</div> <div id="scrollUp...
1 comment
Read more

Today's Quote

This is simply a brain dump.  I'm sleepy, and I want to type out some of the thoughts currently in my head. "Luck is where preparation and opportunity meet."  This is so remarkably true, and today I'm making it a mantra.  I believe that luck is merely an illusion that we perceive, but it truly when we have prepared ourselves for the right opportunity... and then a great opportunity comes along.  Many great opportunities pass us by every day.  Once we begin to recognize them and prepare for them, then we start to experience the thrill of luck. Interestingly, as described in "Good to Great", Mr. James Collins talks about how "Level 5" leaders often attribute their great success to luck .  That's a humble way of saying, "I planned on taking advantage of every opportunity ."   Hmmm...
1 comment
Read more