There was no leap second added at the beginning of 2004!
This is a sign that the Earth's rotation is currently a little faster than expected. In modern times, a leap second is needed every 500 days or so. This keeps our atomic clocks in sync with the earths rotation, which is slightly longer than 24 hours.
But the last leap second we had was at the beginning of 1999. Scientists are a little puzzled by this. In general the earth is slowing down it's rotation. Some theories to account for the current speed up are: position of other planets in our solar system, and current weather changes.
Maybe Global Warming is causing more change to our Earth than I thought!
References / more reading:
International Earth Rotation Service (Bulletin C)
About the Earth's rotation and the Moon slowing it down