Dark energy and I came of age together. In 1998, two competing groups of astronomers discovered that the expansion of the universe was accelerating; the same year, I graduated from college with a degree in physics and began work on a Ph.D. Though much of my research chased down other paths, I kept returning to dark energy. Call it a 16-year on-and-off-again dysfunctional relationship with no end in sight.
Of course, I'm very much not alone in trying to understand dark energy: It stands as one of the central problems in modern cosmology. Even the name “dark energy” is a placeholder for our ignorance, representing the fact of cosmic acceleration without indicating its identity. Astronomers have measured the acceleration rate and determined that dark energy constitutes more than two-thirds of the energy content of the cosmos, but its identity has defeated theoretical physicists.
Despite dark energy’s magnitude, astronomers didn't know about its existence until recently because its effects are subtle. It doesn't noticeably affect the planets in the solar system or the motion of stars in the galaxy. That subtlety enshrouds it in mystery: Scientists are busily determining what dark energy does, and they have yet to reach any consensus on what dark energy is.
Because dark energy doesn't correspond easily to anything in the standard toolkit of physics, researchers have been free to be creative. The result is a wealth of ideas, some that are potentially interesting and others that are frankly nuts. Some string theorists propose that our observable universe is the result of a vast set of parallel universes, each with a different, random amount of dark energy. Other physicists think our cosmos is interacting with a parallel universe, and the force between the two drives cosmic acceleration. Still others suspect that dark energy is a sign that our currently accepted theory of gravity—Einstein's general theory of relativity—is incomplete for the largest distances.
Despite the crazy proliferation of solutions, scientists are not completely in the dark on dark energy. Observations of supernova explosions in distant galaxies seem to indicate that cosmic acceleration hasn't changed in strength over time, and it seems to act at the same rate in all parts of the cosmos. While there's still some wiggle room for small changes over long time spans and distances, the observed constancy of dark energy already rules out some explanations as to its identity.
Dark energy behaves very differently from both ordinary and dark matter. (Dark matter is arguably a much larger mystery, but despite the similar names, they don’t seem to have anything to do with each other.) Matter mutually attracts via gravity, forming galaxies and other structures, while dark energy doesn't seem to “clump” in any way. Even though matter drives cosmic expansion simply by possessing energy, its overall effect on expansion is to slow things down.
Dark energy, by contrast, seems to operate on a feedback cycle. The more the universe expands, the more dark energy there seems to be, and the more dark energy there is, the more it drives expansion. That characteristic indicates dark energy is not created by any known type of particle. Its influence grows as the universe ages, while matter's effects dissipate. If trends continue, the galaxies we see today will grow so far apart as to be invisible from one another in the far future. That would bring an end to astronomy as we know it. Assuming our species (or evolutionary offspring) can survive beyond the end of Earth, any descendants in the distant future would only be able to see vast expanses of blackness beyond the outskirts of the galaxy.
So if dark energy isn't anything normal, what is it?
The most popular candidate is vacuum energy: the energy from quantum fluctuations in empty space. This idea was first kicked around in the 1970s and '80s, long before cosmic acceleration was discovered. Physicists recognized that the same stew of quantum processes that determine the properties of electrons and other particles would grant energy to empty space. From general relativity, any energy has a gravitational effect. In this case, the energy would serve to accelerate cosmic expansion.
The effect of vacuum energy can easily be incorporated into Einstein's theory as something known as a “cosmological constant,” an extra gravitational factor not created by matter. The problem: The amount of energy predicted by quantum theory is much larger than the observed amount of dark energy, something researchers knew even before the discovery of cosmic acceleration.