358 | Solo: Vacuum Energy and the Cosmological Constant

0:00:00.1 Sean Carroll: Hello, everyone, and welcome to the Mindscape podcast. I'm your host, Sean Carroll. In physics, which is a very broad field, it includes atomic physics, plasma physics, condensed matter physics, biophysics, particle physics, gravity, cosmology, all these things. Different fields, of course, have different rates of progress, both because of theoretical ideas and experimental input. If you focus just on fundamental physics, and let's not argue about what that means or should mean, I take it to mean figuring out the most fundamental laws of nature the best we can. So not including things like biophysics and condensed matter physics, not because they're not important, it's just a different thing. I once tried to get the field to change the name fundamental physics to the name elementary physics, the most elementary laws of physics being the idea. But no one agreed with that. That's okay. Fundamental physics is here to stay.

0:00:54.5 SC: In this subfield of fundamental physics, which includes sort of particle physics and gravity, things like that, there haven't been a lot of helpful, experimental, surprising discoveries in the past several decades. Arguably, since the 1970s, there have been a few here and there, masses of neutrinos, things like that. Also things that we expected, like the Higgs boson, gravitational waves. But in terms of true surprises, it's been few and far between. That makes it hard to make progress in fundamental physics because experiments are what drive us to great ideas. And there's one, of course, shining counterexample to this idea that there haven't been many surprises, which is the acceleration of the universe. Back in 1998, we were told by results from two competing astrophysics groups looking at type 1a supernovae, using them to measure the expansion rate of the universe and how it changed over time, that against all expectations, the universe is actually accelerating, not decelerating, in its expansion rate.

0:02:00.0 SC: And of course, this was a big deal. Nobel prizes were handed out and the whole bit. We still don't understand with complete confidence what is going on with the acceleration of the universe, which is great for fundamental physicists such as myself, for theorists who are trying to figure out what's going on. It would be nice if we had more than that one fact, the acceleration of the universe. It would be helpful to our project of inventing theoretical models to explain it, but it doesn't slow us down. We're gonna invent them anyway. There is a leading candidate, the cosmological constant, as proposed by Einstein many, many years ago. But there's other ideas as well. So recently, through a set of circumstances that I won't go into, I fell behind personally in my recording of podcasts and I'm doing traveling and things like that. And so even though I just did a solo podcast on quantum mechanics not that long ago, I figured I can fill in by doing some more solo podcasts.

0:02:58.3 SC: I figured just one solo podcast, but what you're gonna do is you're gonna get two and two separate weeks. What happened was I went to Patreon where I sometimes turn for advice from my Patreon listeners. If you're ever curious about what it would be like to be in the cool in-crowd of Patreon listeners, just go to patreon.com/seanmcarroll and you can sign up. And I asked them what would be a good topic for a solo podcast, and I gave them some suggestions. And I think that a pretty clear tendency, even though there's a lot of diversity in the answers, was for the suggestion of theories of dark energy and the accelerating universe. And in some sense, it's shocking I haven't done that already. This is something that I do know something about. Indeed, my first ever attempt at publishing a trade book, at writing a popular level book and getting it published, was I proposed a book on the accelerating universe and dark energy. And no one wanted to publish it.

0:03:57.3 SC: This was like 2006 maybe, and people were already like, "Well, we've done that already. That's old news. 1998, 2006, the ship has sailed." So, I don't think that was right. I think there's plenty of room for a really good book on that topic, but I never did write it. So, plenty to say about that, and I thought that was a very good suggestion. Choice on the part of my Patreon listeners. So I thought, sure. And then I started sketching it out and I realized, there's actually too much to say about this topic. So I've broken it into two episodes. I know that I have in the past gone on at great length in solo episodes, but I do think that there's sort of a natural dividing line here. You can talk about the cosmological constant as Einstein came up with it and as we really wondered about what it means in relationship to quantum field theory and what the observations are telling us about it. And then you can go beyond that to what if it's not a cosmological constant? What if it's something else? What if it's something dynamical, a modification of gravity or something like that?

0:04:57.6 SC: So those will be the two solo podcasts. I'm gonna talk in this one, that you're listening to right now, about the cosmological constant, the energy of the vacuum, because that's an equivalent idea. They're precisely identical. And then I'm going to do another one on dynamical dark energy, which on the one hand contains a whole bunch of really exciting, interesting ideas and would be a tremendously exciting discovery were we do find evidence for it. On the other hand, there is no evidence for it right now, so it's still completely speculative. We're a little bit less constrained in that part of the world, so we can let our imaginations roam. It's all great physics, it's all cutting edge, it's all very perplexing and confusing. And that's exactly what you like as a working physicist. You want to have problems you don't know the answer to, struggling to figure out what those answers are. So I'm gonna try in these two podcasts to sort of let you in on both what the questions are and why certain answers look the way they do, certain other answers are less promising, things like that, hopefully to give you a roadmap so you can understand better what we're trying to do in cosmology and fundamental physics with this problem and what might happen next. So with that in mind, let's go.

0:06:30.7 SC: When we're talking about the cosmological constant, the energy density of the vacuum or whatever, there's a very obvious starting point for us, which is Einstein. He's the guy who came up with the idea back in the day. And it provides a fun excuse to think historically about how these ideas came about, because it's always useful to think historically because we have different preexisting ideas than they had back then. We have different things that we think are really important, and other people in different historical eras might not have cared about that much, or they might have had other things that they care about that we don't care about very much. So that's very much the case for Einstein, who was absolutely not the shut up and calculate kind of guy. He was someone who had very strong ideas about how things should be, and he thought them through in a very rigorous and careful way. But he was open to changing his mind and realizing that he was wrong, but he definitely had a style that opened his mind to different possibilities.

0:07:34.3 SC: So you picture that around 1915, Einstein had put the finishing touches on the general theory of relativity. The general theory of relativity says that spacetime, which was an idea that went back to Minkowski right after Einstein had figured out special relativity in 1905, his old professor Minkowski had pointed out that it was better formulated as a theory of a unified spacetime. And Einstein didn't like that suggestion by Minkowski, but he eventually came around to realize that was a good idea and in fact, that we could incorporate gravity into the ideas of relativity by allowing spacetime to have a geometry, by allowing it to be curved. And that was the basis for his theory of general relativity. So that's 1915, and I just can't even imagine how much fun it must have been to be a physicist at that time and to have this new playground to run around in and explore new things. Of course, Einstein had the head start on everybody else, so many of the fun ideas that one could explore in the context of general relativity were first explored by Einstein very soon after he invented the theory.

0:08:49.6 SC: He didn't do everything, like the Schwarzschild solution. The metric tensor, the curvature of spacetime for something like the solar system or a black hole or whatever, was found only two years after Einstein invented the theory. But Einstein himself didn't find it. It was Schwarzschild, of course, who did it. One thing that Einstein did do was to think about the universe as a whole. And now it's kind of interesting. This is why I mentioned the philosophical predispositions that different people have. Einstein was heavily influenced by Ernst Mach, who was a physicist and a philosopher of physics, to be honest, in Germany at the time, who had very definite ideas about the relationship between space and time and matter and things like that. I would say that Mach's influence on Einstein was maybe inspirational, like maybe he nudged Einstein in the right direction, to say some good things. But ultimately, the answers that Einstein came up with are completely disconnected from any of Mach's ideas. And it's a weird historical thing that people keep trying to fit it in.

0:09:56.1 SC: People keep trying to have Mach's principle, which connects the rest frame of the universe to the existence of matter in the universe, to general relativity, Einstein's theory, even though they're not really actually very closely connected. But I bring this up because Einstein thought that they were closely connected. And when he first turned his attention to the universe as a whole, he was very influenced by Mach. And there was also in the back of Einstein's mind, and people today forgot about this, but there was this puzzle that people knew about in Newtonian physics, in Newtonian gravity. The puzzle is the following. You can sort of imagine solving Newton's equations for gravity using this way of writing things that's actually due to Pierre-Simon Laplace in terms of the gravitational potential. So rather than just saying that the sun reaches out 93 million miles to the Earth and pulls it via the gravitational field, you can say... Well, I shouldn't even say via the gravitational field, because that's not how Newton talked, via the gravitational force, via action at a distance, in Newton's way of thinking about it.

0:11:09.8 SC: Laplace replaced Newton's equations with a field equation. And then there's no action at a distance. The sun is pushing the gravitational potential field, the Earth moves in it. All that makes perfect sense if you're talking about the solar system. But it was puzzling when you try to talk about the universe as a whole, because it turns out that if you just imagine a universe filled with an absolutely constant density of matter, it is undefined what potential you should have. There are multiple solutions that work equally well. You can add a constant to it, things like that. And it didn't seem to be a good starting point for investigating the universe. So Einstein had that in mind when he started doing relativistic cosmology. And therefore, rather than just assume that the universe was spatially flat, because remember, the whole point of general relativity is it's the geometry of space and time that defines what we think of as gravity, Einstein said, well, what if space is finite? What if space doesn't go to infinity?

0:12:15.3 SC: Then you can both help understand how to get out of this puzzle in Newtonian gravity and you can be consistent, maybe, he thought, with Mach's principle, that it's the matter in the universe that is fixing what we think of locally as our standard of rest and our inertial frames and things like that. So Einstein had these ideas and he was pretty convinced, as far as I can tell. I'm not a super expert on the history here, so maybe someone can correct me. But as far as I could tell, he thought that there were just really good philosophical reasons to think that space is finite, not infinite. Personally, myself, I'd rather be open-minded about that. I don't have any strong preexisting opinions about whether space is finite or infinite. I'll let the data or a better theory decide that. But anyway, he tackled the problem of cosmology and he said, look, let's just start by assuming that matter is distributed uniformly throughout the universe.

0:13:13.0 SC: He had no idea whether that was true or not. There was some indication from cosmology circa the 1910s that maybe there was some uniform distribution of matter in the galaxy, but we didn't even know in 1915, '16, '17 about the existence of other galaxies. That idea didn't... Well, the idea was there, but the knowledge, the data didn't come along until the 1920s. So it was sort of open season for speculating. There wasn't enough data to really tie you down. So Einstein said, okay, let's imagine a uniform universe, one where the density of matter is statistically the same everywhere. He knew the density of matter is not the same on the Earth as in the sun and things like that. You're imagining that you're thinking about very, very large scales throughout the universe and the average density of matter could be the same everywhere. And he said, okay, what if I take a spherical universe... And by spherical, of course, we know that space is three-dimensional on large scales. So that means a three-dimensional sphere. That's no problem to differential geometry and general relativity.

0:14:21.1 SC: So space in Einstein's model is a three-dimensional sphere and time is just a line. Time goes from wherever it goes on into the future. And he said, okay, I'm going to plug this into my equation called Einstein's equation, and I'm going to ask what that implies for the evolution of the universe. A constant density in a spherical universe. And what he found was that two things happen. You plug into Einstein's equation and you get an equation for either the expansion or contraction of the universe. Space wants to change its size. That's not really something that was very well defined in the Newtonian point of view, but in general relativity it makes perfect sense. You have a three-dimensional sphere, and the three-dimensional sphere can get bigger or smaller. And it turns out that both that geometry of space and the existence of matter contribute to this change in the size of the universe. And you can find a solution where at one moment of time, the universe is not changing size.

0:15:25.0 SC: So you can set the sort of the velocity of the universe, if you want to call it that, to zero by balancing the curvature of space versus the matter. What you can't do is keep it there. So basically, what Einstein realized is that the equations he was looking at kind of implied a universe that would start from somewhere. It was all very vague at the time. We're talking about 1917 or so. Start from somewhere, expand, and then reach a point of zero expansion rate, but then just start collapsing again. This is very familiar to modern cosmology students. This is a closed, finite-time universe that we can absolutely think about. Doesn't seem to be the universe we live in, as far as we know. But as far as Einstein knew, he wasn't sure. So he seemed to find... Again, you could find a solution where the universe is not expanding, but it would instantly start either expanding or contracting, depending on how you set up the different numbers. So that was puzzling.

0:16:27.3 SC: And there's this somewhat of a myth that says that Einstein was just philosophically devoted to the idea of a static universe. As far as I can tell, that's not true. What is closer to the truth, though, again, I'm not a historian here, is that Einstein asked his astronomer friends, he said, wait, is the universe expanding and no one told me? And they said, no, no, no, it seems to be constant, seems to be more or less static as far as we can tell. Again, they didn't know there were other galaxies. They were just looking at stars within our galaxy. So Einstein said, I'm going to listen to the data. And he went back and said, can I find a different solution to the equations where the universe is not expanding or contracting? And to do it, he changed his equation. He added a new term to Einstein's equation for the spacetime geometry in general relativity. These days, you might go like, well, that was a little bit of a panic move there, Professor Einstein.

0:17:25.7 SC: You didn't stick by your guns. You went and changed your equation right away. But you have to realize it was Einstein who invented the equation, it wasn't sacred to him. It hadn't been tested and put through decades of hoops that it had all successfully passed. He had written it down. He can write down a new one, why not? It was a very adventurous time in theoretical physics in that sense. So he added a new term to the equation, and he didn't do it just willy-nilly. He said, well, what am I allowed to add to this equation? There's, of course, famously the metric tensor in general relativity. If you want to know more details about some of this general relativity stuff, please pick up my book, "The Biggest Ideas in the Universe, Volume 1: Space, Time, and Motion" where we explain what all these things mean. And he realized that he could just add a term that was proportional to the metric tensor. The metric tensor is what tells you distances and times in the spacetime geometry that you have in general relativity.

0:18:24.1 SC: There's absolutely nothing wrong with adding such a term. It's 100% compatible with the principles of relativity, with the symmetries, with all the language that he had, the notation, et cetera. So it's fine, you can just add it. And there is a constant. There's a new constant of nature, which Einstein called the cosmological term, or we would call the cosmological constant. So he added it to his equation. And very roughly speaking, very informally, the effect of the cosmological constant, if this number, the cosmological constant, is positive, it resists the tendency of matter to pull the universe together. It sort of pushes the universe apart. So by cleverly choosing the value of the cosmological constant, Einstein could find a solution to his equations where space was a three-dimensional sphere, matter was uniformly distributed, and there was a positive cosmological constant that kind of resisted the gravitational field of the matter to give you a static universe over time.

0:19:28.8 SC: And indeed, this solution, which is a perfectly good exact solution to Einstein's equations, is now known as the Einstein static universe. Now, almost right away, people realized this was not a good idea. For one thing, people talk about it as if the Einstein static universe is unstable. That's a little bit not quite accurate. I mean, I get it. I know what they mean. It means something real. See, what you're doing is you're balancing the density of matter versus the amount of the cosmological constant, and you have to balance it exactly. If there was a slight mismatch between the proper value of the matter versus cosmological constant, the universe could be almost static for a long time, but ultimately you begin to expand or contract. That's not quite an instability, but that's just pointing out that any deviation from this very, very precise number, which had no good reason to be true, would lead you to a non-static universe.

0:20:30.1 SC: There's a better objection, which I don't know whether anyone raised at the time, which is the universe is not perfectly smooth. As I said, Einstein knew that the density of matter on Earth is different than in the sun or in interstellar space. And so at best, this assumption that the matter density of the universe is constant is an approximation. It can't be exactly right. There are inhomogeneities in the universe, and those inhomogeneities will generally grow with time if you solve the equations. Overdense regions, regions in which there's a little bit more matter than average, will pull more matter onto them. And that. And that is an instability. Denser regions grow more dense, less dense regions grow less dense, and the universe becomes lumpier and lumpier over time. The only reason that's a problem... I mean, we think that's exactly what does happen in the real world. That's the origin of structure in the universe.

0:21:25.6 SC: But in the modern world, we know that that has only been going on for a finite period of time because there was a Big Bang about 14 billion years ago. Einstein, remember, was proposing a solution which was supposed to last for infinity years. There was no Big Bang in his model. So why hasn't all the structure already formed? Why isn't the universe sort of arbitrarily inhomogeneous? Was a question for him that he wouldn't have been able to answer. So anyway, those sort of puzzles became a little bit less pressing 10 years later or 12 years later, 10 years later, actually, I think, when Edwin Hubble and others put forward the data that said that the universe is not static after all. I mean, Hubble actually did two things. First, he showed that these little fuzzy nebulae that people were taking pictures of in their telescopes are, in fact, whole separate galaxies. They're very, very far away.

0:22:19.3 SC: Then he compared the distances of those galaxies to their redshifts and showed that the universe is expanding. He was very cautious about interpreting these data in the light of general relativity. He was an observer at heart, not a theorist. So he just said, "I have a relationship between redshift and distance." And of course, all the theorists like Einstein, but also people like Lemaître and de Sitter and others instantly knew that what he was seeing was the expansion of space. So there's some controversy in the historical literature about whether Einstein said that this was the greatest blunder of his life to have invented the cosmological constant rather than just saying that his theory predicted that the universe should be expanding or contracting. He could have predicted the expansion of the universe, and then he could have become famous, who knows? But we don't know.

0:23:12.7 SC: That was a story that was apparently told by George Gamow, who liked telling stories. I think it might very well be true. Einstein certainly did say that he should get rid of the cosmological constant once he realized the universe was expanding. You don't need it. The whole point of it was to make the universe static. And if the universe is expanding, you can do that without any cosmological constant there. The problem with that move, the problem with saying, "I don't need it anymore," is who cares whether you think you need it anymore? It's there as an open possibility. Like I said, it was a perfectly legitimate addition to the equations. So the scientific thing to do is not to say, "I don't need it, I'm going to set it aside," but to say, "Is it there?" Ask the question: what would the effects of this weird thing be? How would we know it? Can we put either observational limits on it, or could we even detect it? Or something like that.

0:24:12.6 SC: And really ever since then, ever since Einstein suggested it, there was a sort of a cycle of phases of people suggesting that the cosmological constant could help with this particular problem that we have in observational cosmology and then realizing, "Oh, we don't need it," and it would go away. And so it waxed and waned quite a bit. Okay. So that was the situation. And honestly, cosmology was not at that time, 1920s, even in '30s, '40s, '50s, maybe '60s, not the most respectable part of physics. The ratio of theory to data was quite large, and so it was hard to make progress, just as it's hard to make progress right now in all sorts of questions in fundamental physics. And so people moved on to other things. The really good physicists were thinking about quantum mechanics and particle physics and things like that. And so let's talk about that. Let's talk about quantum mechanics and particle physics and things like that.

0:25:08.8 SC: It was very quickly realized, well, I shouldn't say very quickly, pretty quickly realized by people like Lemaître, Georges Lemaître, a famous cosmologist and really the pioneer of the idea of the Big Bang theory. By the 1930s, Lemaître had thought about the cosmological constant, and he had realized that you don't need to think of the cosmological constant as a change in Einstein's equation. You can just include it in terms of the original equation that Einstein wrote down back in 1915. And this is a point that I think is really perfectly transparent, but people sometimes fumble it, so it's worth getting exactly right here. When Einstein wrote down his equation in 1915, there's a left-hand side and a right-hand side. The left-hand side are some symbols. I'll tell you what the symbols are. It's R_μν - 1/2 R g_μν. These are symbols that are constructed from the metric tensor and the Riemann tensor, and they all have to do with the curvature of spacetime.

0:26:14.8 SC: And then there's the right-hand side to Einstein's equation where you see 8πG T_μν. And 8 and π and G are all constants of nature. There's also like a 1 over c^4 or something like that, but I don't ever remember where the speed of light goes because it's set equal to one in my brain. So to me, it's just 8πG. G is Newton's constant of gravity. It goes back to Isaac Newton and his famous inverse square law. 8 and π are numbers that you are familiar with. T_μν is the energy tensor, the stress-energy tensor or the energy-momentum tensor. Different people call it different things. It doesn't matter. It basically sums up and includes all the different forms of energy and momentum and stress and strain and all those things, pressure and density, that could be causing spacetime to curve. So the right-hand side says that there's matter and stuff. The left-hand side says there's spacetime curvature. And the equation sets them proportional to each other.

0:27:12.1 SC: So when Einstein first invented the cosmological constant, he put it on the left-hand side. He grouped it in with the curvature of spacetime terms. He wrote Λ times little g_μν, where little g_μν is the metric tensor. What Lemaître realized is you can just do math, that is to say, you can subtract. You can move Λg_μν from the left-hand side to the right-hand side of the equation, make it -Λg_μν, and then just identify that as a new form of energy. Indeed, it is the form of energy that would exist even in empty space. So Lemaître says you can think of the cosmological constant as the amount of energy in the vacuum, aka the vacuum energy. And he talked about the different properties of this vacuum energy. It's constant. It never changes by hypothesis. There is a pressure that is associated with the energy density. Really, there's two numbers in cosmology that you need when you have some new source of energy-momentum, you need the energy density and you need the pressure.

0:28:20.2 SC: And for the vacuum energy, Lemaître pointed out that the pressure was negative. That is to say, it's more like a tension. The classic example is if you have a rubber band and you pull it, the rubber band pulls back on you. That is a negative pressure. So pressure pushes, negative pressure pulls you back. And so that was fine. Lemaître realized that you don't need to think of the cosmological constant as a change in Einstein's equation. You can think of it or identify it with the energy density of empty space. And what I want to emphasize is there literally isn't any difference between those two ideas. It's not, and some people still insist that it is, but they're just wrong, it's not as if either you're changing the geometry side of Einstein's equation or you're adding vacuum energy. There's no difference between those two things. They're literally exactly the same thing.

0:29:16.1 SC: So from that perspective, it's once again making the cosmological constant term even more inevitable sounding. Like, okay, now there is a number. There's a constant of nature, the energy density of empty space, and you gotta go measure it. Maybe you can set it to zero for convenience as a simplifying assumption to start, but it is in principle something you should go out and constrain by taking data. Now, the other thing I want to mention is sort of jumping ahead to more modern times a little bit, but this negative tension thing, sorry, negative pressure thing, which is tension, leads to some weird explanations for why the cosmological constant makes the universe accelerate. So you will hear things that sound something like the following: ordinary matter or radiation or things like that have positive energy density and positive pressure, so they make the universe decelerate. They basically pull things together. If you imagine a universe full of galaxies and it's expanding, all of those galaxies are pulling on all the other galaxies, so the expansion rate would naturally want to slow down.

0:30:24.2 SC: And this particular way of saying things, which I don't like, so don't trust me on this, says, "But now the vacuum energy has a negative pressure and therefore it pushes things apart." And you're like, "Wait a minute, where did that 'therefore' come from? After all, you just told me that the idea of negative pressure is tension, which pulls things together. It doesn't push things apart." And then you're told, "Aha, but what I mean is the gravitational effect of negative pressure is to push things apart because the expansion of the universe responds not to just the energy density, but the energy density plus three times the pressure," which is true. And that number three, three times the pressure, is the number of dimensions of space. So it's not arbitrarily chosen. It's because space is three-dimensional that it's three times the pressure. So rho plus 3p, where rho is the energy density, p is the pressure, that's the number that comes into the acceleration equation in general relativity for the acceleration of the universe.

0:31:30.7 SC: The problem with that explanation... I mean, it's perfectly true mathematically, rho plus 3p is negative if you have a cosmological constant because for the cosmological constant, p equals minus rho. The pressure is precisely minus the energy density, and therefore rho plus 3p is minus two rho. So if you have a positive energy density in the vacuum, you have a negative amount of energy density plus three times the pressure, and that's what pushes the universe apart. The problem with that as an explanation is that where do I get any intuition for the claim that rho plus 3p is appearing on the right-hand side of my equation? It's just magic, unless you've actually gone through the math, you're not actually explaining anything to anybody. So I don't like that explanation. There's another way of explaining it, which is the equation that cosmologists actually solve for the expansion rate of the universe. And this is all done post-Einstein.

0:32:31.3 SC: Einstein had the idea that the universe could be a three-dimensional sphere, but then other people like Robertson and Walker, and then followed by Friedmann and Lemaître and others, really generalized that idea to all sorts of different possibilities, and they got a very general equation now known as the Friedmann equation. And the Friedmann equation—in fact, there's several Friedmann equations, but the one that everyone solves sets the expansion rate of the universe, which is given by the Hubble parameter, that we now call it the Hubble parameter because Hubble measured it. In Hubble's plot of velocity versus distance, his equation says that they're proportional. Velocity is a constant times distance for a distant galaxy, and that constant is known as the Hubble parameter. And so that same constant appears in the cosmological version of Einstein's equation, in the Friedmann equation. And it's a way of judging how fast the universe is expanding.

0:33:27.4 SC: So the Friedmann equation says that the Hubble constant squared is proportional to the energy density of the universe. Maybe if you also have a term for the curvature of space, if you want to include that. These days we know that the curvature of space is close to zero, so we don't need to keep talking about that for today's purposes. So anyway, H squared, H is the Hubble constant, we square it, it's proportional to the energy density. That's it. That's the equation. There's no appearance of pressure in there. And you say, "Well, how do I get the difference between accelerating and decelerating universe out of that equation?" And the answer is how the energy density changes with time matters. So imagine a universe that has nothing but vacuum energy in it, nothing but cosmological constant. No matter to worry about. Makes our lives easy. That says that rho, the energy density, is just a constant.

0:34:24.0 SC: That says that H is a constant because H squared is proportional to rho. H is proportional to the square root of rho, and if rho is a constant, so is the square root of rho. So what the equation is telling you is just H is a constant. The expansion rate of the universe is a constant. And you might now get confused for a different reason. You might say, "Wait a minute. You were telling me that this means the universe is accelerating, and now you're telling me it means the expansion rate is a constant. That doesn't sound like accelerating to me." And that's because you haven't grown up with non-Euclidean geometry. Sorry about that. You're thinking of the expansion rate as somehow a velocity, but it's clearly not. In an expanding universe, if you look at a galaxy using Hubble's law, a galaxy a billion light years away is moving apart at an apparent velocity of some number, whereas a galaxy twice that far away is moving away twice as fast.

0:35:15.9 SC: There's different velocities for different galaxies. So the expansion rate of the universe cannot possibly be thought of as a velocity. What it is is a rate of expansion. It's basically the answer to the question, how quickly does the universe double in size? In fact, the Hubble time, which is one over the Hubble constant, is almost exactly that, the doubling time for the universe. And so to say that H is a constant, which is what you get for vacuum energy, says that the amount of time in which the universe doubles in size is a constant. So in other words, from any one size of the universe, it can double in size and then double again, and then double again and double again. So it goes from its initial size to twice that, to four times that, to eight times that, to 16 times that, and so on. That is what we call exponential expansion because you keep multiplying by two at every time step. They told two friends, and they told two friends, et cetera.

0:36:17.4 SC: So a constant energy density leads to a constant expansion rate, which shows up as an exponential expansion to the universe, which is obviously accelerating. So that chain of logic is longer than the negative pressure business, but it actually makes sense, unlike the negative pressure business. The negative pressure business, like I said, it's not wrong. It's completely true. It's just sort of making it sound easier than it really is. Okay. Sorry. I gotta that off my chest because I'm trying to fix all the mistakes that I hear out there in the world. There's probably too many mistakes out there for me to fix all of them, but I can do my little part. Okay. So that's the state of play in the 1930s. Einstein had come up with his equations. He had come up with the idea of the cosmological constant. Lemaître interpreted the cosmological constant as an energy density of the vacuum. But Lemaître wasn't doing quantum mechanics. This is all still perfectly classical.

0:37:15.2 SC: Make sure that we distinguish different ideas in our heads. Lemaître is just pointing out that this number Einstein had invented could be interpreted as what is the energy density in empty space. If I take a cubic centimeter of space and I remove all the stuff, so there's no particles, there's no radiation, there's no dark matter, as we would now say, it is just truly empty, you can ask yourself how much energy is in that empty cubic centimeter of space. And according to Einstein and Lemaître, that's a new constant of nature. That's not something you can just say, "Oh, it should be zero because it's empty." It's something you have to go out there and measure. Now, at this era, the 1930s and so forth, like I said, a lot of the excitement in physics was in quantum mechanics, particle physics, things like that. People knew and loved general relativity, but they didn't think about it too much. It was not front and center in the minds of the leading theoretical physicists.

0:38:09.1 SC: People did know about it. Wolfgang Pauli wrote a whole textbook on general relativity when he was preposterously young, but it wasn't what they were thinking about on a day-to-day circumstances. They were thinking about quantum field theory and quantum mechanics. And guess what? The idea of vacuum energy rears its head in a different way in quantum field theory. And this is again something that people get wrong a little bit, so let me try to explain it the best I can. Forget for the moment about quantum field theory. We'll get back to that. Think about the simplest physical system we can think of: the simple harmonic oscillator. So literally a weight on a spring attached to a wall, bouncing back and forth, simple harmonic motion. In physics, we describe this in terms of a potential energy as a function of x. If x is the location of the weight on the spring, the potential energy looks like omega-squared x-squared, where omega is the frequency, literally the frequency of the thing bouncing back and forth.

0:39:10.8 SC: And so this is a very, very simple physical system with an omega-squared x-squared type of potential energy and the ordinary kinetic energy. It's simple because it's simple to write down, but you can solve it, and it's kind of beautiful and it illustrates a lot of very basic ideas. And so if you take a quantum mechanics class, like I was teaching here at Johns Hopkins last semester, one of the very first things you will do, and one of the things you'll keep doing at higher and higher levels of sophistication throughout your physics career, is study the simple harmonic oscillator. So in classical physics, what that means to study the simple harmonic oscillator is, you have a ball and it's rolling in this hill, and the hill looks like a parabola, omega-squared x-squared. And you say, "Well, if I let the ball go at zero velocity at a certain point on the hill, can I solve its equations of motion going backward and forward?" And the answer is yes.

0:40:03.2 SC: And they look like sines and cosines, sinusoidal functions for the oscillation of the particle back and forth in this potential. Of course, it would be very natural to do the quantum mechanical version of this. What does that mean? That means you don't have a little ball moving in a potential. That means you have the Schrödinger equation, which means you have a wave function in that potential. And you're going to solve the Schrödinger equation in that kind of potential and see what you get. And it's very, very instructive to do that. What you get by solving the Schrödinger equation in the simple harmonic oscillator potential is a series of states with fixed constant energy. You can have any energy you want, but if you want a definite energy... Remember, in quantum mechanics, when you measure something, you're not sure what answer you're going to get generically. If you measure the position of a particle, there could be some uncertainty there. If you measure the energy, there can be some uncertainty there.

0:40:57.9 SC: But there's a special set of quantum states where the energy is perfectly well-defined and known: the eigenstates of energy. And for the harmonic oscillator, these eigenstates are exactly separate, exactly equally spaced in energy. So there's a lowest energy state, and there's a state that has one more energy, whereby one I mean h-bar times omega. Remember, omega is the frequency of the harmonic oscillator. H-bar is Planck's constant. So h-bar omega is the size of the energy step from the lowest energy state to the next energy state. It's also the size of the energy step from the first energy state to the second, and from the second to the third, et cetera. They are exactly equally separated energy amounts between the different levels, as we say, or different excited states of the simple harmonic oscillator. That's just one of the things that makes it a very, very nice system. Other kinds of systems with other kinds of potentials will also have different kinds of energy states, but they won't be equally spaced in energy between them.

0:42:07.1 SC: So all that's very well known. You'll study that to death in your quantum mechanics class. But there's one thing that is kind of mysterious, and it's gonna depend on your professor's inclination how much you're gonna think about it, which is that, the lowest energy state, if you start with an ordinary simple harmonic oscillator with a potential omega-squared x-squared, by the way, it's usually one half omega-squared x-squared, but don't worry about the half. So omega-squared x-squared. What is the lowest energy state? Well, classically, that's the particle's just sitting there at x equals zero, so it has zero potential energy and also zero kinetic energy. It's not moving. That's lowest energy state is zero energy. That's how we're trained to think when we grow up. Quantum mechanically, solve Schrödinger's equation, that turns out not to be right. The lowest energy state of the simple harmonic oscillator doesn't have zero energy quantum mechanically, if it did classically.

0:43:07.9 SC: There is a difference between the classical lowest energy and the quantum mechanical lowest energy, and that difference is one-half h-bar omega. So h-bar omega is the gap between one energy state and the other. One-half h-bar omega is the starting point. So the lowest energy state is one-half h-bar omega, the next one is three-halves h-bar omega, and then five-halves h-bar omega, et cetera, for infinitely far into the higher energy states. This phenomenon, that the quantum mechanical energy of the ground state, of the lowest energy state, is one-half h-bar omega, is called the zero-point energy of the harmonic oscillator. And so, like I said, it's gonna depend on the predispositions of your professor how much you're gonna fret about that. I think that the answer is you shouldn't fret too much. And the reason why is the number one-half h-bar omega is not to be interpreted as the ground state energy of the simple harmonic oscillator.

0:44:11.9 SC: What it's to be interpreted as is the difference between the ground state energy of a classical harmonic oscillator and a quantum mechanical one. So quantum mechanics raises the ground state energy by one-half h-bar omega, but it doesn't tell you what it was to begin with. I could've imagined starting with a simple harmonic oscillator whose potential was not one-half omega-squared x-squared, but rather one-half omega-squared x-squared minus one-half h-bar omega. Nothing can stop me from doing that. And classically, if all you're doing is studying the rocking of a ball in a hill or something like that, adding a constant to the overall energy makes no difference whatsoever. It does not show up in any of the motions of the particle. It has no effect on the physics. It's completely irrelevant. All that matters is how the energy changes from point to point. So, two things. One is the actual zero-point energy is completely arbitrary. I can set it to be whatever I want.

0:45:14.7 SC: It's not one-half h-bar omega; it's whatever I choose it to be. And number two, the number one-half h-bar omega isn't even anything intrinsic to nature. It's a difference between a quantum mechanical theory and a classical theory. But nature is just quantum mechanical. Nature doesn't know about the classical theory. Nature says you're quantum mechanical from the start. Why were you starting with the classical theory and then quantizing it? You should have just started with the quantum mechanical theory, and in that case, the relevant fact is that the energies of different energy states are separated by h-bar omega. Nevertheless, people kind of get hung up about this a little bit. They start thinking that quantum mechanics tells you that there is energy density in the vacuum. Don't get me wrong, there could be. That's what I'm trying to say. It's a constant of nature. It's a fact about reality. What is the zero-point energy of the simple harmonic oscillator?

0:46:11.6 SC: It's not a fact about your quantum mechanical calculations. You can choose the vacuum energy to be whatever you want. Okay. So, why do we care about that? Why do we care about the simple harmonic oscillator? It's a very nice thing to be able to solve things precisely, but we want to get into the messy real world with electrons and protons and stuff like that. So we have to upgrade our game a little bit and start thinking about quantum field theory. At this point, I will encourage you to read volume two of "The Biggest Ideas in the Universe: Quanta and Fields" where we talk about precisely quantum field theory. If you want to know what that means, it just means that there are fields stretched throughout all space and time, and they vibrate and you quantize them, and that's what gives rise to particles and things like that. So since the 1930s, at least, getting into high gear in the 1950s, quantum field theory has been our best attempt at understanding the world at a deep level, at least particle physics and the world other than gravity.

0:47:11.4 SC: Gravity is a special thing but... Okay. That's a more difficult topic to get into, not something we understand very well. Okay. So quantum field theory sounds like a very large journey away from the simple harmonic oscillator. But here's the wonderful news. When you start studying quantum field theory, just like in other areas of physics, you're gonna start simple and you're gonna work your way up. So what does it mean to start simple? It means to start with a field that is just doing its own thing, a field that is not interacting with other fields, not even interacting with itself. And those are all very well-defined statements and you can specify what they mean. So we talk about a free field to be one that is not interacting with other fields or itself. That's one step. Let's think about free field theory. No interactions, just make our lives easy, then we'll put in the interactions later. It's an ancient move. It goes back to Galileo saying, "Let's ignore air resistance, we can put it back in later."

0:48:10.4 SC: We're saying, let's ignore interactions between fields, we can put them back in later. The second move, which is something that is less ancient but is absolutely always what you do when you study fields, is you start thinking about waves. And that means you start thinking about waves of definite wavelength, what we call modes of the field. And again, I'm not gonna get into details if you haven't read the book. If you have read the book, this is all perfectly transparent. But you can think about what is happening in a vibrating quantum field. Rather than thinking about what's happening at each point in space, you can think about what's happening at each wavelength. And this is not a weird thing. This is called Fourier analysis, and it's literally what your ears do when you hear sounds. You hear different pitches because your ears are saying, "Well, how much vibration is there at this frequency and how much is there at that frequency?" and so forth.

0:49:01.4 SC: So let's do that. Rather than focusing on the behavior of the quantum field at one point, let's focus on the behavior of a single mode with a fixed wavelength everywhere of this quantum field. Guess what happens? Miraculously, each mode of a free quantum field obeys an equation which is exactly the same as the equation of a simple harmonic oscillator. That's why we're doing all this. That's why I was wasting my time telling you about the simple harmonic oscillator, because free, non-interacting quantum fields very naturally behave like a collection of an infinite number of simple harmonic oscillators. Not one at every point in space, but one at every wavelength, indeed one at every wave vector, we say, because a mode of the field has both a wavelength but also a direction in space. That's the wave vector. So every wave vector is a mode, and every mode acts like a simple harmonic oscillator for a free field theory. And that means that every mode of the quantum field has the same kind of zero-point energy contribution that the regular harmonic oscillator had.

0:50:17.9 SC: And everything that we just said about the quantum mechanical harmonic oscillator really is still true in quantum field theory. That is to say, it's both true that if you start with zero energy in the classical theory and then you quantize it, you get a new addition to the energy of the vacuum, which is one-half h-bar omega. Now there's every value of omega. So you don't just because there's different wavelengths, wavelength modes of the field, they all have their own vibrations. And what that means is that you don't just have energy density in empty space of one-half h-bar omega. You have the sum, or the integral, if you like, of all possible omegas of one-half h-bar omega. And what does that equal? It equals infinity. So, this is one of the famous infinities of quantum field theory. This is perhaps the most basic infinity of quantum field theory. The energy of empty space, the vacuum energy, is infinity, if you're very, very naive about quantum field theory. But before you get too excited about that, everything I said about the quantum mechanical oscillator is still true.

0:51:29.2 SC: I could choose the vacuum energy to be whatever I want. It doesn't need to be infinity. I can just choose it to be zero. This is one of the steps in what we call the renormalization of the quantum field theory. It's perfectly legitimate. No one forced me to have a certain value of the vacuum energy to begin with. So the lesson is not that there's an infinite amount of energy in empty space in quantum field theory. The lesson is that there's an arbitrary amount of energy in empty space in quantum field theory. And guess what? That's just the lesson of the vacuum energy that Lemaître had talked about. There's a new constant of nature, the vacuum energy, and we have to go measure it. It is a very delicate back-and-forth kind of thing. Quantum field theory says, "Okay. There can be vacuum energy in empty space." It does not say what it has to be, but there's another level that says, "Well, maybe it lets you have a guess as to what it could be." Maybe it lets you sort of estimate a natural value.

0:52:34.9 SC: There's nothing natural about zero. You could set it to zero. You could say, "I'm just gonna say there's zero energy in the vacuum." You're allowed to do that. There's no reason to do that from the point of view of quantum field theory. It's some number, and you can wave your hands and you can say, "Okay. I have sort of natural guesses for how big that number should be." So the first person to go down this road was Yakov Zeldovich, who was a physicist, a Russian physicist, and in the 1960s, he made this point very clear, that the vacuum energy of quantum field theory is precisely what goes into the vacuum energy of relativity, the cosmological constant. And this is the beginning of the connections between quantum field theory and vacuum energy. There were some precursors there. I shouldn't jump right ahead to Zeldovich. Again, it's a fascinating story because people don't know what's going on, so they cling to certain beliefs and they get rid of others, and they learn something but don't put it in the right context and all this stuff.

0:53:37.6 SC: Richard Feynman, of course, became famous for Feynman diagrams in quantum field theory. And Feynman diagrams are little pictures of particles moving, but they're 100% derived from quantum field theory. They're not derived from thinking about particles. The particles come out of quantum field theory as predictions once you quantize these fields. But these ideas don't come out of a vacuum, if you'll excuse the pun. And Feynman had this previous idea with John Wheeler, who was his advisor at Princeton: the Wheeler-Feynman absorber theory of radiation. You may have heard of this. This is part of the package of ideas where Wheeler came up with the idea that maybe positrons, which are antiparticles of electrons, are just electrons moving backward in time. And therefore there's only one electron anywhere that sort of moves forward and backward in time and goes through the whole universe. People love this idea. It's not right. It was a good guess. It was a fun idea. But these days we know, no, that's not right.

0:54:42.0 SC: What's actually right is quantum field theory. The reason why all electrons are the same is not because they're the same electron, because they're all vibrations in the same underlying quantum field. But, okay, they didn't know that. They weren't sure. They were trying new things. That's what physicists do. And so Wheeler and Feynman, in their absorber theory, really tried to think of electrons as particles, not as fields. Even though quantum field theory already existed at the time, they were open to changing that. And one of the things in the back of their minds was they knew about this vacuum energy business. They didn't know it was the cosmological constant. They didn't know, as far as I know, they didn't know it had anything to do with gravity or anything like that. But they were just kind of worried about the fact that when you quantize field theory, you get an infinite contribution to the energy. Again, I think that a more correct attitude is to say that infinity is not the energy, it's the difference in energy between the classical theory and the quantum theory.

0:55:39.4 SC: And it's the quantum theory that is right. So who cares about the classical theory? But still, they were a little bit worried. And so they thought if they could replace quantum field theory with a theory of particles, that would make this vacuum energy problem go away. It didn't work, of course, but that was part of their motivation. It eventually led to Feynman diagrams, which is pretty good. The other example is Phil Anderson. Phil Anderson, Nobel Prize-winning condensed matter physicist, who arguably was the first person to really propose what we now call the Higgs mechanism. The Higgs mechanism says if you have some scalar field, so you have some field with no directionality in space or anything like that, just a number at every point in space. And imagine that it couples in a certain way to other fields like electrons and photons and stuff. And this field that is now called the Higgs field, which Anderson thinks should have been called the Anderson field, imagine that this field gets a non-zero value in empty space.

0:56:39.6 SC: So what you're imagining is that this field, the scalar field, which we now call the Higgs field, has some potential energy function, which says how much energy is in the field simply as a virtue of its value. As opposed to, fields have kinetic energy if they're changing in time, and they have gradient energy if they're changing in space. The potential energy is just how much energy they have because they have a value. So if the minimum energy is not at zero field value, which there's no reason why it should be, this is the famous Mexican hat potential, then the field will roll down to the minimum, which is not at zero, and it will sit there and it will affect other fields. And Anderson knew this, and he sort of mentions it in words in one of his papers, but he didn't pursue the idea. That's why he doesn't really get credit. He doesn't get his name on it for the Higgs mechanism.

0:57:30.5 SC: Why didn't he pursue the idea? Part of it was because he was really a condensed matter physicist, not a particle physicist. That's where his interests lay. But also because I think, this is my impression, he knew that there would be a tremendous energy density associated with that field changing its value in empty space. So you're proposing that there's a field pervading the universe that carries an enormous amount of energy, and no one has ever noticed it before. I'm not sure that he ever, again, connected it to gravity or the cosmological constant, but it bugged him, so he moved on to do other things. That's okay. He won the Nobel Prize for other things. Anyway, all of which is to say, by the 1960s and Zeldovich, people had put the story together. If you just care about the zero-point energy of the quantum field, it doesn't matter. You can make it whatever you want. It's a constant of nature. And if all you're doing is particle physics, it's not even an observable constant of nature.

0:58:29.2 SC: It's just like the ball rolling back and forth in the quadratic potential, in the parabolic potential, the harmonic oscillator. If I lift the potential up and down by adding a number to it, it doesn't change the behavior of the particle. Likewise, if I just change the energy density of empty space, it doesn't change particle physics in any way. What it does change is gravity. Gravity cares about the total amount of energy in the universe, and Zeldovich appreciated that. So he pointed out, "You know what? We have a problem here, folks." Because there is a number, the energy density of empty space. We have no idea what it is. We have no idea what it should be, I should say. But you can sort of wave your hands about natural energy scales, blah, blah, blah. And Zeldovich points out the natural scale for this number is much bigger than it can possibly be given the observations. We're talking about an enormous discrepancy. So let's skip ahead to the modern way of thinking about these things.

0:59:29.7 SC: The modern way of thinking about these things actually puts some meat on the bones of all this talk about natural values and stuff like that. And that framework that we're talking about here is effective field theory. In the effective field theory framework, you admit you don't understand everything. You say, "Look, I don't know the theory of everything. I don't know whether it's string theory or something else. I don't know what's going on at very, very high energies," which in quantum field theory corresponds to very, very short distances in spacetime. Say, "I don't know what's going on." Happily, quantum field theory has this feature that even if you don't know what's going on at very short distances, you can still find an effective theory that only includes particles and fields and phenomena at low energies and long distances. This is called the infrared theory because you think about infrared things as long distances, long wavelengths; ultraviolet things as short distances, short wavelengths.

1:00:29.6 SC: So following Ken Wilson and others, you say, "I have an ultraviolet cutoff. That's basically the shortest distance that I'm claiming to understand. I don't know what goes on at even shorter distances, but I'm gonna bundle all those effects up and ask what they do in my infrared theory." and then there's some rules for how big you expect the parameters in the infrared theory to be, et cetera, et cetera. And basically, some parameters in the infrared theory are what's called relevant. They get more and more important as you go to longer distances and more in the infrared. Some are irrelevant. They're not that important. And the cosmological constant, the vacuum energy, is super-duper relevant. It very much shows up in the infrared. And you can use these techniques of effective field theory to say, "Okay. How big do we expect the cosmological constant to be?" And the answer is hilariously bigger than it can be. And I say this even knowing that in the 1960s, our cosmological observations were not nearly as precise as they are today.

1:01:34.6 SC: They didn't need to be. The discrepancy between the effective field theory prediction for the vacuum energy and what you actually observe it to be is just so hilariously large that you don't need very precise observations. The most dramatic way of saying it is if you run your effective field theory, you put your ultraviolet cutoff all the way up at the Planck scale, and you compare it to our limitations, let's say in the 1980s, before we actually found the vacuum energy, how much bigger is the predicted vacuum energy than the limit? The answer is a famous factor of 10 to the 122 times bigger. One followed by 122 zeros. So not the number 122, that would be easy. We could deal with that. But one followed by 122 zeros. That's 10 to the 122. It's a very, very big discrepancy. That discrepancy is known as the cosmological constant problem. And again, as I keep emphasizing, you're 100% allowed to solve the cosmological constant problem in the following way, just say, "That's the way it is."

1:02:41.7 SC: The cosmological constant in the infrared, the measurable quantity, is something we measure. I mean, maybe you had an expectation for what it should be. Your expectation was wrong. Tough cookies. And that's an attitude that you can have and you can get away with it. It's completely compatible with the data. Maybe that's just how things are. Sometimes physics is like that. I think a lot of theoretical physicists are gonna want to say, "Maybe that's true, but maybe it's not." Maybe this huge discrepancy between our expected value for the vacuum energy in effective field theory and the observed value is a clue to some new physics that we don't yet understand. So even though it's intellectually allowed for you to say, "The cosmological constant just is what it is. I measure it, then I get on with my life," maybe you're missing an important clue that nature is trying to give you to develop a better theory. And I think that's what a lot of physicists these days would think.

1:03:42.0 SC: So that came about in the 1960s, and by the 1980s, once we'd put the standard model of particle physics in place, people began turning their attention to gravity and cosmology. The 1980s were when the first superstring revolution happened, and it also was the time when people really began thinking about the cosmological constant problem very hard because they'd solved all the problems in experimental particle physics. They were moving on to the universe. And so why is the observed vacuum energy... So just to make it super-duper clear, if the cosmological constant or the vacuum energy were anywhere close to its natural value as far as particle physics is concerned, the universe would have exploded apart into nothingness in a tiny fraction of a second after the Big Bang. The expansion rate of the universe driven by such a vacuum energy would be so big that you couldn't make an atom, much less a molecule or a planet or a star or anything like that.

1:04:42.4 SC: So it's not like you have to do some delicate cosmological measurements and maybe you have error bars on those and maybe you're wrong about this. The discrepancy is just so big that it's absolutely, overwhelmingly obvious. Furthermore, you don't even know whether the cosmological constant is positive or negative theoretically, right? You're not making a prediction for its sign. You're making a prediction for its absolute magnitude, if anything. It's just a guesstimate more than a prediction. It could be negative, and that would mean it would make the universe recollapse in a tiny, tiny fraction of a second. And that's also ruled out by the data since the universe is around 14 billion years since the Big Bang. So by the 1980s, theoretical physicists had started thinking seriously about this cosmological constant problem. And I say this because I entered graduate school in 1988, and now for the rest of this podcast, solo podcast, and for the entirety of the next one on dynamical dark energy, it's gonna be a very Sean-centric view of all these problems.

1:05:44.1 SC: Not because I am central to the story of dark energy and the accelerating universe, but dark energy and the accelerating universe are central to me because I did a lot of work on it and wrote a lot of papers on it and things like that. So I'm not gonna try to give you a completely unbiased view of anything. I'm gonna tell you what was happening around me. When I was thinking about these things, because I'm not doing a lot of research for these solo podcasts. Sorry about this. But I do have some experience that I can share with you here. So there's a bunch of things in the late 1980s, early 1990s, where people were thinking about the cosmological constant problem. And I'm gonna mention four ideas, four attitudes, four, let's say, strategies you can take to address the cosmological constant problem. One of them is a little bit anachronistic. It came along later, but it gives you a feeling for what's going on. None of them are very good. None of them are like obvious slam dunks.

1:06:40.4 SC: That's why it's still an interesting problem. One that got an enormous amount of attention is supersymmetry. Supersymmetry in the 1980s especially was incredibly popular. It really just had grown popular in particle physics in the '70s. And in the '80s, people were still very excited about grand unification and also connections of supersymmetry with string theory. So it was like the hot topic in theoretical particle physics circles. And supersymmetry, unlike other symmetries, does have something to say about the cosmological constant. So what do I mean by that? Remember the absolute value of the vacuum energy, as we were talking about for just the harmonic oscillator. It doesn't matter, the absolute value. All it matters is how it changes from place to place, except when gravity comes into the game. So for the rest of particle physics, for the standard model of particle physics, for the electroweak force and the strong force and the Higgs mechanism and all that stuff, you can ignore gravity and you don't need to worry about the cosmological constant.

1:07:48.3 SC: Supersymmetry is different because supersymmetry is a mixture of a spacetime symmetry, or it's a way of mixing spacetime symmetries with what we call internal symmetries. Supersymmetry is sort of a unique way of having a non-trivial interaction between spacetime symmetries and things like gauge invariance that you get in the standard model of particle physics. So supersymmetry, because it relates particles of different spins. Spin zero and spin one-half particles, for example. Spin is a spacetime feature. It's a spacetime property of the quantum mechanical field. So supersymmetry, in some very rough sense, knows about spacetime. If you talk to like supersymmetry experts, they will start telling you things like the supersymmetry transformation is the square root of momentum or something like that. And I'm not sure if that's very enlightening or not, but the point is it is related to spacetime, supersymmetry.

1:08:51.8 SC: So it's related to both particle physics and to gravity. That's very exciting all by itself. And because it's related to spacetime, supersymmetry knows something about the vacuum energy, by which I mean in a given theory of supersymmetry, the vacuum energy is not arbitrary anymore. In regular old quantum field theory, it's arbitrary. It's a constant of nature, the vacuum energy. You're gonna measure it or whatever. In a supersymmetric theory, it's determined by what is going on in the rest of the model. And people very quickly realized that there were certain versions of supersymmetry where the vacuum energy would be exactly zero. You could actually solve the puzzle of why the vacuum energy is small if you had perfect supersymmetry. They also realized you could have negative vacuum energy, which seemed weird, but okay, at the time they didn't bother with that too much. They did, though, instantly realize a problem with this, which is that supersymmetry is not exact in the real world as we know it.

1:09:50.3 SC: If it were, there would be an electron with a certain mass and charge, and there would also be a spin zero partner of the electron with the same mass and charge. No such particle exists. This we call the selectron by supersymmetry aficionados, but it hasn't been found. This is not a huge puzzle. This is just because probably supersymmetry is broken, if it exists. Broken symmetries are super easy to have in particle physics. That's what the Higgs mechanism does. That was something that particle physicists understood very well. So you have supersymmetry and then you break it, and you could even estimate the energy scale at which it had to be broken and things like that. The problem there is that again, if you break supersymmetry, it does care about the vacuum energy, and you generally not only do not get zero for the vacuum energy, but you do get a number that is incompatible with the data. So supersymmetry was intriguing for the cosmological constant problem because it seemed to have implications for it, but the implications seemed to be bad.

1:10:56.3 SC: It seemed to take away your freedom to make the cosmological constant whatever you wanted. Now, that's not necessarily a sign that you're on the wrong track. Maybe you just hadn't filled in all the details yet. And so people put a lot of work into seeing could they come up with a supersymmetric model that explained the small value of the cosmological constant, and the answer was no, they hadn't been able to do that quite yet. So that was one avenue. Another avenue is wormholes and Euclidean quantum gravity. And I mentioned this because this was one of those ideas in physics that it got very popular for a very brief period of time and then it went away. And the person who really put it together was Sidney Coleman, who's a professor at Harvard, who was my quantum field theory professor and was on my thesis committee. And he was a great guy, brilliant scientist. And usually, it's very funny, and Sidney himself was a little bit chagrined about this, like his style as a theoretical physicist, the reason why he's not as famous as Steven Weinberg or Sheldon Glashow, his colleagues at the time, was just that he was very cautious.

1:12:01.1 SC: He was doing well-established calculations in quantum field theory. He was not speculating about some crazy model. And then the one time that he did a little speculation about some crazy model was this cosmological constant stuff, and it got him a lot of publicity, but it wasn't really well-grounded because it was just speculation, and ultimately it went away. But I gotta tell you about it because even though it went away, even though people don't really think it's the right answer, it's really brilliant. I mean, it really worked, and if it had worked, it didn't really work, if it had worked, it would sort of offer an explanation in a super intellectually satisfying way, which is kind of intriguing. So here was the idea. People, like I said, were beginning to think about gravity as well as particle physics. And what is the simplest way to get there? I didn't really plan out this part of the episode that deeply. Maybe I should have planned it out more deeply.

1:13:00.2 SC: But one of the ways you can do quantum gravity... We know that you don't have an exact theory of quantum gravity, we don't know all the details, but we know a lot about quantum mechanics, you know a lot about gravity. You can make some reasonable statements that you think are telling you something about quantum gravity. And that was done by people like Stephen Hawking and Jim Hartle and others, and they did it via what is called the path integral approach to quantum gravity. So basically, the first thing you do, and this is already a suspicious step, but you think of spacetime, but instead of four-dimensional spacetime, you just think of four-dimensional space. This is called doing Euclidean quantum gravity. So four-dimensional space, that is to say all the four dimensions are the same. There's not one picked out to be time and three picked out to be space. And by a path integral, you say you calculate the quantum mechanical probability of the universe being in a certain state by adding up all of the different ways you can get there.

1:13:56.5 SC: This is one of Feynman's ideas, the path integral formulation of quantum mechanics. And so in gravity, since gravity is a theory of four-dimensional spacetime, the proposal from Hawking and others was that you could calculate the wave function of the universe. By summing over all the four-dimensional spaces that have a boundary that is the universe that you're looking at right now. And it was just a mathematical trick. The universe is not really Euclidean. It was just supposed to be a way of calculating the wave function of the universe. So people had this idea, all right, let's sum over all the four-dimensional geometries and calculate the wave function of the universe. It's a good way to spend your time while you're waiting for the particle accelerators to actually give you useful information. And they realized, okay, what's the first thing you do? What's the simplest thing you do? Well, you're thinking of how to approximate an integral or a sum over all four-dimensional geometries. That sounds hard, but let's look at what we're summing up, and it's Einstein's equation.

1:15:00.9 SC: It's Einstein's expression for what we call the action of general relativity. It's a number you can attach to every four-dimensional geometry. And in... Well, I'm not gonna go into all the details, but you want to find a saddle point or an extremum of the action. That's the classical equations of motion. And so what they found was the simplest solution is just a big sphere, a big four-dimensional sphere. That's a simple contribution to the path integral of quantum gravity. Fine, that's easy, we can do that. But then what about the next simplest thing? And some genius, I don't know who it was, but someone said, "Look, if I have not one sphere, but I have two spheres..." So they're both very smooth. The idea being that if you wiggle the sphere, it gets less and less of a good solution to Einstein's equation and therefore less and less important in the quantum mechanical calculation. But this person says, "If I have two spheres, I could connect them by a tiny wormhole."

1:16:01.2 SC: And that tiny wormhole would have almost no contribution to the action because it's this tiny wormhole. So two spheres are almost as good of a contribution to Euclidean quantum gravity as one sphere is. And then once you say that, well, okay, I have three spheres, or I could have two spheres with two wormholes connecting them, et cetera. So there's an infinite number of things that I can add up. And this was exactly the kind of thing that Coleman was really good at. So he and others, Steve Giddings, Andy Strominger, and other people worked on this, Joe Polchinski, they thought about how to calculate the wave function of the universe as a sum over spheres connected by all these wormholes. And people realized that this was a huge problem, actually, because if you have a wormhole connecting two big spheres, it's not just gravity that gets involved. What if an electron falls down the wormhole? What if it goes from one sphere to another? And this is a lot of... I'm gonna try to cut this short because we could get out of hand.

1:17:03.5 SC: Like I said, it didn't really pay off anyway. But the point is, it would seem that this was worrisome for real-world particle physics. It would seem like in the real world, you might be able to have processes where a photon goes down a wormhole and disappears, and that would be bad. And what Coleman was able to figure out was that, in fact, that's not the observable impact of these wormholes. In fact, what happens is you can't just have an electron go down the wormhole because it would look like charge is disappearing in the universe, and that's bad. What you could have is an electron and a positron going down the wormhole and a photon coming out. So that would conserve charge and energy and everything like that. And that wormhole, therefore, looks from far away, where you don't know it's a wormhole, it looks just like an ordinary particle physics interaction. So really, Coleman says, the effect of these wormholes is not to swallow up particles and make them disappear.

1:17:59.9 SC: It's to contribute to all of the existing parameters of particle physics, like the fine-structure constant that tells you how strong the electromagnetic interaction is, like the Higgs mass, like all these other things, and like the vacuum energy. So in other words, Coleman says these wormholes add a new contribution to the vacuum energy. So you have whatever classical contribution you have, you have the zero-point energy, you have the Higgs field, and now you have the wormhole contribution to the vacuum energy. And what he showed by a calculation was the contribution of the wormholes was exactly to cancel out the contribution of everything else. It was truly beautiful. It was very, very convincing and provocative when it first appeared. And this is late '80s, about when I arrived in grad school. Sadly, the whole thing never lasted because, as Coleman himself said very, very clearly, he was not bombastic about it, he says, "Look, this is a house doubly built on sand," because we're talking about Euclidean quantum gravity and wormholes.

1:19:07.5 SC: These are things we don't understand very well at all. And the more and more people looked at it, the less and less sensible and reproducible it all appeared. So it didn't burn out, but it faded away. The idea that wormholes provided the successful resolution of the cosmological constant problem, there were enough obstacles to it really working. Like some people said, it actually doesn't make the cosmological constant small; it makes it big. It was very hard to figure these things out because no one knows what the rules are. So that kind of went away. But again, I'm trying to give you a flavor for what kinds of ideas were being bandied about at the time. Another idea I wanted to mention, which came out much later, I mean, 10 or 15 years later, is something called self-tuning. And this is interesting because I wrote a little paper about this. I didn't invent it. Other people invented the idea. But this is once the idea had come along in the 2000s that maybe we live on a brane, B-R-A-N-E brane, not B-R-A-I-N, brain.

1:20:09.8 SC: So a brane is like a three-dimensional, there's different dimensionality branes, let's say a three-dimensional surface embedded in higher dimensions. So if you have higher dimensions, extra dimensions of space like you do in string theory, maybe we don't live in all of them. Maybe we live on a three-dimensional brane embedded in them. And by specially picking the way in which the brane interacts with the rest of the world, you can find parameters, or you can find dynamics, I should say, that will cancel the vacuum energy on the brane. So basically, if you have, let's say a cosmological phase transition where you have a change of physics from one phase to another, ordinarily that would be associated with a change in the vacuum energy. Like, a change in the average energy in water and ice is different. That's why ice cubes float. Same thing is true for spacetime and quantum fields. When there's a phase transition, the energy changes.

1:21:05.7 SC: And so this self-tuning idea, what they were able to do was to show that they could pick a set of fields, et cetera, with the property that when you had a phase transition on the brane, it would not change the vacuum energy, or it would not change the effective observed value of the vacuum energy that you would measure by doing cosmological experiments. And this was called self-tuning. The vacuum energy sort of goes away. And now, this turns out, again, it didn't work. It's in the dustbin of history. It was a good idea. Like many good ideas, it doesn't quite work. I remember when I gave a talk on it at the University of Texas and Steven Weinberg was in the audience. And Weinberg was an expert on the cosmological constant. He had written an extremely influential review article in the late '80s where he went through all the different ways of solving the cosmological constant problem that had been proposed at the time. And he actually proved a theorem that said basically you can't just invent some new fields and solve the cosmological constant problem.

1:22:05.9 SC: There will always be some fine-tuning involved, at least in the context of ordinary quantum field theory. And so when I explained in my seminar, I was explaining the idea of self-tuning, which again, I didn't invent, but I'll tell you what I did about it in a second. And he's asking questions. He's like, "Well, how do you do this? I thought I had a theorem that said this couldn't be done." And I eventually explained it to him and he goes, "That's not self-tuning. You tuned it." And he was completely correct in the sense that you had to choose other parameters of the theory, not the vacuum energy, but other parameters of the theory, very carefully so that the vacuum energy would cancel in a certain way. My paper with Laura Mersini, which I wrote when I was a beginning professor at Chicago, was not about whether or not it was fine-tuned. I thought it was fine-tuned from the start, but I thought it was interesting because I wanted to know, we have the Friedmann equation.

1:22:58.7 SC: We have this equation that says the expansion rate of the universe feels the total energy density of the universe. It doesn't just feel the energy density of matter and radiation. How does it know to feel the energy density of matter and radiation, but not the energy density of vacuum energy? How does that work? And what we showed was, I sort of guessed this ahead of time and then we showed that it was true, effectively what was going on was that the self-tuning mechanism had replaced the energy density in the Friedmann equation. The energy density is given the letter rho, the Greek letter rho, I don't know why, for historical reasons, with the combination rho plus p, where rho is the energy density and p is the pressure. Now remember, for the vacuum energy, rho plus p is zero because p equals minus rho. The pressure is exactly minus the energy density. For ordinary matter, there's no pressure. So cosmologically, you could get an ordinary-acting universe at late times where we're dominated by matter density, and then the vacuum energy would just disappear from the equation.

1:24:07.7 SC: The problem with that is that there's also radiation. In the early universe, the universe is radiation-dominated. In fact, radiation is more important than matter. And at the moment of Big Bang nucleosynthesis, when we are fusing protons and neutrons into helium and other light elements, the universe is actually more radiation than matter. And if you change the Friedmann equation by replacing rho with rho plus p, radiation has pressure. It has positive pressure. In the case of radiation, the pressure is one-third of the energy density. So you're replacing rho with something like four-thirds rho, and that's not allowed experimentally. I remember Shamit Kachru, the string theorist, was one of the people who first proposed the self-tuning idea. And I gave a talk at Stanford and he was in the audience and he said, "You're telling me that I've made a prediction about something that can be experimentally ruled out? That's awesome." And it was experimentally ruled out, but it was also theoretically not that attractive, so again, it went away.

1:25:10.5 SC: But I'm trying to give you a feeling for what was attempted those days. I'll give you one last one, one last attempt at solving the cosmological constant problem. Of course, you know this one. It's the anthropic principle. And this was suggested by a number of people, but it was Steven Weinberg who really did it carefully. And he showed that if you live in a multiverse with different parts of the multiverse having different values of the cosmological constant, but everything else the same, then you would naturally not get a big cosmological constant in those parts of the universe that were hospitable to life. For exactly the reason we talked about: if the cosmological constant is too big, it blows galaxies apart. If it's negative, it collapses the universe very quickly. So there's a small window, relatively speaking. And to his eternal credit, he made these predictions in the 1980s, before the cosmological constant was discovered.

1:26:02.8 SC: And he said, "Look, the natural expectation, if this anthropic principle is on the right track, is that there is a non-zero value for the cosmological constant that we will eventually discover, because there's no reason, no symmetry, for it to be exactly zero." It turned out to be right, and he was more or less in the ballpark of the correct answer. That doesn't mean the anthropic principle is right, but it does mean that compared to those other things that I was telling you about, it did a better job of making a prediction and coming out right. Okay. So there we were in circa 1990. We had the idea of the cosmological constant. We related it to the vacuum energy. We had tried to explain why it was small without a lot of progress. Now, I come into the game because Bill Press, who was a professor at Harvard at the time, he's at the University of Texas now, he was in the astronomy department. And he was a very well-respected theoretical cosmologist. And he had been invited by Allan Sandage, who was the editor of a journal called Annual Reviews of Astronomy and Astrophysics.

1:27:09.0 SC: Bill was invited by Sandage to write a review article about the cosmological constant. I have no idea why. It was not like a particle physics review article. Weinberg had already done that. This was the astronomy version of the review article. And okay, so Bill for some reason said yes. And then for some reason, I was a grad student in the astronomy department at the time, and Bill asked me to collaborate with him. I mean, I know why that happened, because Bill didn't want to write anything about supersymmetry or wormholes or anything, and he knew that that was important to the story. So he was going to do some cosmology, he was going to make some plots. He was a very numerical guy. He was one of the authors of Numerical Recipes, which was at the time, before you could ask LLMs to do all your coding for you, Numerical Recipes was a very helpful way to do scientific computation. And so Bill did a wonderful job at figuring out what the effect of the cosmological constant would be on cosmological observations, growth of large-scale structure, a million different things.

1:28:09.9 SC: He actually didn't just do numerical calculations. He derived formulas and semi-analytical approximations and things like that that were super useful for working cosmologists. And then we eventually realized that neither he nor I were an expert in the observational side of things. So we asked Ed Turner, who is a professor at Princeton, an astronomer, to help us with that. And here's where it gets a little embarrassing, because what Ed was really an expert in was a technique called statistics of strong gravitational lensing. The idea is that if you have a very far away galaxy or quasar, it can be gravitationally lensed by an intermediate-distance galaxy or quasar. And the probability of that happening is something you can calculate, and you can calculate how that probability changes as a function of the cosmological constant. If you have a non-zero cosmological constant, that can change the statistics of how frequently you will get lensed quasars and galaxies in the background.

1:29:13.1 SC: And when we started writing our article around 1990, this was thought to be the best way to constrain the cosmological constant. Ever since Einstein had come up with the idea, people were trying to measure the value of the cosmological constant. They had not done so historically and notoriously. It was very difficult to get precision measurements in cosmology. This gravitational lensing story was thought to be a promising way forward. Now, as it turned out, within 10 or 15 years after we wrote our article, there were two methods that were really useful for measuring the cosmological constant. One was supernovae distances, and the other one was anisotropies in the cosmic microwave background. We didn't mention either one of these in our review article. This is what happens when you're in the right place at the wrong time. We were a little bit too early to be able to do that. So we were superseded by later articles, but in the meantime, we were useful for quite a time there.

1:30:08.8 SC: So my job was to write about the theory of the cosmological constant. I wrote both about all these things I've just been telling you about, the anthropic principle and wormholes and stuff like that, but also I just tried to explain the issues with naturalness and fine-tuning with the cosmological constant. So there is the idea that the value of the cosmological constant is mysterious. Why is it so small? That's the cosmological constant problem. But there's also the idea, well, what if it's not exactly zero? So this is 1990. We didn't know that it was not exactly zero, but we were open to the possibility. We thought it probably was zero, honestly. In the back of everyone's mind, all the theoretical physicists, anyway, there was this idea, look, I don't know why the cosmological constant is so small, but in the space of all possible theories that I haven't yet thought of, more of them make it exactly zero than make it 10 to the minus 122 times its natural value, but not 10 to the minus 123 times its natural value.

1:31:14.3 SC: Like maybe there's some symmetry or some dynamics or some mechanism or something like that that makes the cosmological constant zero. I can't imagine that it's just beyond the scope of detectability right now. Of course, the one big counterexample to that was the anthropic principle. But what if you had detected? So I and others were saying back there in 1990, what if we do detect the cosmological constant? There was one thing that everyone recognized, which is that if you know how fast the universe is expanding, if you know the Hubble constant, then there's a certain amount of energy density in the universe that would be compatible with the universe being spatially flat. So remember Einstein, ever since Einstein, you can think of the three-dimensional geometry of space changing through time in a cosmological setting. And that three-dimensional geometry of space could be flat, it could be positively curved like a sphere, or negatively curved like a saddle.

1:32:12.9 SC: And that zero curvature option right in the middle is sort of the special point. It's the middle point between the other two things. It's what's called the critical density of the universe. The critical density is the amount of energy you need to satisfy Einstein's equation with a flat universe. So you might, on the basis of theoretical prettiness, prefer the critical density. And in the early 1980s, we had the idea of the inflationary universe scenario from Alan Guth and others. We had the idea that you could explain why the universe is so smooth and homogeneous and isotropic because you can imagine that the universe underwent a period of super-fast expansion very early on that basically smoothed all the wrinkles and flattened everything out. And at the time, 1990, inflation didn't make a lot of predictions other than the critical density should be the density of the universe. The universe should be spatially flat. Meanwhile, astronomers had looked for the density of the universe. They'd tried to measure it.

1:33:17.6 SC: They measured the gravitational density of galaxies and clusters and whatever. They couldn't find it. They kept getting numbers that were close to 0.3 times the critical density. So a lot of astronomers just thought the universe was not spatially flat, it was negatively curved. So one of the motivations astronomically for taking the cosmological constant seriously is, technically, maybe the universe has the critical density, but 0.3 of it is matter and 0.7 of it is the cosmological constant. We knew in 1990 that that would be allowed by the data. It would be a little bit convenient. It would be a good story for inflation. But it would be weird. And I'll tell you the reason why it'd be weird. It's something which we call the coincidence problem. The point is that the universe has been expanding for a long time since the Big Bang, 14 billion years. And as the universe expands, matter and radiation, which we know exist in the universe, dilute away.

1:34:17.6 SC: The density of matter and radiation goes away as the universe gets bigger and bigger. Whereas the vacuum energy does not dilute away. Its energy density stays exactly fixed. So just to compare it to matter, by matter, cosmologists just mean any collection of slowly moving particles, slow compared to the speed of light. So dark matter is matter, but stars are matter, galaxies are matter, et cetera. Matter scales away as the volume of space increases, because the number of galaxies stays the same, but the volume of space goes up, so the amount of galactic matter per volume goes down. Meanwhile, the vacuum energy remains constant. So if you compare the value of the vacuum energy and matter today to what they were in the early universe, whatever they are today, there was an enormously bigger amount of matter and enormously smaller amount of vacuum energy in the early universe, because the matter has to have declined ever since. And in the future, it will be almost all vacuum energy.

1:35:27.0 SC: So if the vacuum energy is of the same order of magnitude, within a factor of 10, of the matter density in the universe today, that means it was completely invisible in the early universe and will be completely dominant in the late universe in the future. And we just happen coincidentally to live in the one time in the history of the universe when these two numbers are comparable. So this is called the coincidence problem. And I was really convinced in this coincidence problem. I made some plots that got into the paper that explained how hilariously unlikely it would be for the cosmological constant to actually be of the same order of magnitude as the matter density. But it doesn't matter how much theoretical uncomfortableness you have, eventually you're gonna have to go look at the data. And that's what people did. So in 1992, the COBE satellite discovered the anisotropies in the cosmic microwave background.

1:36:20.6 SC: So the cosmic microwave background radiation left over from the Big Bang was discovered in the '60s, but between the '60s and the '90s, whenever you measured it, it was the same temperature, the same intensity in every direction of the sky. It looked perfectly smooth. We all knew that it couldn't be perfectly smooth because we knew there's galaxies in the current universe. Those galaxies had to come from somewhere. And that implied there must have been tiny ripples in the universe at early times, which is what the microwave background is reflecting. So eventually we're gonna find them, and the COBE satellite did find them. Again, Nobel Prizes all around. That was in 1992. And I remember it vividly, that this was one of the first times in my life that I got to feel like an insider, because Bill Press, who was an insider and also an inveterate gossip, he knew before the official announcement that COBE had found these anisotropies, and he told everyone at lunch.

1:37:19.7 SC: And then I got to tell people in the physics department, like Sidney Coleman and others, that COBE had discovered the temperature anisotropies in the microwave background. Everyone knew that was a huge deal, and I got to be the bearer of news and feel like I was a little bit ahead of the game. So I'm not above feeling good because of those kinds of things, even though they're pretty meaningless in the ultimate standard of things. But anyway, it was again a kind of an interesting historical moment because people knew about the prospect of finding anisotropies in the microwave background. Jim Peebles and others, and Zeldovich and others, had worked out all the math in the '70s and the '60s, for that matter, for what those anisotropies should look like. But people hadn't really thought it through, like the rest of the community hadn't really absorbed, so what can we do with this data that's going to come in? Okay. We have this telescope, the COBE satellite, which detected the existence of these anisotropies, but it was very primitive.

1:38:21.3 SC: It only just found that they're there, didn't really do a great job of the details. It was not a very highly in-focus image or anything like that. So we knew that future telescopes would be bringing us enormously richer data about the microwave background anisotropies. And soon it was figured out that indeed these data that were going to come in would be very, very informative about cosmological parameters, like the Hubble constant, like the cosmological constant, like the dark matter density, all of those things. That turned out to be true. It took another 10 years between COBE and other ways of measuring the microwave background before we really started to squeeze information out of it. But it was clear, even in 1992, that that was going to be coming soon. The other thing, of course, that was data-oriented, which turned out to be decisive, was measuring the Hubble expansion rate, the Hubble diagram, that is to say velocity of distant things versus their redshifts... No, that's not true.

1:39:23.9 SC: Velocity and redshift are the same thing. Distance of distant things versus their redshifts in a new way, namely using Type Ia supernovae. So I had nothing to do with that except I was plugged in. Like, again, I was in the right place at the right time and my friends were very involved with this. So the idea here is that there's different kinds of supernovae. I don't understand all the different classifications. That's real astronomy. But there are Type IIs and Type Is, and the Type Is have subdivisions, Type Ia, Type Ib, etc. My office mate, Brian Schmidt, who was a graduate student of Robert Kirshner, who was a professor at the Harvard astronomy department, he might have been the chair at that time, he was certainly a chair at some point, Bob Kirshner. And Bob was like the guy in observational supernova cosmology. Back in those days, you don't remember, you kids are too young, but we didn't have that many supernovae.

1:40:22.4 SC: They were hard to find. In a big galaxy, you might be able to see one supernova per century. And so if you don't look at many, many, many galaxies, you're not gonna see any unless you're very lucky or they're very close. So there had been a couple, there had been a few supernovae that had been found, more than a few, but it was not a systematic procedure for finding them. And Brian Schmidt, my office mate, worked as his thesis project on Type II supernovae and measuring the Hubble constant using what's called the expanding photosphere method. Basically, you model the expanding Type II supernova and you use geometry and blackbody radiation and physics to compare the brightness versus the distance and things like that. And it was pretty good as a method for measuring the Hubble constant, but it was not quite good enough. There's just too many uncertainties in the sort of sphericity of a Type II supernova and things like that. The other thing you could do is Type Ia supernovae.

1:41:21.8 SC: There the physics was a little bit less understood, maybe still is, I'm not really up on it, but for some reason, it seemed as if Type Ia supernovae were almost standard candles. That is to say, almost every Type Ia supernova is approximately the same brightness. There is an understanding for why this is true. A Type Ia supernova... Well, sorry. Let's go backward. The Type II supernova happens when a red giant star runs out of fuel, collapses and bounces. And different red giants are different, so that's gonna be different for different Type II supernovae. A Type I supernova, Type Ia supernova is supposed to be when a white dwarf has a companion star that is gradually dribbling mass onto it and eventually it hits the Chandrasekhar limit. It collapses and blows up, and that's a Type I. Unlike the red giant explosions when they give out their nuclear fuel, the Chandrasekhar limit is universal. All the white dwarfs have the same Chandrasekhar limit. So you can kind of see why Type Ia supernovae should be the same brightness.

1:42:27.9 SC: And they're very bright. That's the good thing. Now, they're not exactly the same brightness, and that's an important thing, and I'll talk about that in a second, but they're approximately. So that means you can know approximately how far away they are, at least relative to other Type Ia supernovae. And if you find a Type Ia supernova in a galaxy that you know the distance to already, you can figure out what that absolute brightness is. So in Bob's mind... Bob worked on Type Ia supernovae also, Bob Kirshner, and the graduate student who he put on that task was Adam Riess, who was not my office mate, but was in the floor below us. So Harvard and where I was as a grad student was like the center for cosmological supernova observations at that time. But the focus was on measuring the Hubble constant, not the cosmological constant. The Hubble constant is a natural first thing to try to measure. There was an idea, though.

1:43:25.6 SC: What if we didn't wait around to find supernovae by accident, but we made a dedicated program to go look for them? And this idea is very, very telling historically. It's a very good lesson in the history of science. It wasn't the astronomers who had that idea. It was the particle physicists. Saul Perlmutter in particular was a particle physicist who said, "Why are you just waiting around for these supernovae to happen? Let's get a big telescope, look at a lot of galaxies, and let's in a deterministic, know-aheadable way, find a bunch of supernovae and measure their distances. That would be the better way to do it." Particle physicists think big. They don't want to take pictures of galaxies and hope something interesting happens. They want to be systematic and go over the whole sky. And I remember having a long phone conversation with Saul and Ariel Goobar, who was his co-author on one of the very early papers where they talk about, "Could we measure the cosmological constant doing this?" because I was still a grad student at the time and a nobody, but I had been an author on this review article with Bill Press and Ed Turner that they knew about.

1:44:32.2 SC: So I became friends with Saul, et cetera. And of course I was also friends with Bob and Brian and Adam and everything. And I still think there's a good history of science book to be written, this would not be by me, but by somebody, about everything that went down in that time. Because basically Saul and his collaborators formed the High... Sorry, formed the Supernova Cosmology Project, the SCP, that's what they called it. And they were largely ex-particle physicists who said, "Damn the torpedoes, we're gonna do this." And they had the ability to get a lot of money from the Department of Energy and they had good expertise at computer programming to reduce the data and things like that. They just went out and did it. The astronomers, including Bob Kirshner very noticeably, were very skeptical that it would ever work, for good reasons, not for bad reasons, they weren't just protecting their turf, they knew the pitfalls. They knew the possible ways everything could go wrong.

1:45:30.0 SC: They knew that not all the Type Ia supernovae were the same. They had data saying that. There was a method that Bob and others actually pioneered, and Adam was involved with it, and I think Bill Press was involved with it, that tried to standardize the candles. So not all Type Ia supernovae are exactly the same brightness, but if you plot the amount of time it takes for them to get brighter and dimmer, that's related to their absolute brightness. This is the Phillips relation, after Mark Phillips. And so they were trying to figure out how to reduce a heterogeneous sample of many supernovae down to a homogeneous sample and make them all behave the same. Then they would be actually acting as standard candles. So they're standardizable candles even though they're not standard. And the point was they were so impressed with all the difficulties that they thought that this Perlmutter thing was never gonna work. And my understanding is, and my understanding very incomplete because it's just from my own knowledge, it was Nick Suntzeff and Brian Schmidt.

1:46:39.3 SC: Brian and Adam had both by this point graduated and moved on to other things. But they said, "You know what? I think we can do it." In fact, I think that these are the astronomers saying, "I think we can catch up because we know astronomy better than those particle physicists do." And so Saul and his group started first but needed to learn the astronomy. Brian and Nick and their group, which eventually became known as the High-Z Supernova team—Z is for redshift, so High-Redshift Supernova team, started later but caught up. And they were very good at reducing the data and whatever. And so Adam and Bob and many other people joined that group as well. And what happened is they were both right. Saul was absolutely right in saying that it could be done and starting it. And Brian and Adam and Bob and Nick and all those other people were also right in that you need to be careful and get it right and do it correctly. And they were plugging away. And I remember a very... It's a very cute story.

1:47:38.9 SC: I remember getting in the mail a set of papers that the astronomy group had written, the High-Z Supernova team. So they were advertising their work. We didn't have the archive, but I guess we did have the archive back then, but still, people would mail papers to each other. So they had put a bunch of the papers they had written into like a spiral notebook and mailed it to people who they thought might be interested. And the title page was "The High-Z Supernova Project: Measuring the Deceleration of the Universe." The irony being that what they actually ended up measuring was the acceleration of the universe. But no one expected that at the time. They thought they would measure the Hubble constant and maybe they could measure the critical density, or the density, I should say, the density parameter of the universe, which would tell us, did we have the critical density or not? And so it was in 1998 that both teams came out with their results.

1:48:35.2 SC: But let me tell you one more thing about 1997. In 1997, I was a postdoc at the Institute for Theoretical Physics in Santa Barbara, and I was doing some crazy things with topological defects and extra dimensions and stuff like that. Who knows what I was working on at the time. But Phil Lubin is an observational cosmologist, a microwave background guy who's a professor in Santa Barbara. And he organized a little workshop, December 1997. And basically it was on determining cosmological parameters, like I said, the Hubble constant, cosmological constant, et cetera, from the cosmic microwave background. They didn't have the data to do that in 1997, but they were working on how to do it. So they had a workshop to do that. And Phil asked me, could I give a talk at the workshop? I knew nothing about how to extract cosmological parameters from the microwave background, but he said, "Could you do us a favor and give a talk on measuring cosmological parameters in all the ways that are not the cosmic microwave background?"

1:49:42.6 SC: So from galaxies and supernovae and things like that. So I said, "Sure," and I did that. And I gave this talk. And it was only because I was forced to do it because I was giving the talk, but I went through an enormous amount of papers on modern observational cosmology from very different perspectives. And it was super informative because over and over again... So this is 1997, the favorite theorists' model of the universe was a critical density of matter, the good old flat cosmological universe. The observers' favorite universe was an open universe with only 0.3 the critical density of matter. And over and over again, there were from different angles things in the data saying the theorists' favorite universe can't be right. We didn't know what it was. I gave a talk and I said, "Look, I don't know what's wrong, but there's enough data here already in 1997 to say that the theorists' favorite model of Omega equals one," as we would say, the density parameter of the universe is exactly the critical density, and it's all made of ordinary matter, and that's what inflation predicts and things like that, that model can't be right.

1:50:55.4 SC: Maybe it's because the universe is open, there's not enough matter to make the critical density. Maybe it's because there's a cosmological constant. Maybe it's because dark matter is warm or mixed rather than cold, or maybe there's a tilt in the spectrum of things like that. There's a whole bunch of different possibilities. I didn't know which one it was, but I knew that something was going on. And that was the mood in cosmology at the time. So in February 1998, when the High-Z Supernova team, led by a paper that Adam Riess was the first author on, said, "You know what? We did our measurements. It looks like there is a non-zero cosmological constant." And a couple months after that, Perlmutter and his group came out and said, "Yep, that's what we got, too. We're both getting exactly the same answer." This was a very dramatic claim. Adam and others have explained in interviews that they didn't believe it themselves when they got it.

1:51:49.8 SC: They're like, "It's very easy to make a mistake. Maybe your computer program was wrong. Maybe you put in the wrong numbers somewhere." You don't want to write a paper saying the universe is accelerating unless you know you're on the right track. But eventually, they couldn't make it go away, so they published their paper. And I had a little bit of an inside scoop. I was friends with all these people, so I got copies of the plots and I got to give talks in Santa Barbara telling everyone the news, and everyone was super-duper excited. But they were willing to accept it very, very quickly exactly because it made everything snap into place. It's, again, a super interesting lesson for the history of science. Not all claimed dramatic discoveries are created equal. If you claim a dramatic discovery that is literally flying in the face of everything that we think we know, people are correctly going to be skeptical of it. Whereas if you claim a dramatic discovery that makes everything fit together that we think we know, people are much more likely to believe it right away. Not everyone did.

1:52:54.2 SC: There were some grumpy people who said they didn't believe this supernova stuff. That's okay. There's always a place in physics for a couple of grumpy people, as long as eventually they realize that the data become overwhelming. And then, long story short, it's already pretty long, but in the early 2000s, we did get the cosmic microwave background data from both ground-based telescopes and from the WMAP satellite and elsewhere. And the CMB confirmed the picture of the universe that had been put together by the supernova data, where 0.7 of the critical density is the vacuum energy, the cosmological constant, and 0.3 of it was the matter, just like the astronomers had been saying all along. So that is the still best fit 20 years later: Lambda-CDM cosmology. Lambda is the letter we give to the cosmological constant. CDM is for cold dark matter. And that's the best we can do for fitting the universe, and it fits it very, very well. It's not 100% perfect.

1:53:56.8 SC: We talked to both Adam Riess and Marc Kamionkowski at different times here on Mindscape, and they explained to us that there are things like the Hubble tension that don't completely fit in. But that's okay. That's what drives progress in science. The Hubble tension is something where we don't have a good theoretical model. It's not making everything snap together. So we're not sure whether that's going to go away or not. But roughly speaking, we have a model that fits really well, and the microwave background in the early 2000s confirmed everything that the supernovae had been telling us. So where does that leave us? Well, it's a good fit to the data. The coincidence problem, the fact that we live in the unique era in the universe where both matter and vacuum energy seem to be relevant, is still very much with us. The way I like to put it is you should think about the coincidence problem and the cosmological constant problem in the following way: the best solution that we have on the market right now is the anthropic principle. And that's not a good solution. It's a very messy solution.

1:54:57.0 SC: There are things about it we don't understand theoretically. It involves a lot of speculative physics. It's still better than any of the other alternatives. Again, not to say that it's right, but that is the state of the art. One thing that is still percolating in the back of people's minds is the cosmological constant problem, different than the coincidence problem. Why is the vacuum energy so small? That's just been made harder by the fact that it's not zero. Again, in the minds of most theorists, it was easier to think of something that would set it all the way to zero than something that would set it to 10 to the minus 122 times the Planck scale. But that's apparently where it is. It might be, if you want to think dramatically, it might be indicating that this whole effective field theory paradigm fails when gravity and cosmology are in the game. On the one hand, if you just say it casually, it doesn't sound that crazy. Like, yeah, gravity is weird, quantum gravity is weird, it's not an effective field theory. Okay.

1:55:58.6 SC: On the other hand, even gravity should fit into the effective field theory paradigm from everything that we know, so it's not really clear why it doesn't. On the third hand, some people have boldly said, yeah, maybe that's what's going on, and tried to figure out what that means for the cosmological constant problem. I would say Tom Banks and Willy Fischler, who I mentioned earlier in the previous solo podcast as really pushing the idea that the dimensionality of Hilbert space is a finite number, they've also been saying if you take that whole idea seriously, it completely changes the cosmological constant problem because the dimensionality of Hilbert space is related to the cosmological constant, and it therefore is not an ordinary parameter in an effective field theory. So I'm saying these words out loud without really understanding them or expecting that you understand them, but I'm just trying to let you in on how some physicists are thinking about these big issues right now.

1:56:56.6 SC: It's only one number that we've measured, the cosmological constant, so it's very hard to build detailed models on top of that. We don't have that much guidance, but that one number is spurring us to new ideas. It does imply, if the cosmological constant is truly a constant, it does imply that the... We know about the future of the universe now. Like, when I was your age, we were still taking very seriously the idea maybe the universe would recollapse someday. We don't know. If it's truly a cosmological constant then that is making the universe accelerate right now, there's no reason for that not to continue forever. The cosmological constant doesn't go away. It's a constant. The density of energy in empty space is a number. It is about 10 to the minus eight ergs per cubic centimeter. That will never go away. The universe will get emptier and colder and more desolate and lonely forever and ever. So enjoy the time you have in the universe right now. That would be the lesson.

1:57:58.1 SC: Or the other possibility is maybe the cosmological constant is zero. Maybe we were right all along back in the '80s when we thought that in the space of all possible ideas we haven't come up with yet, it is easier to think of an idea that sets the cosmological constant to zero than to some small number. And you say to yourself, "Well, wait a minute, you just told me that the data don't allow that." But that wasn't quite true. The data say there is something making the universe accelerate, but maybe that thing is not the cosmological constant. The cosmological constant is a very, very good fit to the data right now, but we were surprised when we found it, and so we should be open-minded. And being open-minded means we should think about the possibility that what is making the universe accelerate is not the cosmological constant, but something else.

1:58:46.0 SC: So the term dark energy is the catch-all term for something else, or even including the cosmological constant. The cosmological constant itself is a form of dark energy, but there are other forms. There's other theoretical models for kinds of dark energy that might change with time just a little bit. The cosmological constant is constant. Dynamical dark energy can change just a little bit, and that opens up an enormous number of possibilities. If I were you, I would return for the second installment of this solo exploration of what's making the universe accelerate next time here on the Mindscape Podcast. Thanks for listening. Talk to you next time.