315 | Branden Fitelson on the Logic and Use of Probability

0:00:00.5 Sean Carroll: Hello, everyone, and welcome to the Mindscape Podcast. I'm your host, Sean Carroll. One of the things that I always like to say about science and how it gets done is that science never proves things. This is something that is an important feature of science, especially in the modern world, where what science does, how it reaches conclusions, how trustworthy it is, these are all under contestation by different parts of society. So it's important to understand what science is and how it actually reaches its conclusions. And the claim that science never proves things, which is something that most scientists would go along with me on, comes from a comparison to real proof in mathematics, or for that matter, in logic. Most scientists have taken some math classes, at least enough to know what it means to prove something in the old fashioned sense of Euclid and geometry or Aristotle and logic, proving a conclusion from some well articulated premises. In the philosophical study of logic, this is known as deductive reasoning. You have some premises and you reach a conclusion. And science just doesn't go that way. Right? Science looks at the world, it looks at all sorts of things in the world and it tries to figure out what the patterns are that the world follows.

0:01:20.4 Sean Carroll: Always knowing that tomorrow you might do a new experiment that will overturn your best guess as to what the pattern was. Or maybe someone will do something as simple as just thinking of a better pattern, right? A theoretical physicist coming up with a better idea for what the laws of physics really are. So if science doesn't prove things, if it just sort of comes closer and closer in some sense to getting it right, then what is? What's going on? One very common idea about what's going on is inductive logic rather than deductive logic and inductive logic. We begin to see a pattern. A, B, C, D, E, F, G. The next one is probably gonna be H, right? 'Cause we think that probably you're just mentioning the Alphabet in alphabetical order. But there's all sorts of paradoxes that come up when you do inductive logic. Like how do that it's not A, B, C, D, E, F, Z? That's a sequence of letters that you could have. My old math teacher in college used to hate those SAT questions or standardized test questions that would give you a series of numbers and ask you to guess the next one?

0:02:26.2 Sean Carroll: 'Cause he said, I can make any number I want. I can come with a formula that would give you any number I want after the ones that you already showed me. So philosophers, unsurprisingly, are very interested in making as rigorous and careful as possible. This idea of either induction or whatever should replace induction as the logic of understanding things in science. The names attached here go from old school names like David Hume and John Stuart Mill to relatively newer ones like Rudolf Carnap, Carl Hempel, Karl Popper, for example. And it's still an ongoing thing. So this is something that lives at the intersection of how we think about science, but also how we think about probability, what probability is, how conditional probabilities work, Bayesian logic, all that stuff. And that's what we're gonna be talking about today. Today's guest is Branden fitelson, who is a philosopher at Northeastern University. And it's eye opening to me as someone who is now part time in a philosophy department. Just a huge range of stuff that gets characterized as philosophy, right? Some philosophers are saying what is the good? Others are saying what happens when an observation is made in quantum mechanics.

0:03:42.9 Sean Carroll: And others are like doing pretty hardcore mathy logic. And well, there's mathematical logic, but there's also just big picture questions about logic. How does logic, how do probability, how do these things get used both in a perfectly rational world and also in the slightly irrational world in which we live? We're gonna be talking about both of those things, and it's intrinsically interesting to understand probability and logic better, but also super important to thinking about how science works. So let's go. Branden Fitelson, welcome to the Mindscape podcast.

0:04:32.6 Branden Fitelson: Thank you so much for having me, Sean. I'm a big fan. What you're doing here is super important, especially nowadays.

0:04:39.1 Sean Carroll: I hope you still think those things after we are done talking, but I hope to make it true. So, we're talking about stuff that is dear to my heart. We're talking about increasing the probability that something is right. We're talking about what probability is, how it fits in with learning about things. Obviously, science cares about this a lot. So, let's start at the very high level and you tell me what probability really is.

0:05:05.3 Branden Fitelson: What probability really is. Well, there's many kinds of probabilities. So there's probabilities in science. So for instance, biology has its own conception of probability, which shows up in the theory of natural selection especially. And genetics. Physics, of course, as better than I, has lots to say about probability both in classical and quantum physics. Yeah, so. And in economics we also use probability and in many other special sciences. My interest in probability, I started out as a physicist, so I started out interested in probability and Physics. That's how I got into it. But over the years, I became more and more interested in the role probability plays in thinking about evidence and how strong arguments are, that is, how strong something is as a reason for believing something else. And that's kind of the application of probability that I'm most interested in nowadays. And in that context, there's still many kinds of probabilities, because when you're assessing the strength of an argument, it really depends on the context. So if you're playing a game of chance, say, and you're like poker, and a certain card comes up and you're wondering, well, what effect does that have on my probability of winning this hand?

0:06:25.0 Branden Fitelson: Well, now what probabilities to use. They're given by the probabilities of a game of chance. Each card is equally likely to be drawn. And that allows you then to calculate the probability of any hand given what's left in the deck and so on. And so there it's very clear what probabilities to use to assess the strength of that as a reason for believing, say, that you'll win or lose the hand. In other contexts, it's much more difficult to say which probabilities are the appropriate ones. So, for instance, if we're wondering whether a certain scientific theory is true, in fact, we might even be worried about whether a certain scientific theory of probability is true. And you might have two competing theories of what probability in a certain scientific context is like, well, there. How are you gonna adjudicate how strong the arguments are? Well, you can't. It would be question begging to assume a notion of probability that, say, one of the theories adopts but the other rejects. That would just be question begging. So what you need is some more general notion of probability that will allow you to evaluate arguments even in those contexts.

0:07:28.8 Branden Fitelson: And as a philosopher, I wanna go even more broad than that. I wanna be able to assess arguments for the existence of God maybe, or for ethical claims and so on. And as you get more and more abstract and these contexts get further and further away, say from games of chance, which is kind of the easiest case, it gets more and more controversial. What kinds of probabilities are the relevant ones. But of this like any other science, Sean. Probability theory, it's a theory. And then what you do when you're faced with a certain situation is you have to construct models of the theory. And okay, that's a very complicated process which involves making all kinds of assumptions and idealizations. And that's okay. And the goal there is to try to come up with the best account of which probabilities we should use. So that we're adjudicating this question of how strong the arguments are in a way that's fair and reasonable. And that's really a case by case thing.

0:08:27.3 Sean Carroll: So. I was just a couple weeks ago at the retirement celebration conference for Barry Lower, former Mindscape guest. And there was a talk by David Albert, former Mindscape guest and in part in response to things that I and others have been saying about quantum mechanics and self locating probabilities. And David, he was just unapologetically old school about it. He says the only sensible use of probability is when you have a frequency of something happening over and over again. And you can sort of imagine taking a limit of it happening infinite number of times and the ratio of the number of times where it looks like X rather than Y. That's the probability. And I tried to say, but okay, come on. We certainly use probability in a much broader sense than that. We talk about the probability of a sports team winning a thing, even though we're not gonna do it an infinite number of times or even twice. So is there a consensus about this very basic question about the relationship between frequencies and probabilities versus just a more epistemic, this is my best guess kind of thing.

0:09:35.6 Branden Fitelson: Right. Okay, good. Yeah. So there's lots of what used to be called interpretations of probability, but I would just call them theories of probability. As I say, there are many. The frequency theory, well, it's a very strange theory, actually. It started off as an actual finite frequency theory where the probability of some event is actually just given by the actual frequency of some event in some population. So, for instance, suppose you have a coin and it's been tossed exactly five times, three heads and two tails, and then it's destroyed. Well, according to the actual frequency theory, the probability is three fifths that it's heads.

0:10:16.5 Sean Carroll: Wow.

0:10:17.8 Branden Fitelson: And in fact, if there's any odd number of tosses, actual odd number, then you can't get an even. You can't get one half. So even if you have a fair coin, if it's tossed an odd number of times, then according to the actual frequency view, the probability, it isn't fair. So the actual frequency view was a non starter. Right? That's not going to work.

0:10:35.8 Sean Carroll: I hope so. Yeah.

0:10:36.5 Branden Fitelson: Also, you can't get irrational values. You can't get you can only get rational values. And that seems wrong 'cause physics has all kinds of IR rational value problem. Okay, so then people said, well, okay, maybe what we'll do is we'll talk about hypothetical infinite extensions of the actual experiment. Okay, well, what does that mean? They say things like, well, it's what would have happened had you continued indefinitely that initial sequence of five tosses. And I wanna say, well, that's very hard to understand because there's uncountably many such extensions, and on almost all of them, there's no limiting frequency. So it's true that for any real number, you can get that as the limiting frequency of an infinite sequence. But it's also true that almost all of the sequences don't have. They diverge in their limiting frequency. And so...

0:11:29.8 Sean Carroll: Sorry, this is. 

0:11:32.2 Branden Fitelson: Yeah.

0:11:32.8 Sean Carroll: Sounds like you're paraphrasing some technical result with the use of the idea of almost all. That's a technical math term.

0:11:39.4 Branden Fitelson: Yes, that's right. I just mean. Well, it's not all. There's like a relatively small number of sequences that will converge, but it's sort of like if you pick a real number at random, it's like, what are the chances of getting a rational number? Pretty small. Most of them are not rational by any reasonable measure of most. And the same thing is true here. You have all these sequences, well, which one? And so then you've gotta say, well, which hypothetical infinite extensions are the ones that actually give you the real probability? And I just think this is the wrong way to go. My view is, I like to make an analogy with measurement in general, say in physics. So you might think, might ask you, what is mass? Say, suppose just for the sake of argument, that we're in a Newtonian universe and mass just behaves the way Newton thought it did, just for sake of argument. And then you think, well, what is mass anyway? Well, my view about what mass is in such a universe is it's whatever the theory says it is.

0:12:41.7 Branden Fitelson: It's the functional role played by that concept and all the laws. And that's a very complicated thing. There's no easy way to summarize. It's just whatever Newton's theory says it is. But you might be tempted by a different view. You might think, well, wait, maybe it's just frequencies. Maybe it's just what you do is you make measurements and then you take an average. And maybe if you take infinitely many measurements and you take the limiting value of the average. Maybe that's what the mass of the object is. No if you're lucky then if you were to do that, of course you can't do it. But if you were to do that, then if you're lucky you would get something very close to the actual mass. But that isn't what the mass is. 

0:13:24.8 Sean Carroll: Right.

0:13:25.6 Branden Fitelson: And I wanna say the same thing about probability. Suppose you're doing some quantum mechanical experiment, right? You can make measurements. That's what you do. You make a lot of measurements and you take averages and you do statistics and that's how you estimate the probability that something will be observed in a quantum mechanical system.

0:13:41.7 Branden Fitelson: But that's not what the probability is. The probability is what the theory says it is, and whatever that is. So one property it has is you use Born's rule to calculate what the probability is. Okay, that's a really complicated theoretical story. But the probability isn't any sequence of measurements, it's not any limiting frequency. That's a symptom of this property of probability. But the property is what the theory says it is. So I just think the frequency view's got everything backwards. Frequencies are just the way we maybe know about probabilities, but they're not what the probabilities are. So that's you.

0:14:16.9 Sean Carroll: I'm very sympathetic to that. How does that view fit in with the sort of classic divide between thinking that probabilities are mostly epistemic, they're about our knowledge versus that probabilities latch on to some objective chances out there in the world?

0:14:32.7 Branden Fitelson: Oh, certainly there are objective probabilities. As I said, not just in physics but also in biology. Each theory has its own concept of probability and at least the probabilistic theories do. And what probability is in those systems is whatever the theory says it is. It's just like mass. So that's, I have a very flat footed view of that. And so in quantum mechanics, well, we know how to calculate probabilities. The theory tells us to. In statistical mech, we can also calculate probabilities as well. And we can do that as well but, and now you might wonder about the interpretation of those probabilities. But you can certainly calculate things which obey the laws of probability in statistical mechanics. And so in that sense at least they are probabilities. They satisfy the formal principles of probability. And so yeah, I wanna say certainly there are objective probabilities. No question. I'm a scientific realist. So, if I accept a theory and the theory says there's a thing then there's that thing. That's it. So I'm a realist, so I don't have any problem with that.

0:15:37.1 Branden Fitelson: However, the problem is, and where things get really tricky and this is what got me really interested in other notions of probability. The tricky thing is, as I said, suppose there's a dispute about the nature of probability in some physical context. Right? There's a dispute about that. You have two theories. One theory says probability behaves like this, another theory says behaves like that. And then you do experiments, okay? And you try to use that data to adjudicate. Does the evidence favor the one theory of probability over the other? Well, whatever probability you're using there, you got to be very careful 'cause you don't wanna beg any questions. So you don't wanna use the probability that the one theory says it's correct, but the other says it's incorrect to do the very calculations of how strong the arguments are. That would be question begging. So, in those settings, you need some other notion of probability. And that's where the epistemic notion comes in. You need it in at least these contexts where you're actually trying to adjudicate different physical theories of probability.

0:16:41.1 Branden Fitelson: Say you can't use what one theory says to adjudicate because that would just beg the question against the other theory.

0:16:48.8 Sean Carroll: Yeah.

0:16:49.1 Branden Fitelson: And so in those contexts at least you're gonna need some other notion of probability, something neutral like a judge is impartial. So you need some impartial notion of probability. And this is the kind of notion that statisticians have been trying to come up with ever that early work in genetics, which is where it all really started with Fisher and, and Haldane and all those kinds of people, and Pearson. So, but this is where Bayesianism is helpful, the Bayesian approach, because at least in these contexts where we're not sure, we're uncertain what the correct theory of probability is, we need something that's, it feels like it's gotta be epistemic, at least it's gotta be neutral and it's gotta be something you can use to adjudicate. Does the evidence favor one theory of probability, say, over another? And for that matter, like I said, you wanna adjudicate debates in other areas where who knows what the probability of Newton's theory is? Even if there are objective physical probabilities, it's hard to imagine how they would tell us what the probability is that Newton's theory is true.

0:17:55.0 Sean Carroll: Right. Good.

0:17:55.5 Branden Fitelson: Or, it's just, what would that mean? So, in those contexts where we're adjudicating physical theories, say Newton versus Einstein or two different versions of quantum theory or something else, we're gonna need some other notion of probability. And that's where the Bayesian approach is. Kind of need something like that, because you need something neutral.

0:18:16.9 Sean Carroll: I would have said 15 minutes ago that I don't believe that there is any such thing as objective probability in the world. That there's the world, and we describe the world the best we can. And maybe we have incomplete information, so we appeal to some probability, but there's some exact description of it also. But, and of course, if you're judging between different theories of the world, then you have some epistemic view of probability. But now you're pointing out that. Okay, but there's a notion of a thing that appears in a theory, whether it's quantum mechanics or genetics or whatever, and that thing obeys the laws of probability. It adds up to one and whatever. And we might as well call that objective.

0:19:00.3 Branden Fitelson: Yeah. Just like I would want to call mass objective.

0:19:04.0 Sean Carroll: Yeah okay.

0:19:05.4 Branden Fitelson: I would say the probability in quantum mechanics that's delivered by, the born rule or whatever, however you calculate it, whatever that is, it's some real thing. It's just as real as mass or any other theoretical quantity. It seems to me that the theory implicitly defines through its laws. So, yeah, again, I'm a realist, though, so I have to just fess up to that. But, but as I say, even if you're not a realist, even if you think, okay, maybe there's different kinds of probabilities, but none of them is objective in the relevant sense, still, if you wanna know whether some evidence favors one of those theories over another one, and you want that to be a probabilistic inference, which it is, 'cause it's not going to be deductive. After all, scientific evidence doesn't entail the answer to these questions. It doesn't deductively guarantee that one theory is true and the other is false. It just at best makes favors one over another. And that's gonna have to be, the best way to model that is probabilistically.

0:19:58.4 Branden Fitelson: But then you need a general framework of probability that's gonna have to be, I don't know, epistemic or something less objective in that sense, because otherwise it would run the risk of just begging the question.

0:20:11.1 Sean Carroll: So good. This leads us right into where I wanted to go, which is the idea of induction and how in the early days people tried to hope that inductive reasoning, looking at many examples and generalizing would be a kind of logic that would fit the scientific process. And then other people point out that there are problems with induction. So, pretend we're in the Philosophy 101 class. What are the problems that people have with induction?

0:20:39.6 Branden Fitelson: Well, of course, in philosophy and epistemology generally, you generally start out with the really hard problems like skepticism. And induction is no different.

0:20:49.8 Sean Carroll: Yeah.

0:20:50.6 Branden Fitelson: When you're studying philosophy of induction, you tend to start with these skept arguments. Like David Hume was had a kind of skeptical arguments. He's like, well, okay, you say there are these arguments that are, that don't guarantee the truth of their conclusions if their premises are true. Well, their conclusions, maybe they're quote, unquote, probably true, but they're not guaranteed to be true, like in mathematics. And he, gave this dilemma. He said, well, let's think about how that would actually work. So suppose, you've observed the sun rising, a million times, and you infer that on the basis of that historical evidence that the sun will rise tomorrow. Hume points out that, well, that argument assumes some kind of principle of regularity of nature that, the past, the future will resemble the past. And now if you ask, how are you gonna justify that premise that the future will resemble the past when, well, you can't give a deductive argument for it because, well, how would you do that?

0:21:55.5 Branden Fitelson: Nothing you've observed is gonna entail that the future will resemble the past. In other words, there'll always be some chance that you can't rule out with certainty that the future won't resemble the past. So it won't be a deductive argument. And then if it's an inductive, it just feels like now it's gonna beg the question because, well, wait, what are you gonna do? Reason as follows. In the past, the future has resembled the past. So therefore, in the future and now, you're just, it's now you're just circular. Now it's just a circular argument 'cause you're assuming the very principle that you mean to justify in order to justify the argument. So, philosophy always starts with these skeptical arguments. But you don't have to worry about induction. This happens in every field. Like why believe there's an external world? After all, you can't rule out with certainty that there's an evil demon or that you're in a simulation or etc. So what you gotta do when you're doing philosophy, there's sort of the first thing you have to do in any of these domains is figure out how you're gonna respond.

0:22:53.3 Branden Fitelson: The skeptic, that is. What are you gonna, what are you gonna say? Why do you think that there's an external world for it? Let's start there and then we'll get to induction. Well, why do you think there's an external world? Well, I can only speak for myself. The reason there's an external world is when about everything that I take myself to know, everything I take to be evidence about the world. All my observations, everything I take to be true. And well, what's the best explanation of all of that? To me, I don't see any way to plausibly explain all that stuff without postulating the existence of an external world that is mind independent in many ways. And that's why I think there's a mind independent external world. And now I wanna take the same anti skeptical view about induction. Well, how do you explain, say the success of science or what appears to be the progress of science? Well, I don't know, but it seems hard for me to be able to explain that unless there weren't some principles of when evidence actually does favor one scientific theory over another and does provide reason to believe one rather than the other.

0:24:09.2 Branden Fitelson: And so what I tend to do is think about historical cases of that look like real scientific progress and then what's the best way to explain that? So for instance, when Einstein's general relativity theory of general relativity overtook Newton's theory of celestial motion, there were a lot of experiments that were crucial. One was the motion of Mercury. The motion of Mercury. Mercury moves in this very strange way around the sun. And it was known, that was known for a long time that it had this strange motion. Well, the Newtonians tried their best to give explanations of that and in the past they had had similar episodes, but they were able to explain it by some missing mass that they found, that was in the universe that they didn't know about. And but eventually they realized there isn't gonna be the right hidden mass here. It's, Newton's theory is just not gonna be able to predict this. This is just a thing that Newton's theory can't explain. Can't predict. And then Einstein comes along and gives a theory that explains all the stuff Newton's theory could explain and this thing too, and a bunch of other stuff that it couldn't explain.

0:25:14.4 Branden Fitelson: That just now I wanna say, well, that just seems to make it more probable that Einstein's story is true or at least more probable that, would be the better bet to make, that would be the more acceptable theory. And a probabilistic way of modeling that is just the best way that I know, know to model it. And so again, I just think, well, what's the best explanation of these episodes of scientific progress? And to me part of that has to be, well, there just must be cases where the evidence really does favor one theory over another. Not that it guarantees that one's true and the other's false or anything like that, but it sort of raises the probability of one more than the other. And I just think, I don't know how else to explain science episodes of scientific progress unless something like that is true. So I believe that something like that is true. Now the details of it are difficult to work out, but this is what statisticians, as I said, have largely been trying to figure out how those inferences work. Like when we have an experiment and we think the evidence favors one theory of another, what's the right way to use probability?

0:26:18.9 Branden Fitelson: Right? To model that. And there's a lot of disagreement, of course, in statistics between Bayesians and classical statistics. There's all kinds of different schools, but one thing they all agree on is there are episodes where the evidence favors one theory over another. And probability is an indispensable part of the explanation why they all agree on that much.

0:26:39.8 Sean Carroll: It might be unfair of me, but I do think that it's a very common phase in an individual's philosophical maturation to realize that not everything can be established on rock hard foundations that you agree with 100%. Sometimes you just gotta say this is the best we can do with what we got.

0:27:00.7 Branden Fitelson: Absolutely. most of the time we're kind of in that situation and that's okay. So but that's the nature of these inferences. As I said, it's not like deduction. You don't have the certainty of mathematics in these kinds of inferences. So there's gonna be something that's under determined. It's not gonna exactly determine completely what our attitude should be. There's gonna be some wiggle room, some leeway. So in a way, you're always making something of a leap of faith when you do one of these amplifications or inductive inferences. And I just think you kind of have to live with that and do the best you can.

0:27:37.0 Sean Carroll: And this leads us right into. You're very good at this. You're just bringing us along on the logical train of thought that we need to be on the idea of confirmation. What we're trying to do is to formalize this idea. Like you just said that Einstein's theory is simple, it fits the data. Newton's theory doesn't fit the data. In some sense, Einstein has now become more probably right than Newton. What sense is that? And confirmation is one of the words that gets batted around. I want you to really sort of carefully explain to us what that's supposed to mean, because many people informally think that if you've confirmed something, you know it's true 100%. And that's not how philosophers use the word.

0:28:18.5 Branden Fitelson: No, that's right. So yeah, in, in ordinary language, the word confirmation has very strong connotations. But in the philosophy of induction, confirmation it's actually a very weak claim. And a helpful example. I like to use simple examples, nice example to use is one of diagnostic testing. I always like this example and in a way, it's kind of fully general 'cause in a way you can think of scientific experiments as a kind of diagnostic test where you're testing the world to see whether some hypothesis is true or false. And so when you design an experiment, you really are in a way designing a diagnostic test. But let's think about diagnostic testing. So, for instance, there are many diagnostic tests that are very reliable that you can buy in the store now. So for instance, you could buy a pregnancy test or an HIV test, any of these tests that you buy. If you read the box, you'll notice something very interesting on the box. There's things they tell you and there's things they don't tell you. So one thing they tell you for sure is what they call the true positive rate and the false positive rate of the test.

0:29:28.0 Branden Fitelson: Right? So the true positive rate is something like this. Suppose that you have the disease. Then how probable would it be that you would get a positive result from this test? And then on the other, the false positive rate is, suppose you don't have disease. Then how probable is a positive result? And what the great thing about these diagnostic tests is you can determine those error rates in the laboratory. You don't need to know anything about the subjects, the particular subjects that are using it, and so on. And that's why they can put that information on the box. It's very reliably known. Well, that ratio of the true positive rate to the false positive rate is called a Bayes factor. It's also called a likelihood ratio. And it doesn't determine how probable the hypothesis is given a positive result. It doesn't determine that. In order to know that how probable it is that you have the disease, you have to plug in what's called a prior probability, an a priori probability, philosophers call it. And what is that? Well, that's something like how probable you think it is before looking at the evidence.

0:30:34.5 Branden Fitelson: Okay, well, what is that? Well, of course, the guys who design the test, they can't tell you what that is. That's gonna depend very sensibly on things about you. So for instance, suppose it's a pregnancy test. And if someone takes a pregnancy test and they get a positive result, well, they'll know the likelihood ratio. They'll know that the error rates, false positive and true positive. So they'll know how reliable the test is in that sense. But to get how probable it is that they're pregnant, well, they need to know a lot about maybe their own behavior in recent days and so on, which of course, the designers of the experiment can't know and don't need to know in order to know the error rates.

0:31:13.1 Sean Carroll: Right? So just...

0:31:14.9 Branden Fitelson: Yeah. Go ahead.

0:31:15.9 Sean Carroll: To put an example on this, so if there is a pregnancy test that the likelihood is very high, it is claimed that if it comes out positive, the likelihood that you're pregnant is very large. But if I took a pregnancy test of that form, I am biologically incapable of becoming pregnant. I know that with pretty high probability. So if I happen to get a positive, I would not conclude that my probability of being pregnant is high 'cause my prior is so low.

0:31:46.8 Branden Fitelson: Exactly. In fact, it might even be zero depending on the case, but it'll be very close to zero. And that's exactly the distinction that I wanna make, this distinction between that Bayes factor, that how reliable the test is, which is just the ratio really of those two error rates, that could be really high. But all that tells you is what to multiply the prior by to get the posterior. Basically, it's like a multiplier. So if you start off low, but not that low, and then you get a really reliable test, well, maybe it's a multiplier by a factor of 1000. Well, then you're gonna have a reasonably high probability. But if you start really low, then even if you have a pretty high factor, a multiplicative Bayes factor, still you're gonna end up low. And this people are very bad at making these inferences. This is something that Kahneman and Tversky discovered back in the '80s. They called it the base rate fallacy. And when people are given an example like this where, okay, so you have a reliable test for a rare disease, they're told the disease is rare, like one in a thousand.

0:32:50.1 Branden Fitelson: And then they're given pretty good error rates and they say, well. And then they're asked, how probable is it that the person has disease? And often people give a very high number. In fact, interestingly, the numbers tend to cluster around basically the Bayes factor, if you normalize it to a zero to one scale. And I don't think this is a coincidence. What's happening here is you have two factors. There are two things that are relevant here. There's how probable it is that you have disease, the probability of the disease, and then there's the confirmation. There's how much the evidence confirms, and that's just how much does it change? How much does it raise the probability? And in these cases what you have is low probability but high confirmation. That can be very confusing.

0:33:36.0 Sean Carroll: Right.

0:33:36.0 Branden Fitelson: Because both of these things are relevant to quote, unquote, how strong the argument is. But that is how strong the evidence is as a reason to believe that the disease is present. But they go in different directions. So it can be very confusing. And then you might. There's still a residual question. Well, why would people defer to the relevance to the confirmation number, right?

0:33:57.7 Branden Fitelson: When they're asked about probability? This is not a crazy thing to do at all. As we said, those error rates are objective and invariant in a really important sense. You can just discover them in laboratories. You could just by working with the causal structure of the test and the chemicals you're looking for, you can be pretty confident about those error rates independently of the prior probability. And so there's something more objective about those numbers. And, there's something really ironic about the Kahneman and Tversky research because if you read their own paper, well that's a scientific paper. And so what do scientific papers do? Well, they generally design an experiment and then perform an experiment and the experiment generates evidence. What do they tell you about the experiment? What do they tell you about how to interpret that evidence? Do they tell you how probable their hypothesis is to be true given the evidence? Of course they don't. Just like the diagnostic test maker can't tell you how probable it is that you have disease that relies on this prior information that they don't know. Science is the same way. When you design an experiment, what you're really doing is trying to get maximum conformational power out of the experiment.

0:35:12.0 Branden Fitelson: You want it to be as much of a multiplier of that prior probability as you can. Either a multiplier or a divider. If it's evidence against, then okay, then it's kind of a divider of how probability is it makes it smaller, makes the probability smaller. But the point is it's not probability that you're, you can't maximize the probability that your hypothesis is true. That depends on the prior. And different scientists are gonna have different priors when they, when they look at experiments. So all you can tell people basically is what the likelihood ratio, what that base factor is of your experiment, including Kahneman and Tversky's own experiment. So there's this real irony. They're implicitly criticizing human beings for being bad at doing a thing that their own paper doesn't require scientists reading the paper to do.

0:35:58.8 Sean Carroll: In the big picture, I was a little cheeky. I put this idea as everyone's entitled to their own priors, no one's entitled to their own likelihoods.

0:36:07.8 Branden Fitelson: Exactly. And that's exactly right. And so there's something not irrational here about deferring to the likelihood information. After all, that's the objective. That's the invariant information that we can know. And that's how science works, right? Scientific papers, they basically report base factors or something about whether the evidence favors one theory over another. They don't tell you how probable it is that one theory is true or the other theory is true. They know that's gonna depend on these priors. And they don't know the prior probabilities of their readership depends on what their readership knows.

0:36:41.7 Sean Carroll: And so our self appointed task is to come up with a formal understanding of this idea of confirmation. Like, clearly it's important, maybe you have your own priors, maybe you disagree or maybe you agree about them, but we should be able to quantify how much the new evidence is confirming our theories. And it's also like you say, but maybe it's worth emphasizing it's weaker than entailment from deductive logic. We're familiar from high school, P, and if P, then Q, therefore Q. That sounds solid, that sounds logic to us. And we want a logic of confirmation.

0:37:20.0 Branden Fitelson: Yes. And we can have one. And basically those base factors, they give it to you. One thing that's really interesting about this literature and is this is really what my, this is what I really got interested in when I was in graduate school. I wrote my dissertation on this. If you look in the literature on probability statistics, Bayesianism, any of that literature, there's lots of measures of this confirmation. There's lots of measures of, say, degree of correlation. So correlation is another word for confirmation. It's just when one thing raises the probability of another. Right? There's lots of measures of how strong that confirmation is. One thing you could do is just take the posterior probability and subtract off the prior probability. And you could say, well, how? That's one way of measuring how much of a difference the evidence made to the hypothesis. But there's many ways to do it because it turns out that you can define correlation in many equivalent ways. So one way is the posterior is greater than the prime higher. That's one way. But another way is that the true positive rate is greater than the false positive rate.

0:38:24.6 Branden Fitelson: Right? Or greater than one minus the false positive. So the probability, the evidence given the hypothesis is greater than the probably evidence given the denial of the hypothesis.

0:38:33.1 Sean Carroll: Yeah.

0:38:34.2 Branden Fitelson: And that's equivalent qualitatively. Those are gonna be true at the. But if you define measures based on those inequalities, they're actually different. They don't agree on which thing is better confirmed than which. They actually disagree on orderings of how well confirmed hypotheses are.

0:38:49.3 Sean Carroll: So they...

0:38:49.7 Branden Fitelson: And so they can't be measuring the same thing.

0:38:50.9 Sean Carroll: It's either it's being confirmed or dis-confirmed, but they don't agree on how much.

0:38:55.3 Branden Fitelson: Exactly. And so if you wanna measure it, which of course we do, we wanna know how much, then you've gotta pick one of these many. And there's dozens of measures and they all disagree. And I this is what I survey in my dissertation. And so you've gotta pick one. Now, the good news is that if you're an inductive logician, which is a certain tradition that I'm a member of, you actually have a criterion that allows you to narrow things down to a unique measure. And it turns out to be the Bayes factor, the same thing that people report on the boxes of the diagnostic tests. And it's a very simple criterion. The criterion is, however we're measuring this confirmation, it should be such that it generalizes entailment in the following sense. If the evidence did entail the hypothesis, if it guaranteed that the hypothesis was true, then that should receive a maximal value of confirmation. And if it refuted the hypothesis, entailed that it was false, falsified it, then that should be a minimal value. Just add that as a criterion, and you're basically uniquely down to this Bayes factor.

0:40:02.6 Sean Carroll: Good.

0:40:03.6 Branden Fitelson: And so that gives us, if we're in the framework of inductive logic now, we actually do have a unique way of measuring. And it just turns out, and I'm not sure this is a coincidence, but it turns out it's the very same Bayes factor that they tell you when you buy a diagnostic test.

0:40:17.2 Sean Carroll: Good. Yeah. I'm now gonna look in stores for diagnostic tests that tell me what my priors should be.

0:40:23.5 Branden Fitelson: Right. That's right. It's probability 9/10 that you're pregnant no matter who you are.

0:40:29.7 Sean Carroll: So just this might be a tiny little aside, but I remember when I was young and taking my first philosophy of science course, when we came to Carl Popper, we were taught that his notion of falsification was supposed to be a better thing to think than the old fashioned logical positivist notion of confirmation. I know now that we weren't actually told what that old fashioned logical positivist notion of confirmation actually was, or at least it didn't become clear to me. But what is the difference between those two ideas?

0:41:01.7 Branden Fitelson: Yeah, so that's a great question. So Popper was right in a sense. There is an important asymmetry when you think about degrees of confirmation. So let's think about how strongly does something that refutes, what's the conformational impact of that versus something that doesn't refute? Well, as I just said, our criterion requires refutation to be. That's the worst, that's the most negatively relevant you can be. And so in this sense, this is the kernel of truth of what Popper said. Refuting evidence is more powerful than non refuting evidence.

0:41:41.8 Sean Carroll: Good.

0:41:42.8 Branden Fitelson: As a negative evidence. And that's absolutely true. He's absolutely right about that. That's in fact one of the criteria that we use to get down to a unique, the base factor measure. So Popper's, what he wasn't right about was that all there is is refutation. So Popper had this weird view that there's no such thing as inductive arguments here he was influenced by Hume. He really got hooked on that skeptical argument and he thought well, the only arguments that could be compelling must be deductive. So there aren't any inductive arguments. So well then everything must be refutation that is right.

0:42:15.7 Branden Fitelson: That was all that would be left. You couldn't have disconfirmation in a weaker sense 'cause that doesn't exist. As I said, I'm not a skeptic, I'm an anti skeptic. We know a lot of stuff. We can make distinctions between refutation and just negative evidence that's not refuting. Now of course it's difficult. It's an art. You have to decide on a probability distribution to use to assess these things. Yes, you do have to do that at the end of the day or at least enough constraints on probability so that you can say like what the likelihood ratio is or something like that. You need some probabilistic information to do that. But we can obtain such problems by doing statistics. So I'm not a skeptic at all. I don't have a problem. So I kind of don't worry about the skeptical arguments in epistemology at all, including an induction. But let me just say one more thing. Popper was also right in his criticism, in some of his criticisms of the logical positivism. Carnap was probably the real best exemplar of someone who tried to develop a logical empiricist in inductive logic.

0:43:18.1 Branden Fitelson: And a lot of what he says in his work is great and useful. But there's one key mistake that he makes and that a lot of people have made and that is this, he thought, and many people still think amazingly, that there must exist a single probability function such that every argument strength can be measured with that one function. This is wrong, but I do think...

0:43:42.6 Sean Carroll: What do you mean by a probability function in that sentence.

0:43:45.8 Branden Fitelson: Yeah. So there must be some probability distribution over the relevant propositions.

0:43:51.4 Sean Carroll: Okay.

0:43:52.4 Branden Fitelson: Right?

0:43:52.7 Sean Carroll: Good.

0:43:52.8 Branden Fitelson: Such that for any argument as if there's this they used to call it. Well some people called it like the super babies probability function or something that there's this one probability function that can assess accurately the strength of any conceivable argument. And I just think this is absurd. It doesn't exist. There's no such thing. But I do think there's a weaker claim that is true. I wanna say that for every argument there exists a suitable probability function such that when you use that probability function to assess the strength of argument you get a pretty accurate assessment of how strong the argument is. And so I just wanna reverse the quantifiers. This idea that there's one in the sky that works for every argument. No, that's what Carnap thought. He was wrong about that. But it's true probably that for every argument there's some suitable probability distribution that works that gives you the right assessment of what the evidence favors or how strong the evidence is.

0:44:43.5 Sean Carroll: Is Carnap's idea either identical to or at least related to an idea that we could find the one true set of priors for all these propositions?

0:44:52.8 Branden Fitelson: Yes, that's right. That's another way of thinking about it. If you're a Bayesian then you'll think so called objective Bayesians...

0:44:57.9 Sean Carroll: Yeah.

0:44:58.8 Branden Fitelson: Think that there's one probability function that will rule them all or something like that. And of course that just won't work. You can just. It's very easy to. And this is what Carnap did for about 40. He kept getting more and more sophisticated counterexamples for whatever specification of the single family of probability distributions. And I just think this is a fool's errand. You don't need to do that. The way I think about science is you have a theory. So this theory is just probability calculus with your base factor and your conditional probability. Okay. That's your theory of inductive logic. And now to apply the theory you have to construct models of particular arguments and particular contexts. That is an art and a science. It's gonna involve a lot of statistics. It's usually gonna be empirical. It's gonna involve a lot of extra work. It isn't gonna be knowable a priori. But why should it be?

0:45:52.9 Sean Carroll: Yeah.

0:45:54.3 Branden Fitelson: That was the logical empiricist dream that it had to be noble a priori. And so there had to be just this one probability function you could divine a priori to determine all the answers. And I just think, no, that's not how science works. There uncountably many probability distributions don't tie your hands by not allowing yourself to use ones that science tells you are appropriate. And so that's just. Now that's gonna be empirical matter of constructing models of real arguments. And this is gonna be hard work. And there's gonna be... In many cases, it'll be controversial. But this is the same thing that happens when you're constructing models in science. You gotta make all kinds of assumptions, idealizations, approximations, and it's gonna be controversial how to do that the right way. Way, yeah, that's itself part of science. And who said it was gonna be easy?

0:46:39.1 Sean Carroll: Nobody said it was gonna be easy. That's for absolutely Shora. I don't. I don't think they did, but okay. As someone who lives in Baltimore, home of Edgar Allan Poe and the Baltimore Ravens, I am very fond of what we call the paradox of confirmation. Like, as soon as you have this idea that you're gonna start confirming things, you get in trouble. And the philosophers come along to tell you it's not gonna be so easy either.

0:47:05.8 Branden Fitelson: Yes, there are many. There are many paradox of confirmation, but. You're thinking of Hempel's paradox, the Raven paradox. Hempel's paradox, yeah. This is a classic. So the way this one goes is it involves a specific kind of hypothesis, something like this. All ravens are black. That's a hypothesis we could have. We could formulate. Suppose we hypothesize that all ravens are black. And if you want that to work, the way we usually think we're confirming, that is we make a lot of observations, so we observe a whole bunch of positive instances. And we think by the more positive instances we observe, by and large, the better supported this hypothesis is okay. But that assumes that even just a single instance would provide some support and maybe just a tiny amount, but it'll raise the probability a little bit of the hypothesis, which is a plausible idea. The problem is if you accept that principle, that a positive instance provides some support for a universal claim. So like the observation of a black raven should support a little bit that all ravens are black. Of course, you need many to do a lot of confirming.

0:48:19.0 Branden Fitelson: But one does something, right? That's how you get started. The problem with that is if you accept that and then you accept the following principle, which sounds very plausible that if a piece of evidence supports a hypothesis, then it supports anything logically equivalent to that hypothesis.

0:48:34.4 Sean Carroll: Sure.

0:48:35.2 Branden Fitelson: That seems right. Logical equivalence, that's a really strong form of equivalence. So anything that's evidence for something should be evidence for something logically equivalent. In fact, we would just think they're the same hypothesis. Well, okay, all ravens are black is logically equivalent to all non-black things or non-ravens. And now what's a positive instance of that hypothesis? Well, it would be the observation of a non-black, non-raven. Okay, but now you get the conclusion that the observing non-black, non-ravens confirms that all ravens are black. Okay, that doesn't sound good because it sounds like you can engage in what Nelson Goodman used to call indoor ornithology. You just observe a bunch of shoes, or observe a bunch of white shoes, a bunch of non-black, non-ravens, and you're gonna get a lot of confirmation for the hypothesis.

0:49:26.3 Branden Fitelson: Well, that's definitely a problem, but this is where the quantitative theory of confirmation helps. So yes, let's suppose you get some confirmation. Right? But now that leaves open the following question. Might it not be the case that the amount of confirmation provided by the observation of a non-black number is much less, in the circumstances we think we find ourselves in, than the observation of a black raven? And in fact, given very plausible assumptions about statistical sampling or however you're modeling, the usual statistical models of observing these things, given very plausible assumptions about the world, here's one assumption. There are a lot more non-black things than there are ravens. That seems right. Okay, so that, and if you think that's true, and it's still true, even if you suppose that all ravens are black, that is, that wouldn't affect much the relative proportions, then it just follows that you're gonna get more support of the hypothesis by the observation of a black raven than by the observation of a non-black, non-raven. So this is where the quantitative theory really helps and statistics gives us that.

0:50:39.0 Branden Fitelson: It gives us a quantitative way to estimate, estimate how much of an effect an observation has. And so given very plausible assumptions, it's just gonna be, yeah, you get some evidence, but it's extremely weak compared to the evidence you get from black ravens. And you can make this much more precise and you can show that in general, it's just much more informative to say, sample from the ravens and see if they're black, then sample from the non-black things and see if they're non-ravens. Right? And you can just make this very quantitative using the theory of confirmation. Just these Bayes factors, and given very plausible assumptions about what we think the probability distributions look like, it's just gonna follow that the best way to do the experiment is to sample from the ravens and see if they're all black, as opposed to sampling from the non-black objects and seeing if they're not ravens.

0:51:26.9 Sean Carroll: Well, and for the non-philosophers out there, just to remind them that this notion of confirmation is extremely weak. Right? When you say observing a white shoe confirms that all ravens are black, it's really, it's closer to supports. You even use supports a couple of times there as a synonym. Provides a tiny amount of evidence that might be really tiny.

0:51:47.0 Branden Fitelson: Yeah, it could be. It's just some bump. It just means the probability goes up, but it could go up a tiny amount. And in fact, this is what we think happens when we sample from the non-black things and see whether they're non-ravens, as opposed to sampling from the ravens, seeing whether the black. We just think there's a much larger effect there. So although there's some effect.

0:52:04.9 Sean Carroll: Yeah.

0:52:05.8 Branden Fitelson: It's not like it's totally gives you no information. And by the way, it's plausible that you should get some information because if you observe a non-black, non-raven, then what you've done is you've ruled one object out. You know that there's one object in the universe that can't be a counterexample to the hypothesis. And so in that sense, yes, you've gotten maybe a tiny bit of support, but it's absolutely minuscule compared to what happens when you sample from the ravens and see if they're all black.

0:52:31.2 Sean Carroll: Okay, good. So I'm on board the confirmation train here, but you mentioned in passing the idea of a quantitative measure of this confirmation factor. In one of the papers that you wrote that I actually read, some of you go through different plausible suggestions for what the equation should be for giving you what that confirmation factor is. And there's something called the received view that would you call the received view? Do other people also call it the received view? I don't even know.

0:53:06.0 Branden Fitelson: Well, it's just, it is just kind of the conventional wisdom about how to think about strength of arguments.

0:53:12.6 Sean Carroll: Yeah, right. Okay. And, and that relates this confirmation factor to a conditional probability. And I know that some large fraction of your intellectual life has been thinking about conditional probabilities. So why don't you tell us what a conditional probability is and why it might be related to a confirmation.

0:53:30.4 Branden Fitelson: Yeah, so one thing you definitely wanna know, it's just going back to the disease case. One thing you definitely wanna know, maybe the most important thing you wanna know is how probable is it that you have the disease. Conditional on or given that you get a positive result.

0:53:45.4 Sean Carroll: Right?

0:53:45.8 Branden Fitelson: That's called the conditional probability. And the way it works is you do this, you suppose that you get a positive result and then you ask your yourself, given that supposition, supposing the world is that way, how probable is it that I have the disease? And that's sort of the natural way of thinking about it. And so conditional probabilities are essential to induction. But of course, there's many different kinds. There's many different conditional probabilities. There's the probability of H given E, that posterior probability, that's really important. But there's also the likelihood, the probability of E given H, that true positive rate. And there's also the probability of E given, not the false positive. Right? So there's actually the...

0:54:21.6 Sean Carroll: E and H are evidence and hypothesis?

0:54:22.9 Branden Fitelson: Yeah, E and H are evidence and hypothesis. So E, let's say, is a positive test result. H is that you have the disease. And of course, what you wanna know is how probable is H given E? Right? Suppose E to be true. And then if you learn E, then you update. You update and you accept as your new probability the old conditional probability. That's sort of the Bayesian way of doing things. And yeah, you definitely wanna know that. Of course that's like, that's a very good thing to know. But knowing that requires you to know not just the true positive rate and the false positive rate of the test, but also the prior, the unconditional probability, the probability prior to the evidence before learning how the experiment turned out. And of course that's gonna vary very greatly from subject to subject, from person to person who's judging the evidence. So conditional probability is super important. And I still wanna say that is one of the features that makes something a strong argument. You definitely want the hypothesis to be more probable than not, at the very least given the evidence.

0:55:21.9 Branden Fitelson: If you're gonna believe it, if you think it's a reason to believe it, that's part of the story. But I want us, and that's the conventional view about how strong the received view is. If you wanna know how strong an argument is, just calculate that posterior probability, the probability of H Given E. And that tells you how strong a reason is. E is for believing H. But that can't be right. It can't be right because take you or me, if we take a pregnancy test...

0:55:46.6 Sean Carroll: Yeah.

0:55:47.0 Branden Fitelson: Look, the likelihoods are still the same. If we happen to get a positive, which is of course possible because physics, not because things aren't impossible, we could get a positive result. Okay. But we don't think that's a good reason to believe that we're pregnant 'cause we know we're not. So what that means is there's another dimension to the assessment of the strength of arguments and that is what we've been calling confirmation. And basically I wanna say it's just the, the ratio of those two error rates. It's just the Bayes factor, the likelihood ratio, whatever you wanna call it, that's the way we measure that second dimension of confirmation.

0:56:20.9 Branden Fitelson: And so I wanna say I'm offering a two dimensional theory of argument strength. For an argument to be strong, it's gotta be probable. Sure, yeah, it should be more probable. The conclusion should be more probable than not given the premise. Or in this case the hypothesis should be more probable than not given the evidence. But also the evidence should be relevant. If the evidence is irrelevant, it's not a reason to believe the hypothesis at all. Right? So if you have an argument where the premise is just irrelevant, doesn't affect the probability of the conclusion at all, then I don't wanna say that's a strong argument because that it's not a reason to believe the conclusion at all. Okay? And so this was something that the classical inductive logicians just ignored. Ignored. Not just Carnap, but if you read books on inductive logic, all the way up through Brian Skyrms book, which is one of the state of the art books from the 2000s, they just give you this one dimension, the probability of the conclusion given the premise. But I just think that can't be the full story because relevance confirmation also matters as to whether something should affect your beliefs.

0:57:19.2 Sean Carroll: So let me try to rephrase it 'cause I'm not sure I read my brain completely around it. The classical story would say if the probability of the hypothesis given the evidence is very high, then that counts as confirmation. But what if...

0:57:33.6 Branden Fitelson: Yeah.

0:57:33.9 Sean Carroll: For example, the probability of the hypothesis is just very high? What if we're already convinced of it? Then it could be also high given the evidence. But you wouldn't count that as confirmation. Is that the idea?

0:57:44.9 Branden Fitelson: That's right In fact it could even be highly probable given the evidence. But the evidence makes it a little bit less probable.

0:57:51.8 Sean Carroll: Right.

0:57:52.1 Branden Fitelson: You definitely don't wanna say that's a reason to believe. No, it, if anything it's a reason to believe that hypothesis is false.

0:57:58.5 Sean Carroll: Right, okay.

0:58:00.5 Branden Fitelson: It just so happens that it happens to have still a high probability anyway given the evidence. But that's probably because it had such a high probability to begin with.

0:58:09.7 Sean Carroll: Yeah.

0:58:09.9 Branden Fitelson: Okay. It's not that the evidence is a reason to believe the hypothesis. And so when as logicians what we wanna know is not whether we should believe the conclusion supplicator. But we wanna know how strong the argument is as a reason to believe the conclusion. And that I claim requires both probability and relevance confirmation.

0:58:27.8 Sean Carroll: And supplicator is weird philosopher talk for all else being equal.

0:58:31.0 Branden Fitelson: Yeah, that's right. And sure if the thing is relevant then all that matters is the probability. But if it's not relevant, then it's not a strong argument, I would say.

0:58:41.2 Sean Carroll: Good, so that sounds perfectly plausible. But of course we're gonna wanna know what is the way to know whether something is relevant. Is there is like a Vibes based thing or is there an equation?

0:58:53.9 Branden Fitelson: There's an equation, there's. And it is just that the thing they give you when you buy the diagnostic test, they give you this ratio of the two error rates, the two likelihoods. The probability of E given H and the probability of E given not H and you take that ratio. That's a really good measure from an inductive logical point of view. It's pretty much the only one that's gonna satisfy these desiderata we like. And so that's how I propose. So I'm proposing a two dimension so you can visualize it as like a cartesian space. The X axis is the conditional probability of the conclusion given the premise and the Y axis is that likelihood ratio that is that measures how much impact how relevant the premise is to the conclusion or the evidence is to the hypothesis.

0:59:35.0 Sean Carroll: And sort of...

0:59:36.9 Branden Fitelson: And yeah.

0:59:37.8 Sean Carroll: So there's no one number at the end of the day. It's not like you add those two together or you add their squares together or whatever. It's just you gotta give me both numbers.

0:59:46.1 Branden Fitelson: Yes. And this is a really fundamental thing that's so important to emphasize. One of the real deepest mistakes that was made in the history of Inductive logic was that they thought there'd be a single measure on which you could totally order all the arguments in terms of their strength. A single function that takes a premise and a conclusion and a probability distribution and gives you a single number. I don't think this can be done. What it gives you is a ordered pair.

1:00:12.8 Sean Carroll: Yeah.

1:00:13.2 Branden Fitelson: It gives you a probability and a Bayes factor vector.

1:00:16.5 Sean Carroll: Good.

1:00:17.1 Branden Fitelson: And that's all, in general, that can be said. Now, of course, you can say something. There's a some ordering because if the evidence, if you move up both in terms of probability and relevance, well, then you've gotten stronger because you've gotten stronger in both dimensions.

1:00:30.9 Sean Carroll: Sure.

1:00:31.3 Branden Fitelson: But these mixed cases, this is the problem. Cases where you have improbability but high confirmation, like the base rate fallacy, or cases like the conjunction fallacy, which also involves relevance going one way, confirmation going one way, but probability going the other way. And so these mixed cases, which it's no surprise they led to the Nobel Prize about concerning how quote unquote, bad people are at probabilistic reasoning. It's because the cases are mixed that people get confused. If you ask someone how strong is an argument? Well, if that has two dimensions to it and one of them's high and the other's low, that's ambiguous. The question's ambiguous. And so you might not blame them so much if they're a little confused about those arguments where you have high relevance and low probability or, high probability and low relevance. Those are hard cases to assess.

1:01:27.2 Sean Carroll: Yeah.

1:01:27.7 Branden Fitelson: Right. For most people because they realize both factors are relevant and what they're being asked for is a single summary, a single assessment. But maybe there isn't. Maybe it's ambiguous, maybe it's strong in one sense, but not in the other.

1:01:41.0 Branden Fitelson: And so I in general want there to just be two dimensions. And so, I don't think there's a total ordering a single number you get for any argument and any probability distribution, there's gonna be two numbers, in general.

1:01:53.9 Sean Carroll: And has...

1:01:54.5 Branden Fitelson: And that's one of the mistakes. Yeah.

1:01:56.2 Sean Carroll: Has everyone basically agreed with your impeccable logic here?

1:02:00.0 Branden Fitelson: Well, some people have. So they're, in psychology. So, we did. I had the pleasure of working with some psychologists on these conical reasoning fallacies. And yes, there's a lot of experimental evidence now that it's the mixed cases that are hard. And they're hard because they're mixed. And so if you fiddle with the confirmation, that is the relevance, you fiddle with that Y dimension, it's really gonna affect how good people are making judgments about the X dimension. And so, this is because what people really care about is not just how probable the conclusion is given the premise, they care about how strong is this as a reason to believe the conclusion. And intuitively they know that depends not only on the probability, but on whether the evidence is relevant, whether the evidence confirms the hypothesis. And so there's a lot of psychological evidence now that notion of confirmation really is relevant to explaining what's going on in these cases.

1:02:53.6 Sean Carroll: So let's go through some of these cases a little bit more carefully 'cause I'm sure that people kind of vaguely heard of them. But, it's always good to be clear. The conjunction fallacy you already mentioned, and it is one of my favorites 'cause I was not fooled by it when I first saw it, but I saw why I could be fooled by it. So I'm sympathetic.

1:03:13.1 Branden Fitelson: Yeah, let me. That's a great one. So what the way that one works is you're given some evidence about a woman named Linda. You're basically told that she went to Berkeley in the late '60s, she participated in anti-nuclear demonstrations. She very active politically and so on and so forth. She was like a flower child and so on and so forth. And that's the evidence you're given. And now you're asked, this is years later you're asked, okay, now I have two hypotheses I'm gonna give you about Linda nowadays. Either she's a bank teller or she's a feminist bank teller. And you're asked which is more probable given the evidence that I gave you? And back in the day, a lot of people said feminist bank teller was more probable given that evidence. Of course, that's impossible because feminist bank teller entails bank teller. So every possible world in which he's a feminist is a world in which he's a bank teller. And since probability is just a measure of, how big a class of possible worlds is, it couldn't possibly be that the conjunction is more probable than one of its conjuncts.

1:04:20.1 Sean Carroll: Right.

1:04:20.4 Branden Fitelson: That would just violate basic logical and probabilistic principles. So that can't happen. So what's going on? Well, what we showed in a paper that we wrote, and there's been a lot of research on this since then, is that two very simple assumptions, if two very simple assumptions hold, which I'm gonna give you in a second, then it's just guaranteed that while, yes, the bank teller hypothesis is gonna be more probable than the feminist bank teller hypothesis, the evidence will actually confirm the feminist bank teller hypothesis more strongly. It'll be more relevant to that control conjunction than it is to the first contract. And here are the assumptions, they're very weak. First assumption, the evidence isn't positively relevant to whether she's a bank teller. That seems plausible. Okay. Second assumption, suppose she is a bank teller. The evidence I gave you still positively relevant to some degree to her being a feminist. Maybe it's only a tiny amount, but still somewhat relevant to her being a feminist. Those conditions entail that for any way of measuring confirmation, for any of the measures, it turns out the evidence will confirm the conjunction more strongly than it confirms the conjunct.

1:05:31.9 Branden Fitelson: And so these are cases, they're mixed cases. You have a case where probability goes one way, bank teller's more probable, but bank teller's less relevant.

1:05:40.8 Sean Carroll: Right.

1:05:41.2 Branden Fitelson: It's less well confirmed by the evidence. And it's again, no surprise that just like in these rare diagnostic testing cases, rare disease cases which are called the base rate fallacy, cases which we already discussed, just like in those cases, these cases involve one of the dimensions of assessment, probability going one way, and the other dimension of assessment of the strength of argument, confirmation or relevance going the other way. And I'm not at all surprised that people defer to relevance.

1:06:10.1 Sean Carroll: Right.

1:06:10.7 Branden Fitelson: It makes sense. We already saw relevance is in many ways more objective. It's more invariant. It's sort of the language of science. The way science understands evidence, it usually thinks in terms of terms of how much the evidence confirms, not how probable the hypothesis is. That depends on all these idiosyncrasies about prior probabilities. So I'm not at all surprised that people do any of these things. So I say it kind of makes sense that when the confirmation goes one way and probability goes another way, deferring to the confirmation kind of makes sense, since there are many ways in which confirmation is just more important, more informative, more objective than probability is.

1:06:49.1 Sean Carroll: So I have a slightly different, or I had for a while after hearing about the experimental results, slightly different hypothesis about what was going on, but I'm not sure if it's slightly different. So let me explain it to you, and you tell me if it's different. I'm wondering whether or not when people hear the evidence, which in this case is Linda went to Berkeley, she was a flower child, she was an activist, and then they're given the two hypotheses. She's a bank teller or she's a feminist bank teller. Implicitly, they assume that being a bank teller means that you're a typical bank teller, and being a feminist bank teller assumes that you're a typical feminist bank teller. And the typical bank teller is not feminist. So there's some sort of interference or tension between the hypothesis that she's a bank teller and the evidence that she was a flower child. It's still a sort of a mistake with the question, phrased as it was. But that would be a way of psychologizing why we make the mistake. I'm not sure if it's the same as your way or different.

1:07:53.5 Branden Fitelson: Well, yes, so there have been many proposals for different things that might be going on. One of them that was received a lot of attention early on, which is similar in some ways. Maybe you can tell me. it's related to what you were saying is it was originally postulated that actually people were hearing the question slightly different. They're hearing it as feminist bank teller versus non-feminist bank teller.

1:08:17.7 Sean Carroll: Yeah.

1:08:18.9 Branden Fitelson: And actually there's definitive psychological research that that's not what's happening. So I can point you to papers that are just absolutely stunning on this by some of my psychological colleagues, though. Okay. So there are experiments where first they teach people how to do deductive inferences, they teach them how to infer conjuncts from conjunctions, they teach them all this stuff and then they have them bet, they have them do betting, and they still, a lot of people bet more on the conjunction, even though they know that the thing follows, they've actually gone through the logical exercise of it, following logically.

1:08:56.5 Sean Carroll: Right.

1:08:57.3 Branden Fitelson: That one hypothesis entails the other. So this has been controlled for, in my opinion, this particular hypothesis actually there's a lot of evidence against it now. So I find the relevance approach, the confirmation approach, more plausible given all the evidence. But of course, this is the active area of research. There's some even more recent research trying to refine the notion of relevance to go beyond confirmation and take into account other pragmatic kinds of relevance as well. That's really fascinating research. But there's pretty strong evidence now that this, the second dimension I'm calling it of argument. Strength is making a significant difference. There may be many other things that are making a difference, but it's pretty clear it's making a difference.

1:09:39.2 Sean Carroll: I kind of love the intersection of the actual psychology experiments with the philosophical reasoning. Most abstract level, it does, the rubber does hit the road at some point.

1:09:51.6 Branden Fitelson: Oh, absolutely. To me, that's one of the most interesting areas of research in general is that borderline between the descriptive and the prescriptive.

1:10:02.3 Sean Carroll: Yeah.

1:10:02.9 Branden Fitelson: That's a really. It's such a difficult area, but it's such an important area because after all, what we're interested in is evidence for humans. It's like, this is another weird thing about logical empiricism. Who cares about evidence if it's just some purely formal logical relation between things? How does that actually bear on what we ought to believe? So that's another problem with the whole kind of logical empiricist way of thinking. It's very disembodied and abstract, and it's just unclear why it would ever have any purchase on humans.

1:10:35.3 Sean Carroll: Okay, so let's... One more example might seal the deal here. And you suggested the four card problem, which I do remember. Look, I looked it up. You have your paper, but your paper is full of all these equations and things, so I just looked it up on Wikipedia to remind me what it was. And I do remember coming across a four card problem and that one I did get right, just because I've done probability problems before. But I see the similarity here, but the argument plays out in a slightly different way. So why don't you tell us what the problem is?

1:11:09.3 Branden Fitelson: Yeah, so there's this famous case of the waste and selection task is what it's called. And there's... So the way it works is there's cards and there's different variants of it. So I'm trying to remind myself of the version that we actually worked on, 'cause I don't wanna talk about a version that I don't...

1:11:32.1 Sean Carroll: I actually don't. I wrote it down. If you want me to give the problem and then you can explain the solution.

1:11:36.5 Branden Fitelson: Yeah, could you do that? And then I can. Yeah. Let me.

1:11:38.7 Sean Carroll: The version that I know from your paper is that there are these cards and that there is a number on one side of the card, a letter on the other side of the card you know that. And you're shown four cards. One says the letter D, the other says the letter K. I don't know if this is for Daniel Kahneman or not. I don't know where these letters came from. Then it shows the number three and the number seven. Okay, so DK37. So obviously you showed the letter side of two of them, the number side of the other two. And then the hypothesis is all cards that have D on one side will necessarily have three on the other side. And the question is which cards you have to flip over to most efficiently test that hypothesis, that if D is on one side, three is on the other side and you're shown DK37.

1:12:32.0 Branden Fitelson: Yes. And so, yeah, so this is a great, great case. So we wrote this paper a while back. Me and Jim Hawthorne wrote this. Really, it's my favorite paper I've ever written still to this day. And so it's about this waste and selection task, which people make a certain kind of mistake in, tend to, and its relation to the paradox of confirmation, which we already talked about. So you remember back in, when we were talking about the paradox of confirmation parents, that it's a better strategy to sample from the ravens and see whether they're black than it is to sample from the non-black things and check whether they're non-ravens.

1:13:13.0 Sean Carroll: Right.

1:13:13.9 Branden Fitelson: It's just more conformationally powerful to do to sample from the ravens and check and see if they're black. This turns out to be an isomorphic problem. This problem is basically the same problem. Okay, because... So what hypothesis are we being asked to test in this case? So we've got the four cards, DK 3 and 7. And what hypothesis are we being asked to test?

1:13:39.6 Sean Carroll: If D is on one side, then 3 is on the back.

1:13:44.0 Branden Fitelson: That's right. So all D cards are three cards.

1:13:48.2 Sean Carroll: Yes.

1:13:49.6 Branden Fitelson: Or you could just say all Ds are threes.

1:13:51.6 Sean Carroll: Yep.

1:13:53.0 Branden Fitelson: Okay, now all Ds are threes. Same structure as all Rs are Bs. All Ravens are blacks, and the same kinds of things happen. So what you wanna do is, if you think about, back to the Raven case, what did we say the best strategy is? Look at the ravens and then check and see whether they're black. The analogous thing here would be check the D card and then turn it over and see whether it's a three on the other side. That is exactly the analogous thing. And the same models will show that's the most efficient way to respond to this. And in fact, if you just use some very weak assumptions about probability and you use this confirmation measure that we were talking about, then you can actually rank the strategies in terms of their conformational power and it'll turn out, given very weak assumptions about what's going on, that D turning over the D card is the best. Then next, turning over the three card. Oh, sorry, no, that's what people actually do. Sorry, right. That's what people actually do. So what people actually do. This is great 'cause I just actually did it.

1:15:05.9 Branden Fitelson: What people actually do is they turn over the three card. That's the second best strategy.

1:15:10.4 Sean Carroll: 'Cause they're trying to confirm.

1:15:11.0 Branden Fitelson: That isn't the second best strategy.

1:15:12.3 Sean Carroll: Right.

1:15:12.4 Branden Fitelson: Yeah.

1:15:13.3 Sean Carroll: They think that they're trying to confirm, but that's not the best way to learn.

1:15:16.9 Branden Fitelson: Yes. What you should be doing is looking for counterexamples. Right. Next. So you should turn over the seven card, right, and see whether it's a D. Yes, exactly. And this is exactly. We show, because we actually show that the two cases, the paradoxical information and the waste desk, are actually isomorphic. They have basically the same structure and you can use the same kinds of probability models to model them. And when you do, you get exactly the prescriptions in both cases. You get best thing, sample from the Raven, see if they're black. Next best thing is look at the non-black. Look at. Right, the non-black things and see if they're ravens. Right. Look for counter examples. Right. Same thing here. But what people actually do in this waste and task, which is really interesting, is they reverse those. The second and third strategies. So what they do is they say D first but then they'll say three. Yeah, they'll say no, turn over the three card. When that's definitely less informative. And here the paparian intuition really is correct. You should be trying to refute.

1:16:21.1 Sean Carroll: Right.

1:16:21.4 Branden Fitelson: Next you should be looking at the seven card. And as I was saying, the paparian thing, the kernel of truth of pauper comes out in this paper because basically you can just show that after sampling from or sampling the D card and looking to see whether it's a three, the next best thing is looking at the seven and seeing whether it's a D. That's Popper's intuition, basically. And people aren't pop parrying, it turns out, because they think turn over the three cards better than turn over the seven card. But actually it's very easy to show just using very weak assumptions about probability, that that's wrong. And so in a way, it's a Bayesian vindication of Popper. That's one of the things I like about this paper. It tells you the kernel of truth in the Popperian falsificationism, that in this case, going for the falsification is better. It's the second best thing and not the third best thing, which is what people tend to think it is.

1:17:20.3 Sean Carroll: Yeah, but the thing that. So, but the thing that people tend to do, they reason, if it can be called that, they think. Well, your hypothesis is that if there's a D on one side, there's a three on the other. If I flip over the three and I see a D on the other side, that will confirm. That will give some evidence for this in the space of all possible cards, that's a more likely thing to see.

1:17:43.9 Branden Fitelson: Yes, and it will confirm. But because refutations are always more powerful than non refutations.

1:17:50.2 Sean Carroll: Exactly. Yeah.

1:17:50.7 Branden Fitelson: That's the Paparian insight, and that's why Popper was critical. So, yes, you're absolutely right. It's a kind of confirmation bias. And in our paper we actually prove, given very weak assumptions, that the only way to get that ordering is if you come into the experiment with a confirmation bias. That is you think you're more likely to see positive instances rather than counterexamples.

1:18:13.4 Sean Carroll: Exactly right. Good.

1:18:15.6 Branden Fitelson: And you can just prove that that just follows from the very weak modeling assumptions we have, that the only way to get that ordering is gonna be if you come in already thinking that you're more likely to get confirming instances rather than refuting instances, which is sort of the classic confirmation bias.

1:18:31.8 Sean Carroll: It is a bias, and that in this case the parameters are sufficiently clean that doing D and 7 is clearly the right strategy here. But the real world of science is complicated. Right? I guess we're getting late in the podcast. We can let our hair down and think about less completely logically rigorous deductions here. Are there lessons for how we should do science? Scientists are constantly arguing about what experiments are the best ones to do. Obviously it has to do with the probability that your different hypotheses are true, your priors, which of course we don't agree on. But also you would argue the relevance of that experimental result to changing your beliefs.

1:19:22.9 Branden Fitelson: Absolutely. A great way to think about experimental design is to think what you're doing is you're trying to maximize the conformational power of the evidence generated. And that could be. So that's neutral as to whether it's negatively relevant evidence, which it might be or positive, relevant. But what you wanna do is maximize the conformational power. And that's the framework of this Weiss and Hempel paper that we did.

1:19:48.7 Sean Carroll: Yeah.

1:19:49.4 Branden Fitelson: Where we're basically just a very simple measure of conformational power. It's basically just the absolute value of this, confirmation measure that we have. And if you just try to maximize that, then you can just figure out which strategies are going to do that. Now, just to your more broad question, just speaking a little bit more philosophically here, zooming out a little bit. So, as I said, this is a very elegant theory of inductive logic now. And when you're actually applying a theory, you have to construct models. And this is where all the hard work comes in. You gotta really come up with not necessarily an exact probability distribution, but you have to have enough constraints on your probabilities to be able to decide whether the evidence in your experiment favors one hypothesis over another.

1:20:34.4 Branden Fitelson: And you might not need to give an exact, numerical probability distribution over everything, but you'll need enough constraints to determine whether favoring occurs, in which direction it goes in. And how do you do that? Well, it's going to be quite difficult in many cases. It's gonna involve a lot of science, measurement, statistics, a lot of also just theoretical arguments and just trying to, it's. So it's partly an art. Modeling is not just a pure science. But this is true in all branches of science. So what I wanna say is inductive logic is no different than any other science. It gives you a theory, but in order to apply that theory, you have to construct models. And that's really. You've gotta get in the trenches and do a lot of really difficult science, a lot of statistics, a lot of measurement, a lot of idealization, whatever is suited to assessing that argument. And it's gonna be case by case. It's gonna be each context. We have to do the best we can to come up with the most plausible constraints to tell us what the evidence favors.

1:21:32.4 Branden Fitelson: That's all we can do. So I really think it is a case by case thing of constructing models and doing the best we can, just like the rest of science. People often say all models are false, which I agree with, but that doesn't mean the theories are false. So, when you take general relativity and you try to model actual situations with it, well, what do you do? Well, you have to make all kinds of approximations 'cause you can't solve the equations. And then you gotta make all kinds of auxiliary assumptions and all kinds of measurements you gotta do. And you got to do kinds of statistics there to figure all these things out and get parameters right and all that. Okay. Those models of course are false because they all involve idealization and approximation and so on. But the theory might be true. It certainly could be at least a really good framework for constructing models. And this is how of the framework I'm offering for inductive logic.

1:22:23.0 Sean Carroll: Yeah.

1:22:23.4 Branden Fitelson: With its two dimensions of assessment. In order to apply it. Yeah. You've gotta fit in some adjustable parameters. You gotta tell me what the premises are conclusion is. And then you gotta tell me enough about the probabilities over those things so that I can get a judgment as to whether the evidence favors the conclusion or not. Is the evidence relevant to the conclusion? You may not be able to say how probable the conclusion is, but at least you'd like to say how relevant is the evidence. Get some assessment of how relevant it is.

1:22:53.6 Sean Carroll: Yeah, I guess I'm trying in real time here and not quite succeeded. To put this in very down to earth terms. My favorite example of a non-frequentist probability is the dark matter a weakly interacting massive particle, a wimp, or is it an axion that's another candidate for the dark matter or is it something else? The third category, something we haven't thought of before. So obviously this is not a frequentist kind of question. Right? This is something that we have some priors we're going to update. But now what I'm presuming is that your way of thinking about this would help me answer the following question. If I had a certain amount of money to build an experiment and one experiment would confirm, like detect the wimp, right? Detect that it is that. But the other experiment would tell me that it is not an axion or something like that. Could I somehow. I'm truly not able to answer the question in real time, but could I somehow judge which is more useful depending on what my priors were for those different hypotheses?

1:24:03.7 Branden Fitelson: Yeah, you could. What you would need, you may not even need your priors. What you're gonna need are the likelihoods you're gonna need. Okay, how probable is it that we would have observed this evidence.

1:24:16.3 Sean Carroll: Right.

1:24:16.6 Branden Fitelson: Given the one hypothesis versus given the other hypothesis. So you're gonna have to be able to compare those likelihoods.

1:24:22.0 Sean Carroll: Yeah.

1:24:22.5 Branden Fitelson: At the very least, that will give you some information about the relevance dimension. Like does the evidence favor one over the other? It may not tell you the probabilities 'cause for that you're going to need priors. But still it can give you a good amount of information and it can tell you something that the experiment's doing something valuable. It's giving you evidence that favors one of those hypothesis over the other 'cause it's more relevant to one than it is the other. Even if you don't know how probable they are, that's fine. You may not know how probable they are, so you may not know whether to accept or reject, but you still can say, hey, this evidence seems to favor the one hypothesis over the other. And that's generally how scientific experiments actually work. As I was saying before, when you're designing experiment, you can't determine how probable things are gonna be. You can, given your priors or something, if, you could yourself determine. But what you can do generally is you can design the experiment such a way that it provides evidence that favors one thing over another or is relevant to the experimental question.

1:25:24.4 Sean Carroll: There's a claim out there that I'm a little sympathetic to that scientists should be more open about what their priors actually are. Like when we do an experiment, like we turn on Large Hadron Collider and scientists said, well, we could find all these new particles. Tell me what the probability that I will actually find these new particles, which physicists at least never ever do. I don't know if people in other fields actually do that. Do you think it would be good that they put their money where their mouth is in that way?

1:25:48.6 Branden Fitelson: Well, the great thing about. So I've been thinking a lot about different sciences 'cause I'm working on this project with a couple of colleagues on the replication crisis in science. And the different sciences are very radically different in terms of how they're dealing with replication and what problems they have. Particle physics is sort of like the gold standard. The experiments they do, the evidence they generate is so confirmationally powerful that it almost doesn't even matter what your priors are.

1:26:14.1 Sean Carroll: Right.

1:26:14.6 Branden Fitelson: It really doesn't. It basically just swamps completely. There's such large likelihood ratios that you get from those experiments that, come in with whatever prior you want. You're gonna basically come out pretty sure that these particles exist if you're paying attention to the evidence. And so particle physics is this is really a great example of where we're designing experiments that are so confirmationally powerful that it almost doesn't even matter what your priors are. But other sciences are not like that. Other sciences, it's much more sensitive to your priors as to what attitude you're gonna come out after looking at the experiment. And also it's even more controversial whether you have really relevant evidence or not. Even that is controversial in a lot of the special sciences. Whereas in particle physics, no, you know the evidence is very relevant. It's extremely relevant. And so, I view that as kind of one of the easy cases. And like any theory, it's gonna have cases, it's really good at explaining and it's gonna have anomalous cases. And that goes for the Bayesian theory that I'm of inductive logic that I'm offering.

1:27:19.7 Branden Fitelson: It's a pluralist Bayesian. It's not saying you should use a particular probability, but it's some probability function. Right? It's Bayesian the sense that I'm willing to put probabilities over all the hypotheses. Right, okay. Which non-Bayesians aren't to willing to do. But in any event, look, it's a theory and it's gonna have limitations just like Newton's theory wasn't able to explain, in any really plausible way the motion of Mercury. I'm sure there are gonna be cases that we can find in science where the theory I'm offering, it's gonna have be really challenged to come up with plausible models that explain how much confirmation there is in that case. And, but that's the nature of science. And so even in... So I like to think there's a spectrum of, there's easy cases like particle physics or games of chance. These are easy cases. And then you go down the spectrum and there's really much harder cases and much more controversial cases. But that's true pretty much of any science.

1:28:16.2 Sean Carroll: Okay, well we have confirmed that this is a fun thing to talk about, but maybe we haven't because my prior was so big that it wasn't actually relevant the evidence we collected here. But in any event, Branden Fitelson thanks very much for appearing on the Mindset Landscape podcast.

1:28:30.9 Branden Fitelson: Thank you so much, Sean. What a pleasure.