284 | Doris Tsao on How the Brain Turns Vision Into the World

0:00:00.0 Sean Carroll: Hello everyone, and welcome to the Mindscape Podcast. I'm your host, Sean Carroll. Most of you probably know that I love science of all sorts, but my favorite kind of science questions are the ones that become slightly existential, right? That sort of bump up into issues of meaning and reality at a super deep level. And some such questions are straightforwardly physics or cosmology, right? You know, where did the universe come from? Why is there a universe at all? What are the fundamental laws of nature? Could the universe have been different? Are there other universes? These all make you think about the world in which we live and how they could have been different? But there's a whole nother realm of science questions that also have that somewhat existential character, which is, of course, the mind, how we think, consciousness, but not just consciousness in this sort of philosophy question about it.

0:00:58.8 SC: What is it like to be something, but even just how it all works, right? I think of myself as a person with opinions and emotions and desires and values. I can also, if I'm consistent and I believe my own rhetoric, I have to believe that I can also be described as a collection of cells, neurons and other kind of cells that are interacting with each other in these interesting ways. Or even just of course, as a collection of particles or quantum fields or whatever. And somehow my selfhood, my awareness, my ability to think about the world and model it and interact with it. It's gotta emerge out of all that basic stuff. And there's a lot of juicy philosophy here, obviously, and I'm a big believer in that. But natural philosophy, as I think of it, is the intersection between philosophy and science, right?

0:01:54.3 SC: It's the kind of philosophy that engages very strongly with the empirical information we get about the universe. So today we're going to, we'll dabble a little on the philosophy side, but mostly we're gonna be hard-nosed natural scientists today, thinking about the brain and how it works. Our guest, Doris Tsao, is a major neuroscientist, multiple award winner who specializes in vision, the visual cortex. In fact, her most recognizable work is in how we recognize faces. The human visual field or the human, not just human. When animals see things, they don't apprehend the world pixel by pixel or you know cells in our retina, one at a time, right? We put together pictures, we construct models of what is around us, and we pay more attention to some things than others. We pay a lot of attention to faces, faces of other human beings, of course, but also faces of other animals.

0:03:00.2 SC: And indeed, it doesn't take that many strokes of a pen to draw a symbol, even if you're not a great artist, that everyone will recognize as a face. Understanding faces, recognizing faces is of obvious usefulness in biological, evolutionary history, right? We need to be good at interpreting that. It is, therefore, in some sense, no surprise, that there are parts of our brains that are devoted to this task. Their job is to recognize faces and help interpret them. And Doris has done pioneering work in that and identifying usually in monkeys, but they're not that far away from us biologically. Exactly what parts of the brain are doing what, when we're looking at faces. And it turns out this is one of those nice conversations that actually goes to very fun places that I was not smart enough to anticipate ahead of time, thinking about what's going on when we are recognizing faces or whatever, segues quite naturally into other questions about how the brain works in building models of the world around it, abstract thought, consciousness, all those kinds of things. So this is a, to me I'm glad that Doris Tsao was in the spirit of it. It's a very mindscapy kind of conversation where we both dig into the science and we get to talk about the big ideas along the way. So let's go.

[music]

0:04:36.9 SC: Doris Tsao, welcome to the Mindscape Podcast.

0:04:40.4 Doris Tsao: Thank you. Good to be here.

0:04:43.7 SC: So I'll tell you why specifically this podcast happened. I mean, obviously we were both at Caltech at the same time. I knew who you were doing great research in something, but someone a couple years ago, one of my podcast listeners emailed me to chide me that I was always talking about like consciousness and stuff from this philosophical point of view. But we should talk to people who actually study the brain in a more, down to earth experimental science way. And your name was an obvious choice.

0:05:17.6 DT: Oh, that's funny. I mean, I was hoping to talk to you about some... Your thoughts about theories of consciousness, but indeed, yeah, our lab is very interested in understanding the neural mechanisms underlying consciousness. This is one of the big questions that brought me into neuroscience and specifically to start studying monkeys because I think monkeys are conscious and very likely, at least their visual consciousness is very similar to ours. And so we can really get at the mechanism.

0:05:46.9 SC: Yeah. And the visual part there is, I guess, you know, where you've really made your money. And that's what I wanna try to focus on here today. So correct me if I'm wrong, we're gonna start very, very simple since I'm a poor theoretical physicist. But I get the impression that a lot of people know about cameras and video recorders and things like that, and the idea of like pixels in a detector screen. And so they kind of think vision is like that. You know, we just see the pixels and we interpret them like that. But I remember I learned the hard way that the visual system is much more elaborate than that. We don't just detect pixels directly and then interpret them.

0:06:27.0 DT: Yeah. You know, so your eye is basically like a camera, right? The light falls goes through the lens and falls on your photoreceptors, and then all those signals from the photoreceptors get sent via the optic nerve into your visual cortex. And then the visual cortex is this incredible piece of machinery. In monkeys, it probably takes up about a third of the brain. It's a giant piece of machinery. And within this visual cortex, there's dozens of different areas that are specialized for processing different aspects of the visual world. So it's like a whole factory that's transforming these pixels into your perception of objects in space. And the first really big insight, like the eureka moment in our understanding of visual cortex came from Hubel and Wiesel, right?

0:07:27.3 DT: So they were postdocs with famous nerve physiologist, Stephen Kuffler and Kuffler had been recording from retinal ganglion cells, right? These are cells in the retina. And they had this property of center surround, which is already very interesting. So center surround means that it'll, it likes spots of light, but if you show a very diffuse pattern of light, then the cell doesn't respond because it's inhibited by the so-called surround region. Okay. So they're doing this like redundancy reduction already in the early stages of the retina, and then Hubel and Wiesel they thought, okay, what happens next? Right? And so they made this decision, let's just go and follow the anatomy and see where those retinal ganglion cells go inside the brain. And so the first place that they go is this structure called the lateral geniculate nucleus.

0:08:19.0 DT: And there the cells responded pretty much like the retinal ganglion cells. They just showed center surround, and then they went one stage further into primary visual cortex, and there they uncovered this whole new world, right? The cells suddenly didn't respond to spots of light at all, but they required edges, right? And different cells respond to edges at different locations in the visual field. And so it was this whole new, this dramatic transformation in how the visual information's represented. And that was just a lightning bolt. It's like all of a sudden, wow, there's this machinery that's actually transforming the pixels and you know, what happens after edges. And so that launch this whole field that I'm very lucky to be part of.

0:09:02.3 SC: And that does, it can't help but remind one a little bit of like a deep learning network, right? Where you have different layers that have different jobs.

0:09:14.9 DT: Yeah. I think there's like argument about this, but I think the neuroscientists firmly believe that deep neural networks were inspired by Hubel and Wiesel's discoveries about visual cortex.

0:09:24.2 SC: Good. I'm happy to give them the credit. And you said one provocative thing there. I want to follow up on the visual cortex is the a one third of a monkey's brain. I'm guessing that it's less than a third of a human brain.

0:09:39.6 DT: Yeah. I don't want to put any number out there, but if you ask what fraction of the human brain will respond to a visual stimulus, it's basically the whole brain. Jack, my colleague Jack Allen puts people inside FRS scanners and shows them movies and the whole brain lights up. So, and you can you know, certainly a lot of that is like multimodal. So it'll also respond to text and to audio and so on. But it's definitely responding to visual stimuli. So I would argue part of the broad visual cortex. Yeah.

0:10:12.3 SC: And one other thing, which I, again, I think is true, but you're the expert, and you correct me if I'm wrong, this fact that certain neurons beyond the first level are responding to lines or motion or whatever, rather than just pixels. Does that help explain optical illusions? We sort of fill in things if the right neurons light up.

0:10:35.4 DT: Yeah. The fact that we have these specialized neurons I mean they can, yeah. So there's, I don't know if this getting too much into the weeds, but there's this very cool illusion called reverse phi where you can make a bicycle look like it's constantly moving just by setting the contrast in the correct way. And that is beautifully explained that by the properties of the direction selective cells in the motion processing part of the brain. So, yes.

0:11:06.6 SC: So very roughly speaking, you know, our brain has been designed to see the kinds of things we typically see. And so you can trick it if you show it things that are not the kinds of things we typically see.

0:11:16.8 DT: That's right. And I should mention on the topic of optical illusions, my lab has done a lot of work on face processing. And one of my favorite optical illusions, and I strongly encourage everyone to go look at this, is called the Thatcher illusion. And that's this image that you create of a face where you basically you turn all the features upside down. Okay? So you keep the frame of the face upright, but you like turn the eyes and the nose and the mouth upside down. And so you can imagine that looks very freaky, right? And now you just turn this freaky face upside down and suddenly it doesn't bother you anymore. It looks like a normal face. And the reason why I love this illusion is that what's happening when you turn it, that face upside down is that you're essentially inducing a lesion in your... You're causing your face areas to become silent, 'cause they're just not wired to respond to upside down faces. And so you can experience what it's like to have brain damage. And what's remarkable is that you feel like you see everything. You don't feel like there's something missing, right? And so it's kind of... I like to think that it's reassuring, when I lose my mind at least I won't know about it.

0:12:28.9 SC: Well, but it does bring up the fact that for a physicist studying, the origin of the universe, it's a fairly impersonal line of research, but you're studying the brain and vision, and you have a brain and you have vision. So there must be moments when you realize like, oh yeah, that's what my brain is doing.

0:12:44.5 DT: Oh, every time. I mean, that's one of the most beautiful things about being a vision scientist. Like, you open your eyes and you see the miracle that you're trying to explain, like we're all experts on vision in some sense.

0:12:58.5 SC: Very good. Yeah. Okay, good. In fact, let's dig a little bit more deeply. 'cause I think you've already suggested this, but what we call the visual cortex is subdivided, right? There's V1 and V2 and V3, and what roles do these different folks play?

0:13:13.3 DT: Yeah. I'm impressed that you've heard about V3 'cause there's been like just like five papers written about it. But yeah, there's all these different areas and the fact is we know very little. So because, and one of the reasons is that I know there aren't that many visual neuroscientists, and they all like sort of congregate. So there's lots of lab studying V1, and so everyone is studying V1. So, what are these different areas doing? I mean, we have some cartoon picture that V1 is doing this early processing of edges, and then it goes to some intermediate stage that maybe it's like doing segmentation. So, you know, picking out the borders of different objects, figuring out the surface properties, the textures of different objects, and then it the processing goes forward.

0:14:00.8 DT: And I should say that there's a very important principle that there's actually two streams of visual processing. There's this dorsal stream and this ventral stream, and they do different things. So the ventral stream is specialized for object recognition. Just recognizing that is a leopard, that's my mom. And then the dorsal stream is really the pathway to our motor cortex, right? So it's involved in knowing where things are and what their 3D shape is so that we can grasp them in the correct way. And it's very interesting that the brain has actually dissociated these two functions, right?

0:14:38.6 SC: I mean, it is very interesting. It's more than interesting. It's amazing to me because how did it know to dissociate those two different functions? So you're saying, when you say dorsal stream, ventral stream, does this just refer to like literal pathways in the brain down which information is flowing?

0:14:54.7 DT: Yeah. There're two different pathways. And the way that we know that there's this division of labor is that people have lesions and they can become selectively unable to recognize things, but they can still like, grasp them just fine. Or they you know, they can't grasp them, but they can recognize them just fine. So physically distinct.

0:15:14.2 SC: That's very interesting because I wasn't actually thinking of talking about consciousness that much, but it certainly does reinforce the idea that there are individual pieces of the brain doing very specialized tasks and whatever we call consciousness is somehow knitted together out of all these various things working in concert.

0:15:33.1 DT: Yes, that's right. That is one of the huge puzzles. When I was in grad school, I remember my classmate pointed out this mystery, and it still bothers me. And what he pointed out was that if you look, say, if you just look at an edge, okay, any edge, it's so sharp and so fine. And that the only part of the brain where the neurons are, have the high enough resolution to represent the edge is probably V1, right? Like primary visual cortex. And yet you're conscious of it. Like, how is that possible? Does that mean that like V1 is conscious? And if it is, like, you know, like what is it that makes it conscious, right? Because all the things that we understand about V1 are, I would say quite, I mean, the processing seems quite mundane, right? Like you have an edge detector, you can program that, you know, in my lab, like, what makes that conscious? Like what is that thing? And so I think that's very... That's the big mystery.

0:16:30.1 SC: Well, and it reminds me of not quite an optical illusion, but there are these pictures that are kind of indistinct and people will say, what do you see like in the cloud? Do you see a giraffe or a face or whatever? And if they... Whatever they say, first is what I end up seeing right? It ends up...

0:16:46.0 DT: Yes.

0:16:46.5 SC: It goes into my brain and my brain can't unlock itself from that suggestion.

0:16:50.8 DT: Exactly. So there's this one idea about consciousness that I don't think explains the mystery of consciousness. I've never come across any theory that actually like, explains the mystery of consciousness and we can get into that later 'cause I'm interested in your thoughts about that. But the one theory of consciousness that I've just like as a mechanism, like what is this... I mean, some people call it the predictive coding theory or the generative model theory. But the basic idea is that everything that we're conscious of is generated by the brain. So all this stuff goes in and we process it, and it gives us the input to know what to generate. But then what we actually are consciously aware of is the top down process that actually recreates the world.

0:17:42.8 DT: And I think this is a really beautiful theory for multiple reasons, but one reason in particular is that it explains this mystery of why our conscious perception is always like reluctantly consistent across different levels of representation, right? So, if you show me this illusion, the space vase illusion, I think everyone has seen that, right? It's like you can see this thing either as two faces, profile faces or a vase. And when you see it as a vase, it's not only your high level percept of the identity that's a vase, but also all the details, right? As you look at the little piece of edge at the vase, it's always owned by the vase and not by the face. And those little details, we know from decades of neurophysiology that's coded in a different area than the area that's coding the identity.

0:18:37.8 SC: Ah, okay.

0:18:38.5 DT: Right? So how do you get this synchrony? Right? And also when you see it as a vase, like both edges are always consistent. You never see like one side's profile, and the other is like the vase, 'cause that wouldn't makes sense. And so how does that work? Because like this area that's coding the edge ownership, V2, like the neuron coding the left edge of the face is not directly talking to the neuron coding the right edge. So how do they always come in agreement? Like always in agreement, right? That demands an explanation. And this predictive coding theory, whereby consciousness is generated through top-down feedback, beautifully explains this because it says that it's generated to be consistent, I know it's a base and I generated those two edges. So of course they have to be consistent.

0:19:19.5 SC: That does make sense, actually. I mean, we talked to people like Andy Clark and Neil Seth who've worked on these kinds of things. But my own understanding nudges forward incrementally. So I won't say that I have any grand theory of this myself. But basically, if I can sort of rephrase, you're saying we have concepts, right? We have models of the world in our brain. And rather than just having an image that we keep pixel by pixel, we fit it into the box that is given by the concepts we have there. And then we sort of keep up that box. This is a face or this is a vase until something pushes us out.

0:19:56.8 DT: Yeah, exactly. That's a beautiful way of thinking about it. Another way I think about it is that, the brain has like something like a video game engine. Like there's just a space of reality that it has to live in, like your percepts have to live in. And all the inputs are doing is turning the knobs and then you generate that. Reality.

0:20:14.3 SC: Good. And I guess the one other thing at this detailed level I wanted to get on the table is that it's not just a one-way flow of information, right? It's not just that we see the photons in our eyes, they go to V1, they go to V2, then somewhere else we construct reality. But there's like feedback and feedforward going on.

0:20:31.3 DT: Oh, yeah. There are so many feedback connections. Yeah. It's almost every area. I don't know. I think there's like one. Connection like between IT cortex and the striatum that only goes from IT to the striatum. But yeah, it's like, I don't, almost every single connection in the brain is bidirectional.

0:20:52.2 SC: Is there something specific about, if we sort of naively think of V1 as detecting light and dark and just shapes and things like that? How would that be affected by other parts of the brain? It seems like it has a job and should just do it.

0:21:11.4 DT: Yeah, well, so the idea of this predictive coding model is that, you know, it's like a whole cascade of dominoes. Like you think you see a face in the clouds, then that's going to bias V2 to generate these edges. And then it's going to go back to V1 and say, "Oh, that should be like a little bit darker because that's the eye." And V1 would actually be filling that in.

0:21:29.4 SC: And how, that sounds great. I love it. How well do we know? That that's happening? Are we like looking at individual neurons in V1?

0:21:38.0 DT: As someone, you know, our lab has spent like the past five years, you know, trying to find evidence for this predictive coding generative feedback hypothesis. I would say we still don't know, the jury is still out.

0:21:55.7 SC: That's perfectly fair. And it's very good that we could admit that, right? I always say physics is the right science to go into for people with short attention spans, but So neuroscience and biology require an enormous amount of knowledge and uncertainty in your brain at any one moment. And so then, okay, you mentioned something. Again, I want to dig it into, IT cortex. I take it that that is the inferior temporal cortex.

0:22:17.2 DT: Yes.

0:22:18.5 SC: Which I just looked up on Wikipedia a little while ago. So don't be too impressed that I know what it is. But maybe you can explain to the audience what that is.

0:22:25.8 DT: Sure. So your brain has, what is it, occipital, temporal, parietal, frontal, four lobes, right? And so the temporal lobe is this thing right next to your temples, your ears, that region of your face. And so that's the temporal lobe. And in humans, there's actually two big, I don't know, sulci. Those are like hollows in this, folds in this cortex. And so the bottom part of that, that's inferior temporal cortex, and that cares about vision. And then the stuff above that. It is really to language, actually, in humans. And in monkeys it also responds to auditory cortex. And when we record from, you know, inferior temporal cortex, we often, our electrode goes through this stuff above it. And I would know because I would like shake my keys and I would hear it responding. I'm like, okay, we're not in visual cortex yet. And then we get to inferior temporal cortex where the visual cells are.

0:23:22.5 SC: So I think that you just changed my life a little bit because as a physicist, I always thought of temporal cortex as having something to do with time. Because it's temporal, but I think that you are implying that it has something to do with being close to our temples.

0:23:34.7 DT: That's right. It's more perfect.

0:23:36.9 SC: Okay, good. So this inferior temporal cortex is doing a more abstract job than the visual cortex?

0:23:45.5 DT: Ah, yeah. So we talked about the ventral stream, right? This is the pathway that starts in V1 and it goes to inferior temporal cortex. And that's the part of the brain, the pathway that's responsible for object recognition, right? Knowing, again, that's your mom, that's your dad. How do you do that? You know, those are keys and that's a pencil, like just recognizing objects and recognizing them invariant to how they're presented, right? I need to recognize, my son, if he's looking at the profile or looking at me or even the back of his head. So how do you do that? That's a huge computational problem. And I think inferior temporal cortex is the part of the brain that's solving that problem of object recognition.

0:24:26.6 SC: Over evolutionary history? Did that come later than the visual cortex? It sounds like a higher level process.

0:24:34.1 DT: Yeah, yeah. That very interesting question. We've actually just finished the study on tree shrews because we're exactly interested in that question about evolution. So tree shrews are these animals that I think they're native to Indonesia. And they're one of the closest relatives to primates that are not primates. And they have really, really good visual systems. So there's like tons of labs studying vision in mice, but mice are like terrible at seeing. There's a tiny fraction of their brain is responsible for vision. But tree shrews have giant eyes and they really see very well, have high acuity. And so we were interested in exactly that question. Does inferior temporal cortex exist in tree shrews? And we found this very surprising result that V2, which you mentioned earlier, the area right after V1 seems to have many of the functions of IT cortex. And there's even face cells in tree shrew V2. So, yeah, it's interesting to think about how this evolved. But we think that it started from a very shallow hierarchy and became deeper.

0:25:34.9 SC: I see. So maybe the tree shrews don't have an inferior temporal cortex, but they do the same job elsewhere. And then over history, it sort of gets differentiated.

0:25:47.1 DT: Yeah. That's what it looks like.

0:25:49.1 SC: The people who are better at it, the shrews who are better at it become better survivors. And I guess we didn't even talk about the audio input. Way back when I had David Eagleman on the podcast and he told me this story. I would love to hear your picture of it. But if you see someone dribbling a basketball and it takes longer for the sound to get to you of the basketball hitting the street than the vision, obviously, speed of light. Faster than the speed of sound, but your brain matches them up. So it looks like they're happening at the same time until the person juggling the basketball gets so far away that that becomes unrealistic. And then suddenly you snap into this mode where the two are unrelated to each other. Does that sound familiar?

0:26:32.1 DT: Yeah, that sounds very reasonable. And I think that also gets, I think it supports this idea of predictive coding in general, the fact that what you see is generated, right? So you get all these signals, they're mismatched in time. And there from that, you build this inference. And actually, getting back to consciousness. So, we think that inference step where you put all this together and then build what you consciously see, we think that occurs in discrete time points. It's actually, even though consciousness feels like it's continuous, we think it's happening discreetly. And in between, that's when you're taking all these measurements. So there's a very interesting story there about how you perceive time, but I think it's constructed, it's definitely not like when the signal comes in, that's when you perceive thing as happening. And David Eagleman has certainly has beautiful demonstrations of that.

0:27:26.1 SC: Well, okay, but this is fascinating. I'm going to go off script here, because I want to hear more about this. Now, I understood that there was a delay in time, takes time for the brain to put that picture together, right? But you're saying something very interesting that I don't think I've heard before that our perception of time is kind of like a film strip, right? Where there's a discrete set of frames a little bit different from each other. There's so many of them that it seems continuous to us, but there is some number, the discreteness between our conscious experience of different moments of time.

0:28:00.5 DT: Yeah, that's right. There's discrete frames. And I think I'm saying something even more radical than that, which is that there's discrete frames and those discrete frames, unlike in a film strip, are not consecutive. Like there's like spliced with stuff like where you're basically unconscious, and then you become conscious again. The film strip goes on. But because you were unconscious, you didn't even realize that this thing happened in between where you're not conscious. And so all you're aware of is the. You know frames where you are conscious right so we think that's what's happening.

0:28:30.2 DT: There's there's a going back to illusions there's like do you know the wagon wheel illusion? Like if you just take a disc where you have, I don't know, you paint like different frequencies of like white and black at different radii, if you spin it it'll look like part of it's moving backwards because of aliasing, and if you and if you spin it and you take frames like that with a film strip then it looks like part of it's going backwards right like it's aliasing. But you can actually see this like in real life. And Dale Purvis, who's a developmental neuroscientist who became interested in vision, he made this amazing leap of inference, like from the fact that you can see this wagon wheel illusion in real life to the conclusion that our consciousness is discrete. It's like sampled, right? Just like a movie strip. Yeah.

0:29:17.0 SC: But when you say it that way, yeah, now it makes perfect sense. That is actually evidence for it empirically. Good. So, I got to ask, how long am I unconscious for? What is the space in between two moments of conscious perception?

0:29:29.0 DT: I think it depends on the stimulus, like how fast it's coming in. But in the situation where we've studied it, we find epochs up to a few hundred milliseconds where you're not.

0:29:40.2 SC: A few hundred milliseconds. That's a lot of...

0:29:41.8 DT: But where the neurons are not representing what you're consciously seeing.

0:29:43.0 SC: Wow, that's a lot of unconsciousness. Okay. Does everyone agree with this or is this sort of cutting edge speculation?

0:29:53.6 DT: This is. We haven't even published it yet.

[laughter]

0:29:53.6 SC: Okay, good. That would be that's very cool stuff. I want to hear about that coming in. But, okay, so this is a lot. Let's pause and take our breath here. I mean, it sounds like maybe I just don't understand computers well enough or AI well enough, but it sounds like what the human brain does is kind of more subtle, right? There's a lot of different streams doing different things, matching on to various templates and so forth. It's kind of a remarkable picture that I guess makes sense from an evolution point of view, that different capacities become relevant at different stages in the biological evolution.

0:30:35.4 DT: I'm not sure I resonate with that. I think my hope is that the visual system is this beautiful piece of machinery and we'll be able to understand the fundamental principles that it's implementing.

0:30:49.3 SC: Oh, that I 100% agree with. I'm just saying like the actual computers that we have seem to be more simple minded than this beautiful brain. I'm very much a physicalist, a mechanist about the brain. I can imagine that we build a computer that does exactly what the brain does. But, we intelligently design our computers. So we make them sort of direct and as straightforward as possible, whereas the brain grew up over millions of years to lump together different capacities in. Useful ways.

0:31:18.7 DT: Yeah, I mean, as an experimental neuroscientist, I would have to say, I don't know, the brain just astonishes me with how precisely it's organized.

0:31:28.3 SC: Yeah. Oh, yeah. Okay. Very good. Well, and let's get on to one of its most impressive abilities, which is recognizing faces. This is something that you've been very active in. So is there a, I mean, it's obvious why we would want to be able to recognize faces. Is there a specific part of the brain that does that, or is that also distributed through different parts?

0:31:50.2 DT: Yeah. This is probably, you know, our ability to recognize faces is probably one of the functions that's been most clearly ascribed to a specific piece of cortex. And so the earliest evidence for this came from, again, lesion studies, people with strokes. And suddenly they can't recognize faces anymore, but they can recognize everything else just fine. And that just shouts that there's a piece of cortex that's dedicated to representing faces. So that was known. And then in 1997, Nancy Kanwisher at MIT published this landmark paper. It's like the most cited paper in the Journal of Neuroscience, where she reported that human patients, just human subjects, normal human subjects that she scanned inside an MRI scanner, and she showed them pictures of faces and objects. They all showed this region in their temporal lobe. In the right temporal lobe that responded much more to faces than other objects. And it was like the size of a blueberry. And it was like in the same place in every single subject. And it just showed this huge signal in response to faces. And so that was like really strong evidence that there is a piece of dedicated cortex representing faces.

0:33:03.3 SC: Does it have a name?

0:33:05.2 DT: Yeah. So she called this the fusiform face area because it's in this part of the temporal lobe called the fusiform gyrus.

0:33:10.7 SC: Okay, very good. It's amazing to me that was only 1997, that this is all so new and fun, really.

0:33:17.2 DT: Yeah. I was a grad student then. I remember reading her paper, and it seemed like so weird to me. Why does a brain have a piece of cortex for faces? Faces don't seem that different from anything else. And so, yeah, little did I know how deep I would go down that rabbit hole.

0:33:32.1 SC: Right, but you have. So we've been, since 1997, learning a lot more about how this works. And is it another story where we're discovering substructure in the fusiform face area that, you know, there's different parts doing different things.

0:33:46.2 DT: Yeah. So it's been an incredible story figuring out the details of how this area works. So, when I was a grad student, I was scanning monkeys, actually studying 3D vision. And then I decided to show them faces and other objects, look for a face area just for fun, see if monkeys also have a face area, you know. Not a problem if they don't, because fMRI experiments are very easy to do. And they did. And not only did they have a face area, they had six of them. And so it was like, whoa, there's these six regions. What are they doing? And because we're working in the monkey, we can insert an electrode into each of these regions and study what the neurons are doing.

0:34:26.8 DT: And you asked if there's functional specialization. And indeed, each of these six patches seems to be performing a different function. So in particular, you know, the most posterior one, it seems to like really like eyes, right? Just like a dark disc inside this outline. And then the next one, you know, cares about faces, but at specific views. And then you go even more anterior. It responds to faces in a mirror symmetric way. So it likes left profiles and right profiles or faces looking up straightened down. And then the most anterior patch, you have these incredible cells that respond in a view invariant way. So they don't care which way the face is pointed, but as long as it has the same identity, it'll respond. So there is this remarkable division of labor.

0:35:13.4 SC: So there's a part of my brain that is just noticing eyes.

0:35:18.6 DT: Well, and other features, but eyes are so prominent that, yeah, a lot of cells, like 70% of the cells in this one face patch are tuned to like the size of an eye. You can even show just a simple cartoon face and you change those two pics regions and they start responding more and more. Yeah.

0:35:37.6 SC: And I have to ask, are you showing the monkeys pictures of monkey faces or human faces or does it not matter?

0:35:43.5 DT: Yeah, it turns out that it doesn't. I mean, it does matter, but they're using the same principles to code all these different faces. And they even respond to cartoon faces and faces in clouds. And yeah.

0:35:55.1 SC: Well, this is why I was going to get there. I mean, does this help explain why a few dashes of line in a cartoon can be so expressive if they're supposed to look like a face? Like our brains are sort of tuned to notice these tiny differences?

0:36:09.7 DT: Yes, exactly. Yes.

0:36:11.8 SC: And probably there's some story of how those parts of the brain are talking to more emotional or romantic, I don't know, parts of the brain.

0:36:23.2 DT: Yeah, we know much more about how the identity is represented than how the expression is represented. But indeed, we have the specialized machinery for seeing all these different dimensions, right? Because faces are multidimensional. You need at least like 50 or 200 dimensions to create a really good likeness. And we can see all those dimensions simultaneously. So that's really remarkable, right? Like color has three dimensions. And I don't know if you've heard of Chernoff faces. It's like this method for visualizing multi-dimensional data sets where you map them onto faces and then you can see oh yeah there's like three dimensions that are changing because you can see that in the faces.

0:37:01.1 SC: Wow I should know about that okay very good and I gotta get one vocabulary word in there apparently, the name for a cluster of neurons in this part of the brain is a glob.

0:37:12.3 DT: Yeah we call it a patch. So there's also yeah there's like globs and blobs. Yeah, we... There's these colored globs, so that's in a different part of the brain. In V4, there's these specific sub regions that care about color, and those are called color globs.

0:37:33.2 SC: Globs, globs and patches. And is the fusiform face area part of the visual cortex?

0:37:38.5 DT: Yeah. So, the fusiform phasers in the human brain and then monkeys, there's we think the homologous region and then the fusiform face area is, yeah, it's part of the ventral visual cortex. It's like high level visual cortex so it's beyond your V1, V2.

0:37:53.9 SC: Good. Okay. And when was this research that you just mentioning being done? How old is this?

0:37:58.5 DT: Oh so recording from all these different face patches, that was like 2000... We published the paper, I think in 2010 on the this hierarchy.

0:38:10.1 SC: Okay, good. And do we, maybe this is an unfair question. Recognizing faces, or even... Sorry, let me let me back up. Is the job of the fusiform face area more to recognize whose face this is, or is it to recognize this is sort of the aspect of the face? This is a frowny face, this is a smiley face. This is a friendly face.

0:38:34.0 DT: I think it's not processing emotion per se. It's processing the physical features of the face. Like what is the inter eye distance? What is the texture? What is the shape of the face? It hasn't yet reached the level of the explicit identity. We think that happens later on.

0:38:50.8 SC: That happens somewhere else. Okay. Very good. Are we down to looking at the jobs of individual neurons, or is that a technological challenge?

0:39:00.3 DT: Yeah, all our recordings are of single neurons. We put these very thin wires. They're insulated everywhere except the very tip. And the tip is like 10 microns wide. And so we can pick up the electrical activity from many hundreds of single neurons now. So yeah, we can ask what is the selectivity of single neurons?

0:39:14.1 SC: Let's be... Most of the audience listening is probably not professional scientists. So let's dig into the experiment itself. You mentioned FMRI, and now you're mentioning probes. These are two very, very different ideas, yeah?

0:39:28.7 DT: Yeah. So FMRI, people say it's measuring the brain's plumbing. It's just measuring where the blood is flowing, right? And it turns out that when you use a part of the brain, there's more blood flowing to that part of the brain. It's like homeostatic mechanism, so yeah. And so it's very coarse. It's measuring like neural activity at a scale of a millimeter cube, and that contains like a 100,000 to a million neurons. The other technique that we use, like once we find these like face areas or color areas or so on, then we wanna know what the details are. How are the neurons actually representing the face? You can't figure that out by studying the brain in millimeter size pixels. You need to study the single neurons. And so to get at that detail of information, then we insert these electrodes that are... Let us pick up neural activity. And these days, we're using these electrodes called Neuropixels Probes. And the idea is sort of like they let you watch the TV of the brain, right? So Neuropixels. And these have like 4,000 contacts. So they're using this silicon fabrication technology. And so there's these silicon probes and so you can actually record from hundreds of neurons simultaneously, which is a really exciting thing.

0:40:41.1 SC: Okay. And can we kind of go backwards? Can we just look at what the neurons are doing, and work out for ourselves, what is being looked at?

0:40:49.4 DT: Yeah. So there's... Many labs are motivated by that goal to decode the neural activity and try to recover what are you looking at? What's happening in the world? And so we've done this in the realm of these face areas. And that was a really satisfying accomplishment to be able to take just the neural activity from 200 neurons, and from that be able to reconstruct precisely the face that the monkey was seeing and creating a likeness that you couldn't even tell which one was the real stimulus and which one was the reconstruction that we made from his neural activity.

0:41:26.5 SC: So there's an obvious technological application of this to computer brain interfaces?

0:41:34.3 DT: Yeah.

0:41:34.9 SC: But only if we can stick probes inside our brains.

[laughter]

0:41:37.4 DT: Unfortunately, yes.

0:41:41.1 DT: Okay. But it is a little... Even though I am a thoroughgoing physicalist, it is still slightly spooky to me that if someone could probe all of my neurons, they could figure out what I'm thinking. But that's what you're on the road to doing. So we'll have to learn to deal with that. So does this help us explain, you mentioned lesions before. Does this help us account for various disabilities? I know that some people have trouble recognizing faces.

0:42:08.6 DT: Yeah, that's right. So people have studied these so-called prosopagnosia who have trouble recognizing faces. And some of these people have lesions, but I think there's like 4% of the population, some very high percentage didn't have any stroke, but they're just like really, really bad at recognizing faces. And indeed, if you study the the brain activity in these people, some of them have poor selectivity in their face areas or small face areas things like that. Yeah.

0:42:36.2 SC: Is there hope for improving it, or is that too ambitious right now?

0:42:39.3 DT: Oh, that's very interesting. Maybe, I mentioned that this face area in the human is in the right hemisphere, and the corresponding piece of cortex in the left hemisphere is actually responsible for recognizing letters. And so there's this remarkable plasticity in the human visual system. And so I don't know, maybe with some kind of training.

0:43:04.2 SC: Okay, I'm a little bit...

0:43:05.5 DT: But I think it happened early on. Yeah, I'm not sure.

0:43:07.6 SC: No, that's great. But now I'm amazed at... So when you say that there's a little part of my brain size of a blueberry whose job it is to recognize faces, I nod along and say, yes, that makes perfect sense why that would evolve. Now, there's another part of my brain that is similar, that is recognizing letters, that seems a lot more culturally contextualized. Like, what about before we had letters, what was that part of the brain doing?

0:43:37.7 DT: Yeah, we think it was recognizing faces. So like monkeys, the face areas are perfectly bilateral. And what's amazing is that in illiterate people who don't recognize letters, they also have bilateral face areas.

0:43:49.6 SC: Wow. So the obvious conclusion is that we have repurposed part of the brain that was helping us recognize faces to help recognize letters.

0:43:58.4 DT: Yeah.

0:44:00.4 SC: That's amazing. Okay, very good. 'Cause that means that, yeah, who knows what we'll be repurposing next. Are we worse? Are are people who are literate worse at recognizing faces than people who are illiterate?

0:44:12.3 DT: I don't know, but it seems like a logical conclusion. Someone should test it. Yeah.

0:44:18.2 SC: Yeah. Okay. Good. But it also brings us to some of the more provocative things that I've read in your work and in discussions of it. In some sense, we can build upon these ideas to try to help us understand abstract thought or the sort of origin of symbolic thought in the brain. I'm probably saying this badly, 'cause I'm not sure I completely understand it.

0:44:43.2 DT: Oh. Well, the way I... I'm tackling this problem of how abstract thought arose, it's actually totally independent, this work on faces. And we're going to the dorsal stream, right? So faces are a process in this ventral stream. I mentioned at the beginning, there's this dorsal stream that's the basis for action, right? And so how do we know how to act in the world? I think we had to have a compressed, symbolic representation, like an event based understanding, right? We really... So those are the objects. This is what they allow us to do. So I'm gonna go pick up that banana now. And so I see the big challenge to understanding how symbolic thought arose as understanding this very concrete problem of segmenting and tracking, right?

0:45:33.5 DT: So we have all these pixels going into our eye. How do we transform that into objects, into persistent objects, right? And once we do that, then we can assign labels to them, and we can associate them and we can think about them. But before we know that that's Sean, and I can walk around, Sean is still in the room. Before we have that concept of objects, we can't even think about anything, right? It's just like this sensory blur. So I think that's like the key step. And we don't understand that at all.

0:46:10.3 SC: Good.

0:46:10.4 DT: In my opinion. Like how the brain actually goes from all this sensory stuff and all these features to these discreet objects. So yeah, we're tackling that.

0:46:20.2 SC: I did have Judea Pearl on the podcast, the Causality Guide, and he says that what babies do is, they work as hard as they can to construct a causal map of the world. They keep poking at things and seeing what happens and seeing what leads to other influences and so forth. And that doesn't say what is happening in the brain, but it fits in with the story that we're kind of trying to construct this model out of our experiences.

0:46:47.3 DT: Yes, exactly. Yes.

0:46:48.7 SC: And this brings us also, you wrote a recent paper with your father, which I thought was very charming, that you wrote a paper with your father. What does your father do for a living?

0:47:00.4 DT: He's a mathematician.

0:47:00.5 SC: Okay. Because the paper was very mathy. I was impressed at the amount of differential geometry in this paper. I don't know if you usually have that in your papers or you were leaning on your collaborator.

0:47:10.0 DT: No, it was kind of funny. I think I was an undergrad. Yeah, undergrad at Caltech. I took this class on differential topology. I brought my textbook home with me, and my father studied it, and he realized like they had applications to these ideas and vision, and it all worked out. Yeah.

0:47:28.9 SC: So if my very brief understanding of this paper is you're trying to understand something you just mentioned, that sort of how we know that there is the same object in the world, even though it temporarily walks behind an obstacle and we can't see it and it walks out the other side, but in our brains, that's a continuous chain of being there.

0:47:49.0 DT: Yeah. Or like if I walk around you, I know that you're the same person. How can I solve that? And it turns out like people in computer vision, they say you just put lots of samples, just lots of training data, and then you just magically learn, okay? And what we say in the paper is that no, you don't need any training data. It's actually, it's a really elegant mathematical problem that goes to the very definition of a surface, right? Like a surface is this collection of charts, overlapping charts and so that overlap is the key, right? So as I walk around you, I compute this chart, and I change my perspective either because I have two eyes or I move, and then I find this overlapping chart, and so that's part of the same surface, and I keep doing that. I can walk all around you, and so I can form this equivalence class of charts, where the equivalence relation is to overlap. And so it's like, it's a really beautiful mathematical theory of how objects can arise.

0:48:46.2 SC: This is great. You've learned all these buzzwords that are in the chapter one of my general relativity textbook when I'm teaching people differential geometry, so that is cool. So, would it be an exaggeration to say that effectively what the... The point is that it's kind of simpler for the brain to conceptualize the object as being the same object, even as you're looking at it from different points of view than to sort of imagine distinct objects at every step along the way?

0:49:16.6 DT: Yeah, absolutely. Yeah. The biggest job of the visual system is to solve this in variance problem and figure out what is corresponding to the same thing, 'cause the information is changing so much, right? Just like move your head. Like my hair's within all the pixels change. You have to like be able to counteract that. Yeah.

0:49:37.3 SC: So in some sense, it's like a compression problem, right?

0:49:41.3 DT: Yes. That's right.

0:49:43.5 SC: Yeah. And again, stab in the dark here, does this have anything to do with the Bayesian Brain Hypothesis? We talked to Karl Friston once on the podcast.

0:49:52.8 DT: Yes. I think it has a lot to do with it. So the way our theory works, like the way you actually compute these overlaps, right? Like these diffeomorphisms is, so we're asking, so the mathematical problems, how do you know that this view of a patch, a visual patch is a transformed view of this patch? Okay? And the way we figure that out is to introduce these things called dynamic receptive fields, which essentially, you can... So the measurement that the brain makes is basically like the projection of the image patch onto a receptive field function, right? Projecting just inner product. And so you can actually transform, you can set up a dynamical system to transform the receptive field functions to counteract the change in the image.

0:50:42.8 DT: So a very simple example, like say the left eye image patches has shifted, compared to the right eye image patch by like 10 pixels. Well, if I shift my receptive field function by 10 pixels, then I'll get the exact same measurement, right? And so that's the idea. Like we introduce these dynamics on the receptive field function, so we can compensate for these transforms. And to my understanding, that's exactly what's happening during Bayesian inference. So you have this top down s signal like that's trying to predict your sensory signal, right?

0:51:22.0 SC: Yeah, so.

0:51:22.2 DT: Every incoming Sensory signal.

0:51:23.2 SC: Again, I'll try to sort of re-say it, so I think that I understand it. If I think of what I'm looking at right now, as my visual field as being represented by a number of pixels with different values, there's an enormous amount of information in there. And if I change the direction in which I'm looking by just a little bit, I have two choices. One choice is to completely rewrite that, and replace it with a different, huge amount of information, or the other choice is to say, it's almost the same thing you were looking at, but shifted by this amount.

0:51:56.6 DT: Yeah, exactly. Right. So yeah.

0:52:02.7 SC: And that's an enormous sort of saving in terms of energy and information processing and all those other good thing. And is this nudging us? It is your fault, 'cause you brought up consciousness, or maybe I did, I don't remember. But is this nudging us toward a better understanding of consciousness of who we are, like our ability to conceptualize the world at this more abstract level?

0:52:22.1 DT: I hope so. I think if we can solve the nuts and bolts of how neural activity is representing what we see, we will be a long ways to understanding consciousness. And I wanted to ask you a question.

0:52:40.5 SC: Sure. Good. We're late in the podcast now, so we can let our hair down and go wherever you want.

0:52:46.2 DT: I wanna get your take on this. Okay. Like, we evolve to survive, right? Our genes don't care at all if we're conscious or not. Like that doesn't affect selection, right? So it seems to me that either... So they just care about our behavior. So if we're like a pea zombie, our genes would be propagated exactly the same as if we're conscious. Therefore, I would argue that either consciousness is like, we're just incredibly lucky, and it just happened that brains with our behavior also happen to be conscious or any kind of system, any kind of complex system that does what we are capable of doing and represents the world with as sophisticated way as we can and can see moving cars and people and can track them and navigate. Anything that has all of our behaviors is likely going to be conscious. Would you agree with that?

0:53:38.3 SC: I would agree 100%, yes. I'm entirely on your side. There are plenty of people, let's just say, I don't wanna characterize how many. There are plenty of people who would disagree, right? That's the whole origin, the impact of the zombie argument in consciousness studies is, people... It was popularized by David Chalmers, but it goes back to other people. And the idea is, I can conceive of a being that act in exactly the same way that I do, but doesn't have inner conscious experiences. That is what I label a zombie. And if I can conceive of that, it must follow that whatever inner conscious experiences are, they can't be reduced to the behavior of the neurons or the atoms or whatever in my body. There must be something other than the physical behavior that we're talking about.

0:54:26.7 SC: But my response is more or less exactly along the lines of what you were hinting at. I would say no, if you really take seriously, the idea that the zombie is behaving exactly like a conscious creature would, including when I ask it, "Are you conscious?" It says "Yes." When I tell it a sad story, it starts crying. Like all of those things, then we would just call that conscious. To someone who is a physicalist and does think that consciousness is sort of an emergent, higher level way of talking about things, there's not more to it than that.

0:55:03.4 DT: Yeah. Okay. I'm so glad that you agree. So you wrote this very interesting essay about like, can physicists explain why there's something rather than nothing?

0:55:13.7 SC: Oh, yeah. Okay.

0:55:13.8 DT: I forget. What was the title of your essay?

0:55:15.5 SC: It was literally just, Why is There Something Rather Than Nothing?

0:55:18.8 DT: Yeah. And I love that. And I think that, am I correct that your message was that they can't, they actually can't explain that?

0:55:29.3 SC: In fact, I would go, again, I would be even more radical in your own words. I would say it's not even an answerable question. It's not. Not that we don't have the ability to answer it, but it's the kind of question, why is there something rather than nothing that literally does not have an answer. There is no why. There's no reason that we're going to discover someday, why the universe exists rather than doesn't.

0:55:51.7 DT: Great. So, okay, let me ask you then. 'Cause I think of consciousness in much the same way. I think that we are gonna discover exactly how a set of neurons needs to be configured to be conscious or not. Like that's a scientific question. But to me, it seems like consciousness itself is something given. Does that resonate at all with you?

0:56:12.0 SC: Well, we're gonna have to interrogate what you mean by the word given, but to me, it is certainly a way of talking about what is going on both in our brains and then in our macroscopic behavior, right?

0:56:25.3 DT: I mean that it's something like the existence of matter. Like, you just have to accept that certain complex systems are conscious. You can't ask why is it conscious. The fact that it's conscious, it's like subjective experience is as much something you have to accept as objective experience. And you can ask like how you can transform it and create different types and all of that is like scientific questions, but like, fundamental fact that subjective experience exists is like something you have to accept.

0:56:54.4 SC: I don't know. I'll have to think about that one. I think it's an interesting perspective. It's not exactly what I would've said automatically. What I would've said is, if I had never heard of the idea of consciousness, but I was an anthropologist from Mars as Oliver Sacks once imagined. And I came down, I interacted with human beings, I would notice that they seem to react and behave in ways that indicate they are aware of different things, sometimes unaware of things. Other times, they have what I would label as mental states that help me explain their behavior and things like that. So I think we would've invented the idea of conscious states and behaviors, even if we didn't know about it. I think that it's a useful description of this incredibly elaborate emergent thing we call a human being.

0:57:45.7 DT: More about that.

0:57:48.2 SC: I did write a paper that I thought was gonna be the one you were gonna reference called Consciousness and the Laws of Physics. But really, what I always say about these things is, I don't know anything about consciousness. All I know is you don't need to invent new laws of physics to describe it, because we understand the laws of physics, much better than we understand consciousness. It's very cart leading the horse to think that we should change laws of physics to help understand consciousness.

0:58:12.6 DT: Oh, that's interesting. Okay, 'cause this is the last question I wanted to ask you. To me, something really mysterious about consciousness is that like, as a physicist, and chemists and biologists, we explain these systems at different levels, right? Like you explain at this like, very fundamental level, and we explain at the level of there's like photoreceptors and visual cortex and stuff, but we think that there's just different levels of explanation. Everything is consistent, and you can explain everything. You can predict everything about the system at your level, and then it's just like more, it's just more simple to explain it.

0:58:48.6 SC: More coarse grain at higher level. Yeah.

0:58:49.3 DT: More coarse grain at a higher level. And it seems to me like there's a... The fact that we're conscious speaks against that. The fact that we're conscious of red and the objects around us, that seems to suggest that there is like a correct level of interpretation of a physical system. Like, it's not... You can't just think of it as like random atoms bopping around. You have to think of it at that level where your conscious percept exists. Does that make sense?

0:59:20.0 SC: Well, I don't know. No, so I would think, I would suggest that both levels are perfectly good, and they stand on their own feet independently. So if I were Laplace's demon, if I had this magical ability to understand the complete state of every atom and electron photon in my body, then I think I could successfully predict what my body was going to do next, or over some period of time without ever using words like consciousness. Or for that matter, words like entropy or temperature, or any of those other higher level words, I would only talk about the atoms and what they're doing, but...

1:00:00.8 DT: Yeah. But you couldn't explain the internal consciousness, 'cause you're... You know what I mean?

1:00:06.9 SC: Maybe I know what you mean. Maybe I don't. Can I explain what a table is, just by listing all of its atoms and how they're interacting with each other?

1:00:15.6 DT: Yes, you can.

1:00:16.9 SC: Okay. So...

1:00:17.0 DT: But you can't explain whether there's this, like, the table's not... When it comes to the brain, it's not just the brain. Like you have this conscious experience, and that seems to exist at a specific level.

1:00:28.4 SC: Right. So my view is that consciousness is just like the table. It is a useful collective way of describing what happens at that level. But that level is completely compatible with saying there's another level where I don't use those words at all, and still have a complete description. I'm as always happy to be talked out of these things. So let me put it in yet another way. And maybe this is vibing with the question about why is there something rather than nothing?

1:00:55.4 SC: The hard problem of consciousness is supposed to be over and above the physical behavior of the brain, right? The whole point of people who love the hard problem, how do we explain what it is like to be something, to have this inner first person subjective experience, is that they would claim that I could know everything there is to know about what the neurons do, how they interact, how they push the body around. And still, I have not accounted for what it is like to be a bat or a human being or whatever. And my attitude is that just like the why is there something rather than nothing question, we're not gonna solve that problem. It's just gonna dissolve away. As we understand better and better what the neurons are and what they're doing, we will say, well, there isn't anything extra. It's just that when these neurons are doing this kind of thing, we call that the brain is experiencing the redness of red.

1:01:49.4 DT: I agree completely with that. And I think that as AI matures and becomes conscious we'll be able to create these new quality and we'll really discover the laws that govern this relationship. And like you say, it's gonna completely dissolve away. And we'll see that consciousness is, this basic property of complex systems? Yeah.

1:02:13.2 SC: In that case, I cannot possibly think of a better closing line than that one that you just gave. So this was, I expected this to be super interesting. This was even way more interesting conversation than I hoped it would be. Doris Tsao, thanks so much for being on the Mindscape Podcast.

1:02:25.4 DT: That was so much fun. Thank you, Sean.