278 | Kieran Healy on the Technology of Ranking People

0:00:00.5 Sean Carroll: Hello, everyone. Welcome to the Mindscape Podcast. I'm your host, Sean Carroll. I wanted to start today's podcast by reading a paragraph from a new book called The Ordinal Society by Marion Forcad and Kieran Healy. So it goes like this. The idea of modernity has long been seen as having two contending aspects. On one side, the side of social organization, is the domain of rationalization and control. This is the modernity of bureaucracy, science, technology, and planning. It is the technocratic, sansimonian vision of a society run on rational principles and devoted to the elevation of humanity in the abstract. Here, the administrative task of modern organizations is to know and manage their subjects. On the other side, the side of the individual, is the domain of experience and expression.

0:00:52.6 SC: This is the modernity of the Romantics, of the full and authentic realization of the self and all its powers. Here the existential task of modern individuals is to know and create themselves. I like this paragraph because, you know, there's a little bit of historical resonance there, but also the contrast or the dilemma here is very real between the organizational systems-oriented view of the world and how that can bring about tremendous real benefits versus the romantic individual, you know, ignoring the system and going their own way. I think that for whatever reason, in the modern world, we prefer to personally identify with the romantic individual. But, you know, as we will talk about in this podcast, the modern world offers all sorts of conveniences and services that are only available to us if we kind of do agree to participate in the broader system.

0:01:50.8 SC: I recently noticed when you go to the Apple App Store and you want to download an app, they tell you what information the app gathers about you, your location data, your information from other websites or whatever, and where it sends it to. So in principle, you could just not download any app that collected information about you that you didn't want it to. I'm betting that in practice most people go ahead and just download the app, right? 'cause the app is useful. It's not like there's no point to it. I bet that most people use Google Maps when they want to go somewhere, even if that means Google knows where they're going. I've noticed that Google Maps will sometimes tell me where I parked my car. That's, on the one hand, a little weird that Google's keeping track of where I park my car. On the other hand, super convenient because I am often not very good at keeping track of where I parked my car.

0:02:44.6 SC: So today's guest is Kieran Healy. He's one of the co-authors of the new book. And the idea is the ways in which the modern world not just keeps track of us, but classifies us, right? The ordinal society is one in which people are characterized and ranked in all sorts of different ways. Ranking people has been something that has been going on forever, of course, but technology has enabled it to happen at an enormous rate from very simple things like a credit score to hyper finely divided ways like what ads you get served up when you go to Amazon or Google or YouTube or what have you. And these forces are somewhat invisible, but all pervasive. And apparently they really matter to our lives. A lot of people, when you buy a new dishwasher, you buy a smart dishwasher and it sends information to the dishwasher company about how often you're washing your dishes.

0:03:45.2 SC: And where do we draw the line? Where do we decide how much convenience is worthwhile versus how much individuality and romantic experience is worthwhile? I think these are questions we're gonna have to be struggling with more and more because these systems of surveillance and classification are not going away anytime soon. So let's go.

[music]

0:04:27.5 SC: Kieran Healy, welcome to Mindscape Podcast.

0:04:28.6 Kieran Healy: Delighted to be here.

0:04:29.6 SC: I think we got to start with a question you will find embarrassingly simple. What is the word ordinal mean to you? You have it in the title of your book.

0:04:39.3 KH: Right. Yes, I know. It's funny. I've gotten kind of, when I've been talking about the book online or to other people, immediately the mathematicians and the physicists come out and I get a lot of quite abstruse jokes about what is this.

0:04:55.7 SC: I'm biting my tongue, yeah.

0:05:00.5 KH: Right. What does this mean? Yeah. So in this case, it's fundamentally focused on the idea of ranking, although it's twofold, right? So first of all, and most importantly, it's the idea that... We live in a world where the pairing of kind of massive data sets with various processes, algorithmic, broadly conceived, statistical, mathematical, written in code of some kind, have made inroads into every social institution. And techniques of optimization and data collection are deployed to kind of streamline and organize processes across those institutions, a whole wide range of things. And the way they work is to take kind of information or data in computationally and spit out scores and especially rankings, orderings out the other side.

0:05:54.8 KH: And so fundamentally, the idea of ordinality or an ordinal society is one that's based around and justified by the idea of kind of measurement and ranking. I will say too that there is a second piece to it a little bit, which comes, so Marion Foucault, who co-authored this book with me, is French, originally, she teaches in Berkeley now. But in French, the word for computer is ordinateur. And the reason that it is that word is when in 1956 or so, IBM launched the IBM 650, which was its first really mass-produced machine. It surprised them how much the demand for it was amongst businesses. And when IBM France came to sell it, they had to decide what to call it, what to call this class of device. The natural choice would have been calculator, which is the direct translation of computer.

0:07:08.5 KH: But they consulted in a very French way. You have to be careful about what word things are gonna call. They consulted with this guy, Jacques Perret, who was a professor of Latin at the Sorbonne. He objected and he said, what about ordinateur? He said, it's a correctly formed word, it's in the dictionary, and its ultimate root is to do with ordination, the word ordination, like a religious sense of a God who brings order to the world. And he says, but that theological usage isn't frequent, so you could call it an ordinateur. And yeah, and so that's what IBM picked. And so the idea of kind of calculation as an ordering in the kind of just ordinary mathematical sense of first, second, third, fourth, but then also as a device that brings order to the world. That's what we have in mind.

0:08:05.4 SC: I love the idea of Google consulting a Latin professor before... It's a different world.

0:08:12.4 KH: Right. Very much so. Yeah.

0:08:14.4 SC: So but of course we have ranked people and classified them all the time, I mean, you and I are academics, we live in a world where half of our time is spent deciding who's better than you know who else. So you're specifically focusing on how much better we are at it now because exactly of the computer age.

0:08:33.6 KH: Or how much more pervasive it is, yeah. One of the things we do want to kind of emphasize in the book is that it's not the case that like the fact of ranking and ordering people is a sort of fundamental aspect of human social organization. And we do that whenever there's differentiation, you implicitly have the chance of some sort of ordering or ranking of classes or categories of people comes out. So that is not new at all. In fact, it's kind of an endemic feature of just how human societies are organized. And then there's also a long history of devices and methods and techniques that we have for doing this that goes all the way back to the beginnings of... All the way back to double entry bookkeeping or that bring an ordering of things to businesses to sort of mundane devices like the filing cabinet or the card index in the 19th century, which really were kind of revolutionary in their way.

0:09:37.3 KH: But what's new now is, and what's really sort of transformed not just in sort of scale, but also in scope over the last 50 years or so, is that the ability to do this has both become much more fine-grained and much more widespread. The scope that we can, the degree to which we can sort of apply these ideas and processes is sort of much wider, a whole range of kind of forms of social life that were just not within reach of any kind of measurement, certainly not any kind of real-time measurement, has really expanded. And then the degree of kind of granularity of that has also been transformed as well.

0:10:24.8 SC: I will mention for those of you who are just listening over audio, that Kieran in the background has an original Apple Macintosh computer here. And is it a working model?

0:10:35.6 KH: Oh, absolutely. It works. Yeah.

0:10:38.1 SC: So when we talk about the history here, you know, you and I both lived through a lot of it anyway and you know what I'm talking about.

0:10:42.2 KH: Yeah, yeah. No, we're in that sort of, I think, just about in that intermediate generation that is cursed to explain computers both to people older than us and people younger than us.

0:10:54.1 SC: There are worse curses than that. But just so the audience gets in mind what we're talking about here, it's not just rankings that you're concerned with in the book. I mean, it's just the very, it's kind of a classification ability as well.

0:11:08.9 KH: Yeah. Yeah. So, and those two things are very kind of closely connected, right? That there's two processes to, before you can rank things, you must name them first, right? And so there's, so there's a process of classification that, that takes place, which is sort of identifying categories and deciding which things fall into which categories. So the sort of nominalizing thing that this is an instance of that, this other thing is an instance of the second thing. So that's sort of, you know, so as we say in the book, machines classify because people do in the same ways and they rank because people do. But there's always this kind of, there's always a very strong tendency then for nominal classifications, which should be unordered or which are often unordered to turn into rankings and then positioning people within those things.

0:12:07.2 SC: Plato famously said that one of the jobs of philosophy is to carve nature at its joints, right? So we're handing that job over to our computers now in some way.

0:12:17.6 KH: Yeah, I mean, to a large degree. And certainly kind of the... One of the... We're handing it over to them and they're extremely powerful and fast. And the legitimacy that we invest or that these systems often come to have derives in part from this idea that they are in fact, that this is a real, that you're really picking up on something real in the world and that you're doing it with a degree of precision and accuracy that hasn't been possible in the past. And of course, we know, like anybody who's worked with data of any kind in a quantitative form, is gonna be well aware of just how difficult it is to kind of cleanly collect information that truthfully reflects the way things are. And then with social data in particular, there's all kinds of additional complexities about the degree to which you're imposing your framework.

0:13:21.3 KH: When I'm in talks with this stuff, I do have, to go back to the carving nature at the joints thing, I sometimes refer to that idea and I have a little slide from an American cookbook that I have and a French one showing the very different cuts of beef that exist in different, in France and the United States. And so, even butchers carve nature at the joints quite differently. So there's heterogeneity, even if ultimately there's still a cow under there. There are different ways to cut it up.

0:13:53.1 SC: I don't know if you noticed just today, and I think it was in Vox, they had an analysis of the US House of Representatives. And to your point of the data are hard to sometimes get, and therefore you can get better results. But also we have the categories that we testified to, that we claim we're members of, and then we have the categories that are actually borne out by our actions, right? So in this article, they went through all the voting patterns of different members of Congress and grouped them into eight groups on the basis of affinity of that voting. So on the flip side, you can actually reveal categories that were there all along, but we didn't mention them, yeah?

0:14:37.5 KH: Yeah, and that's one of the main sources of the kind of power and legitimacy of these kinds of classifications and methods of data collection and analysis. Because like I say, for the longest time, organizations of all kinds, whether it's businesses or in the longer term, the state, have been very interested in discovering information about either the customers that they have or the citizens that make up or the people whom they govern. But it turns out, for a long time, it was extremely difficult to do this at any kind of reasonable speed. You have a census every 10 years, or maybe you collect sample surveys. Indeed, the idea of statistics, the word stat in there, is closely related to the state. It's information about a population.

0:15:31.8 KH: The promise of these new methods is that with the expansion of all of these methods of data collection, initially kind of on the desks and then ultimately in your pocket of... You're getting a record increasingly of individual and social activity that isn't just dependent to the promises. It isn't just about kind of asking people what it is they think or what it is that they... How it is that they would want to be classified, you also get this essentially, the promise of essentially behavioral data about people, which you want to say is kind of more truthful. And so that leads immediately to this idea that there's a, you know, at the level of the classifiers, a lot of legitimacy associated with, look, I'm capturing what you're actually doing.

0:16:31.7 KH: If I ask you how many steps you walked today, you might tell me one number, maybe you're bad at guessing it, or maybe you prefer to think it was a little higher than it is, but your watch on your wrist knows. And so that behavioral data is both very valuable and it does kind of capture something that's much more accurate in some sense about what people are actually doing, but then it also provides this kind of tremendous legitimacy to the classifications that you develop from it and it introduces the possibility of saying, well, the things that flow from you being in one of these categories are really your own fault or your own virtue, depending on what... If you're well-classified, you experience it as being kind of a virtuous sort of sense of that you're correctly classified. And if you're in a sort of poor... If you're in a poorer category, the idea is that then you're to blame for your own social situation.

0:17:27.0 SC: Well, there's definitely an idea lurking in the background of the book. I mean, maybe you say it very explicitly and I just sort of glossed over, but we have an idea in the modern liberal world that we forge ourselves, right? That we create who we are and we have an image of ourselves and maybe we don't always live up to it, but okay, we're trying, good for us. And what you're driving home is that every corporation, Amazon and Apple and Facebook and whatever, has an image of us. Like they know who we are in maybe a very different way. And it's disconcerting to think both that they have that and that it's not who we think we are.

0:18:07.7 KH: Yeah. And then there's that point where those two things coincide, right? And so yes, like one distinctive feature of modernity, of the existing in the world kind of that we're in now is the idea that people are... That individuals are kind of autonomous agents with their own preferences and rights, who make their own decisions in the world, who are going about sort of choosing to do things and are kind of empowered, active, agentic is the word often that sociologists will use, imbued with this kind of sense of a busy little person going around and making their own choices. And then on the other hand, you have this idea of kind of this sea of recorded information about us yields this sort of set of digital traces that produce this kind of shadow of ourselves, a data double of ourselves, if you like. That's a phrase that Dan Book, as a historian coined, that kind of represents us truthfully in some sense.

0:19:22.0 KH: And that kind of organizations can know about us. And so our sense of agency, our sense of being a kind of active person in the world is very closely bound up with our feelings of authenticity, like of being true to ourselves, this kind of romantic conception of being true to ourselves. But then the measurement and the representations in numbers and records that exist of us is very closely connected to sort of our need to be authenticated formally by... And those two processes coincide. The world we're in, on the one hand, you're enjoined to be really yourself in the authenticity sense, but then also organizations are very concerned to know whether you're really who you say you are. That is, in the sense of it's necessary that you be authenticated as you, as an identity, rather have somebody impersonating you or, and so on.

0:20:23.2 SC: And maybe the scale of this kind of operation makes it hard for people to visualize what's going on. I mean, if I worked at Amazon, I mean, clearly somewhere buried in the bowels of a data center is some list of correlations of all the things I've ever purchased or shopped for or whatever. But could a person working at Amazon sit at a terminal and call up a profile of somebody? It seems impractical. There's a lot of customers out there.

0:20:52.3 KH: There are a lot of customers out there, yeah. Ultimately, they probably could. Somebody can. It is true though, that it's not a question. Most of the time, organizations are not specifically interested in you or me. We're not particularly interesting or important enough, which is one of the reasons, perhaps, that your Amazon recommendations might be weird or governed by other things. One promise of all of this stuff is that, again, speaking to the language of personal authenticity and tailoring, that everything could be personalized to you and that the recommendations that Sean Carroll gets on Amazon would just be perfect. But then it's very common for us to have this experience of saying, oh, I've been an Amazon customer or a customer of some other similar organization for a couple of decades and they recommend these things to me and I don't know why I get them.

0:21:47.1 SC: It's still terrible.

0:21:47.2 KH: But again, this is the actual kind of the demands of doing this practically are quite strong. Organizations know and very strongly feel that they should be collecting this kind of level of granular data about you. And they will boast sort of internally or they will organize themselves. We call this kind of the data imperative. They know that this is something they should be doing. And there's a whole infrastructure, there's a whole set of occupations of people who tend data lakes and who manage data infrastructures about individuals. And as Maciej Czajkowski has said, the whole imagery of that, the metaphorical imagery of it is like a kind of accident waiting to happen.

0:22:34.4 KH: This data lake is dammed up behind a barrier that could crack at any time or be overflowed and get released into the world. So on the one hand, it's very difficult in practice to get that kind of granularity and ease of access to data, about a particular person have it be useful. On the other hand, there are the leading edge of this are the best institutionalized versions of these scores really do exist and are used all the time.

0:23:04.1 KH: The credit score is the most obvious one where you have, in the United States, where you have exactly that kind of a single number that characterizes your behavior in a way that you are sort of morally responsible for, that reflects your actual behavior when it comes to paying your debts or not, and that any shop assistant can call up and decide whether to make you a store credit offer or something like that, and whose initial usage in relatively restricted circumstances has blossomed out into kind of much like the driver's license becomes an effectively a national identity card, a credit score becomes the gateway to having a harder or easier time in areas where it was never really initially designed to be applied at all.

0:23:55.5 SC: Well, let's... Yeah, this says many juicy things to talk about here that we've leapt ahead. But you do open the book with some history and it's fascinating. In the halcyon early days of computers and the internet and so forth, the expectations of where these capacities would go were very different than where they ended up ending up.

0:24:20.9 KH: Yes, very much so. I mean, there's this period beginning... The prehistory of this in the 1960s and '70s when computing is, as we know it, is kind of becoming established and just getting off the ground, is this strange fusion on the one hand of kind of what we think of as the more Dr. Strangelove almost elements of computers come out of the war, of code breaking, of defense systems, and command and control methods for missiles, and all of that kind of stuff. And at the same time, really from the beginning, you also have this kind of like hacker culture amongst engineers who want to tinker and experiment and mess around with. In that sort of familiar kind of scientifically sort of, let's push this and see where it can go, that's quite flat and sort of libertarian in its way, where people just want to be left alone and blue sky research type stuff, just want to be left alone and do their thing.

0:25:22.7 KH: And as computers take their kind of modern form through the hobbyist era of the 1970s into their kind of expansion into business and society at large in the 1980s and 1990s, those two tendencies kind of continue to coexist. And so by the '90s, when the internet and the World Wide Web in particular is developed and becomes widely available as a protocol on the web. There is this kind of, that's the sort of high watermark in a lot of ways of excitement about the kind of pure freedom associated with just setting out on your own, setting up your website, these homestead dreams, we call them, and that language of kind of digital homesteading, Howard Rheingold used that term at the time.

0:26:13.1 KH: People just kind of, this is a place where we can be free of all of the, in effect, all of the world of ranking and of the world of local status and of the suffocating kind of, what John Perry Barlow calls the weary giants of flesh and steel, right? They had this image of cyberspace sort of being a kind of free for all, a new frontier where the dead hand of kind of post-war suburban industrial society would no longer touch you. And you could just be yourself. Again, the romantic image of a homesteader empowered by technology. And that was really kind of the beginning of... Yeah, that's where we started a very different kind of set of associations having to do with what this new networking technology, what these new protocols would enable.

0:27:07.3 SC: Well, and it's very common that people trying to predict what the impact of a new technology is going to be, get it wildly wrong. And maybe it's just because of wishful thinking or whatever. But looking back on examples of this, and with some sociological wisdom in the background, you know, are there systematic ways that people get these futuristic scenarios wrong? Or should we be better at predicting? Is there some equilibrium we're always gonna go to?

0:27:37.3 KH: Yeah, I mean, as the kind of conventionalism goes correctly. Nothing defines an era better than its vision of the future. And there are moments when... This sense of kind of a set of possibilities that existed and then were in some sense closed off or that's not how things turned out. It's extremely common.

0:28:04.1 SC: It's a common story.

0:28:05.8 KH: It is a very common story. And the main thing that happens, I suppose, is people project their own desires about what an ideal world would be onto whatever the sort of social change, often a technological change, but not always, you know, what that seems to enable. And then as it sort of, as it goes on, and things don't quite work out that way, one of the things that can happen is that the initial, especially for the sort of utopian visions of things, there can be a kind of immense disappointment amongst the utopian vanguard with everyone else and their failure.

0:28:44.5 SC: The rest of the world let us down.

0:28:51.2 KH: Yeah, I think that's a very common feature. Sort of revolutionaries end up with kind of a disappointment verging on contempt for the peasants that they have liberated. And one of the things that happened in the development of the World Wide Web was this transition from the kind of homestead era, people sort of said, well, this is great. I love being online. I love talking to my friends. I love being able to be in touch with people who are like me. I didn't know there were often people just like me and lots of them that I can... That really was a kind of liberating feature of these kinds of technologies. However, I don't want to run my own website or I would rather not administer my own servers or could you just... I need to find people more quickly and more effectively, can someone take care of that for me?

0:29:41.9 KH: And there is a sort of tendency to think, and this is something that's out in the world, but it's also kind of a feature of kind of social criticism and theorizing about kind of the internet. There is this tendency to think that the kind of much more suburbanized, centralized, perhaps hierarchical internet that we ended up with was imposed on people very much against their will. It was certainly imposed on some people against their will, the original homesteaders. But a lot of it was very much kind of demand driven where people kind of preferred the convenience of somebody else taking care of these things for them in order to get what they wanted, which was often the kind of sheer sociability, but not all of the associated kind of system administration.

0:30:28.0 KH: And they would prefer people to take care of that. And so it is, it's tempting to think that, oh, we could have had nice things, but then the corporations came along and sort of made this world terrible for us. Now, it's not that there's nothing to that critique, but it is the case that people do want different things. And one of the things people really wanted was convenience and the ability to just get to the kind of fun social part, which in part led to the concentration of infrastructure that we kind of now have with a small number of companies and platforms facilitating just that sort of thing. And there were many cases where companies try to impose their way of doing things on an individual and failed. And we have a lot of failed giants in the first dot-com era and afterwards. So it's not that people are duped, but it's a more complicated process because on the one hand, companies are trying to guess what people want. But on the other hand, people then always tend to kind of overflow or do things with technologies that the people seeking to kind of run them don't expect a lot of the time.

0:31:46.8 SC: Well, clearly, we're at another moment right now with LLMs and artificial intelligence, where there's a new set of utopians coming in to promise us things. And so I'm just trying to figure out, like, how do we avoid making the same mistakes again? Clearly, one feature of humanity is that there will be a bunch of rapacious capitalists, or whatever the version of people who want to accrue power to themselves. And on the other hand, there's going to be a large number of people who will vote for convenience over freedom every time. And so does that help us guess what kind of future AI will bring about?

0:32:28.6 KH: I think that there's a lot of similarities to, I mean, OpenAI is its own, I don't know, OpenAI, the specific company, but the large language models and artificial intelligence generally is its own sort of technology with its own distinctive features. And so things don't happen exactly the same way. But the way that things are rolling out with a lot of this stuff is quite similar to things that have happened before. And so we get sort of, one of the main ways this tends to happen is that this transition from a world of initial technology with seemingly infinite possibilities to one where there's a smaller group where those possibilities seem to narrow and then we're stuck with things that we have difficulties with, is that the... So the initial technology really is amazing and delightful and astonishing to people.

0:33:38.9 KH: And I think, again, this is sort of something that's easy to underplay if you're a cranky social critic who's sort of just sick of that... We start with somebody... Someone our age might think of the first time that they used the World Wide Web or first time that they saw a webpage load. Personally in my case, it was as an undergraduate in my friend Owen's physics lab. He was a master's student and they had a deck alpha running NCSA Mosaic and we downloaded pictures of Mars from the JPL. We didn't have any reason... I didn't have any reason to. I was a social science student. I didn't have any reason to. I didn't need pictures of Mars. But just the fact that you could do it, that there was this, you could talk to this computer in Pasadena and it would serve up these things to you.

0:34:31.7 KH: That was incredible. And then similarly, a decade later, you have a phone in your pocket that can render a map of your current location and show you things around. And you're just holding it. That's amazing. And a decade after that. You take a descendant of that phone out of your pocket and you touch some buttons and you can summon a car to take you wherever you wanna go. And now it's kind of like, oh, you take that same device out of your pocket and you ask it a question and it speaks, or it can generate sort of text. So I don't underestimate at all or discount that degree of kind of delight. Something else will be this... In another 10 years, something else will have that effect on us. Now, how do those things kind of play out?

0:35:18.4 KH: How does that moment become kind of the infrastructure that we end up with, whether it's the web or everybody in the world who can afford a smartphone owning one, or the platformized world of labor for Uber drivers, with all its exploitative dimensions and undertones and entrenched ratings, and now again with kind of artificial intelligence? Well, the first thing that happens is that we get given this for free as a gift, so to speak.

0:35:45.4 SC: The first is free.

0:35:46.4 KH: It's given away to us. And so one of the reasons that it's delightful is that it's sort of given to us as a... And as any sociologist or anthropologist will tell you, gifts set up these expectations of a return. What we give in return is information about ourselves. And we give information about ourselves, our location, or behavior, or who we're interacting with, what we want to know, the questions that we ask, and so on. And it's from there then that these organizations then seek to take that information and make it profitable or take that knowledge and make it profitable. And at the beginning, with the web especially, that first stage with things like Google Search, that was a real revelation. It took a while for people to figure that out.

0:36:33.8 KH: That the digital traces and the logs left behind was actually kind of potentially tremendously valuable. And so with OpenAI now and similar companies, the world of large language models, yeah, the question is kind of... I would expect the same sort of... The thing that tends to keep happening is the same sort of concentration of service provision amongst... That you just get a couple of competitors, really not, maybe not directly competing, but we saw that with smartphones, like with... You have Apple and the Google Android platform, and that's kind of it. And we see it with these other platforms as well. The thing that's distinctive about the world of artificial intelligence, or one of the things that's distinctive about its most widespread use cases is that they're now kind of... Having trained themselves on the free gift of everything that is the World Wide Web and all its content that was available.

0:37:45.0 KH: Now they're in the process of kind of emitting the effluvia of AI-generated output back into that environment. And it's not clear to me what's going to happen. Because again, one of the things that happened with a lot of the sort of the first 20 years of social activity on the web has gradually declined in terms of its kind of public accessibility. And there's still just as much social activity, more than ever really, taking place, broadly speaking, online. But much of it has retreated either to platforms where you can't see, unless you're, or whether it's with things like messaging or content. But down to thing as mundane as having a substack rather than a website, being in a Discord or a Slack rather than in a web forum or blog comments and so on. And so if all that's left in public is the sort of slop of AI output, that might pose problems for this technology in the future.

0:38:48.7 SC: Maybe, I think we skipped ahead a lot to talk about this data collection and its implications just because we all know that our data is being collected. We've all seen Amazon serve up its recommendations. But you've thought about this a lot more carefully than most of us. I mean, what is your overview of the ways in which the data is being collected? Some of them are obvious, but probably some of them are less so to the people who aren't thinking about it all the time.

0:39:16.3 KH: Yeah, there's a couple of different dimensions to this. Like the... What's happening? There's such a volume of data that's available to companies now just because of the gradual expansion of kind of all of these ways for monitoring and tracking individuals. That's often kind of presented just in terms of surveillance, let's say, and that it's just people spying on you. But I think that sort of tends to underestimate the degree to which kind of social life in general as a whole is taking place in these environments where you're being kind of monitored.

0:39:57.9 KH: So it's not quite that kind of, it's not like street cameras, although that's a part of it too, like spying on a real world of social activity that's taking place and then kind of collecting data about it. It's more that the social life itself is now kind of taking place mediated through these technologies. And that's tremendously kind of powerful and a kind of qualitatively different feature of how the world is now. Because there have always been, or for a long time, there have been kind of specific sort of settings where whatever is happening is essentially happening as a flow of numbers or as a flow of data in things like financial markets, for example, stock trading.

0:40:47.9 KH: But usually, for the first 100 years of technology along these lines, those were tremendously specialized, narrow environments. The idea of capturing kind of every conversation or every joke, every sort of interaction seemed both kind of pointless and was impossible. And so that's changed. The breadth of data has really sort of expanded. Then the degree to which kind of people, that's changed a couple of things. And one is that at the level of individuals, it's changed how people kind of think about themselves and their public visibility or their visibility to others. And so there's a whole set of questions along those lines about kind of how we think of ourselves as having an identity online. And again, I think this stratifies quite a lot by age, probably. We know less about this than I would like, actually, that again, there's a kind of naive version of this that says, oh, there's old fuddy-duddies and there's digital natives. But often the digital natives, what makes them kind of native is not their deep understanding of how these technologies work, but more that they're kind of comfortable swimming around in this environment like fish in the sea without thinking too much about water and how it works.

0:42:14.3 KH: So that's one set of issues. And then on the other side, organizations, companies, and states have also been sort of transformed by what they're doing or what this data or what they seek to do with this. And just to pick one kind of classic example, one thing that's happening a lot with the kind of embedding of software and data collection devices in everything, is that it pairs very well with this sort of broader logic of financialization, this idea that kind of what we're interested in doing, what businesses are interested in doing is turning every potential transaction into a stream of income. And that a rent as economists would call it, right? And so to think in practice, what does that mean? Well, in a simpler way of doing things, if you buy something, the transaction, you buy a refrigerator or a car or a tractor and you buy the thing and then you're done, right? Now you own the thing.

0:43:24.4 SC: Those were the days.

0:43:25.2 KH: And maybe, yeah, right. And maybe the first step is to sort of think, well, again, some of these things go back a long way, the company says, well, we could maintain a relationship by selling you a warranty or by giving you a loan to do it. And so again, these ideas are not new in that sense, right? Because what the company is interested in is sort of some ongoing stream of income that it can then turn around to its own shareholders and say, here is the steady, we know that we're gonna be getting this every month for the next five years from the customer. With the arrival of data collection, that now there's like a little computer in your car or your fridge or your tractor, then suddenly this whole range of possibilities gets opened up that connect very nicely with what things like financial markets are interested in. Again, the simplest case is, well, now if we're lending, if we've leased the car to, or we have a loan to you, Sean, and you stop paying your monthly fees, well, maybe we can just kind of remotely turn it off. And so this is not something that is happening right now, but you see Ford and others have filed patents kind of combining a kill switch with data, and you just connect that to the financial records. And you're just like, well, you know, you were, and so you could cut off your car in the same way that your cable service could be cut off, something that we just take for granted if you stop paying every month.

0:45:06.5 KH: The next step up from that would be, well, how good a driver are you? And your car knows much more about that. Now, in the past, to get car insurance, you might get asked a polite series of questions by an insurance agent saying, how much driving do you do? What kind of driving? And then a couple of crude measures of predictors, essentially, of are you over 25? Are you a man or a woman? Which part of the country do you live in? What's the weather like there? That sort of thing. But now, cars are talking to, will talk to their manufacturers all the time. And you may have, if you have a relatively late model car, you may already have been, I don't know, have you ever been scolded by your car for not keeping your hands on the wheel, for example? Or that's the thing that...

0:46:02.1 SC: Not that, but certainly if I don't wear my seatbelt or if I'm coming too close to the car in front of me, which is partly helpful, but a little bit annoying sometimes.

0:46:10.6 KH: Yeah, yeah, and so those things can all become kind of inputs into an individualized price for insurance, in your case. So that's the sort of second level, which itself transforms kind of what insurance is, conventionally, right? Because for insurance to work, you have a pool of people of varying degrees of risk, and then you spread that risk across individuals. But if you have individualized data on people, well, then you can kind of have a different version of an efficient market, where you can price discriminate perfectly, ideally, at the limit. And so then it becomes, it's more like, it's bad insurance. It's more like dental insurance, where, because like health, there's health insurance, or in more civilized countries, there's health, where you have a full displacement of risk across the population. But in sort of American dental insurance, it's not really insurance, you're just prepaying for something that they know you're gonna do. And so car insurance might become like that, which would transform the insurance market. So that's the sort of second level. But then there's like, that's just the beginning with this world of kind of data, because then, I said tractors earlier for a reason, like John Deere has been prepping its shareholders for a while for the idea that like, look, we sell these combine harvesters, we sell this fleet of tractors. Sure, we have a kind of a business with a leasing company and loans and so on.

0:47:37.5 KH: But look, we know now when these farmers are going out and plowing and when they're planting, we know kind of a whole range of things. Well, that means we could become sort of a provider of market intelligence to people, that we could become a sort of, we could take this information and not just use it to sort of serve our direct customers, but bundle it up and sell it to people who might be interested in it, not just advertisers, but people interested in futures markets for various products.

0:48:07.3 KH: And so this tendency to make data collection more and more granular fits really nicely with this tendency for finance to wanna make sort of products that are more and more abstracted, layered and layered up and homogenized so that potentially kind of every manufacturer becomes a software company and every software company becomes a provider of software-as-a-service. And then every service becomes something that can be sliced up and bundled where you can look at tranches or categories or classes of users and then sell their information or sell information either about them directly or sell the information they are generating to interested parties. And this is something we see kind of right across again, everything from your smart fridge, and so in some cases, like in some settings, this seems sort of ridiculous to us now, like the idea of your fridge, for example, knowing what's inside it and first telling you, you need to order more milk, it's been in here for two weeks.

0:49:22.2 KH: Your fridge having moral objections to what you're eating. That seems silly, but with the car market, we see this just perhaps beginning to happen where car manufacturers are like, wow, we could really transform our finance branch, which is in many cases, the most profitable part of the company. And then there's areas where we already take it for granted, like game consoles, for example, where you're signed up, you have a PlayStation or an Xbox and you buy a piece of software, but in order for the software to run at all, it's a multiplayer game, it's talking to servers, it's a subscription service, there are seasons for games, and the company is collecting, the people running all kinds of data about you, some of which is used in a way that you like, some of which is explicitly ranked, like for example, if you're playing some multiplayer game, they all run some ELO-like ranking system to make sure that you're matched against people who are kind of competitive with you, but who won't destroy you in games, or who won't be too easy for you to defeat. So in that sense, the rankings are just super useful for you as a way to enjoy your service, but then also provide a global view of the whole system about who plays the most, what kind of people, and so on. So this is already here for certain kinds of products, and then it's continually expanding for many others.

0:50:54.7 SC: Is it true that if I have a late model refrigerator that it will know what's inside, and will it send that info to the refrigerator manufacturer?

0:51:04.0 KH: No, but, or at least, there are smart fridges, I need to get back to, I need to look more carefully like at specific cases, but yeah, Samsung and others have started to, you can buy fridges that have a little camera in them and try to identify and help you with your shopping by kind of paying attention to what you're buying and deciding what it is that you need. One interesting question about those two is the extent to which behind the scenes, and I haven't seen anything specific about this particular case, but the thing that occurs to me right away is if you have a fridge that has kind of a monitoring capability, perhaps through a camera, is this fully sort of machine learning or is there somebody in India or the Philippines who's looking inside your fridge and helping you? As we saw recently with Amazon, right? That with their Just Walk Out stores and so on, where they shut that down, but initially it was all this hype about it being AI, but then it just turned out to be a bunch of people. Poorly paid overseas who were kind of tagging everybody to make sure, so that's the kind of thing that happens.

0:52:09.7 SC: I do remember in your book, you mentioned this fact that I had seen before. The General Motors is mostly a bank now. They make more money out of their auto loans than they do off their autos.

0:52:19.8 KH: Yeah, right, exactly. The finance division of, again, the financialization of products generally extends to companies that we think of as manufacturing hardware of consumer goods, but really where the profit lies is in their financing divisions. And that's true. Take, I'm sitting in front of an Apple computer. Apple is probably the last, you know, it's the last kind of major Valley company, the Silicon Valley company surviving from the '70s that primarily is about making its own hardware.

0:52:56.0 SC: Hardware.

0:52:56.3 KH: That it makes a software, but it sells, makes most of its money from selling hardware. But it too, over the last decade in particular, has both increasingly, has been getting most of its growth from the rise of services of various kinds, subscriptions and deals with Google and others. And then also has been expanding into an Apple credit card, and the general expansion into kind of, you know, paying for anything, companies want to get a little piece of that one way or another, or whether they're, if they don't manage it directly, they want to get a cut for providing the customer. But that really is where a lot of the long-term sort of stable profit, a stream of income, that you can differentiate by categories of consumer in terms of how much money you can make from them. That's where the profit is.

0:53:53.5 SC: I know you're not mostly a self-help book here, but is it of any use at all to turn off cookies and, you know, not let Google keep my search history and things like that, or is that just a little window dressing?

0:54:11.2 KH: Well, I think that it's important for people to think about these things. We're, in the book, mostly concerned with kind of how it is that, in a big picture way, how sort of the phenomenon of social order generally is being kind of created and the rise of kind of categories of people are being kind of maintained across institutions. One feature of that is that the tendency to think about how people think about these problems at all is also relentlessly individualizing. And that it comes down to questions. People naturally ask these questions, well, what can I do to protect my own data, to make sure to opt out of these systems, to make sure that things remain private for me and so on. Those are all very reasonable questions. But to the extent to which that becomes the terrain on which most debate about this is happening, well, then you're kind of losing sight of the broader kind of institutional phenomena. So that like, and this comes out in kind of policy debates in various ways, like in the EU, for example, one of the main, the main kind of, with the GDPR regulation a few years ago, their idea was exactly let's empower individuals to have the choice to accept or reject cookies or tracking in the websites that they visit.

0:55:49.4 KH: And so what we're doing is putting in the hands of individuals this ability to make those, exactly those kinds of decisions and to think of their own kind of internet hygiene or search hygiene in that way. And similarly, if you live in California, you'll see, even if you don't, you'll see the Do Not Sell My Information button on many websites. Now, what that does of course in practice is it leads to people just automatically clicking accept all and just the fact that...

0:56:22.4 SC: Whatever is quickest.

0:56:22.5 KH: The choice is, yeah, the choice is constantly kind of given to you or they install, we just have the kind of, okay, I have to click the button here and then they click it. So, yeah, at that level, that's the sort of wrong way to think about the broader questions. And then the other thing that happens is that people who are really serious about avoiding all of this stuff can try effectively to eliminate all of this stuff from their lives. And I know people who do this and it's not easy. And one of the consequences of it is that you're in danger of kind of exiling yourself from your own society and your own culture, which is a price some people are happy to pay, right?

0:57:12.1 SC: Some people.

0:57:12.7 KH: Because they have nothing but contempt for it. It's a thing that you can do, but it does, but it's not kind of without its costs. So trying to become invisible, the logic of all of these systems, this goes back to what I was saying earlier about authenticity and authentication. The logic of all of these systems is to incorporate, in that sense they're democratic. In that sense they're expansive and inclusive. It's not a world where we're saying these people who are not worth paying attention to and we just ignore them, we deny their existence as social beings, it's not that kind of classification system where you just ignore their existence. Instead, the idea is to incorporate and then stratify, and to incorporate you must measure and track.

0:58:00.2 KH: And then you can sort of just re-rank and properly classify the individuals once they're in, and then so to be outside of that sort of system then is to be at a double disadvantage because you're not even kind of classifiable and you end up kind of... It'd be like, there's an irony here. It's like trying to pay everything, trying to pay for cash, trying to use cash for everything is increasingly difficult because you're not incorporated into the banking system, and for many years, for much of the 20th century, one of the big policy problems was this question of exclusion because people were un-banked and such people still exist, but in the United States in particular, but there's this massive expansion of the banking system beginning the 1970s that incorporates...

0:58:53.8 KH: And that was driven by very laudable kind of ideas about being able to let people into this system that they had previously been excluded from, but what happened, the system that replaced it was one where banks essentially figured out how to make money from poor people through things like late fees and overdraft fees, and so this is a real tension that you can walk away perhaps, if you have the means, but even that is increasingly difficult because of the being invisible, and in that way has all kinds of consequences that many people don't want to bear.

0:59:33.8 SC: Well, I do wanna talk about the sort of implications of all this for our social orders, and that is after all what the book is about, in some sense, if I'm now classified by thousands of different companies in all of these invisible to me, classification schemes why do I care? Why does that affect my life?

0:59:55.8 KH: It affects your life chances, as sociologists would say, it affects the opportunities that are offered to you and what those things will cost, so to speak. And so, yeah, so one feature of this whole world is, as you're saying that it is very... It's very highly differentiated, like what it means, your stock, so to speak, of, we call it "I Gain" capital in the book, this digital representation of you through data. What it means varies according to the market that we're talking about, or the setting that we're talking about. So it's not that, like, we're not sort of arguing in the book that everybody has just reduced to a number, a single number, and that this determines your entire life, it's more that the principle of the basic logic of social order and the creation of social structure is increasingly mediated by these and carried out through these processes, and so the reason that you care then as a result is that the kind of person socially that you are is very closely tied to how you are classified by these institutions, and for some things you may reasonably say I don't care, in the sense of if this or that company classes me as a good or bad customer or something like that, but for other institutions where these scores are also...

1:01:26.3 KH: And these methods are also increasingly being used, in healthcare, in the law, the legal system, and education.

1:01:32.5 SC: Hiring.

1:01:33.0 KH: And in hiring practices and so on, you may very much care, and people increasingly have come to accustom themselves both with varying degrees of voluntary assent to being subject to this kind of data collection and classification as a condition of entry or membership in society, broadly speaking, but in specific cases. And so then that range, so the range of reasons to care can start directly from how much will this cost me right now, will I be charged an additional fee because I've failed a credit check or failed to meet a threshold in some check to get a phone that solves, to internet service, to sign a rental agreement and so on, that kind of thing really does matter for a kind of stratification and where people end up in life, all the way through to sort of, what if I feel like I mis-classified or how can I... If an institution is seeing me in a way that I strongly disagree with, what resources are at my disposal to fight that? And is it just a matter of the only option is to exit the system and thus pay the price of that, or are there ways that I can...

1:03:01.1 KH: Are there ways that I can change my classification, and those things are... Those are hard questions because the very kind of basis or logic of a lot of this stuff is nominally, even if the systems don't really work this way, even if they're laden with error or they're badly implemented as statistical measures or that they reflect bias in all kinds of terrible ways, the kind of cultural logic of them is very much that you're getting what you deserve because it's your behavior, your decisions, it's not just... Those are the things... What happens is that those get parsed as choices that you made, as decisions that you took, and so in one sense of all of the things that sociologists and social scientists generally conventionally think of as social structure, where you came, from what your opportunities were growing up, like what is constraining you in the world, people's opinions of you and so on.

1:04:03.0 KH: Those all tend to get stuffed through if you like, the behavioral channel, they get recorded as choices, and then you get judged on the basis of those choices you apparently made, even if at the time, you may have felt, well, I didn't really have an option here, I had to do this, or this was a constraint that I faced that really wasn't of my own doing or on the flip side, the same thing applies in reverse to people who benefit from these things, like the idea of being born on third base thinking you had a triple, you take upon yourself all of those virtues, you think about all those advantages become sort of experienced by you as personal virtues of similar choices that you made beginning with your excellent choice of parents.

1:04:47.9 SC: Just what you deserve. Yeah, and you emphasize in the book how even if you thought of all of these classification and ranking systems as purely objective and quantified, it bleeds over into normative questions, you do get judged as better or worse, like it or not.

1:05:06.0 KH: Yeah, definitely, yeah. And even if they get... Even if you think of them as classification schemes that are not intrinsically, that are not trying to rank you, so to speak, that you're just trying to classify, there are very few cases of nominal classifications, unordered classifications that people don't try to then turn into rankings. In part because... You might ask why is it that that happens? Again, where there's differentiation, there's stratification in human societies, but then also people find these like... It's a two-sided process, if you're being judged by these systems, if you're been classified by them, that can be a very unpleasant experience if you're on the sharp end of them, it can be very gratifying if you're well classified.

1:05:56.2 KH: But on the other side, if you're looking to make a decision, if you're trying to organize something, these technologies are just tremendously powerful because they just are heuristics, they simplify decision-making immensely, and they make it possible to do things, and so people demand in that sense, the ability to make these kinds of decisions, to rely on these rankings or to rely on these scores because they just cut through, they really are extremely powerful methods. In that sense, it's just that then we would also prefer not to be subject to them ourselves a lot of the time, and a lot of the social struggle that goes on around these things is exactly who is it that's predominantly taking advantage of these measures and then who is that gets to avoid being subject to them at key moments in their lives.

1:06:46.3 SC: In a recent solo podcast, I sort of off-handedly speculated about how a lot of the modern condition was affected by the fact that because of the connectivity of the world, we're connected everywhere. So the structures we're dealing with are very, very big, it's not like a local coffee shop, it's an international chain, and they have all this data that they can turn into action very, very quickly, the efficiency of extraction of our wealth, etcetera, becomes super high. So I speculated that this just makes us sad because we can't ever feel like we're getting a good deal, we're paying as much as we would possibly be willing to pay for everything we do, and we can't get any human response from the systems we're stuck in. This is not even a question, but is that kind of what's going on?

1:07:41.0 KH: I think that's an excellent point. I think it comes out in different ways, one is insofar as these systems do reach their limit of efficiency in that sense, the result economically, is exactly what you described, which is in its way a perfectly efficient market, just not the kind that... It's not the traditional kind where supply and demand grope their way towards a kind of balance and there's a single price that clears the market, it's a perfectly price discriminated market where everybody pays exactly what they're willing to pay, and so that means then yet that sense of kind of, Hey, I got... What the economists would call consumer surplus, the sense that, "Hey, I got a good deal here, 'cause I really would have paid more than this if I had to, for this thing" can evaporate. And yeah, that's really something that... That feeling is quite real, and it's also... It comes out in a more social way too. I think another way it makes you sad is that freedom in the book we call this interstitial liberty, that there's a kind of freedom that comes from the institutions that organize our lives being so relatively poorly connected to one another, that they're not really able to transfer information efficiently or to communicate with each other, that they get stuck. In the older times, there's a file over here that needs to...

1:09:12.9 KH: That's out of data that needs to be sent in the post or the bureaucracies mesh poorly. And so the freedom that comes with that, that kind of freedom, this interstitial liberty that kind of bubbles up out of the cracks between organizations was the freedom to move somewhere else and not have anybody know who you were or be able to find out. To start your life somewhere with a clean slate or something, or just to move through society without the sense of that there's the possibility that everything about you, relevantly, could be known easily, and as these systems have expanded, the benefits that you get supposedly, are the ones that have to do with kind of an experience tailored to you personally and you think, Oh, that would be nice. But that's a bit like being asked, if I could go back and live at any time in history, what would I pick? People think of themselves, I would be the king, right? Of course, it would be great. So maybe what's tailored to you, what the system thinks you deserve may not be what you think you deserve, and so you gotta sort of benefit, but then you also lose this freedom that came with the friction that previously existed between institutions. Yeah, you have more opportunities in that sense, because there was so much more of a... There were gaps, there were cracks that you could live in, in a way that's increasingly difficult now.

1:10:45.1 SC: So sadly, we're winding up on a relatively downbeat note here, I'm wondering, projecting...

[laughter]

1:10:50.7 KH: What's that joke that, you should end on a positive note... I don't have a positive note, would you take two negative notes.

1:11:01.4 SC: But can we project into the future? Everything that you're describing, seems to me to be... Everything, these things are definitely happening, but I can imagine them happening even way more, so I'm guessing that none of this is gonna go away.

1:11:19.1 KH: Yeah, I would say... It is a little pessimistic in that sense, I would say though, one of the main commitments in the book, or one of the feelings that we have in the book, and that we think that's kind of come out empirically over and over again, is that people... Just because the system is pervasive, just because of the way that life is organized is everywhere now, it doesn't mean it's totalizing in the sense that it completely dominates and fully dictates every aspect of everyone's existence. So a thing can be, like the world we're describing we think is real in the sense that it really is kind of expanding in its scope and scale in the way that we describe, but it's also true that any human system, any sort of set of social institutions is intrinsically... People tend to overflow the boundaries of the systems built to enclose them, or that we build to enclose ourselves, and because social life is messy, things happen at random, there's noise in the system, things break.

1:12:40.5 KH: So there's always this possibility, it's inevitable really, that things just don't go according to plan, things sort of spin out, and when they do... It isn't always a bad thing. And the very things that kind of helped create this whole system were exactly that, it was sort of people creatively over-using early web form, early web discussion forums suddenly become kind of communities where people discovered each other and sometimes those communities were wonderful, we might judge normatively and others, it turns out that, Oh, look, all the white supremacists can find the exit, can meet up as well, so it really is a truly messy process in the sense that it's not kind of... There isn't a nice moral story about how everybody is wonderful deep down or anything like that, but it is the case that these systems, they don't exist forever. And so there's both good old solid kind of policy, we can kind of architect these a little bit in ways that can push them in one direction or other, but there's also just the sheer fact of human sociality and the randomness, intrinsic and messiness of human social existence that tends to overflow whatever boundaries get put on it sooner or later, what happens after that, what's next is, I don't believe in the idea that it's necessarily better, but it's not inevitable that we're stuck, we're never stuck forever in a particular way of organizing things, new things come along.

1:14:09.4 SC: I guess that's a good, slightly positive. Good, thank you for at least trying there. I appreciate the effort. Look, I will note that I've noticed that on YouTube, the ads that I get served up are just terrible, they're just like tawdry, no relationship to me, and I clicked like trying to ban an ad and it said you've turned off your search history so we can't target your ads.

1:14:29.8 KH: That's right.

1:14:30.6 SC: And I'm like, Yeah, well...

1:14:32.7 KH: That's exactly the kind of... That's one of the ways that... There's a very good example of the price you pay for not wanting to be incorporated because... And the price you pay for not wanting to be incorporated is that you get the worst, they're like, Okay, we'll have to serve you up the lowest common denominator.

1:14:53.8 SC: The lowest common denominator.

1:14:56.4 KH: So it's like, I'm not on Twitter anymore, but any time I go back there now and it's like, oh, it's just like lyricking, these weirdos who are... And it's the same sort of thing it's like, well, I turned off all of my information, you don't give. This is, we will call the mostly bargained, you don't give to the system, and so it doesn't give back to you, and so you have to, then to the extent that you're still watching YouTube, you have to suffer through these terrible ads because you didn't give back in the way that it wanted you to.

1:15:26.5 SC: It's a first world problem, but it's a problem that I care about. So that's what I have to deal with. Kieran Healy, thanks so much for being on the Mindscape Podcast.

1:15:32.0 KH: Thank you, Sean.

[music]