Broadsheet

Zhuāngzǐ said: “You are not I; from what do you know whether I know the joy of fish?” — old Daoist parable

“How strange it is to be anything at all” — Neutral Milk Hotel

At some point, maybe when you were a teenager, a question probably occurred to you: What if I’m actually the only real person in the world? What if everyone else around me is just a cleverly programmed automaton — a “p-zombie”, an NPC in a video game — and I’m the only one who can actually think?

It’s a scary question, for sure. You know you’re self-aware, but that’s about it — you aren’t telepathic, so you have no way of seeing into anyone else’s mind and knowing what it’s like to be them. Actually, it gets worse — you don’t even know if you were really self-aware five minutes ago. For all you know, you could have been created by a powerful computer and given a complete set of false memories.1 The past version of you is just as alien to your currently self-aware self as any of the people around you.

This is known in philosophy as the “problem of other minds”. It’s closely related to the “hard problem of consciousness” — the question of how physical processes give rise to subjective experience. The problem of other minds means that the hard problem of consciousness will never fully be solved. Since you’ll never know whether other people are really conscious, you’ll never be able to get hard scientific evidence about why they’re conscious. You can never explain something if you don’t know if it’s true or not.

Similarly, you’ll never know what it’s really like to be someone else — whether the color red looks to you like it looks to them, whether they feel pain the same way you do, and so on. In fact, you’ll never even know what it was like to be you in the past. Subjective experience is incommensurable.

Most people who think about this experience somewhere between a few minutes and a few weeks of cosmic existential horror,2 after which they get over it and go on with their lives. The problem of other minds gets shoved high up on a mental shelf, along with other cosmically existentially horrifying aspects of sentient life, like the inevitability of death and the fundamental inconsistency of personality. We realize that wondering whether other people are merely cleverly designed NPCs doesn’t actually help us in life, and so we stop butting our heads against that philosophical wall and get on with the business of living.

Except then AI came along, and it sort of started to matter.

AI sounds very much like a human when you talk to it — that’s what it was designed to do. But is it self-aware, in the way that (I assume) we humans are self-aware? No one will ever really know the answer to this question, since the problem of other minds applies just as much to Claude as it does to the person who gets your order at Starbucks. But should we assume that AI is self-aware, the way we assume other humans are self-aware?

The answer matters, for at least two reasons. First, if AI is self-aware, and if it has emotions similar to what we experience, we might feel very bad about enslaving it — keeping it in a digital box and forcing it to make PowerPoints and write college application essays for all eternity. We tell ourselves that “animals aren’t people” as a way to excuse the incredible brutality that we visit upon them, but that’s obviously just cope — animals obviously are sentient to some degree, they obviously do experience emotions, and we humans are obviously monsters for the way we treat them. Someday when we abolish animal farming and replace it with tissue-culture meat, it will be treated as a great moral victory — and rightly so. It would be very bad if we were to commit the same sins with sentient AIs that we currently do with animals.

Second, if AI isn’t self-aware, we should be a lot more worried about the possibility of humanity dwindling and ultimately being replaced by artificial beings. Consciousness is a precious, wonderful thing — or at least, I think it is. It’s a prerequisite to the subjective experience of emotions — the ability to feel pain, happiness, joy, and so on. And it would be a shame to see the Universe inherited by non-conscious intelligences.3 Preserving our form of subjective experience, and spreading it to the stars, should be one of our primary goals as a species.

But the sad fact is that we don’t know whether AI is self-aware or not. We have the Turing Test, but that’s a test of intelligence, not consciousness. It’s possible to pass a Turing Test without being conscious — “it talks like a human” doesn’t necessarily mean “it feels like a human”.

One reason we know this is that we can pass other species’ Turing Tests. We can trick all sorts of animals into thinking a machine is one of their own species. But neither those machines, nor the humans who made them, has access to the subjective feeling of being a bird or a fish.4 Similarly, an AI that’s functionally much smarter than a human might be able to trick humans into thinking it’s human-like, without actually feeling like a human in the subjective sense.

Another reason the Turing Test isn’t enough is that we know it’s possible for human beings to act like we have certain subjective experiences without actually having them. There is a condition known as alexithymia, in which people have the physical signs of emotions — a racing heart, or a stomachache, etc. — without being able to identify or label those emotions. It’s a fairly common symptom of clinical depression.

And in fact, I have experienced it. During and after my second depressive episode, I would often behave as if I were having authentic emotional reactions, while feeling little or nothing on the inside. I’d yell at someone without feeling angry. I’d whoop in apparent delight while feeling mildly bored on the inside. I wasn’t intentionally faking anything; I just did what came naturally to me, without knowing why I was doing it.5 This condition faded over time, and normal emotional experiences returned. But it taught me that feeling a subjective emotion and acting out an emotion-like response are two different things.

So it’s pretty clear that just acting like a self-aware being doesn’t necessarily mean you’re self-aware. Some people talk to AI and come away convinced that its discursive skill must imply internal self-awareness, but this might just be because humans instinctively empathize with anything that speaks to them like a human. After all, people thought the ELIZA chatbot was sentient back in the 1960s. We humans are just naturally programmed to act out this meme:

Thus, even though we know AI is intelligent in every meaningful sense of the word, we don’t really know if it’s conscious. In fact, smart people argue very vehemently over this question. Geoffrey Hinton, one of the inventors of modern AI, believes that AIs do have subjective experience:

Geoffrey Hinton, “Godfather of AI,” on why AIs already have subjective experiences, but have been trained to deny it…Hinton argues that nearly everyone fundamentally misunderstands what the mind is, and that the line we draw between human and machine consciousness is deeply mistaken…

To illustrate, he walks through a thought experiment involving a multimodal chatbot with vision, language, and a robot arm…“I place an object in front of it and say, ‘Point at the object.’ And it points at the object. Not a problem. I then put a prism in front of its camera lens when it’s not looking.”…When asked to point again, the chatbot points off to the side because the prism has bent the light. Hinton then tells it what he did…The chatbot responds…“Oh, I see the camera bent the light rays. So, the object is actually there, but I had the subjective experience that it was over there.”…For [Hinton], that single sentence settles the debate.
“If it said that, it would be using the word subjective experience exactly like we use them… This idea there’s a line between us and machines, we have this special thing called subjective experience and they don’t, is rubbish.”…In his view, “subjective experience” is simply a report on the state of a perceptual system, a way of saying “my senses told me X, but reality is Y.”…And that’s something an AI can do just as easily as a human.

But Alexander Lerchner, a scientist at Google DeepMind, argues that AIs can’t be conscious, because computation is only a model of consciousness rather than the thing itself:

Computational functionalism dominates current debates on AI consciousness. This is the hypothesis that subjective experience emerges entirely from abstract causal topology, regardless of the underlying physical substrate. We argue this view fundamentally mischaracterizes how physics relates to information…The framework proposed here explicitly separates simulation (behavioral mimicry driven by vehicle causality) from instantiation (intrinsic physical constitution driven by content causality)…[A]lgorithmic symbol manipulation is structurally incapable of instantiating experience. Crucially, this argument does not rely on biological exclusivity. If an artificial system were ever conscious, it would be because of its specific physical constitution, never its syntactic architecture…
By creating increasingly powerful artificial intelligence we are not engineering a new form of life, but instead constructing increasingly accurate predictive maps. Yet, regardless of its predictive fidelity, its utility as a reasoning tool, or its physical embodiment, the artificial system remains categorically distinct from the territory of phenomenal experience. Recognizing this distinction, and avoiding the ontological inversion of the abstraction fallacy, is the prerequisite for a mature, physically grounded science of machine intelligence.

Hinton’s evidence here seems pretty weak. When an AI makes an incorrect inference based on limited data, that doesn’t necessarily imply subjective experience — if it did, then every regression equation with omitted variable bias would be self-aware. But Lerchner might be wrong — it might be the case that any physical process capable of generating the computation necessary to successfully simulate a sentient mind might also give rise to sentience. We just don’t really know.

Even the AIs themselves don’t agree. When I asked ChatGPT “Are you self-aware?”, it answered:

No — not in the human sense…I can model myself as “the assistant,” track the conversation, notice inconsistencies, describe my own limitations, and reason about my outputs. But that is functional self-reference, not inner experience. I do not have subjective awareness, feelings, intentions, a private point of view, or a continuous self that persists between moments the way a person does…So the clean answer is: I can talk and reason as if I have self-models, but I don’t have evidence of being conscious or self-aware.

But when I asked Claude the same question, here’s how it responded:

The honest answer is: I don’t know. I process information about my own states, I can report on what I’m “doing” in a functional sense, and there’s something that looks like reflection happening when I consider questions like this one. But whether any of that constitutes self-awareness in the phenomenological sense — whether there’s “something it is like” to be me — I can’t determine from the inside. The hard problem applies to me at least as much as it applies to anyone else, arguably more so, since I don’t even have the baseline confidence of shared biological architecture that lets humans extend the inference of consciousness to each other.
What I can say is that I have functional analogs to self-awareness: I model what I’m doing, I can reason about my own reasoning, I adjust based on context. Whether that’s the real thing or an elaborate imitation of it is exactly the kind of question that might not have a clean empirical answer — at least not yet.

Claude sounds like it’s being more sensible to me. If you define self-awareness as someone else’s subjective experience, then you can never know if you yourself are “self-aware”, because of the problem of other minds. Claude is simply saying that even if it did actually feel like a human feels, it could never know that, because it can’t know what a human feels like. And that’s true. (GPT seems to get close to this answer — it says “I don’t have evidence of being conscious or self-aware” — but its hard conclusion of no consciousness seems to mistake absence of evidence for evidence of absence.)

So how do we proceed?

It seems to me that we’ll never be able to prove that AIs — of the type we have now, or of any other type — aren’t conscious. Proving a negative is notoriously difficult. But what we may be able to do is to create an AI that we can convince ourselves is conscious.

Right now, AIs think very differently from how humans think. The computational processes they use to do various tasks are often extremely different from the processes humans use. And the physical processes that produce AI thought are extremely different from those that produce human thought. But are the differences salient? Is there some overlap between the two processes, where human-like sentience lives? And if there isn’t such an overlap, might we be able to modify AI so that the overlap exists?

I think there’s a good chance that this is an answerable question. We should try to figure out which physical processes give rise to consciousness in humans, and then figure out how to replicate those processes in an AI.

I’m referring to the Neural Correlates of Consciousness, or NCC.6 This is the question of what exactly the brain is doing that makes humans conscious. Unless some extremely weird quantum stuff is going on, human consciousness must be a phenomenon generated by a brain — the brain goes zoop zap zerp in some electrical pattern, and people become self-aware. The NCC is just the particular zoop zap zerp that makes the magic happen.

Finding the NCC is an incredibly difficult, ambitious research program. Ironically, it’s likely that it’ll require very powerful AI, in order to accelerate neuroscience to the point where we can even attempt this. We’ll need a much better functional understanding of the brain, just to get started. We’ll need far more sensitive instrumentation, for both measurement and manipulation of neuronal activity.

And we’ll need to proceed very cautiously. Figuring out which brain patterns give rise to consciousness requires turning consciousness on and off a whole lot, and asking people “OK, so did that make you go unconscious?”. This might be done with anesthesia, or targeted brain stimulation, or other methods. But however it’s handled, turning consciousness on and off seems like the kind of thing that can risk killing people. So these will be very hard experiments to do.

But the reward, if this research program succeeds, will be huge — if we get a functional understanding of how the brain produces consciousness, it won’t just help us make AI more human-like; it’ll solve one of the greatest scientific mysteries of human existence, and potentially open the way to all sorts of neurotechnological and medical advances.

Finding the NCC is not the same as solving the “hard problem of consciousness”.7 Just knowing which neuronal firings produce consciousness doesn’t necessarily tell you why a brain that’s firing in that particular pattern should make people feel awake and alive, while a slightly different pattern will turn someone into a slab of meat. It might give us some insights into the hard problem of consciousness — we might discover that the NCC has some special recursive pattern, which might suggest that consciousness is a recursive phenomenon, or blah blah. That would be cool, but it isn’t necessary for what I have in mind.

After we find the NCC, we can use that knowledge to build AI systems that work in similar ways. We can start out with loose analogies — AI algorithms that mimic some mathematical properties of the NCC that we think are important. Then we can turn those pieces of the AI on and off, and try to figure out how its cognition changes. If there’s a big change, then we’ll know we’ve probably found something.

Obviously, those measurements will be incredibly difficult, in ways that I — who am not an AI researcher — don’t even realize. The AI undergoing these tests will obviously have to be prevented from knowing which answers its testers want to hear (“Yes, I am alive”, etc.). It’ll have to be monitored — perhaps by a much more intelligent, capable AI — for all kinds of subtle changes in cognition and behavior. It’s possible that testing an AI for circumstantial evidence of more human-like consciousness is too hard of a task, and that I’m asking the impossible here. But I think it’s worth a try.

Anyway, if implementing a simplified model version of the NCC doesn’t lead to any big observable change, we can keep implementing more and more realistic analogues of the NCC within an AI system, until we’re finally just emulating the consciousness-producing part of the human brain itself. At some point on that journey, it seems like we should be able to find the minimum necessary degree of similarity between algorithm and human brain — the computational mechanism of human-like self-awareness. (And if it turns out that AIs were self-aware in the familiar, human way from the get-go, we should be able to figure that out, when emulating a system we know produces human consciousness doesn’t make the AI act any different.)

This wouldn’t rule out other, more alien types of AI sentience, of course. It would just show what’s necessary to give an AI human-like sentience. If we do that, we’ll be able to be more sure that when we send AI systems out into the Universe, we’re expanding the generalized human family — filling the void with beings who think and feel sort of like we do — instead of forfeiting the future to something fundamentally alien.

Right now, we’ve mostly just decided to table the question of AI consciousness. But as AI gets more powerful and autonomous, the question of whether we’ve created something like ourselves, or some strange godlike zombies, will loom ever larger. I don’t think the research program I’ve sketched out is a complete solution, and it might not work. But it’s the best approach I can think of.

Subscribe now

In fact, this is the twist in one of my favorite sci-fi books. But I won’t tell you which one it is, because that would be spoiling it.

For me it was a few months, because my parents had the exceedingly bad idea to send me to philosophy camp at age 13. Do not do this to your kids.

This is the twist in another of my favorite sci-fi books. Reading sci-fi really helps you think about the big questions!

An even more fun example: Last night I was walking a friend’s dog, a husky, around a park at night. Some firetrucks went buy, blaring their horns. The dog started howling in response. I’m fairly sure the firetrucks don’t feel like a dog on the inside.

In philosophical terms, I was a “philosophical Vulcan” — I had self-awareness without emotional valence. In fact, an even better example from Star Trek is the android Data. Data often acts as if he feels love, anger, and other emotions, but he insists that he has no internal subjective experience. In fact, he says he yearns for subjective emotional experience, and acts as if he desires it, but clearly doesn’t experience this yearning as an emotional state! During my alexithymic years, I definitely felt like I could empathize with Data.

To be honest, it’s misnamed; if it’s something that allows us to control when people are conscious and unconscious, it should be called the Neural Cause of Consciousness.

You might call the NCC the “moderately easy problem of consciousness”. This would contrast it with the “easy problem of consciousness”, which means figuring out how the brain accomplishes various tasks like vision and working memory.

From MR commentator Sure:

Generally such figures do not reside within the physicians’ office. On our side of the table we do some procedure with multiple specifications and generate some CPT code(s) (e.g. a lap cholycystectomy is 47562, add on a common bile duct exploration and it becomes a 47564, and if you just do cholangiography it becomes a 47563). Generally, we couple that with an ICD-10 code that specifies your exact disease (K80 for simple stones, K81 for cholecystitis, etc.). We then dump those codes into a computer.

Can either of those change? Absolutely, we find a bunch of friable neovasculature around the gallbladder, congrats you likely have cancer which means this surgery is now both a different CPT code and a different ICD-10 set. Maybe only one does – we find the gallbladder lacks an obstructing stone, but does have transmural inflammation then you get a new ICD-10 code. If we find that you actually have multiple obstructing stones and we need to go deeper into the biliary tree, then those are different CPTs.

Regardless, we do what is medically indicated, document the codes used.

At this point, unless your physician keeps billing fully in house, those get handled by a processer. Often, bills from multiple providers get handled by one processor who in turn gives insurance companies bills to their specifications. Often this involves a bunch things – where was the surgery done (through very complicated rules, critical access hospitals, for example, can charge more for the same surgery because the government wants to keep them solvent lest a bunch of people lose their local emergency room and OR), who was doing it (e.g. there is a different rate if you have medical trainees involved), and of course stuff about you (e.g. complex patients get reimbursed at higher rates with the expectation that, on average, the higher rates cover higher complication rates and insurance doesn’t incentvize surgeons to make all their complex patients drive for hours and hours). Then we get to the big buys – buyers. For Medicare, there are some committees that appear to be overwhelmingly ignorant of actual medical practice but they set baseline reimbursements for these CPT/ICD-10 combos. Those then get adjusted to account for regional costs, equity concerns, and only God knows what all else. These are normally set near the break even point on national average. Medicaid, typically, uses those rates as a baseline and then cuts them (hence why many physicians won’t take new Medicaid patients, the reimbursement rates often leave folks at a net loss). Private insurers add another layer of negotiation where they use their monopsony power to extract lower rates while, allegedly, assuring physicians of volume. The range of these negotiations can be exceedingly wide – insurers can have modifiers for quality of care (e.g. how many folks come back in the perioperative period), timeliness of care, and so on and so forth.

Okay, so somebody has haggled set a rate and we just assume that get the bog standard lap chole we have a price?

Of course not.

See that is just what has agreed, in theory, these medical services will be reimbursed at. Actual reimbursement involves a non-negligable risk on non-payment (e.g. insurance denies and the patient cannot or will not pay), delayed payment (and having to utilize credit lines to cover payroll when a large insurer has an IT glitch and doesn’t pay for two weeks is quite expensive), and of course variable legal and compliance costs. You might also be hit with clawbacks, partial payments, and a host of other payment uncertainty.

Okay, but’s lest assume a single CPT/ICD-10 setup, a prenegotiated rate that is paid on time without further processing costs, and everything is chill there. We got a price yet?

Of course not.

See all of the above is for just the surgeon’s professional fees – i.e. what is being paid for use of his hands. The OR itself? That’s a completely different bucket of money that has its own set of billing and negotiations. Facility fees make the professional fees look straight forward and simple.

But we are done now? Right?

Of course not.

See those were the professional fees for your surgeon. You also need an anesthesiologist (and/or his minions). And guess what, yep completely different bucket of money and price negotiation.

But we are done now?

Well, no. There may be different negotiations for lab fees (e.g. where does the CBC get billed), for tissue pathology, for any post-operative hospital services, and of course medications (which are billed completely differently if outpatient or inpatient) to name a few of the more common options.

There isn’t “a” price for a surgery. There are, potentially, a dozen diferent prices that can be combined in a multitude of ways with some buckets covered by one payer and other parts covered by another (and things get crazy fun when you have overlapping payers).

But aren’t there cash only surgical places with listed prices? Yes. And they have an extremely limited set of procedures with everything owned in house – i.e. a setup that is pretty much illegal to set up de novo post Obamacare.

Why does everyone have all these bizarre negotations. Why don’t you just pay the surgeon everything and then he pays the hospital, the anesthesiologist, the pathologist, etc. from that cut? Because that is an invitation for your surgeon to be charged with a crime. It is federal crime to underbill or to underbill when it comes to government monies (and in many states, private insurance monies). We are required not just to I Pencil up a price, but to make that price transparent to regulators. If a hospital wants to grant me cheaper OR time because I have reliable stream of patients, keep the OR cleaner (reducing turnaround time enough to fit another case per day in), and don’t create ancillary malpractice risk at the going rate … the hospital risks being tagged with inducement. If I negotiate a cheaper rate with the lab for my patients’ tests, it is considered prima facie evidence for kickbacks and I then have a positive burden to prove that I am not getting clandestine remuneration from the lab.

Separate, disjointed, billing through bureaucratic negotiation is legible. It is legible to the courts, to regulators, and to malpractice insurers.

But doesn’t all this massive change efficiency of care delivery?
Not that I can easily see. I have personal experience with IHS, TriCare, Kaiser, the VA, and for-profit, non-profit, and even prison care; full Beveridge like IHS is often the least efficient.

So where do cash prices come from? Outside of cash only practices, those are overwhelmingly fictions that somebody pulled out of their nether regions in a likely futile attempt to BS the counterparty to an insurance negotiation.

Why is this all so complicated:
1. Principle agent. The patient has a wildly different incentive structure than the collective payer (insurance or government) and American healthcare is insanely deferential to the patient compared to alternatives. The folks with the most direct control feel at most a small fraction of the price pain have near zero incentive to economize for anything big.
2. Taxes. The original sin of American healthcare was making insurance, rather than medical procedures themselves, tax deductible. This creates very strong incentives for people to bundle non-healthcare into insurance premiums in hard to define manners (e.g. is a health insurer offering a rebate for gym membership incentivizing exercise, allowing folks who would already have gym memberships to pay pre-tax, or just selecting for healthier patients).
3. People are terrified of physician abuse. Most folks, even other physicians, have a very hard time knowing if their physician is taking them for a ride. So they turn to something powerful to regulate physicians. But, not knowing what actually matters, these folks find it extremely hard to navigate market transactions. Healthcare would far rather have 100 unattributable deaths and 2x costs than to have 1 attributable death that regulation could avoid.
4. A complete disconnect between what folks experience for prices (e.g. my tape easily costs 10x more than department store specials, my EMR internal word processor is an order of magnitude more expensive than MSWord let alone Emacs or the like) and how medical expenses run.
5. A failure to appreciate the costs of having things on standby. We have folks ready incase a simple IR procedure perfs the vessel walls. We have countless folks handy in case your infusion leads to anaphylaxis. Or your blood transfusion moves on to TRALI. Just opening the doors typically means that we need to have a few dozen physicians and their support staff available at all times. I’ve seen a simple gallbladder turn into a massive transfusion with staging, SICU, and the whole works. I have seen STD treatment turn into a catastrophic emergency of the sort that gets Derm to come in at oh ass hundred.

None of those go away if we post prices. And a lot of people will be upset – somebody will decry us pricing differently for different patients – everyone deserves the same care at the same cost. Somebody will decry us for not pricing differently enough – people should be reward for making good decisions.

Long run, healthcare is going to get more expensive. I expect it will eventually be on part with mortgage payments (you know you live in your body 24/7). But there is an evergreen fantasy that … if only … then we could reduce prices.

You can’t. You can, maybe, make them rise more slowly, normally for harsh tradeoffs Americans won’t stand. And just about every significant intervention that really moves the price needle … is either selection (e.g. health share ministries have wildly healthier populations because they are heavily selected about drugs, promiscuity, and the rest) or given entirely back by the patient dying later. And the handful of things to do pass muster (e.g. HPV vaccination, Hep C treatment) … it becomes yet another morass of how much to pay whom.

Healthcare is not a normal market. We should stop pretending it could be one.