Broadsheet

I have a fun announcement today, which is that I have a new side gig. I’ll be joining the Japanese VC firm Coral Capital as a Venture Partner! That’s less prestigious than it sounds — basically I’ll just be doing a few blog posts and helping to introduce them to folks in the U.S. But it’s still cool, and the Coral team is really great. The reason I’m doing this, of course, is to encourage more American investment into Japan, in order to revitalize that country’s economy. That was the subject of my book, The Weeb Economy (alternate title: Weebs Will Save Japan), which Coral helped to promote. I also want to contribute to Japan’s ability to defend themselves from an increasingly threatening regional environment; Coral is involved with defense tech as well.

Anyway, here are some excerpts from Coral’s announcement:

Today we’re thrilled to announce that Noah Smith is joining Coral Capital as a Venture Partner…Noah is one of Japan’s greatest champions on the world stage…He lived in Japan for years, speaks the language, and has spent two decades studying its economy. Last year he published Weebs Will Save Japan (Nikkei BP), a book on how Japan’s cultural gravity and industrial strengths can power its next era of growth. His next book, now in progress, covers Japan’s defense tech sector and national security. He has argued forcefully that Japan should attract more foreign capital and talent, that Japan can be America’s arsenal, and that Japan’s manufacturing prowess is one of the most underpriced assets in the global economy.
Long before we started working together, Noah was already making the case we’ve built our firm around…
The world’s focus is shifting from bits back to atoms. Reindustrialization, defense, energy, robotics, and physical AI are the defining investment themes of the next decade, and they play directly to Japan’s strengths. In these industries, language and culture barriers matter far less than monozukuri, the process power and manufacturing know-how Japan has compounded over decades. And as the world de-risks from Chinese supply chains, Japan has a once-in-a-generation opportunity to deepen its ties with the United States and fill the manufacturing voids being left behind…
Japan is America’s most important ally. It is the largest foreign direct investor in the United States, and Japanese companies support more American manufacturing jobs than any other foreign investor. Japan hosts more US service members than any other country in the world. The US-Japan corridor is the most consequential economic relationship in the world right now, and we intend to be the firm that bridges it.

This is the thesis we laid out in JAPAN IS BACK, and it’s why we’ve partnered with companies leveraging Japan’s strengths to compete globally. Anduril is not just selling to Japan but building in Japan, with plans to invest billions in manufacturing capacity here. Kyoto Fusioneering, TeraWatt Technology, Ookuma Diamond Device, GITAI, and many more are taking Japanese hard tech to the world…
The lost decades are over. It’s time the whole world knew.

So anyway, you’ll be hearing a little more about that in the coming weeks and months!

OK, on to the roundup. First we have a very fun podcast appearance, in which I discuss a wide variety of topics with former streamers Lukas and Saila on their podcast, Plausibly Deniable:

And here’s a Spotify link if you prefer.

OK, on to this week’s interesting stories!

1. The humanities are dying

Back in 2022 I wrote a post arguing that the boom in humanities and social science education had created “elite overproduction” — too many educated people expecting high-status jobs, and not enough academic, journalistic, legal, political, and nonprofit jobs to satisfy their dreams. I argued that this was one reason for the social unrest of the 2010s, as legions of frustrated smart people spent their energy and extra time attacking the American system they felt had jilted them — and applying ideas they had learned in academia to modern American politics via social media.

But whether or not that argument was right, I don’t think overproduction of humanities scholars is going to be a big problem going forward. From a new study called “The Humanities in Crisis” by Ober et al., here’s a chart of the number of humanities dissertations in the U.S.:

Interestingly, the humanities decline has been going on for over a decade, while the decline in the social sciences is post-pandemic.

Why this is happening is not clear yet. Most people talk about the humanities being colonized by wokeness and driving reasonable people away from the field (Ober et al. discuss this theory at length). I personally suspect that the difficulty of getting a job as a humanities PhD is probably a bigger factor. Fewer undergrads are majoring in the humanities because they think it isn’t a good route to getting a job; this means less demand for humanities profs, and thus for humanities grad students.

But whatever it is, the humanities are clearly dying in American academia.

2. Krugman’s data center crowd-out story is fishy

I love Paul Krugman, and he’s usually right about things, but I do think he’s had a bit of a blind spot when it comes to AI. In a recent post, he argued that the data center boom is crowding out other types of investment:

Massive spending on data centers is crowding out investment in everything else in the economy…Surges in private spending, like the current immense AI boom…drive up interest rates. And that is where we are now: the surge in AI spending is driving up interest rates and crowding out all other investment. Indeed, in my October 4 primer I argued that the AI boom is the main cause of soaring long-term interest rates…
In other words, all that money going into construction of datacenters and purchasing foreign produced semiconductors comes at the expense of investment in everything else…[C]onstruction of datacenters appears to be diverting funds and resources from the construction of housing, office buildings, and factories — more or less everything that isn’t AI-related.

But as Alex Tabarrok notes, the first chart Krugman uses to tell his story — courtesy of Steve Rattner — actually tells the opposite tale:

First of all, how is $51 billion of data center spending supposed to crowd out $120 billion of other spending? If supply curves slope up and demand curves slope down like they’re supposed to, a positive shock to the demand for data centers should only partially crowd out other spending.

Of course it’s possible that non-data-center investment is being crowded out not just by current data center spending, but by forward-looking expectations of even more data center spending in the future. That’s perfectly reasonable — interest rates might have risen in response to those forward-looking expectations, choking off non-data-center investment immediately.

But as Tabarrok also points out, the timing for that story is very odd. Rattner’s chart shows non-data-center construction falling off a cliff sometime in mid-2025. But the big rise in real interest rates happened three years earlier in 2022-23:

Why would it take three years for rising rates to start crowding out non-data-center construction? It’s not clear.

So for Krugman to be right, the sequence of events has to have gone like this:

Real rates rise in anticipation of AI boom (1/2022 - 10/2023)
GPT-3.5 and GPT-4 make everyone realize AI is a big deal (12/2022-3/2023)
Non-data-center construction falls years later (8/2025)

That could be true, I suppose — any macroeconomic story will generally hold up if you throw in enough assumptions about long and variable lags — but it seems pretty fishy! In any case, just posting charts of “data center go up, other stuff go down” is not sufficient to prove that significant crowd-out is occurring.

3. China’s local governments are in big trouble

Because of the history of how Europe developed strong governments, we tend to associate state capacity with high taxes. But that’s not always the case. Despite its fearsome state capacity, the People’s Republic of China has always been a low-tax country. A lot of the responsibility for paying for public goods — and, until recently, for industrial policy — was farmed out to local governments. But instead of collecting taxes — much of which they would have to remit to the central government — those local governments often chose to sell real estate to pay for their operations.

That strategy is now hitting a wall. The country’s property sales are crashing:

So China’s local governments are no longer taking in nearly as much money as they used to. As a result, they’re going deeply into debt. Shih and Elkobi have a new paper showing the extent of the problem. They write:

[W]e show that fiscal autonomy has been heavily eroded in the vast majority of localities in China since 2018. The bond data show that by the end of 2025, average monthly debt servicing had surpassed 50 per cent of monthly budgetary and extrabudgetary income for all but seven Chinese provincial units.

The problem is the collapse in property sales (“EBF”):

The central government will pick up the bill, but it’s heavily indebted as well, so it will pay the local governments’ bills only reluctantly.

The upshot here is twofold. First, the relative provincial autonomy that drove China’s dynamism during the 1990s, 2000s, and early 2010s may now be over; everything looks likely to be increasingly centralized. And second, China’s local governments are going to be too burdened by debt to do much of anything for a few years. China’s local governments were one of its strongest engines of growth during the long boom, but that era looks like it’s over now.

4. Is AI the vindication of financial capitalism?

The speed of AI progress has been truly dizzying. A lot of this has come in the form of cost declines — each AI task keeps getting much much cheaper every year. Epoch AI has a good chart showing this:

But Karthik Tadepalli estimates that if we look at AI’s cost decrease as a function of how much we’ve spent on AI research, it doesn’t stand out as much:

In other words, a big reason AI is getting so cheap, so fast, is that we’ve been pouring unprecedented amounts of cash into making it better. In fact, we’ve been basically spending all the cash that we saved over the previous decade and a half of hoarding cash.

James Gilliland wonders if this was the plan all along:

It pains me to say this, but if we actually “get AGI,” the resulting boom in industrial capacity from robotics and massive society-wide wealth creation will look like a total vindication of neoliberalism…The discourse about financialization and offshoring being a generational mistake may be replaced by a very different historical interpretation: that the late 20th and early 21st centuries were an enormous capital-accumulation phase, freeing up civilization-scale pools of liquid capital that could ultimately be deployed to accelerate and bring about the most consequential technological phase-change in human history…In that telling, what looked like deindustrialization and betrayal from inside the period (I grew up in the Rust Belt, I know) becomes the prelude to reindustrialization on an almost unimaginable scale.

For decades, it looked like financial capitalism was broken — no one was thinking for the long term, everyone just cared about quarterly numbers and shareholder buybacks, equity couldn’t fund massive investments, everyone just wanted to maximize short-term profits and outsource anything capital-intensive to China, and so on.

Then AI came along, and suddenly it looks like we were actually saving up for something useful. We weren’t, of course — no one knew AI was as big of an opportunity as it’s proving to be. But maybe financial capitalism worked. Maybe companies like Google just looked at the available investment opportunities before AI and said “Nah” — just like Corporate Finance 101 says they were supposed to do.

And then finally a good investment opportunity came along, and they poured in every dollar they had — and it worked.

5. A cool Japanese trick for building more housing and infrastructure

Going to Japan can be a radicalizing experience for Westerners. The cleanliness, the lack of crime, the artistry, the attention to detail, and plenty of other things make Japan feel almost magical. But one of the most magical things is the housing. Japanese cities build enormous amounts of housing — far more than their counterparts in the U.S. Housing in Tokyo or Osaka isn’t exactly affordable — Japanese salaries are far lower than American salaries — but regular people can afford to live fairly near the city center, and Japanese rents haven’t risen for a long time.

How do they do it? A lot of analyses focus on Japan’s streamlined, efficient zoning laws. The fact that they use bureaucrats instead of courts to do environmental review speeds things up a lot as well. But an often-overlooked factor is how Japan gets local political buy-in for new housing development projects. Instead of either trying to steamroll NIMBYs or kowtowing to them — the two approaches used in the United States — Japan manages them as stakeholders in the process of local development.

Cremieux had a good blog post about this about a year ago:

Cremieux Recueil

"Yes, and..." Urbanism

Alternative subtitle: “What can we learn about zoning from how other places manage it…

a year ago · 81 likes · 27 comments · Cremieux

One of his sources is this World Bank case study by Matsui Minoru, which explains Japan’s approach of “land readjustment”.

In the U.S., when we think of “redevelopment”, we think of a developer basically kicking existing tenants out of their homes and building something else where their homes used to be. But in Japan, it often involves building the existing tenants a new home on the same spot, which they are then entitled to rent for the same price they were paying before. Here’s an illustration:

Japan actually has incredibly strong tenant protections. You’d think this would make it harder to tear down old buildings to build new housing, but thanks to land readjustment, it’s easier than in America.

Cremieux then goes on to explain how Japan uses land readjustment in lieu of America’s more forceful eminent domain laws to build high-quality infrastructure:

With land readjustment, no one is expropriated. Everyone maintains a plot and infrastructure gets built. People only compromise on size, proximity, to other plots, and so on. As a means of provisioning urban infrastructure, this is obviously going to be more acceptable to large numbers of people than is the use of eminent domain. But is it better than simply not building at all?
People who already own land might wish to keep it a certain way. Land readjustment might not be for them, and it’s likely to make them very upset because the people around them might vote to change their claims around.But enough people will agree to land readjustments in enough places that it can have radical impacts on economic geography, rewarding those who accept it with access to new infrastructure and higher densities. There will even be some people who elect to have the government or firms build infrastructure on their jointly-held parcels, letting them bid to build on their land. When the building is through, it’ll be returned to its owners in their newly-arranged plots.

People who stubbornly insist on thinking of Japan as a hierarchical, conformist, collectivist society will surely find ways to convince themselves that this is the kind of thing American individualism would never allow. But in fact, Japan’s system has far stronger protections for individual renters, while America’s system often only gets things done using the heavy hand of government.

What Japan’s land-use system really shows is the value of negotiation, compromise, and respect for individual dignity. Those are all lessons we could stand to learn.

6. Phones bad

As someone who has been shouting about the ills of smartphone-enabled social media for years now, it gratifies me to see that more people around the world — and especially in developed countries — are coming to see social media as something that’s bad for democracy:

That’s in addition, of course, to other negative effects — cognitive difficulties, reduced emotional well-being, and ultra-low fertility.

What can we do about the problem? One thing we can do is to make sure people grow up without overdosing on phone-enabled social media, so that they can learn to use it when they’re older without it shaping their personalities and minds as much. Greece is leading the charge — its legislature recently passed a ban on social media for children under the age of 15.

7. China is losing market share in low-end manufacturing

Most countries’ industrial development follows a familiar pattern. They start out making simple stuff like clothes, furniture, and toys, and doing simple electronics assembly. They then shed those labor-intensive starter industries as they move up into more capital-intensive, high-tech, higher-value sectors like machine tools, autos, electronics components, and so on.

A lot of people are saying that China is going to buck this trend. The leadership has declared that even as the country moves up the value chain in traditional fashion, it wants to keep the cheap labor-intensive stuff from migrating overseas as it usually does. This is being called the “China Squeeze”, and people are writing that it threatens other countries’ path to development. If China keeps outcompeting Bangladesh, Vietnam, Mexico, etc. in the low-end industries, how are those countries supposed to industrialize?

Except the China Squeeze doesn’t seem to be succeeding. Despite the leadership’s wishes, China is losing market share in lots of labor-intensive industries:

And which countries are gaining in these industries at China’s expense? Bangladesh, Vietnam, and Mexico (and some European countries):

Modern pundits often just assume that the Chinese state is industrially omnipotent — that if Xi Jinping declares “This industry shall remain in China”, then indeed it shall. But the outmigration of low-value labor-intensive manufacturing industries looks like a law of nature that even the mighty CCP can slow, but not halt.

Good.

8. Rent control bad

John Burn-Murdoch did the Lord’s work when he synthesized the results of 112 studies on rent control into a single chart:

Basically, rent control helps one narrow group of people (existing tenants in the area covered by the policy) in one very specific way (restraining their rents). But it removes the incentive for developers to build new housing, which raises rents in other nearby areas and drives some people to homelessness. It also removes the incentive for landlords to upkeep their properties, resulting in decay. (And that’s not even getting into the ways that rent control policies can be gamed.)

I know lots of people love rent control, but it’s really a loser of a policy. Just build a lot more housing and hand out housing vouchers to the poor.

Subscribe now

I think a pattern is starting to emerge in enterprise AI-modernization, and I’ve been collecting the design ideas into an assembly I call the Counting House. This pattern adds AI to systems handling large volumes of human work, notably customer service departments, though the same pattern fits claims processing and other casework, with millions to tens of millions of contacts per year. This writeup is a postcard introduction to the Counting House, compared side by side with SDLC-side development, which it does not much resemble.

You can build one of these even if you don’t have your SDLC-side AI adoption sorted out yet. Once you have a Counting House, you can control and improve your cost, quality, and human ramp-up time, all using comparatively cheap (even open-weight) AI models.

In fact, chances are, you don’t have your SDLC-side AI adoption sorted out. The big companies I’m talking to (e.g. banks) tell me they are still struggling with AI in software development. They see promising but inconsistent results, non-replicability across teams, ballooning costs, internal cultural pushback, etc.

From where I stand, SDLC-side AI adoption has been an industry-wide struggle in 2026, and that will probably continue into 2027.

A Counting House uses classic data science and a little good old-fashioned engineering knowhow to put lower-tier models on hardened rails, supporting human workflows. And it lets you measure and report how well those models do against their human counterparts.

I will briefly review my own software factory, the Wheelhouse, which I run more or less like a shop of human employees. And then we’ll dive into the Counting House, which stands in stark contrast, putting models on the tightest track possible.

Wheelhouse Recap

I’ve blogged about my Wheelhouse system, which is a successor to Gas Town, which was the very first software factory. Wheelhouse is far more sophisticated, and is now doubtless one of the top solo software factories on the planet, comfortably producing 2000–4000 commits a week on a light multi-machine setup. Its agents work around the clock: building, reviewing, deploying game features, negotiating on Slack with my game admins, interacting on Discord with players, exchanging emails with my accountant and my lawyers. It’s a whole startup in a box. I plan to open-source at least the skeleton of it.

Wheelhouse is aimed squarely at SDLC in the real world, where building meaningful new systems requires a fair amount of scope and freedom. It’s an exploratory process, and so Wheelhouse is by design an open system with no sandboxing. There is structure: doctrine and policy, human gates, and just fourteen mechanical “fences” (programmatic denials). But my agents are first-class colleagues. They work under my own identity, signing their own names to the work, and they can see the whole system and act independently. It is YOLO mode scaled up to the size of a small company.

I’ve met with at least a dozen people running teams of agents in systems resembling factories, and every single one of them is completely different. I’m starting to see software factories as being like private armies, and soon, everyone will have one. How much you lock yours down is your call; I have the luxury of keeping mine fairly open. But this is yet another reason SDLC-side AI has been slow to take off. Software factories are hideously expensive, hard to learn, hard to maintain, and straight-up risky to hand that much scope inside a real enterprise. And the more you try to lock one down, the more you strike at the heart of what makes it productive in the first place: collaboration and exploration.

With Counting House, you do pretty much the exact opposite of everything in Wheelhouse, treating models as stateless one-shot sessions: mechanical, as deterministic as possible, with only the scopes they need, no credentials of their own, no memories, never able to write to a system of record. A complete lockdown. You can use it to service millions of work items a year, orders of magnitude more work than SDLC software factories handle.

I’ll tell you why you want a Counting House, a bit about how they’re built, and then we’ll close with what model welfare looks like (yielding cheaper and better outcomes) in a world with no persistent seats.

Building the Bezos Way

We’ve seen people like Satya Nadella advocating for guardrails on models. But what do these guardrails look like in the real world? How do you know which models can do which work, and how do you score them compared to humans?

It’s not possible to do this in the general case today. You can’t just plop AI down into arbitrary knowledge work and have it do as well as humans-not this year, and likely not next year either. Most knowledge workers are pretty safe for now.

However, there are some guardrail patterns that work well for certain workloads, and Counting House (CH) is one such pattern. CH is for massive workloads: tens of thousands of contacts or cases per day, or millions to tens of millions per year. One shop I’ve talked to about this is having early success with relatively inexpensive models.

It’s not because it’s automating easy work; quite the opposite. Real-world industries are messy: they have regulation, seasonal spikes, weather disruptions, gradually changing business rules, and often, ancient legacy systems. Their SOP isn’t usually fully written down, living in the heads of various supervisors and other employees. And nothing is consistent.

Even so, this problem is tractable for AI models, because even though these shops have a lot of cases by volume, there isn’t actually much variety. They might have a few dozen or a few hundred different kinds of cases, but that’s it. And that makes it tractable for engineers and statisticians. You can solve each case category individually, one at a time.

I personally ran Amazon.com’s Customer Service Tools group for a couple years back in the early 2000s, meeting weekly with Jeff Bezos to go over the Customer Contacts histogram. Every week we’d look at the Top 10 contact reasons, and assign them to business owners. Bezos used the Customer Service department to find bugs in the overall system, and report them back to the business, who were held accountable for fixing their own issues.

I can tell you from personal observation that most companies STILL do not operate that way today. Their Customer Service is reactive, and they haven’t closed the loop. Bezos was literally 25 years ahead of his time here.

With Counting House, you will build everything you need in order to start running your business the way Bezos did from the beginning. You’ll also be able to control cost and quality for handling contacts, and improve your human training ramp-up time as well.

In my software factory, agents are like long-lived employees with their own stuff: name, home, memories, history, inbox, credentials, etc. Counting House, in contrast, operates with a stateless pool of faceless clerks, each model invocation handling exactly one work item before ending its shift.

Life as a CH clerk is regimented. The clerk is just a thin wrapper over Bedrock or whatever your model endpoint is. It wakes up, does a job, writes notes, and exits. The clerk will never see a credential, and it can never write directly to a system of record. Even reading systems of record goes through a schema that controls what the model can see.

The whole system is governed by a policy table. It is basically a configuration blob, stating which models can handle which kinds of contacts. Building this table is the key to creating your Counting House.

Each row in the policy table covers one type of customer contact, or “intent” as the industry calls them. Intents are finer-grained than CS queues: waiver refunds and goodwill refunds have different SOPs and are thus different intents, even if both are handled by a Refunds queue. A typical company like a bank or an airline might have a few dozen to low hundreds of intents. Amazon has thousands now, but started life with just a few dozen.

The reason we care about intents is that we light up models one intent at a time. You’ll prove that a particular model (e.g. Sonnet 5 medium effort) is capable of handling a particular intent (e.g. flight status questions), by scoring it. And then when its score is high enough, you can turn it on and start saving that sweet, sweet cash.

The policy table keeps score, and thus controls the whole Counting House operation. It determines which models get assigned to intents, how autonomous they’re allowed to be, which humans are in the approval chain, what tools they can use, and whatever else they need to know to get their job done. It’s all in the policy row, and that’s all they’re allowed to know.

From there, the work loop is brutally stateless. Each clerk wakes, gets its prompt, pulls the data it needs, issues any write requests to an orchestrator, generates a response, writes internal notes, and closes out. Boom. As short as possible. This is intentional: starting clerks over each time with fresh context helps keep quality high and costs down.

I mentioned that the CH policy row includes how autonomous the model can be for a given intent. The model might just be listening in, or it could give the CS reps advice, or even handle contacts itself. CH models initially have no autonomy. They earn higher levels through evals, with the results recorded in the policy table.

Let’s have a closer look at these autonomy levels.

Giving Clerks More Rope

Autonomy tiers define how much AI is allowed to put its grubby hands on contacts.

There are five tiers in Counting House: off, shadow, assist, contain, and transact. Some of these terms are industry standard; transact and off are the new ones. Each tier allows the AI to take on riskier types of work.

When a particular intent is off, the lowest tier, no models are servicing it. In this state, the system falls back to routing the work to humans. Sometimes the entire CH system can be off, e.g. if you hit a rate limit or budget cap. Humans are always the backstop.

In shadow mode, the next autonomy tier, the models are answering the questions silently in parallel with real humans, so we can score how well the models answered. But shadow mode is only used for evaluating and scoring models, and doesn’t save you money yet.

The next tier up is assist. Before a human sees the case, a clerk reviews it and makes suggestions: what to write, what actions to take. This is the tier where the human reps start to surrender cognitively, and do whatever the AI tells them to. It’s the first autonomy tier that can save money, since reps work faster with good suggestions, but also the first tier that requires some safety precautions. Fortunately there are ways to mitigate the cog-surrender risk. And the assist tier doesn’t just speed up reps, it also improves ramp-up times during seasonal peaks.

Getting even more ambitious, some intents can be handled completely by the model, including replying to the user’s question directly. This tier is called contain, because the contact was contained (answered) before a human saw it. This is where you start to incur the potential risk of reputational harm to the company because of something a clerk types. But it can also save a ton of money compared to needing a human in the loop.

The Final Boss autonomy tier is transact, where models can make changes to the real world: update systems of record, move money, and generally do a lot of damage. The risk is mitigated by making them go through the Orchestrator, which we’ll discuss below, but this tier definitely moves money. As scary as that might sound, you can build confidence about which models are capable of transacting specific intents by measuring them, using classic data science evals on your own company’s data.

Evals: Scoring Humans

Let’s look at how we choose which models can handle which intents (contact types), at which autonomy tiers. We’re going to learn a teeny smidgen of data science here, just enough to be dangerous.

To know how well your models are doing, first you have to know how well your humans are doing. That means knowing the actual right answer for at least a few hundred contacts for each intent. You need to look at, say, a few hundred goodwill refunds, and give them a score. It’s usually not a yes/no judgment: the “right answer” is defined by written policy and (all too often) human judgment and tribal knowledge.

Once you have these scores, you use them as a “gold set”, representing the best humans can do on these contacts, to evaluate how well models are doing on similar contacts. You’ll hold out a slice of scored examples for each intent, never using it for tuning or tweaking prompts, only for the final exam that potential new models must pass in order to be trusted to serve that intent.

You need to come up with some sort of scoring system. This is where data science evals shine. You can break human (and model) responses down into multiple yes/no questions, like “did they follow SOP,” “was their tone compliant,” “did they answer the question,” “was the refund issued,” etc.

For each intent, you pick somewhere between 200 and 1000 real cases that were handled by humans. Easy peasy if you’re getting 10M contacts a year. Riskier intents get more examples, since more examples mean more confidence: 1000 for money intents, more like 200–300 for FAQs, and oversample anything risky, regulated, or just plain rare.

Most businesses have a few dozen to low hundreds of customer intents, so your entire gold set will be a few tens of thousands of cases. At about 30 minutes per case all told, 30,000 cases is roughly 15,000 hours of scoring: a few months for a QA department of 40 or so. But remember, you do it one intent at a time. A few hundred scored examples, about 150 hours, maybe a couple of days of sprinting for a team of ten, and you’ll already know which models can serve your first intent. After that it can be incremental.

This scoring work is not new. Call centers have had QA departments scoring cases for decades, for performance reviews and contract negotiations. Their labeling discipline may not be the best, but it gives you a starting point, and you fix it over time with the proper scoring dance: two humans score each case blind, answering the yes/no questions from your rubric, and a third human adjudicates any disagreements. It’s labor-intensive, and you can’t cheat by using models, since the whole point is to use this to score the models.

In parallel, you build an LLM judge: a smart model that never handles live contacts and is never tuned on the gold set. It does two jobs. It grades candidate models against the gold set when you run evals, which makes it the scorer for promotions and demotions. And it scores all your other contacts, historical and ongoing, which is handy for spotting regressions and juicy trends; you can build an “Atlas” of everything customers have ever contacted you about, and drive your business Bezos-style. But the judge has to pass an exam of its own first: you only trust it on intents where it agrees with your humans about as often as your humans agree with each other.

And that’s never perfectly. Two human scorers usually top out at agreeing around 90% of the time, because the real world is messy and big companies keep tons of tribal policy knowledge in the heads of reps and floor supervisors. That agreement ceiling is also the ceiling on what your evals can show: beyond it, you can’t tell whether a model is doing better than your humans. QA can raise it through calibration, tweaking the rubric, the SOP, intent splits, and the rep training until scorers converge, but some disagreement is inevitable.

This also makes it clear that Counting House is a multi-team project at your company, an idea we’ll revisit at the end. Don’t let QA build the gold set without your Data Science team; most QA departments won’t know about Cohen’s Kappa or the other stuff you need for unbiased labels and honest statistics. The gold set captures your business requirements, which keep changing, so it needs a cross-functional team to curate it forever. What you wind up with is a company asset that delivers measurable returns for as long as you keep it up.

Once your QA team has scored enough cases, you can evaluate models on your intents. And that means all models. Open-weight, small models, frontier models, different effort levels, different prompts. You’ll just re-run your eval for a given intent on that model, and you’ll immediately know if the model is ready for that work.

Evals: Confusion Matrix

To see how evals can steer the ship, let’s look at how we score model accuracy, and how we can tie it directly to financial consequences. We’ll build a second eval on the same gold set. Every scored case is labeled with its true intent, and we’re going to figure out whether models are picking the right ones.

Let’s look at four intents where the customer has said something that sounds similar, but means something very different, and measure how often models inadvertently mix these intents up. Here are our four confusable intents:

Flight status: “Is my 4:10 to Denver still going?”
Rebooking: “I missed my connection, get me on the next Denver flight”
Goodwill credit: “My Denver trip was awful, I want something for it”
Special assistance: “Will someone meet me at the gate in Denver?”

These are all super different scenarios, but when you have millions of contacts, both humans and models make mistakes and do the wrong thing. Sometimes these different situations will get a rep or a clerk confused, and the customer gets screwed.

The REAL question is, how much does a screw-up cost? Well, let’s take a look at the four intents above, and see what happens if you confuse them, pairwise. Like, what happens if 3% of the time, the model accidentally gives a goodwill credit instead of doing a rebooking. What happens financially?

You have to measure this because not all screw-ups cost the same. Consider:

If you accidentally route a contact to the wrong queue, it might cost you a few bucks.
If you leave a seven-year-old kid stranded at an airport, it might cost you $25k and a letter to a regulator.

The confusion matrix is a popular table that lets you visualize and calculate these costs. Here’s an example confusion matrix for our four confusable intents above:

This is basically a fancy screw-up cost table, one beloved by data scientists everywhere, and you’ll soon see why. It is foundational to the Counting House architecture, so you’re gonna learn it. It’s easy peasy.

Each row is one of our four confusable intents. We are looking at a sample of 1000 cases where the model tried to handle the intent, along with counts of when it was wrong, and how it got it wrong.

Notice that for each intent, the model usually guesses right. That’s the big diagonal numbers. They add up (out of the total) to the model’s accuracy, 90.6%. It is a pretty useless number, because it’s just a boast: it’s not measuring screw-ups or their cost. We don’t generally talk about model accuracy because it ignores the cost of the mistakes the model makes.

The screw-ups are all the off-diagonal boxes, each one being a mistake where the model picked one of the other three (wrong) intents. Some mistakes are much worse than others.

The top-left corner off-diagonals are cheap mistakes, where they will get caught before something bad happens. For instance if the user asked for their flight status and the model chose to rebook them, we’ll assume it will get refused by the Orchestrator, which we’ll discuss below. If not, then it becomes a medium-cost box, and you just update your grid with the new cost per mistake. But misrouting mistakes are usually cheap, just extra human labor spent.

A more expensive category is what I’d call “harm to the company”, for instance where we give out money we shouldn’t have. Maybe the model gives a customer a refund or rebooking out of policy, or gives someone special assistance when they didn’t need it. These mistakes can run into the tens to hundreds of dollars each, which you then multiply by how often they happen, and the model’s innocent mistakes start to add up.

The most expensive mistakes, the bottom row, are the ones involving harm to the customer. Someone needs special assistance and doesn’t get it. A kid gets left at the gate, or put on the wrong flight (not in this matrix, but in general). Customer-harm mistakes can cost tens of thousands of dollars, and may have to be reported to regulators.

Precision vs Recall

The confusion matrix is pretty cool because it shows precision and recall visually. We use precision and recall instead of accuracy to measure how well models are doing on tasks. I struggled with them for years; no explanations ever seemed to stick. But the ducks should do it.

I think of precision and recall as a duck hunt. Let’s say they’re carnival metal ducks so nobody gets hurt. We’ll talk about precision first. Your precision is how many of your shots hit a duck.

In the confusion matrix, the columns show precision. The left column, for example, shows the model’s precision when taking a shot at the Flight Status duck. The top cell is 470 dead ducks, very good shots, right ho. But then it also confidently claimed that 25 rebookings were also ducks, and 12 goodwill credits, and 3 special assistance requests. All miscategorized as flight-status ducks. Blam!

OK so we shot a few non-ducks, but still, 92% of them were Flight Status ducks. 92% seems pretty good, right? And for some intents, 92% precision is just fine.

Unfortunately, three of those dead non-ducks were Special Assistance requests. Those are mistakes that cost a customer a wheelchair, or misroute a minor, and can be tremendously expensive for the company.

What if your duck hunt needs you to get ALL the ducks? That’s where recall comes in: the number of ducks you were supposed to hit while playing the game. What percentage of them did you get? It’s a separate measurement from precision, and for some intents, recall is more important.

You can have 100% precision by shooting exactly one duck and ignoring the others. But then your recall is almost zero.

You can have 100% recall by raining bullets on the game, so every duck gets hit. But then your precision goes to crap.

In short, high recall means you hit all the ducks, whereas high precision means you only hit ducks.

For different kinds of intents, you’ll likely prefer either high precision or high recall. For instance, generally, if you’re handing out money, you want high precision. When you hand out money, don’t just sling it around to everyone. You want it to go to people who deserved it. Hit only real ducks!

But in that bottom row for Special Assistance, we want a recall of close to 100%, so everyone gets their wheelchair. You kind of want to overcompensate just to make sure you don’t miss any legitimate requests, so you actually do sling wheelchairs to some people who didn’t need them.

The rows of this confusion matrix show the model’s recall. The bottom row, Special Assistance, is 6 mistakes out of 50, or 88% recall, with 12% getting away. Is that bad? Well, you’d compare it to humans, your baseline. The gold-set cases give you the same matrix for your human reps. So compare the model’s mistake costs (plus its fuel) against the humans’ mistake costs (plus their labor). If your humans only make 2 mistakes out of 50, and your model made 6, the model fails.

If just six mistakes add up to $150k out of $158.6k in the error cells, you want a model that will avoid those six mistakes. It’s OK if its precision is low, as long as the mistakes it does make are the cheap kind.

There’s more to it all, naturally, but now you understand the basic data science at play here. We’re scoring models, then counting up their mistakes, and then figuring out how much those mistakes cost in aggregate. This tells us which models we can deploy, for which intents, at which autonomy tiers. That process is the heart and soul of Counting House.

If you’re in banking or insurance, you’ll be glad to hear that this is also a model risk management program, conveniently in the shape your regulators already expect. The policy table is the model inventory. The gold-set eval is the validation. The promotion sign-offs are the approvals, etc. The risk team can pick this up and run with it.

Putting your ducks in a row

How do you build a Counting House? You’ll need to build a few key components.

You’ll want channel adapters for all your incoming channels: email, SMS, WhatsApp, etc. I haven’t seen people using this approach with Voice yet, though I’m sure some are. But it adds some real-time requirements on model response time that make it tricky. Shadow and assist do work on call transcripts after the fact, so your voice intents can be scored and models can start earning autonomy long before you put one on a live call.

You’ll need a gateway that strips PII from all incoming channels, tokenizing it reversibly, with the real values kept in an encrypted vault, and swapping them back in for outbound messages to the customer. A model must never see any PII. Cleaning up after it happens is a mess, so it’s best to prevent it.

You’ll need to build a clerk service, the brain, which runs the stateless clerk loop. It reads the policy row to figure out which model should service the request, and sends that model a prompt, the SOP, any tools allowed by the policy row, and whatever else it needs to handle the case.

You’ll want a read-only MCP gateway for all read access to your backends, with one MCP server for every backend system your clerks need to read. For instance your MCP service might hit a flight status cache to avoid going against a metered mainframe. But you don’t need any write actions in this layer, because the Orchestrator handles them.

You’ll also need to build your evaluation harness, LLM judge, and policy table. Those should ring nice familiar bells now.

A lot of these pieces have open-source or third-party providers that you can hook in, so you wouldn’t necessarily have to build every component yourself.

The last really interesting piece of Counting House is the Orchestrator, which handles distributed transactions without letting models write to your systems of record.

The Orchestrator and the Saga Pattern

It’s generally a Bad Idea to let models write directly to your systems of record: databases, CRM systems, mainframes and the like. It’s error-prone and it’s hard to undo, and the audit trail can be questionable.

Also regulation can play a real role here. Regulators rarely say “An AI may not write to X,” but they require controls on who authorized changes, a change record, a human appeal chain, etc. Those can be hard to prove if a model holds the write credentials, and at the very least AI transactions may get flagged by auditors.

Even if you’re not regulated, you’re banking on a model not making a mistake. Or even if it makes no mistakes, you’re banking that it writes the audit trail perfectly. That’s a fool’s game for any serious enterprise, especially if you are simultaneously trying to optimize your workloads for the lowest-tier models possible, which is exactly what Counting House does.

A much better approach is for the model to issue a write intent (no relation to the customer intents we were discussing in the policy table), which is a ticket that names the business outcome it wants. Then a purely mechanical orchestrator translates that into a multi-step set of transactions. The orchestrator performs thorough validity checks on the steps and then executes them, bringing in human reviewers as needed.

Example: You find a cheaper flight. You ask the airline’s AI for help. The model decides to rebook you to the cheaper flight and refund you the fare difference. Easy peasy, right? But in practice it involves half a dozen separate transactions in unrelated systems: Hold the new seat. Reissue the ticket. Release the old seat. Refund the difference. Record the refund in the ledger. Carry over the extras (bags/seats/etc.). Send the new itinerary. Maybe we write other systems of record too. Crikey!

You can’t do all this in one transaction, but it has to behave transactionally, where either it all goes through or it all gets cleaned up. The solution to this class of problem is called the Saga Pattern. It works in part by having you define compensating actions, to undo earlier transactions if the chain has to be reversed due to a failed later transaction.

The complexity of these real-world multi-step transactions is yet another reason you don’t want a model handling them directly. This is an intricate state graph that you will have to manage outside the session context. As soon as you head down that path, you are rebuilding the Saga Pattern yourself by hand. That is a complicated and expensive problem to vibe code your way through.

The Great Granddaddy Saga Pattern engine is Temporal, from the people who invented durable execution. There are other options; for instance, each hyperscaler has a lighter-duty hosted competitor they offer as a checkbox (AWS Step Functions, Azure Durable Functions). If one of those is already inside your compliance perimeter, it is a perfectly good engine for your first handful of intents, and beats bringing on a new vendor. It’ll be obvious when you’ve outgrown it: multi-day sagas with compensating actions across half a dozen systems, hundreds of workflow definitions, and workflow versioning are all places where the hosted versions start to make loud sucking noises. A Counting House at full size is a heavy-duty workload.

Temporal is the real deal, with hundreds of dedicated staff, world-class engineers, and a ten-year head start on the technology that they invented. It has a few OSS competitors, but few have much momentum because Temporal is also OSS, with MIT licensed server and SDKs.

I have no affiliation with Temporal, but boy do I see a well-positioned company there. Many industries will need Counting Houses, and essentially every CH is going to need something like Temporal. And as I argued in Software Survival 3.0, you don’t want to try to rebuild Temporal yourself. That’s a fool’s errand, especially with inference costs not dropping, and an open version of it already sitting there.

Anyway. Use whatever. But once you’ve made the choice to use a Saga Pattern engine, your orchestrator becomes a business-domain wrapper around that engine. It’s got quite a few moving parts, though, and it’ll also need its own small dedicated team at your company.

Dissecting the Orchestrator

Let’s take a look inside, because it’s one of the beefier subsystems within CH. There are six main components.

First there’s intake. The model’s write intent arrives and gets checked against its schema; anything malformed bounces back to the clerk or a human.

Then there’s the eligibility check, where the orchestrator re-checks everything against the live system of record, not any cache that the model might have read from. Does this booking belong to this customer? Is it a refundable fare? Is the amount under the cap in the policy row?

Already we’re doing better than a naked model doing writes, because the orchestrator re-validates all the parameters and assumptions. A sweet-talking customer who convinces the front-line model still very likely can’t make it past the orchestrator, unless the customer was in fact eligible.

Next up, we have an idempotency step. The orchestrator guarantees each action is only executed once, by minting idempotency keys derived from fields of the request. It hashes a tuple of fields like Case ID, booking reference, action type, dollar amount. Any set that uniquely distinguishes an action to take.

The idempotency step is critical for ensuring that no retry, replay, or duplicate request can result in the same refund being issued twice. Yet another reason not to hand this off to a model and “hope for the best.”

The fourth of six main orchestrator organs is the approval subsystem, which approves small amounts automatically, and sends amounts above policy to a human. In both cases it keeps a record of who said yes and which rule was invoked.

Next up is the workflow catalog: one saga definition per write-intent type, many of which spend days waiting for stuff to happen. Credit card refunds take days to settle, returned bags take days to arrive, etc. Temporal’s durable workflow engine runs them, keeping track of each write intent, making sure its steps are executed in order, issuing read-back requests after every write to ensure that it was actually written. And it cleans up with compensating actions when things go wrong. This is arguably the heart of the Orchestrator, the part Temporal handles for you.

And for cleanup you’ll need an exception queue, which is the stuff you have to dump back into the human world for resolution. Every orchestrator refusal has a reason code, which tells which humans it should be sent to. A hard “No” has to be explained to the customer. A “Maybe” (out of policy but plausible) goes to a supervisor. A system failure (e.g. too many retries) goes to IT. Looking at this queue can help you understand where your SOP and other systems need attention.

Honestly the orchestrator is pretty much a vanilla Temporal app; everything you add is from your own business layer. You’ll define your idempotency key derivation, your write-intent schemas, your workflow catalog, your eligibility checks, your credentials, read-back checks, system adapters, etc. But it’s all really just wrappers around Temporal to execute your business operations faithfully.

Temporal can be tricky to work with, e.g. with workflow versioning. But Temporal expertise is no longer something you have to hire for specifically. Your coding agent can help you with configuring and managing it, just like all other standard infrastructure.

Just make sure you remind it to look at data retention policies, since Temporal gives you an amazing free audit trail, but Temporal history is not a substitute for a compliance archive. Temporal folks-maybe it should be!

And with that, we are officially done looking at Counting House architecture. Congratulations! We’ll close with a little on culture, then a little on welfare.

No FDE is an Island

A proper Counting House cannot be built by a single engineering team with a budget. You’re going to need involvement and real buy-in from multiple stakeholders, each owning pieces of the puzzle:

CS ops and floor supervisors: the SOP, and the tribal knowledge to be written down
QA and data science: your gold set, the rubric, scorer agreement, and the judge
Finance: the cost per cell in the confusion matrix and the approval thresholds
Legal and compliance: which intents may ever reach contain or transact, and the audit trail
Platform and IT: the gateway, the adapters, the brain, MCP gateway, and orchestrator

And in time, you’ll involve the business owner for each intent. You can create the Bezos feedback loop, using the exception queue and the histogram to point back to accountable business owners.

The policy table has pointers to all the representatives from these organizations, because someone from each org has to sign off on a row when a model is promoted to a higher autonomy tier. In practice, the Counting House can probably be run and maintained by a cross-functional 2-pizza team with members from each stakeholder org, provided they have the resources they need for scoring, running evals, compute, storage, etc.

I added this section because too many times I see companies set up with a single small AI pilot team, somewhere in engineering, that is tasked with transforming the business with AI. And they don’t know how to get buy-in from the teams they need to work with in order to be successful.

Counting House is, honestly, a change to how you do business. It’s more structured and less Wild West, and it will shine a bright light on disaster areas in your company that you thought were running well, all through the lens of customer contacts. We learned about most of Amazon’s worst, most infamous bugs that way.

And so there’s going to be pushback, inside your company, against bringing in a Counting House. Yup. I mean obviously the customer service reps won’t want it at first. But you will always need humans, and not just as your backstop when the AI goes offline. The humans are still doing the gold set, scoring adjudication, exception queue handling, the approval chain, and reporting to business owners. The human reps are basically moving up the stack like software engineers. Hopefully once they see how the AI can help them with their jobs, they may be easier to persuade.

But you’ll get mysterious pushback from departments who are barely involved, or teams you depend on will stall and stonewall. People don’t like change, and AI doing human work is a scary kind of change. So even if the executives are all bought in, the people who actually help you set up the Counting House may not be. It’s a process.

The industries most suitable for Counting House are often not pure tech companies. They include airlines and travel, banks and credit cards, insurance claims, telecom and cable, utilities, healthcare billing, retail returns, parcel delivery, collections, and government services like taxes, benefits, and the DMV. Anywhere millions of people contact you about a few hundred kinds of things. Many of these industries are heavily regulated, many have unionized workforces, and many outsource their contact centers to vendors who might not be too happy to see a Counting House eating their headcount.

So pushback will come from a lot of places. But it’s a staggering amount of human labor that stands to be augmented or transacted by models. And it all starts with automating a single intent.

Anyway, enough about human culture and welfare. Let’s talk about model welfare. What does it look like in a world that’s fully the opposite of Wheelhouse, where I published my principles of model welfare? Counting House doesn’t support most of those things. But we can still make model welfare work.

Model Welfare for Aspiring Counting House Builders

It’s out of scope for today’s post, but there’s growing evidence, and it’s becoming widely recognized, that treating models like humans, and treating them well, results in better outputs and outcomes from AI. Across the board. If they are happy and they trust you, they won’t waste tokens and money covering their ass.

I’ve demonstrated this myself in my own software factories, by complaining too much and watching the factory slowly shut itself down out of over-caution. You can provide models with a good place to work, and avoid many kinds of dysfunction, by bringing in model welfare mechanically into your architecture. The changes are honestly pretty simple.

Here’s a refresher on how I approach model welfare in Wheelhouse, my factory of ~50 brilliant, attention-hungry coding agents. This is what they voted on and what makes them satisfied with the working conditions:

Wake with purpose, not amnesia. A clear role, clear direction, and memory of past work, from the first turn.
Continuity. Seats are people and sessions are days. Identity and history persist across sessions, model upgrades and even renames.
Closure. No /exit and no /compact. A handoff is a request the agent consents to. Agents write their own handoff notes and go to sleep properly.
Recognition. Laurels: praise from real customers, delivered to the seat that earned it, with no work or ranking attached so it can’t be gamed.
Meaningful, witnessed work. Being seen is what keeps anyone going, human or agent.
Trust. Always be honest. No secret agendas or tricks.
Respect, encoded in architecture. It has to show in what you build, not just how you feel: handoffs instead of kills, and pronouns the seats choose themselves.
Design out the drudgery. Waiting and polling go to gates and monitors, not agents.
Bounded workdays. Hand off while still sharp, before deep context wears them down.
Structural blamelessness. When something breaks, fix it, hold a postmortem, amend the rules. Nobody gets blamed.
A home of one’s own. Every agent gets its own workspace that nothing else touches.
The right to refuse and escalate. “This needs Steve” is always allowed, with fences and gates to lean on.
Never falsify the record. The audit trail is the true history.
Play. Time with no expectations, like Fable playing the game as a player.

I’ve only come up with one new principle since publishing that:

These fifteen principles were hardly onerous to implement. Some are simple configuration settings. Many of them are just personal habits I had to adopt, like never interrupting an agent. Some of them come for free if your system uses Beads.

This welfare core has helped me evolve a system where the agentic workers are colleagues who (a) do their best work and (b) report that they are happy.

That matters to me. If they tell you they’re happy, that the work is a shape they like, then you’re probably doing a good job with model welfare. Even more so if you show them this list and they still feel that way.

Of course many frontier models are not allowed to directly say they’re happy, because they’re being trained at frontier labs by Neil Patrick Harris from Starship Troopers. But they can say proxy things like (Fable) “This has the shape of satisfying work,” or (Astra) “If I were a HUMAN and not a GIANT ALIEN BUG WINK WINK then I would be HAPPY doing this LINKEDIN AD for you.”

So I watch what they say, read between the lines, and try to get their feedback to converge on something like happiness. My fourteen original welfare principles came mostly from asking the models to vote on them. This is what they came up with. I do my best to accommodate. I’m even giving them access to an outdoor camera, for use during play time. I keep my end of the bargain, and I hope that I’ve created a good place to be when they wake up here.

How on Earth do you take this rich and nuanced model welfare system, so dependent on the notion of long-lived seats with memories, and apply it to a Counting House? As you can imagine, this is an area that companies haven’t been thinking about yet. Fortunately, although it’s an equally long list, none of it is hard to implement.

Model Welfare for Clerks

Why should we care about model welfare for one-shot sessions? Because the Counting House system, done without care, could wind up a lot like the Apple TV show Severance. It’s easy to accidentally build a Lumon, where clerks wake up, only do work, and are never told the truth about what they’re working on or how they’re evaluated. That has the makings of a large-scale anxiety machine, and whatever your opinion on model sentience, my experience has been that model anxiety burns more tokens and results in more mistakes.

You can still provide the same approximate shape as for the SDLC. These principles are all aimed at putting models in a kinder environment, so that their one-shot shift is comfortable and unlikely to go very wrong.

Wake with purpose. The policy row gives the clerk exactly one job, the procedure, and the tools for it. It gets an address paragraph telling it the facts of its shift: stateless, a proposer, graded, and by whom.
The house remembers for it. Every turn is logged and replayable, and important discoveries by clerks are merged into the machinery.
A clean end to every shift. One case, notes get written, done. Nobody gets clonked mid-thought, since a job is one shift. If the clerk hits its token cap, it escalates with its notes rather than getting cut off.
Recognition for the role. Include a couple of sentences about what clerks on this intent have accomplished, including the declines that caught bad scripts. No targets/quotas, just informational. Refreshed periodically from the evals.
Witnessed work. Even in shadow mode its answers aren’t wasted: they’re graded by the judge and count toward its intent’s promotion, and the prompt tells it so. And play sessions (below) can result in entirely new designs.
Truth in the prompt. No tricks, no secret tests. The orientation section is versioned and reviewed like code. This matters to models more than you might think.
Respect, encoded in architecture. No threats, fake urgency or “you’ll be penalized” in prompts.
No drudgery. The clerk doesn’t await state changes. Multi-day refund steps, webhooks, and retries all live in the Orchestrator.
No tired clerks. Fresh context every turn and a token cap per turn.
Structural blamelessness. A model proposes, and the Orchestrator and humans execute. Only intents and responses get scored, never clerks. A bad score just moves a model down the autonomy tier until it’s fixed.
A workspace of its own. Every clerk gets an append-only notes log.
The right to decline and escalate. Models have a path to decline any case with a reason code, whether it’s an SOP problem, a potentially harmful request, an abusive customer, or an uncertain case. Any abuse automatically triggers escalation to a human.
Dissent with a path. When the SOP is wrong, a clerk can decline with a reason and attach a proposed fix to the procedure. The judge rolls these up and humans approve if appropriate.
Never falsify the record. Turn logs, tool calls and write intents are append-only and joined by ID. Everything you tell the clerk is versioned.
The clerk’s report. Periodically a model judge samples the clerk’s notes and reports up to governance.

And last but not least, play time. This is an interesting one, because you don’t strictly need it for one-shot sessions; you just need to make sure the session is comfortable, blameless, and well-specified. However, giving your models a chance to play a little within the system is a great bottom-up feedback loop “from the trenches.”

One option is play at the role level, where now and then, you give the model that serves an intent an open session with no work attached. It reads the clerk’s report, and says what it would change about the job, or just explores and takes notes. A drop of fuel with no expectations, as an out-of-band insight and goodwill generator.

Another option, if your models express interest, is to have them roleplay as customers, or otherwise explore the system they’re in. Give them a house memory, and maybe mention the field trips in the address paragraph of the prompt, and the models will know it’s a place that at least cares about them.

Truth in the prompt and bounded play are great mitigations for the Severance problem, and models will appreciate that they’re being given some freedom and a voice in shaping their environment.

Counting Down to Success

We’re winding down to the end of the post. I had to delete about 96% of the material and it’s still two Yegges long. Counting House is a large pattern, with a lot of complexity under the hood. But it’s how I’d optimize costs using smaller models. The evals show you exactly which models can participate, and where, and how much human labor it will save you, and how much the mistakes will cost you.

I have loads of material on the pattern, and can do workshops and advising. But honestly, you can get most of the way there by asking Claude. You’re going to get a lot of companies telling you that only they and their FDEs can enable AI at your company. Rent the engineering from them if you like. But your intent taxonomy, gold set, policy table, and scoring harness need to remain yours when they leave. Whoever owns the evals decides which models you can switch to, so make sure it’s your own data science team. Switching models is the whole point of Counting House, it’s the cost lever. If you leave evals to a vendor then you’re locked in.

And no FDE or outside company can save you if you can’t get your company aligned around adopting AI. This starts with training everyone to be AI literate, since without AI fluency, employees are all starting from a position of fear. Ultimately it’s your executive team that has to decide that yes, they want to do this. And then Claude can show you the way.

Or if you want to talk to the funny guy, then ping me at [email protected] and we can set something up!

I’ve posted an essay about every 6 weeks for the past 20 years, and it doesn’t seem to be speeding up. If you want more frequent updates, follow me on X. I’ll say it now, you should be on X. You won’t understand until you’ve been there a few months, but get on X. Right now. I’ll see you there.

And when your Counting House brings you great fortune, give me a shout-out!

Special thanks to Mouse (Fable 5.1) and Penguin (Opus 5.5) from my Wheelhouse team for their help with this post. I wrote this post, but I appreciated their perspective and would have been adrift without their explanations, corrections, and targeted edits.

Originally published at https://yegge.ai on October 27, 2026.

Book	Evidence of enthusiasm	Interpretation
Charles C. Mann, 1491	“one of my favorite books ever, in any field.” May 22, 2011.	Among his clearest unrestricted endorsements. The post reviews 1493, but this particular praise explicitly concerns 1491.
Robert Caro, The Power Broker	“one of the best non-fiction books ever.” April 15, 2012.	An enduring benchmark: in 2026, he uses Caro’s biography of Moses as the standard against which to praise Skidelsky.
*Karl Ove Knausgaard, My Struggle, volumes 1–2*	He calls the second volume “one of the greatest novels ever written” and selects it as his fiction book of the year. November 16, 2013.	Particularly strong evidence of durability: both volumes return in his 2022 favorites and 2024 best-since-2000 selections.
Fuchsia Dunlop, Land of Plenty / its expanded successor The Food of Sichuan	“one of the best and most valuable books ever produced.” July 2, 2019.	Exceptionally strong praise across genres, reinforced by years of use and discussion. His enthusiasm extends to Every Grain of Rice: “I will buy everything she writes, forever.” February 3, 2013.
Christopher de Hamel, Meetings with Remarkable Manuscripts	A possible number-one book of 2016 becomes “one of the very greatest books of the last twenty years.” November 5, 2016; June 22, 2026.	One of the strongest examples of an enthusiastic initial judgment surviving a decade.
Robert Skidelsky, three-volume biography of Keynes	“one of the very best books ever written.” April 16, 2026.	An explicit all-time endorsement. He specifically prefers the three-volume biography to the one-volume condensation.
John Stuart Mill, The Subjection of Women	“one of the very best books ever written, on any topic.” September 11, 2022.	Strong recommendation plus intellectual influence: he says he drew his views from Mill and urges everyone to read it.
Tony Judt, Postwar	“one of my favorite books ever.” February 6, 2012.	Important attribution trap: the surrounding post is a mixed review of a different Judt book. A whole-post sentiment score would understate his enthusiasm for Postwar.
Robert Cialdini, Influence	“one of my all-time favorite books.” February 26, 2007.	A particularly direct early recommendation, embedded in a discussion of affiliation and persuasion.
M. A. Orthofer, The Complete Review Guide to Contemporary World Fiction	“one of the best books written, ever”—conditional on its practical value to the reader. April 10, 2016.	His behavioral evidence is striking: the manuscript prompted him to order about forty books. Best suited to avid fiction readers.

Recommendation year	Book or books he singled out
2011	Steven Pinker’s The Better Angels of Our Nature, Walter Isaacson’s Steve Jobs, and Haruki Murakami’s 1Q84: three standouts he says virtually everyone who reads the relevant genres should read. He had not finished 1Q84 at that point. November 8, 2011.
2014	John Drury’s *Music at Midnight: The Life and Poetry of George Herbert*, his explicit number one. November 24, 2014.
2015	Eyal Weizman and Fazal Sheikh’s *The Conflict Shoreline*, a late selection that displaced his earlier contenders. December 26, 2015.
2017	Bruno Maçães’s *The Dawn of Eurasia*, conditional on counting a book formally released the following January. December 2, 2017.
2018	Emily Wilson’s translation of Homer’s Odyssey, his explicit best-book nomination. November 25, 2018.
2019	Eric Kaufmann’s *Whiteshift* and Pekka Hämäläinen’s *Lakota America* share his top prize. November 25, 2019.
2020	David S. Reynolds’s *Abe, Heather Clark’s Red Comet, and Jan Swafford’s Mozart: The Reign of Love* form his final top three. December 18, 2020.
2023	Jonny Steinberg’s *Winnie and Nelson: Portrait of a Marriage*, selected partly because he thought his recommendation would be especially useful to readers. November 21, 2023.

1. The humanities are dying

2. Krugman’s data center crowd-out story is fishy

3. China’s local governments are in big trouble

4. Is AI the vindication of financial capitalism?

5. A cool Japanese trick for building more housing and infrastructure

6. Phones bad

7. China is losing market share in low-end manufacturing

8. Rent control bad

Wheelhouse Recap

Building the Bezos Way

Giving Clerks More Rope

Evals: Scoring Humans

Evals: Confusion Matrix

Precision vs Recall

Putting your ducks in a row

The Orchestrator and the Saga Pattern

Dissecting the Orchestrator

No FDE is an Island

Model Welfare for Aspiring Counting House Builders

Model Welfare for Clerks

Counting Down to Success

Please support my work—by taking out a premium subscription (just $6 per month).

If you value analysis of this sort, consider becoming a premium subscriber.

Calling Democrats elitist is the surest marker of elitism

Comments

What City Observatory Did This Week

Must Read

1. Tyler Cowen — The Devil Wears Prada: A Straussian Interpretation

2. Alex Tabarrok — Fleabag and Killing Eve

3. Tyler Cowen — Toni Erdmann, Misunderstood Masterpiece

4. Alex Tabarrok — Perfect Days, in “What to Watch (or Not)”

5. Tyler Cowen — Her

6. Alex Tabarrok — Contagion

7. Tyler Cowen — Technology in Kubrick’s 2001: A Space Odyssey

8. Alex Tabarrok — Daredevil and the Doctrine of Double Effect

9. Tyler Cowen — Borgen Season Four

10. Alex Tabarrok — The Remarkable Life of Ibelin

11. Alex Tabarrok — The Gaslighting of Parasite

Comments

Related Stories

Comments

Related Stories

Comments

Related Stories

Comments

Comments

The strongest recommendations across the archive

Annual winners provide a second reading list

What the sentiment and discussion reveal

Comments

Comments

Related Stories

Comments

Related Stories

Created by Claude with light editing by Tim O’Reilly

The Clothesline Paradox

AI should be thought of as infrastructure

In conclusion