Broadsheet

Ad by Proctor Silex via Wikimedia Commons

“Her belly may be full, but her spirit will be empty.” — Captain Picard

Usually, these “GDP is actually good” posts start with a big disclaimer — an acknowledgement of all the things GDP doesn’t measure, all the reasons that measuring GDP is an inexact science, and all the ways that we need to improve society other than just making the GDP line go up. If you want a standard wonky explanation of why GDP is a useful number, here’s one that I wrote four years ago:

Today I’m going to do something a little different. I’m going to tell you what I think the debate over GDP is really about.

Free trade usually raises GDP. Immigration, done right, raises GDP.1 Rightists in America want less free trade and less immigration. But every time they propose restricting trade and immigration, someone — either libertarian business/econ types on their own side, or moderate liberals on the other side — says “That will make America poorer!”. So they want some way to neutralize this objection, so they can do things that will, in fact, make America poorer.

So America’s right borrowed an argument from the European left. The European left favors degrowth, and another term for degrowth is “making GDP go down on purpose”. So naturally, they’re always trying to find reasons to denigrate GDP as a metric of human flourishing (see here, here, and here for examples). The American right is simply tweaking these arguments to make them more appealing to their own base.

JD Vance, who has emerged as the consensus leader of the New Right, makes a bunch of these anti-GDP arguments in his new book. For example, he uses the example of Japan to point out that unobserved quality differences in non-traded products can make it difficult to compare GDP across countries:

If you’re focused on GDP, a $6 pint of Japanese strawberries is no different from a $6 pint of American strawberries. If you’re focused on dollars and cents, each contributes equally to the economic indicators. But if everyone in Japan eats better strawberries than everyone in America, the economic indicators have failed to measure something meaningful.

This is actually a good argument, and I’ve made it myself many times in the past. I’m in Japan right now, and there actually are a lot of little things that make Japanese products and services a bit nicer than their American counterparts — clean tables at Starbucks, slightly better-tasting food, and so on. Economists who try to adjust for quality differences end up catching some of these things, but probably miss most of them. That ends up creating a problem for GDP comparisons between countries. And it’s only one of many such problems. Comparing lifestyles in countries where life is very different is just a difficult thing to do.

But instead of simply noting that economics is hard, JD Vance uses this good argument as a reason to bash the entire field of economics:

When I got back home, a friend asked me if I learned anything on my trip to Japan. “Yes,” I replied snidely. “Maybe economics is just fake.”

When you read some of Vance’s other arguments against GDP, his agenda becomes clearer:

[A]s the decline of Christianity has left us without a shared moral language, economics has stepped into the vacuum. We pretend there are scientific answers to questions of values. Take one of the major issues of the 2024 campaign and a significant focus of our time in the White House: Should our trade policy be oriented around protecting domestic industries and jobs or around ensuring a short-term supply of cheap consumer goods?

This idea — that economists urge values of base consumerism on society, and ignore other moral considerations — is common in European leftist discourse. But instead of urging us to care more about inequality, power, and so on, as European leftists do, Vance wants us to care more about spiritual elevation, morality, community — i.e., things that the American right cares about. He goes on to write:

[W]e now live in a society almost blinded to considerations outside of the economic. This way of thinking is inherently opposed to the Christian way, which demands more focus on people…Take, for instance, the time we spend with our children…Domestic labor—that done by moms and dads—if unpaid, is uncounted in measures like GDP. When I leave work to spend time with my children, when I cook them dinner or argue with them about eating their carrots, I am engaged in economically unproductive work. No money changes hands, so it doesn’t show up in our national figures. By contrast, if I left for dinner at 6 p.m. and returned to work until midnight while paying a total stranger to look after my kid, my contribution to GDP would be much higher.

and:

If you step away from the glory of economic statistics, so much of American life has gone wrong. An influx of prescription opioids became a flood of synthetic opioids, which has led to tens of thousands of deaths each year and a declining life expectancy among a substantial portion of our society. We have made great progress on reducing infant mortality, but we send our children into a world—even in the physical security of their own homes—that bombards them with images and influences that have left them isolated, depressed, and increasingly at risk of self-harm. We are more disconnected, lonely, and isolated, even in the midst of historic levels of material comfort.
All this economic abundance coexists with intense spiritual misery. We orient people toward a life of consumption. We tell them to find meaning in the home they buy, the money they earn, the prestige of their job. We bombard them with all manner of creature comforts, and add their consumption—price club mega-size junk—to our national GDP. We use that GDP as a yardstick for our broader society, which is why it’s possible for false prophet economists to argue the American Dream is healthy even as suicide and addiction rates soar and the laughter of children fades from our streets.

Some of these arguments are — in my opinion — reasonable. A culture of overwork can boost GDP, at least in the short term, at the expense of quality time with family. This is actually a common argument of the center-left, which is why liberals have long fought — often successfully — for more paid family leave and other policies that reduce GDP slightly in exchange for more quality time with family. Whether this has increased birth rates isn’t clear — the evidence is very mixed — but it’s a very popular policy.

Other arguments are clearly mistaken. Over-prescription of opioids has clearly reduced GDP, by a massive amount. Yes, selling a bunch of opioid painkillers to Americans raises GDP by a few billion dollars, but this is vastly outweighed by the trillions of dollars of GDP that we lose from having a bunch more people addicted to painkillers, heroin, and fentanyl. Here’s the Philadelphia Fed in 2023:

There is growing evidence that the opioid epidemic has harmed many aspects of the real economy, including the labor market, consumer finance, and municipal finance. According to analyses from the Council of Economic Advisers’ 2019 report, the annual (nominal) economic cost of the opioid epidemic, including the cost of lives lost, is estimated at about $700 billion (roughly 3.4 percent of GDP) in 2018 alone, and over $2.5 trillion from 2015 to 2018. [emphasis mine]

So if you care about GDP, you should view curbing opioid and opiate abuse as a huge priority! Vance is simply not thinking about this very clearly.

But Vance’s real problem is that he conflates correlation with causation. In words that could have come straight from the mouth of a European degrowther, he rails against “creature comforts”, “consumption”, and “price club mega-size junk”. But nowhere does he explain why depriving Americans of these creature comforts would give them closer-knit families, a stronger sense of morality, stronger communities, reduced loneliness, and so on.

Why would taking away Americans’ large houses, SUVs, big-screen TVs, or central air conditioning make them spiritually richer? Modern Europe — which JD Vance spends much of his time railing against — lacks most of these things. And yet America has higher fertility rates than Europe, we go to church much more, and we have a much more robust social conservative movement. Europe has also been far more restrictive of speech that criticizes Islam, as Vance repeatedly notes. Yes, America has been trending away from social conservatism and Christianity in recent decades, but so has Europe, and the gap remains. Other developed countries in East Asia — most of which are moderately poorer than the U.S. in GDP terms — are extremely secular.

What about America’s past? We were much poorer in the 1950s, yet we went to church a lot more, had larger families, and so on. If you took away the material gains we’ve made since then, would we go back to tradwives and bowling leagues and lawn parties and Sunday church and 4 kids per family?

Perhaps, but it’s doubtful. Remember that the 1950s and 1960s were the culmination of a long upswing of community, religiosity, and so on in American society — something the sociologist Robert Putnam has documented extensively. Church attendance rose:

Fertility was on the upswing too:

And if you believe Putnam’s numbers, social solidarity increased all throughout the early 20th century:

Source: Robert Putnam via Jefferson Educational Society

It’s important to remember that this all came during the most robust and rapid period of GDP growth that America has ever seen. Over the period in which our social solidarity was soaring, our GDP per capita nearly tripled:

It was during this time that Americans got many of the “creature comforts” that Vance despises — the single-family homes, the cars, the televisions, the lawns, and so on. The image of that material prosperity, depicted in glossy ads and paintings from the time, is a powerful part of 1950s nostalgia.

In fact, many economists argue that one big cause of the Baby Boom was the fact that economic growth — bigger houses, better medical care, new labor-saving devices like washing machines, refrigerators, and vacuums, and so on — made it easier and cheaper to raise kids. This is from a relevant Works in Progress article by Anvar Sarygulov & Phoebe Arslanagic-Little:

Parenthood rapidly became much easier and safer between the 1930s and 1950s. The spread of labour-saving devices in the home such as washing machines and fridges made raising children easier; improvements in medicine making childbirth safer; and easier access to housing made it cheaper to house larger families…
[H]ousehold electrification paved the way for other technologies, including home refrigeration…By the 1940s, electric washing machines were becoming normal in middle class homes…Between 1936 and 1956, America’s maternal death rate fell by 94 percent, from 51 deaths per 10,000 live births to under 3…[M]edical advances, which were being made across the West, radically reduced the most serious potential cost faced by prospective mothers: life itself…
Alongside strides forward in household and medical technology…[I]t became easier to secure a home in which to raise children. The number of houses built soared across the West after World War Two…This house-building bonanza led to sharp rises in homeownership rates.
The golden thread linking the phenomena that comprise the triple mechanism we describe above – advances in household technology, progress in medical technology, and easier access to housing – is that they together sharply reduced the cost of having children. [emphasis mine]

If you like the kind of society we had in 1960, you can’t ignore the story of how we got to 1960. The answer was “economic growth”. This, combined with the examples of Europe and Asia, is why there’s no reason to believe that forcing Americans to be poorer — taking away the “creature comforts” Vance despises — would lead us to suddenly rediscover the value of community, family, and religion.

Now it’s worth noting that if you were to decrease America’s GDP to poor-country levels — below $15,000 per person, as opposed to over $90,000 today — you might be able to raise fertility. That’s how low you have to go before most countries have fertility above replacement level:

Chart by Mikael Häggström via Wikimedia Commons

Countries with a basically pre-modern standard of living — where many women can’t read or write, and infant mortality is so high that families have to have many kids as a form of insurance — tend to have above-replacement fertility (though some don’t). But even this law is weakening, as fertility rates in Sub-Saharan Africa plunge, so even that extreme level of GDP reduction would probably fail to restore high fertility over the long run. Also, I kind of doubt that JD Vance wants to force Americans to live lives similar to those lived in Sub-Saharan Africa.

What about JD Vance’s preferred policies — trade protectionism and immigration reduction? Would those restore American community, family, and religion, at the expense of a bit of GDP? As I said earlier, that’s actually what I think this whole debate is really about.

On trade, you’ve seen Trump explicitly make the argument that Americans are going to need to suffer a bit of material deprivation in order to achieve the administration’s goals:

But how will doing this restore community, family, etc.? Presumably you could make an argument that protectionism will bring back good manufacturing jobs, which will then give men the confidence and social standing they need to get married and have kids. There’s just one big problem with this, though: Trump’s trade policy doesn’t actually increase the number of good manufacturing jobs. We’ve lost manufacturing jobs since Trump took office last year!

In fact, Trump’s tariffs are hurting the U.S. manufacturing sector, by raising the cost of intermediate goods. Economists understand this pretty well; JD Vance, who thinks “economics is just fake”, does not seem to understand it.

How about immigration? Rightists will endlessly cite Robert Putnam’s finding that diversity reduces social trust in American communities. But as Bryan Caplan and many others have pointed out, the effect size is tiny — in Putnam’s research, going from zero diversity to maximum diversity reduces social trust by the equivalent of 1 point on a 100-point scale. This suggests that all the mass deportations in the world won’t move the needle on American community and togetherness.

In other words, JD Vance’s crusade against GDP is a cargo cult. Sure, GDP doesn’t measure “the beauty of our poetry or the strength of our marriages,” to use Robert F. Kennedy’s famous words. But that doesn’t mean that making Americans poorer will make their poetry more beautiful or their marriages stronger. Nor does it mean that policies that also happen to make us a bit poorer, like immigration reduction or tariffs, are any more likely to strengthen our society.

In a famous episode of Star Trek: The Next Generation — my favorite TV show of all time — Captain Picard castigates an alien for turning his society into a fascist empire. When the alien responds that the fascist government’s forced modernization program raised his daughter out of poverty, Picard responds with the beautiful quote at the top of this post: “Her belly may be full, but her spirit will be empty.” But was Picard arguing that it was the full bellies themselves that emptied the people’s spirits? Was he merely arguing that the fascist empire ought to become a poorer fascist empire, in order to restore the virtue of the people? Only a fool would think so.

Look, I also want America to have a stronger society. I want us to have more kids. I want us to have more stable families. I want us to have closer-knit communities, better moral values, etc. But that doesn’t mean the New Right knows how to get us there. So far, the New Right has built nothing — no new community organizations or institutions, no religious revival, nothing that would knit our society together. It has merely thrashed and thrashed against modernity, with no plan for a replacement. JD Vance’s crusade against GDP is simply more of the same.

Subscribe now

“Done right” means getting immigrants who, on average, earn more than the native-born. This raises GDP by a composition effect — you have richer people on average. It also probably raises GDP by other means — increasing market size which increases returns to scale, boosting innovation and entrepreneurship, and so on. If you get mostly low-skilled immigrants, things get dicier — the composition effect reduces GDP because you’re importing poorer people, but the increased market size may still cancel that out. But in general, if you want higher per capita GDP, you should be selective in terms of who you let into the country. Total GDP, of course, is a different matter — if you want a bigger country, in order to be more powerful in military terms, then letting in tons of low-skilled immigrants may be worth it even if they reduce per capita GDP. And of course, there are distributional reasons to allow in low-skilled immigration — eldercare and so on. But basically, if you let in high-skilled immigrants, your society gets richer by pretty much every metric.

This morning I released sqlite-utils 4.0, the 124th release of that project and the first major version bump since 3.0 in November 2020. In addition to some small but significant breaking changes (described in this upgrade guide), this version introduces three major features: database migrations, nested transactions (via a new db.atomic() method), and support for compound foreign keys.

Database schema migrations using sqlite-utils

Schema migrations define a sequence of changes to be made to a SQLite database, plus a mechanism for tracking which migrations have been applied and applying any that are found to be pending.

Migrations are defined in Python files using the sqlite-utils Python library, which includes a powerful table.transform() method providing enhanced alter table capabilities that are not supported by SQLite's ALTER TABLE statement.

(table.transform() implements the pattern recommended by the SQLite documentation - create a new temporary table with the new schema, copy across the data, then drop the old table and rename the temporary one in its place.)

Here's an example migration file which creates a table called creatures, adds an additional column to it in a second step, then changes the types of two of the columns in a third:

from sqlite_utils import Migrations

migrations = Migrations("creatures")

@migrations()
def create_table(db):
    db["creatures"].create(
        {"id": int, "name": str, "species": str},
        pk="id",
    )

@migrations()
def add_weight(db):
    db["creatures"].add_column("weight", float)

@migrations()
def change_column_types(db):
    db["creatures"].transform(types={"species": int, "weight": str})

Save that as migrations.py and run it against a fresh database like this:

uvx sqlite-utils migrate data.db migrations.py

Then if you check the schema of that database:

uvx sqlite-utils schema data.db

You'll see this SQL:

CREATE TABLE "_sqlite_migrations" (
   "id" INTEGER PRIMARY KEY,
   "migration_set" TEXT,
   "name" TEXT,
   "applied_at" TEXT
);
CREATE UNIQUE INDEX "idx__sqlite_migrations_migration_set_name"
    ON "_sqlite_migrations" ("migration_set", "name");
CREATE TABLE "creatures" (
   "id" INTEGER PRIMARY KEY,
   "name" TEXT,
   "species" INTEGER,
   "weight" TEXT
);

The _sqlite_migrations table is used to keep track of which migration functions have been run. The creatures table above is the schema after all three migrations have been applied.

To see a list of migrations, both pending and applied, run this:

uvx sqlite-utils migrate data.db migrations.py --list

Output:

Migrations for: creatures

  Applied:
    create_table - 2026-07-07 17:58:41.360051+00:00
    add_weight - 2026-07-07 17:58:41.360608+00:00
    change_column_types - 2026-07-07 18:01:15.802000+00:00

  Pending:
    (none)

If you don't specify a migrations file, the sqlite-utils migrate data.db command will scan the current directory and its subdirectories for files called migrations.py and apply any Migrations() instances it finds in them.

You can also execute migrations from Python code using the migrations.apply(db) method, which is useful for building tools that manage their own database schemas over multiple versions. My own LLM tool has been using a version of this pattern for several years now, as shown in llm/embeddings_migrations.py.

Prior art

My favorite implementation of this pattern remains Django's Migrations, developed by Andrew Godwin based on his earlier project South. Fun fact: Andrew, Russ Keith-Magee, and I presented our competing approaches to schema migrations for Django on the Schema Evolution panel at the very first DjangoCon back in 2008! My attempt was called dmigrations, developed with a team at Global Radio in London.

Django's migrations can be automatically generated from model definitions and include the ability to roll back to a previous version. The sqlite-utils approach is deliberately simpler: unlike Django, sqlite-utils encourages programmatic table creation rather than a model definition ORM, so there isn't anything we can use to automatically generate migrations.

I decided to skip rollback, since in my experience it's a feature that is rarely used. With a SQLite project, an easy way to achieve rollback is to create a copy of your database file before you apply the migrations!

Migrating from sqlite-migrate

The design of sqlite-utils migrations is three years old now - I had originally released it as a separate package called sqlite-migrate, which never quite graduated beyond a beta release.

I've used that package in enough places now that I'm confident in the design, so I've decided to promote it to a feature of sqlite-utils to make it available by default to all of the other tools in the growing sqlite-utils/Datasette/LLM ecosystem.

I made one last release of sqlite-migrate, which switches it to depend on sqlite-utils>=4 and replaces the __init__.py file with the following:

from sqlite_utils import Migrations

__all__ = ["Migrations"]

Any existing project that depends on sqlite-migrate should continue to work without alterations.

Everything else in sqlite-utils 4.0

Here are the release notes for this version, with some inline annotations:

The 4.0 release includes some minor backwards-incompatible fixes (hence the major version number bump) and introduces three major new features:

Database migrations, providing a structured mechanism for evolving a project’s schema over time. (#752)

I think of migrations as the signature new feature, hence this blog post.

Nested transaction support via db.atomic(), plus numerous improvements to how transactions work across the library. (#755)

sqlite-utils has long had a confused relationship with database transactions, partly because when I started designing the library back in 2018 I didn't yet have a great feel for how those worked in SQLite itself.

Adding migrations to the core library made me determined to finally crack this nut, since transactions make migration systems a whole lot safer and easier to reason about.

I ended up building this around a db.atomic() context manager which looks like this:

with db.atomic():
    db.table("dogs").insert({"id": 1, "name": "Cleo"}, pk="id")
    db.table("dogs").insert({"id": 2, "name": "Pancakes"})

SQLite supports Savepoints, and as a result db.atomic() can be nested to carry out transactions inside of transactions. It's pretty neat!

Support for compound foreign keys, including creation, transformation and introspection through table.foreign_keys. (#594)

This came about when I asked a coding agent to review all open issues and PRs for things that should be included in a 4.0 release since they would represent breaking changes if I added them later, and it correctly identified that compound foreign keys were exactly that kind of feature.

I started with a breaking change to the table.foreign_keys introspection method, and then decided to see if Claude Fable 5 could handle the more fiddly job of integrating compound foreign key creation into the library. The API design it helped create felt exactly right to me - consistent with how the rest of the library worked already.

Other notable changes include:

Upserts now use SQLite’s INSERT ... ON CONFLICT ... DO UPDATE SET syntax, detect existing table primary keys automatically and reject records that are missing required primary key values. (#652)

This was the change that first pushed me to consider a breaking-change 4.0 version bump. I built this to help support sqlite-chronicle, which uses triggers to keep track of rows in a table that have been inserted, updated or deleted.

db.query() now executes immediately and rejects statements that do not return rows; use db.execute() for writes and DDL.

Probably the most disruptive breaking change - I've had to update a few places in my own code to switch from db.query() to db.execute() as a result.

CSV and TSV imports now detect column types by default, while inserts into existing tables preserve those tables’ column types. (#679)

The sqlite-utils insert data.db creatures creatures.csv --detect-types flag was a later addition to allow column types (text, integer, real) to be automatically detected based on the data in a CSV. It should be the default, and releasing a 4.0 means I can make it so.

table.extract() and extracts= no longer create lookup table records for all-null values. (#186)

The oldest issue addressed by this release - the underlying bug was opened (by me) in October 2020.

See Upgrading from 3.x to 4.0 for details on backwards-incompatible changes.

The detailed release notes for the features and fixes shipped during the 4.0 pre-release cycle are available in 4.0a0, 4.0a1, 4.0rc1, 4.0rc2, 4.0rc3 and 4.0rc4.

The upgrade guide was entirely written by Claude Fable 5, Claude Opus 4.8 and GPT-5.5. The same is true of the release notes.

This is the kind of documentation I've slowly become comfortable outsourcing to the robots. It doesn't need to convince people of anything, or express any opinions - its job is to be as accurate and detailed as possible. I've reviewed the release notes closely and can confirm they are accurate and comprehensive.

Claude Fable 5 helped a lot

I released the first alpha of sqlite-utils 4.0 over a year ago. I've been dragging my heels on the stable release because of the amount of work it would take to track down and clean up the many other minor design flaws that a major version number allowed me to take on.

Assistance from Claude Fable 5 (and to a lesser extent Opus 4.8 and GPT-5.5) gave me just the boost I needed to overcome inertia and make the most of the time I could afford to spend on this library.

Fable has really good taste in API design, and is relentlessly proactive if you give it a more open goal. My most successful prompt was a review task that I issued against what I thought was the last release candidate:

review the changes on main since the last tagged 3.x release - I am about to ship them as sqlite-utils 4.0, a stable version that promises no backwards-incompatible fixes for a very long time.

review the changelog and upgrade guide, and write yourself scratch scripts to try out all of the new features in v4 - save those scripts but don't commit them

I tried this with GPT-5.5 xhigh in Codex Desktop and Fable 5 in Claude Code.

GPT-5.5 wrote 5 Python scripts and didn't turn up anything particularly interesting - its final report is here.

Fable 5 wrote 12 scripts, identified 4 release blockers and 10 additional issues in its report, and built a neat combined repro script, which, when run, output the following:

=== 1. Failed db.execute() write leaves an implicit transaction open ===
  in_transaction after failed write: True
  BUG: table 'other' silently lost when connection closed

=== 2. Leading ';' bypasses the query() first-token scanner ===
  BUG: raised OperationalError: no such savepoint: sqlite_utils_query
  BUG: row persisted despite rollback (count=1)

=== 3. Rejected write PRAGMA via query() still takes effect ===
  BUG: user_version=5 after 'rejected' statement (docs say no effect)

=== 4. Implicit compound FK resolves pk columns in table order, not PK order ===
  BUG: other_columns reported as ('b', 'a'), should be ('a', 'b')
  BUG: transform of valid data raised IntegrityError: FOREIGN KEY constraint failed

=== 5. ForeignKey (now a dataclass) is no longer hashable ===
  BUG: cannot use 'sqlite_utils.db.ForeignKey' as a set element (unhashable type: 'ForeignKey')

=== 6. Mixed ForeignKey objects and tuples in foreign_keys= rejected ===
  BUG: foreign_keys= should be a list of tuples

=== 7. insert --csv into an EXISTING table transforms its column types ===
  BUG: existing zip '01234' is now 1234 (column type: int)

=== 8. insert(pk=, alter=True) regression: InvalidColumns before alter runs ===
  BUG: InvalidColumns: Invalid primary key column ['id'] for table t with columns ['a']

=== 9. migrate --stop-before an already-applied migration applies everything ===
  BUG: m2 was applied despite --stop-before m1 (m1 already applied)

=== 10. ensure_autocommit_on() silently commits an open transaction ===
  BUG: row survived rollback (count=1) - transaction was committed

I found myself agreeing with almost all of them. Here's the PR with 16 commits where we worked through them in turn.

There's no doubt in my mind that sqlite-utils 4.0 is a significantly higher-quality release than if I had built it without the assistance of the latest frontier models.

Tags: schema-migrations, projects, sqlite, ai, sqlite-utils, annotated-release-notes, generative-ai, llms, ai-assisted-programming, anthropic, claude, agentic-engineering, claude-mythos-fable