Thousands of people gathered outside the John F. Kennedy Center for the Performing Arts Friday evening to protest Trump’s takeover of the building and his threat to shut it down or even demolish it unless his name goes onto the facade. U.S. District Judge Christopher Cooper of the District of Columbia has ordered the board of the center to give 30 days’ notice before making any major physical changes to the building.

Recognizing the leeway the courts have given him over anything that falls under “national security,” Trump this morning decided to take a proactive approach to making sure the courts don’t stop his triumphal arch. He announced on social media:

“At the strong request of the United States Military, and for National Security purposes, I have agreed to convert the magnificent Triumphal Arch, planned since the Civil War Era many years ago, at the Receptive Circle adjoining the Arlington Memorial Bridge, into a top grade Military Complex/Triumphal Arch, to house, store, and have the rapid ability to use large numbers of drones, plus Snipers, on both the roof and plaza areas, and additionally have and hold large quantities of sniper ammunition in storage. There will be no facility like this anywhere in the World. Out of the 59 top cities and Capitals, Washington, D.C., IS THE ONLY ONE IN THE WORLD THAT DOES NOT HAVE A TRIUMPHAL ARCH, but it will now and, by far, the greatest of them all!”

This is the same justification he made for his ballroom where the East Wing used to be. When it became clear that his claim to need an entertainment space up to his aesthetic standards would not fly, he began to claim that the ballroom is imperative for national security. We have since learned that there already is a secure space under the White House, built in secret during President Barack Obama’s administration.

Dan Diamond reported in the Washington Post last Wednesday, September 16, that officials from the National Park Service have agreed that the construction will adversely affect Arlington National Cemetery, the Lincoln Memorial, and several dozen other historical sites, but urged those concerned about historic preservation to sign on to the project through a memorandum of agreement that will stand as approval by noon tomorrow anyway.

The document asserts that the arch is supposed “to celebrate 250 years of American independence by honoring America’s founding principles.” Now Trump says the arch is imperative for “National Security purposes” and will have drones and snipers.

Rebecca Miller, executive director of the DC Preservation League, responded to the request to sign the document: “This is not consistent with NPS practice, NPS regulations, or the intent of the National Historic Preservation Act.”

While Trump suggests he will defend the United States from atop his triumphal arch, our European allies are much more concerned with Russia’s drone strike last Sunday on a passenger train traveling from Kyiv, Ukraine, to Warsaw, Poland. The train was just over a mile—two kilometers—from the Polish border when the drone hit at a border crossing. The drone hit just after a convoy carrying former leaders and high-level diplomats, including former U.K. prime minister Boris Johnson, had passed through.

The train operators were warned of the coming drone and no one was hurt, but European leaders widely interpreted the strike so close to Poland as, in the words of Kaja Kallas, the European Union’s high representative for foreign and security policy, “clearly a message to the western world.”

Poland is a member of the North Atlantic Treaty Organization (NATO). Twelve nations founded NATO in 1949 to resist Soviet aggression. After the Soviet Union collapsed in 1991, NATO’s mission shifted toward crisis management, and then to resisting the aggression of Russia’s president Vladimir Putin.

Over the years, NATO has grown to include 32 countries, 30 of which are in Europe (the U.S. and Canada are the two NATO countries in North America). The success of their collective defense system rests on Article 5 of the North Atlantic Treaty, which states that an attack against any one member of NATO will be treated as an attack against all of them. So a threat to Poland threatens all the NATO countries.

Russia has been pushing NATO countries with what European leaders call “hybrid attacks.” Rather than launching classic military attacks, Putin has been allowing drones to go into NATO airspace, cutting undersea telecommunications networks on which European countries depend, and possibly placing bombs on a train in France.

Russia has been a bad actor for years, destabilizing NATO countries by flooding them with disinformation that affects elections, paying for political parties, bribing politicians, and so on. But the increase in Russian provocations now seems to be related to Putin’s attempt to undercut European support for Ukraine as Ukraine’s drones have brought the war home to Russia’s oil industry and to the Russian people.

In a speech on Wednesday, NATO Secretary General Mark Rutte, former prime minister of the Netherlands, noted that “Russia wants to stop us helping Ukraine—but their actions will only lead us to do more for Ukraine.”

Kurt Volker, who served as U.S. ambassador to NATO under President George W. Bush, told NPR’s Elissa Nadworny that President Trump “has largely looked at all of this and said, that’s a European problem. He doesn’t feel a threat to the United States from Russia, and he doesn’t see a need for the U.S. to increase its pushback on Russia at all.” “I disagree with this,” Volker said. “I think that we have a vested interest in freedom and free people everywhere being safe and prosperous. It creates a better world for us.”

Trump’s stand is interesting in light of news that Russia has been helping Iran in its fight against the United States, sending upgraded munitions as well as satellite imagery and targeting data to hit U.S. forces in the Middle East.

Last week, CNN reported that fourteen Russian spy satellites moved over the Prince Sultan Air Base in Saudi Arabia, used by the U.S., two days before a devastating March 27 attack by Iran. A CNN investigation showed that the spacecraft had access to a large spectrum of data. U.S. officials and international intelligence sources told CNN that the information the satellites gathered helped Iranian forces conduct some of its most precise attacks against American forces and assets.

Two months later, a Russian close to Putin footed the bill for a three-day celebration for Donald Trump Jr.’s wedding to Bettina Anderson. According to Justin Elliott, Brett Murphy, Joshua Kaplan, and Alex Mierjeski of ProPublica, Umar Kremlev spent hundreds of thousands of dollars on the event, held on private islands in the Bahamas.

On Tuesday, September 15, the Department of Justice announced charges against five people working for Russian intelligence “to conduct attacks and murders around the world, including in the United States.” The indictment alleges that a network of Russian hit men has operated since at least 2024 and has recruited U.S. citizens to surveil and murder people, intending to target “all the countries that are helping Ukraine.” As Daniel Boffey and Andrew Roth of the Guardian note, it was an unwritten rule of the Cold War that Russia would not attack people in the U.S. (although Russian agents are widely believed to be behind the murder of U.S. journalist Paul Klebnikov in Moscow in 2004).

On September 16, Congress passed a major sanctions package targeting Russian officials, the energy sector, financial institutions, and the “shadow fleet” of tankers that get around sanctions by using shell corporations and flying under the flags of countries with a loose regulatory system. The measure is named the Lindsey O. Graham Sanctioning Russia and Iran Act of 2026, in honor of the late South Carolina senator.

But on Thursday, September 17, Anton Troianovski of the New York Times reported that Trump is suggesting Putin will be more willing to end his war if the U.S. makes business deals with Russia regardless of whether it stops its assault on Ukraine. In February, Troianovski reported that Gentry Beach, a Texas investor with ties to the Trumps, signed an agreement last fall with one of the biggest energy companies in Russia to develop Alaska’s natural gas resources.

According to a September 14 piece by Dexter Filkins in the New Yorker, the Trump administration has essentially privatized U.S. foreign policy for their own interests. The “generally held view,” a former U.S. official told Filkins, is that Russia has offered administration officials and their allies business deals in exchange for their supporting Russian demands against Ukraine. A classified report a European intelligence agency provided to Filkins lists Trump, his son-in-law Jared Kushner, special envoy Steve Witkoff, former envoy Richard Grenell, and Gentry Beach as people who might participate in any of ten money-making ventures on the table.

The world, and the place of the United States in it, is changing fast. In June, Defense Secretary Pete Hegseth called it “shameful” that European allies did not help the U.S. attack on Iran and said the U.S. would review its military commitments to Europe. On September 17, Dan De Luce, Gordon Lubold, Courtney Kube, and Abigail Williams of NBC News reported that the Pentagon is weighing a plan to pull tens of thousands of troops and weapons, ships, and planes out of Europe despite the warnings of NATO allies about Russia’s increasingly aggressive moves.

In Wednesday’s speech, Rutte outlined what he called NATO 3.0. If NATO 1.0 was the phase of the organization that pushed back against the Soviet Union and NATO 2.0 was the period from then until now, he said, NATO 3.0 is one in which the weight of NATO shifts away from the United States and toward the United Kingdom, Europe, and Canada.

—

Notes:

https://www.pbs.org/newshour/politics/thousands-join-protest-outside-kennedy-center-expressing-concern-that-trump-might-tear-it-down

https://www.pbs.org/newshour/arts/judge-orders-kennedy-center-to-provide-30-days-notice-before-making-any-major-physical-changes

https://www.washingtonpost.com/politics/2026/09/16/trump-officials-say-planned-arch-will-harm-historic-sites-must-go/

https://www.washingtonpost.com/documents/b432bfa4-ff64-4fed-8543-337490a98a5a.pdf

https://www.nato.int/en/news-and-events/events/transcripts/2026/09/16/annual-lecture-by-the-nato-secretary-general

https://www.militarytimes.com/news/your-military/2026/09/15/russia-and-iran-deepen-weapons-partnership-amid-wars-in-ukraine-middle-east/

https://www.washingtonpost.com/national-security/2026/03/06/russia-iran-intelligence-us-targets/

https://edition.cnn.com/2026/09/16/world/video/russia-spy-satellites-iran-intelligence-strikes-us-assets-digvid-invs

https://www.npr.org/2026/09/19/nx-s1-5974199/worries-swirl-that-russia-is-planning-to-test-nato-with-direct-attacks

https://www.propublica.org/article/donald-trump-jr-wedding-bankrolled-russian-oligarch-umar-kremlev-putin

https://www.nytimes.com/2026/09/17/us/trump-putin-russia-sanctions.html

https://www.nytimes.com/2026/02/19/us/politics/trump-russia-deals-novatek.html

https://www.nbcnews.com/politics/national-security/pentagon-weighs-pulling-nearly-third-us-forces-europe-rcna597722

https://www.thetimes.com/us/news-today/article/us-troops-europe-nato-3tpm5hqw2

https://www.newyorker.com/magazine/2026/09/21/jared-kushners-dollar-diplomacy

https://www.justice.gov/opa/pr/members-russian-intelligence-services-network-charged-conspiring-finance-terrorism-and

https://www.theguardian.com/world/2026/sep/20/the-rules-have-changed-has-russia-begun-to-order-assassinations-on-us-soil

https://www.rferl.org/a/us-congress-sweeping-russia-iran-sanctions-bill/33857946.html

https://www.theguardian.com/world/2026/sep/14/ukraine-kallas-not-scared-russian-strike-train

Trump’s Truth:

statuses/41811

In 2016, I removed Google Analytics from this blog to avoid being complicit in feeding the biggest machine for harvesting personal data. Instead, I relied on GoAccess to analyze my server logs.¹ For the past couple of years, the statistics have made no sense, despite my attempts to filter bots: AI scrapers inflate the number of visitors to around 2,000 per day. Eventually, I settled on GoatCounter, an open-source, privacy-friendly web analytics platform. I replaced the JavaScript client to filter bots more aggressively and added a CSS fallback. To improve reliability, I implemented a local proxy running on each of the five web servers serving this blog. The rest of this post details how these pieces fit together and how I deploy them on NixOS. ❄️

Why GoatCounter?
Custom JavaScript client
Local proxy
Deploying on NixOS

Why GoatCounter?

GoatCounter does not collect personal data: instead of storing the reader’s IP address or relying on cookies, it creates a session identifier valid for 8 hours from the user agent and the IP address. Its feature set is modest but sufficient for a blog. If you want to look at the interface, GoatCounter’s author runs a public instance for his site. A hosted version lets you try it before running your own instance. With a single binary and an SQLite database, GoatCounter is one of the lightest self-hosted solutions. Privacy-friendly alternatives, in increasing order of complexity, include Umami, Plausible, and Rybbit.

GoatCounter dashboard showing some statistics from my blog, including an
article on the spanning tree protocol with 1,224 views for the past week, the
referrers, and the breakdown of browsers (52% Chrome, 32% Firefox, with 16% for
Firefox 155 and 6% for Firefox 156)

Custom JavaScript client

GoatCounter includes a small JavaScript client—2,189 bytes minified and gzipped. It ships some features I don’t use: a visitor counter, tracking clicks, configurable settings, etc. I replace it with this function to register a hit:

const count = ({ event, title } = {}) => {
  const params = new URLSearchParams({
    p: event || location.pathname,
    t: title || document.title,
    r: document.referrer,
    q: location.search,
    s: document.documentElement.clientWidth,
    e: !!event,
    rnd: Math.random().toString(36).slice(2, 7),
  });
  fetch(`/count?${params}`, { keepalive: true }).catch(() => {});
};

To filter bots,² I go the extra mile by requiring a user interaction—an idea I stole from Bear Blog.

let sendHit = () => (sendHit = () => {}, count());
["touchmove", "mousemove", "keydown", "pointerdown"].forEach((eventName) =>
  document.addEventListener(eventName, sendHit, {
    once: true,
    passive: true,
  }),
);

If a reader has disabled JavaScript in their browser, I record the hit using a CSS image. The :hover pseudo-class loads it only after an interaction, another trick stolen from Bear Blog. About 2% of my visitors fit into this bucket.³

<!DOCTYPE html>
<html lang="en" class="nojs">
  <head>
    <script>
      // The JavaScript code for this blog requires ES6
      if ("noModule" in HTMLScriptElement.prototype)
        document.documentElement.classList.remove("nojs");
    </script>
  </head>
  <body>
  <!-- ... -->
    <style>
      .nojs body:hover {
        border-width: 0;
        border-image: url('/count?p=/en/blog/2026-kpi-goodhart&t=Building...&r=NoJS&e=false');
      }
    </style>
  </body>
</html>

Where GoAccess reported around 2,000 visitors a day, GoatCounter counts fewer than 200 humans.⁴ I assume AI scrapers use a low-effort approach: if the content is available without barriers, as on this blog, they don’t spawn a complex mechanized browser that could trigger a page view. Even crawlers running JavaScript, like Googlebot with its headless Chromium, do not interact with the page and never trigger the events I listen to. The interaction-based “proof of humanity” I use is likely to keep working.

Local proxy

Five servers ~~across the world~~ in Europe and in North America serve the content of this website, but GoatCounter runs on only one of them. To avoid losing track of visitors when GoatCounter is down, I run a local proxy listening on the same /count endpoint. On each server, it stores the hits in memory with a buffer large enough to survive several days of downtime. It sends them in batches to the upstream backend using the /api/v0/count authenticated endpoint.

Servers on a map. web02 is in Paris, web03 in Helsinki, web04 in Nuremberg,
web05 in Ashburn, web06 in Chicago.

I proposed the code for the proxy in pull request #909. GoatCounter’s maintainer declined to maintain so much code for such a niche use case. As a fellow open-source developer, I often hold the same position for my own projects: a one-time contributor effort may translate into a long-term maintainer commitment.

I expose the endpoint for the proxy on the domain of this website to evade ad blockers. This sounds like I don’t respect the reader’s choice, but as GoatCounter is privacy-friendly, I find it acceptable.

location = /count {
  access_log off;
  proxy_pass http://127.0.0.3:8087/count;
  proxy_pass_request_headers off;
  proxy_set_header Accept-Language $http_accept_language;
  proxy_set_header User-Agent $http_user_agent;
  proxy_set_header X-Real-Ip $remote_addr;
}

Deploying on NixOS

My web servers run NixOS, a declarative Linux distribution with built-in configuration management. I manage this small fleet with Colmena, a stateless deployment tool for NixOS. My configuration is available on GitHub.

Deploying applications in containers

For better isolation, each application runs inside an ephemeral lightweight container, powered by systemd-nspawn. Each container runs a stripped-down NixOS instance. A module wraps NixOS’s containers options to avoid repeating the same options for each application.⁵ The containers share their network namespace with the host: the additional isolation is not worth the increased complexity. For a smaller footprint, I also disable a few non-essential services.

{ config, lib, ... }:
let
  cfg = config.luffy.containers;
in
{
  # User-configurable settings for our custom module
  options.luffy.containers = lib.mkOption {
    default = { };
    description = "Ephemeral containers sharing the host network.";
    type = lib.types.attrsOf (lib.types.submodule {
      options = {
        config = lib.mkOption {
          type = lib.types.deferredModule;
          default = { };
          description = "NixOS configuration of the container.";
        };
      };
    });
  };

  # Translate our options to NixOS containers
  config = {
    containers = lib.mapAttrs
      (name: container: {
        ephemeral = true;
        autoStart = true;
        privateNetwork = false;
        extraFlags = [ "--resolv-conf=replace-host" ];
        config = {
          imports = [ container.config ];
          networking.firewall.enable = false;
          system.stateVersion = config.system.stateVersion;
          systemd.services = {
            console-getty.enable = false;
            systemd-logind.enable = false;
            systemd-oomd.enable = false;
          };
        };
      })
      cfg;
  };
}

To configure a GoatCounter instance running in a container and listening on 127.0.0.4:8088, we import the module⁶ and declare the container in the config.luffy.containers attribute set:

{ pkgs, config, ... }: {
  imports = [ ./modules/container.nix ];
  config.luffy.containers.goatcounter = {
    config = {
      services.goatcounter = {
        enable = true;
        address = "127.0.0.4";
        port = 8088;
        proxy = true;
      };
    };
  };
}

As the containers are ephemeral, we need to keep persistent data in directories on the host. We add a mounts option and ask NixOS’s containers to expose the configured directories through the bindMounts option.

{ config, lib, ... }:
let
  cfg = config.luffy.containers;
in
{
  options.luffy.containers = lib.mkOption {
    type = lib.types.attrsOf (lib.types.submodule {
      options = {
        mounts = lib.mkOption {
          type = lib.types.listOf lib.types.str;
          default = [ ];
          description = "Host directories mounted read-write at the same place.";
        };
      };
    });
  };

  config = {
    containers = lib.mapAttrs
      (name: container: {
        bindMounts =
          lib.genAttrs container.mounts (path: { hostPath = path; isReadOnly = false; });
      })
      cfg;
  };
}

For example, to persist GoatCounter’s database in the /var/db/goatcounter directory on the host, we add the directory to the mounts option and alter the service definition to tell GoatCounter where the database is.

{ config, ... }:
let
  databaseDirectory = "/var/db/goatcounter";
in {
  config.luffy.containers.goatcounter = {
    mounts = [ databaseDirectory ];
    config = {
      services.goatcounter = {
        extraArgs = [ "-db=sqlite+${databaseDirectory}/db.sqlite" ];
      };
    };
  };
}

A container may also need some secrets. Colmena can upload secrets without storing them in the Nix store. We add a keys option to our containers. It takes an attribute set mapping secret names to the commands to populate them. Then, the module declares the required secrets to Colmena in the deployment.keys option, makes the container depend on the presence of the secrets, and exposes them to the container.

{ config, lib, ... }:
let
  cfg = config.luffy.containers;
in
{
  options.luffy.containers = lib.mkOption {
    type = lib.types.attrsOf (lib.types.submodule {
      options = {
        keys = lib.mkOption {
          type = lib.types.attrsOf (lib.types.listOf lib.types.str);
          default = { };
          description = "Secrets, as a command to run locally. They are mounted in /etc.";
        };
      };
    });
  };

  config = {
    # Colmena uploads each secret in `/var/keys` and make them available
    # to the group "keys".
    deployment.keys = lib.concatMapAttrs
      (_: container: lib.mapAttrs
        (_: keyCommand: {
          inherit keyCommand;
          group = "keys";
          permissions = "0640";
          destDir = "/var/keys";
        })
        container.keys)
      cfg;

    # The container can only start if the required secrets are available.
    systemd.services = lib.mapAttrs'
      (name: container:
        let
          units = map (key: "${key}-key.service") (lib.attrNames container.keys);
        in
        lib.nameValuePair "container@${name}" {
          requires = units;
          after = units;
        })
      cfg;

    # Mount each secret inside the container.
    containers = lib.mapAttrs
      (name: container: {
        bindMounts = lib.mapAttrs'
          (key: _: lib.nameValuePair "/etc/${key}" {
            hostPath = "/var/keys/${key}";
            isReadOnly = true;
          })
          container.keys;
      })
      cfg;
  };
}

For example, GoatCounter needs credentials to download the GeoIP database. I provide a local command to fetch the secret from my password manager and expose it inside the container through the /etc/goatcounter.env environment file.

{ pkgs, config, ... }: 
let
  keyCommand = variable: [
    "${pkgs.runtimeShell}"
    "-c"
    "pass show personal/nixops/secrets | grep '^${variable}='"
  ];
in {
  config.luffy.containers.goatcounter = {
    keys."goatcounter.env" = keyCommand "GOATCOUNTER_GEODB";
    config = {
      systemd.services.goatcounter.serviceConfig = {
        EnvironmentFile = "/etc/goatcounter.env";
        SupplementaryGroups = [ "keys" ];
      };
    };
  };
}

GoatCounter server

Nixpkgs already packages GoatCounter. By overriding the src and vendorHash attributes, I reuse its definition for my custom version with the proxy:

{ goatcounter, fetchFromGitHub }:
goatcounter.overrideAttrs (_: {
  src = fetchFromGitHub {
    owner = "vincentbernat";
    repo = "goatcounter";
    rev = "feature/proxy";
    hash = "sha256-dJRlQlFu3tjcEgabT1LEbyFrasJlhmYu4L/T7EkoNcY=";
  };
  vendorHash = "sha256-c9Q5OrbZR+q6pD3SgPPWe8JUzcZco1AVUKGaV61k5DE=";
})

I wrote a NixOS module to encapsulate GoatCounter: the container definition, the service definition, and the secrets. The module accepts the following options: package, serve.enable, serve.listenAddress, serve.port, and serve.databaseFile. I already detailed the container configuration in the previous section. In the end, I chose not to reuse the GoatCounter module from NixOS: it’s small, so it’s better to insulate my module from unexpected future changes.

{ config, pkgs, lib, ... }:
let
  cfg = config.luffy.goatcounter;
  databaseDirectory = builtins.dirOf cfg.serve.databaseFile;
  chown = "${pkgs.coreutils}/bin/chown -R";
in {
  config.luffy.containers.goatcounter = {
    config.systemd.services.goatcounter = {
      description = "GoatCounter Web Analytics";
      wantedBy = [ "multi-user.target" ];
      serviceConfig = {
        EnvironmentFile = "/etc/goatcounter.env";
        SupplementaryGroups = [ "keys" ];
        DynamicUser = true;
        Restart = "always";
        ExecStart = lib.escapeShellArgs [
          (lib.getExe cfg.package)
          "serve"
          "-listen=${cfg.serve.listenAddress}:${toString cfg.serve.port}"
          "-tls=none"
          "-db=sqlite+${cfg.serve.databaseFile}"
          "-automigrate"
        ];
        # Transfer database ownership to dynamically assigned user "goatcounter".
        ExecStartPre = "+${chown} goatcounter:goatcounter ${databaseDirectory}";
        ReadWritePaths = databaseDirectory;
      };
    };
  };
}

The following snippet configures GoatCounter to listen on 127.0.0.4:8088:

{
  luffy.goatcounter = {
    serve = {
      enable = true;
      listenAddress = "127.0.0.4";
      port = 8088;
    };
  };
}

The last step is to configure nginx to expose GoatCounter on the Internet. I disable the /count endpoint as the local proxy handles it.

{ config, ... }:
let
  cfg = config.luffy.goatcounter.serve;
in
{
  services.nginx.virtualHosts."goatcounter.luffy.cx" = {
    forceSSL = true;
    locations = {
      "/" = {
        proxyPass = "http://${cfg.listenAddress}:${toString cfg.port}";
      };
      "= /count".extraConfig = ''
        return 404;
      '';
    };
  };
}

GoatCounter proxy

The same NixOS module configures the local proxy, with the following options: proxy.enable, proxy.listenAddress, proxy.port, and proxy.site—the site receiving the batches of page views. The local proxy has no persistent data, but it needs the API key to authenticate to the main GoatCounter instance: its container uses the keys option but not the mounts option.

{ config, pkgs, lib, ... }:
let
  cfg = config.luffy.goatcounter;
  keyCommand = _: [ "…" ];
in
{
  config.luffy.containers.goatcounter-proxy = {
    keys."goatcounter-proxy.env" = keyCommand "GOATCOUNTER_API_KEY";
    config.systemd.services.goatcounter = {
      description = "GoatCounter Proxy.";
      wantedBy = [ "multi-user.target" ];
      serviceConfig = {
        EnvironmentFile = "/etc/goatcounter-proxy.env";
        SupplementaryGroups = [ "keys" ];
        DynamicUser = true;
        Restart = "always";
        ExecStart = lib.escapeShellArgs [
          (lib.getExe cfg.package)
          "proxy"
          "-site=${cfg.proxy.site}"
          "-listen=${cfg.proxy.listenAddress}:${toString cfg.proxy.port}"
          "-ratelimit=10/1"  # 10 requests per second per IP
        ];
      };
    };
  };
}

For each server, I enable the local proxy with the following snippet. The nginx configuration shown earlier exposes the /count endpoint under the same domain as my blog.

{
  luffy.goatcounter = {
    proxy = {
      enable = true;
      site = "goatcounter.luffy.cx";
      listenAddress = "127.0.0.3";
      port = 8087;
    };
  };
}

Backup of the SQLite database with Litestream

Litestream is a streaming replication tool for SQLite databases. It compresses the changes committed to the write-ahead log (WAL) next to the database and sends them to a remote destination. I encapsulate its configuration in a NixOS module, which takes an attribute set databases mapping a name to the path of the database to back up.

Litestream also runs in a container. I mount the databases to replicate, as well as the secrets to push the backups to a Hetzner storage box using SFTP:

{ config, pkgs, lib, ... }:
let
  cfg = config.luffy.litestream;
  databaseDirs = lib.unique (map builtins.dirOf (builtins.attrValues cfg.databases));
in
{
  config = lib.mkIf (cfg.databases != { }) {
    luffy.containers.litestream = {
      mounts = databaseDirs;
      keys."litestream.env" = [
        "${pkgs.runtimeShell}"
        "-c"
        "pass show personal/nixops/secrets | grep '^SQLITE_BACKUP_'"
      ];
    };
  };
}

Inside the container, I configure Litestream through NixOS’s services.litestream options:

full snapshots every day, kept for 15 days,
three levels of compaction for transaction files: 5 minutes, 30 minutes, and 3 hours,
auto-recovery,⁷
replica stored in a directory matching the host name, and
credentials read from /etc/litestream.env and exposed through variable expansion.

{ config, pkgs, lib, ... }:
let
  cfg = config.luffy.litestream;
in
{
  config.luffy.containers.litestream = {
    config = {
      # The databases belong to dynamically allocated users, whose UID is
      # not known here, so Litestream runs as root.
      systemd.services.litestream.serviceConfig = {
        User = lib.mkForce "root";
        Group = lib.mkForce "root";
      };
      # Use NixOS service.
      services.litestream = {
        enable = true;
        environmentFile = "/etc/litestream.env";
        settings = {
          auto-recover = true;
          snapshot = {
            interval = "24h";
            retention = "360h";
          };
          levels = [
            { interval = "5m"; }
            { interval = "30m"; }
            { interval = "3h"; }
          ];
          dbs = lib.mapAttrsToList
            (name: path: {
              inherit path;
              replica = {
                type = "sftp";
                host = "\${SQLITE_BACKUP_HOST}";
                user = "\${SQLITE_BACKUP_USER}";
                password = "\${SQLITE_BACKUP_PASSWORD}";
                host-key = "\${SQLITE_BACKUP_HOSTKEY}";
                path = "${config.networking.hostName}/${name}";
              };
            })
            cfg.databases;
        };
      };
    };
  };
}

To back up GoatCounter’s database, I declare a goatcounter attribute in luffy.litestream.databases, set to the database path:

{ config, ... }:
let
  cfg = config.luffy.goatcounter.serve;
in
{
  luffy.litestream.databases.goatcounter = cfg.databaseFile;
}

On the SFTP server, we can inspect Litestream’s work, with the compacted transactions and the full snapshots:

❯ ls web02/goatcounter/ltx
web02/goatcounter/ltx/0
web02/goatcounter/ltx/1
web02/goatcounter/ltx/2
web02/goatcounter/ltx/3
web02/goatcounter/ltx/9
❯ ls -lh web02/goatcounter/ltx/1
29.1K Sep  5 01:25 0000000000003f2a-0000000000003f2b.ltx
72.4K Sep  5 02:03 0000000000003f2c-0000000000003f2d.ltx
63.3K Sep  5 02:24 0000000000003f2e-0000000000003f2f.ltx
[…]
❯ ls -lh web02/goatcounter/ltx/9
 8.5M Sep  5 02:00 0000000000000001-0000000000003f2b.ltx
 8.5M Sep  6 02:03 0000000000000001-0000000000004008.ltx
 8.6M Sep  7 02:03 0000000000000001-00000000000043a8.ltx
[…]

We can restore the database from the backup with a few shell commands. First, we stop the containers. Then, we move the damaged database away, invoke litestream restore from the right environment, and restart the containers.⁸

# systemctl stop container@goatcounter container@litestream
# mv /var/db/goatcounter/db.sqlite{,.old}
# ( . /etc/nixos-containers/litestream.conf ; 
>   set -a ; . /var/keys/litestream.env ; set +a ;
>   $SYSTEM_PATH/sw/bin/litestream \
>     restore -config $SYSTEM_PATH/etc/litestream.yml /var/db/goatcounter/db.sqlite)
# ls -lh /var/db/goatcounter/db.sqlite
-rw-r--r-- 1 root root 20M Sep 20 07:33 /var/db/goatcounter/db.sqlite
# systemctl start container@goatcounter container@litestream

Ten years after removing Google Analytics, JavaScript-based analytics is back on this blog, but without storing cookies or IP addresses, and without involving a third party. I still write for myself first, notably because it lets me dig into a topic and refer back to it years later. But knowing a bit more about my fellow human readers is a nice bonus, even the ones disabling JavaScript. 🐐

Nginx scrambles IP addresses before storing them, thanks to the ipscrub nginx module. ↩
GoatCounter already filters some bots based on the user agent or the IP address. But AI scrapers lie about their user agent and hide behind residential proxies. ↩
Without JavaScript, I cannot send the real referrer. I insert “NoJS” instead. ↩
I am missing the humans reading the RSS feed. I am not comfortable adding a tracking pixel, and bots are likely to fetch it, compromising the statistics. I’ll live without counting these readers. ↩
A NixOS module is a function receiving the configuration of the whole system as config and returning three attributes:
- imports adds other modules to import,
- options declares user-configurable settings, each with a type and a default value, and
- config sets values for options declared by any module, like containers from NixOS.
If a module does not need to declare options, you can return the config attribute set directly. When a module does not require any argument, you can define it as an attribute set instead. ↩
Most of the time, you don’t need an explicit import. NixOS automatically imports the modules shipped with Nixpkgs. For my own modules, some machinery also imports them automatically. ↩
Litestream warns against using the auto-recover option as it can cause data loss. But I don’t properly monitor my servers and I prefer uninterrupted backups to a slight chance of losing the last few records. ↩
In my case, the process is slow: around 20 minutes for a 20 MiB database. You can test by restoring to a copy with the -o option, but you still need to stop the Litestream container. ↩

Comments

Related Stories

Downloads

December 21, 2023March 19, 2024June 20, 2024September 22, 2024

References & Resources

You may also be interested in:

Discover More from NASA Earth Science

Subscribe to Earth Observatory Newsletters

Earth Observatory Image of the Day

Earth Science Data

Black boxes are back in fashion

Unconventional uses for Jev

Open weight recreations

Perseverance’s View of ‘Turquoise Bay’

Description

Downloads

PIA26815 (Natural-Color)

PIA26815 Figure A (Enhanced-Color)

Discover More Topics From Photojournal

Comments

Related Stories

Comments

Orbital flight is at hand

Comments

Related Stories

September 11, 2026

Comments

Related Stories

Downloads

2000-2009

2010-2023

References & Resources

You may also be interested in:

Discover More from NASA Earth Science

Subscribe to Earth Observatory Newsletters

Earth Observatory Image of the Day

Earth Science Data

Who is “we”, what does it mean to “beat” China, and why should we even care?

Many hegemons eventually beat themselves

Why GoatCounter?

Custom JavaScript client

Local proxy

Deploying on NixOS

Deploying applications in containers

GoatCounter server

GoatCounter proxy

Backup of the SQLite database with Litestream

Comments

Comments

Related Stories

Friday, October 31, 2025 Market Design Impact Award to Hassidim, Romm, and Shorrer

Comments

Related Stories

Comments

Related Stories

Comments

Related Stories

December 21, 2023
March 19, 2024
June 20, 2024
September 22, 2024