drawtext filter, and it works —
until the caption is a real sentence. An apostrophe, a time like 3:30, or a
comma gives No option name near '30:fontsize=48:...' and exit code 234.
A % is worse: ffmpeg prints Stray % near '', exits 0, and the video has
no caption at all. And Korean (or any non-Latin text) may come out as a row
of empty boxes.
Below is each failure reproduced on my Mac, a working way to escape the text
if you must use text=, and the fix I ended up with: put the caption in a file
and use textfile= with expansion=none — textfile= alone still fails on %.
Every command here renders one frame from a generated black background
(-f lavfi -i color=...) to a PNG, so you can try them without a video. -y
overwrites the output file without asking — run them in an empty folder.
No font that has Korean glyphs: boxes. Without fontfile=, the static
ffmpeg build I used couldn’t load a fontconfig configuration, fell back to a
font without Hangul, and drew one box per syllable. Exit code 0:
$ ffmpeg -hide_banner -loglevel error -f lavfi -i color=c=black:s=800x160:d=1 -vf "drawtext=text='안녕하세요':fontsize=48:fontcolor=white:x=20:y=50" -frames:v 1 -y 1-tofu.png; echo "exit $?"
Fontconfig error: Cannot load default config file: No such file: (null)
Fontconfig error: Cannot load default config file: No such file: (null)
exit 0
The PNG shows □□□□□. Pointing fontfile= at a font with Korean glyphs
fixes it; macOS ships one at /System/Library/Fonts/AppleSDGothicNeo.ttc. (A
Homebrew ffmpeg with a working fontconfig setup may pick a different default
font; I only tested this build.)
An apostrophe inside text='...':
$ ffmpeg -hide_banner -loglevel error -f lavfi -i color=c=black:s=800x160:d=1 -vf "drawtext=fontfile=/System/Library/Fonts/AppleSDGothicNeo.ttc:text='it's 오후 3:30':fontsize=48:fontcolor=white:x=20:y=50" -frames:v 1 -y 2-quote.png; echo "exit $?"
[AVFilterGraph @ 0x6000039ec000] No option name near '30:fontsize=48:fontcolor=white:x=20:y=50'
[AVFilterGraph @ 0x6000039ec000] Error parsing a filter description around:
[AVFilterGraph @ 0x6000039ec000] Error parsing filterchain 'drawtext=fontfile=/System/Library/Fonts/AppleSDGothicNeo.ttc:text='it's 오후 3:30':fontsize=48:fontcolor=white:x=20:y=50' around:
Error opening output file 2-quote.png.
Error opening output files: Invalid argument
exit 234
The ' in it's closes the quote, so the : in 3:30 is no longer quoted and
ends the text option. ffmpeg then reads 30 as the next option, and it has
no name=. Without quotes, a plain text=오후 3:30 fails with the same message.
A comma:
$ ffmpeg -hide_banner -loglevel error -f lavfi -i color=c=black:s=800x160:d=1 -vf "drawtext=fontfile=/System/Library/Fonts/AppleSDGothicNeo.ttc:text=네, 맞아요:fontsize=48:fontcolor=white:x=20:y=50" -frames:v 1 -y 3-comma.png; echo "exit $?"
[AVFilterGraph @ 0x600001d8a4c0] No option name near '48:fontcolor=white:x=20:y=50'
[AVFilterGraph @ 0x600001d8a4c0] Error parsing a filter description around:
[AVFilterGraph @ 0x600001d8a4c0] Error parsing filterchain 'drawtext=fontfile=/System/Library/Fonts/AppleSDGothicNeo.ttc:text=네, 맞아요:fontsize=48:fontcolor=white:x=20:y=50' around:
Error opening output file 3-comma.png.
Error opening output files: Invalid argument
exit 234
In a filtergraph a comma separates filters, so everything after 네 became a
second “filter”.
A percent sign — the silent one:
$ ffmpeg -hide_banner -loglevel error -f lavfi -i color=c=black:s=800x160:d=1 -vf "drawtext=fontfile=/System/Library/Fonts/AppleSDGothicNeo.ttc:text='할인 50%':fontsize=48:fontcolor=white:x=20:y=50" -frames:v 1 -y 4-percent.png; echo "exit $?"
[Parsed_drawtext_0 @ 0x6000006802c0] Stray % near ''
exit 0
Exit 0, the file is written, and the frame is plain black: the whole caption
is dropped, not just the %. drawtext treats %{...} as an expansion (for
things like %{pts}), and a % that doesn’t start one makes it give up on the
text. If your script only checks the exit code, it won’t notice.
The caption is parsed three times before it’s drawn, and each parser has its own special characters:
-vf): , ; [ ] separate filters and links;
' quotes; \ escapes.: separates options; ' and \ again.% starts %{...}; \ escapes.(The shell is a fourth if you type the command, which is why the escaped
text= versions you find online are hard to read.)
text=: escape for each parserEscape in the reverse order the parsers read the text: drawtext first, then
the option value, then the filtergraph. This Python version passes the filter
as a list item to subprocess.run, so there’s no shell layer:
# escaped.py -- the text= route: escape for three parsers, in this order
import subprocess
FONT = "/System/Library/Fonts/AppleSDGothicNeo.ttc"
def drawtext_escape(s):
s = s.replace("\\", "\\\\").replace("%", "\\%") # 1. drawtext's % expansion
s = s.replace("\\", "\\\\").replace(":", "\\:").replace("'", "\\'") # 2. option value
for c in "\\'[],;": # 3. filtergraph
s = s.replace(c, "\\" + c)
return s
caption = "it's 오후 3:30, 할인 50% [끝]"
vf = (f"drawtext=fontfile={FONT}:text={drawtext_escape(caption)}"
":fontsize=48:fontcolor=white:x=20:y=50")
print(vf)
subprocess.run(["ffmpeg", "-hide_banner", "-loglevel", "error",
"-f", "lavfi", "-i", "color=c=black:s=800x160:d=1",
"-vf", vf, "-frames:v", "1", "-y", "escaped.png"], check=True)
$ python3 escaped.py
drawtext=fontfile=/System/Library/Fonts/AppleSDGothicNeo.ttc:text=it\\\'s 오후 3\\:30\, 할인 50\\\\% \[끝\]:fontsize=48:fontcolor=white:x=20:y=50
That rendered it's 오후 3:30, 할인 50% [끝] correctly, and so did
"따옴표" C:\temp; a=b (quotes, a backslash, a semicolon). It works, but look
at 50\\\\%: four backslashes for one percent sign. It’s easy to get one layer
wrong, and the % mistake doesn’t raise an error.
textfile= plus expansion=noneWrite the caption to a file and give drawtext the file’s path. The filtergraph and option parsers only see the path; the caption itself is never parsed by them.
But textfile= alone still fails on %, because drawtext’s expansion also
runs on text read from the file:
$ printf '%s' "it's 오후 3:30, 할인 50% [끝]" > caption.txt
$ ffmpeg -hide_banner -loglevel error -f lavfi -i color=c=black:s=800x160:d=1 -vf "drawtext=fontfile=/System/Library/Fonts/AppleSDGothicNeo.ttc:textfile=caption.txt:fontsize=48:fontcolor=white:x=20:y=50" -frames:v 1 -y 7-textfile.png; echo "exit $?"
[Parsed_drawtext_0 @ 0x600002720fd0] Stray % near ' [끝]'
exit 0
Black frame again. Add expansion=none and the text is drawn exactly as it is
in the file:
$ ffmpeg -hide_banner -loglevel error -f lavfi -i color=c=black:s=800x160:d=1 -vf "drawtext=fontfile=/System/Library/Fonts/AppleSDGothicNeo.ttc:textfile=caption.txt:expansion=none:fontsize=48:fontcolor=white:x=20:y=50" -frames:v 1 -y 8-textfile-none.png; echo "exit $?"
exit 0
The frame shows it's 오후 3:30, 할인 50% [끝]. (expansion=none also turns
off %{pts} and friends; if you need a timestamp in the same caption, that
needs text= and escaping.)
In a script, a temporary file per caption:
# textfile.py -- the textfile= route: no escaping of the caption at all
import os, subprocess, tempfile
FONT = "/System/Library/Fonts/AppleSDGothicNeo.ttc"
def render(caption, out):
fd, path = tempfile.mkstemp(suffix=".txt") # e.g. /var/folders/.../T/tmpab12.txt
try:
with os.fdopen(fd, "w", encoding="utf-8") as f:
f.write(caption)
vf = (f"drawtext=fontfile={FONT}:textfile={path}:expansion=none"
":fontsize=48:fontcolor=white:x=20:y=50")
subprocess.run(["ffmpeg", "-hide_banner", "-loglevel", "error",
"-f", "lavfi", "-i", "color=c=black:s=800x160:d=1",
"-vf", vf, "-frames:v", "1", "-y", out], check=True)
finally:
os.remove(path)
render("it's 오후 3:30, 할인 50% [끝]", "textfile.png")
render('"따옴표" C:\\temp; a=b', "textfile2.png")
print("ok")
$ python3 textfile.py
ok
Both frames showed the captions exactly, with no escaping code. Write the file
as UTF-8 (encoding="utf-8"). A trailing newline (echo instead of
printf '%s') made no visible difference with this ffmpeg.
The path is still parsed. The file name goes through the same filtergraph
and option parsers, so a : in it breaks the command the same way:
$ printf '%s' "안녕" > "a:b.txt"
$ ffmpeg -hide_banner -loglevel error -f lavfi -i color=c=black:s=640x160:d=1 -vf "drawtext=fontfile=/System/Library/Fonts/AppleSDGothicNeo.ttc:textfile=a:b.txt:expansion=none:fontsize=48:fontcolor=white:x=20:y=50" -frames:v 1 -y colon.png; echo "exit $?"
[AVFilterGraph @ 0x6000005a0230] No option name near 'b.txt:expansion=none:fontsize=48:fontcolor=white:x=20:y=50'
...
exit 234
Use paths you control, like the ones tempfile.mkstemp() returns, rather than
a name built from the caption.
$FONT:text is not what you typedOn macOS the default shell is zsh. Putting the font path in a variable and
writing fontfile=$FONT:text=... fails with a different error:
$ FONT=/System/Library/Fonts/AppleSDGothicNeo.ttc
$ ffmpeg -hide_banner -loglevel error -f lavfi -i color=c=black:s=800x160:d=1 -vf "drawtext=fontfile=$FONT:text='안녕하세요':fontsize=48:fontcolor=white:x=20:y=50" -frames:v 1 -y 5-zsh.png; echo "exit $?"
[Parsed_drawtext_0 @ 0x600003a84420] Either text, a valid file, a timecode or text source must be provided
[AVFilterGraph @ 0x600002684380] Error initializing filters
Error opening output file 5-zsh.png.
Error opening output files: Invalid argument
exit 234
zsh read $FONT:t as “the last path component of $FONT” (the :t modifier),
so ffmpeg received fontfile=AppleSDGothicNeo.ttcext='안녕하세요' — no font
path and no text option. Write ${FONT}:text=... and it works (exit 0, text
drawn). bash doesn’t have this problem, but braces work in both.
fontfile= a font that has the glyphs
(/System/Library/Fonts/AppleSDGothicNeo.ttc on macOS).No option name near '...' with a caption: a :, ' or , in the text
ended the option or the filter early.Stray % near ... with exit 0: the whole caption was dropped. Don’t rely on
the exit code; look at a frame.textfile=<path>:expansion=none.
textfile= without expansion=none still fails on %.:, ', , (a temp file is fine).text=, escape for drawtext, then the option, then the
filtergraph, and pass the filter as one argv item (no shell).${VAR}: before a colon, not $VAR:.Tested on macOS 26.5.2 (Apple silicon) with ffmpeg 7.1, the static arm64
build shipped in the imageio-ffmpeg 0.6.0 Python package (built with
libfreetype, libharfbuzz and fontconfig), from zsh; Python scripts run with
Python 3.8.2 (/usr/bin/python3). The font-fallback result is specific to
that build; filter addresses like 0x6000039ec000 differ between runs.
ps shows it alive and idle. You want to know where it’s
waiting without adding prints and restarting it.
On macOS you don’t need to install anything for the first look: /usr/bin/sample
attaches to a running process for a few seconds and prints where every thread
is. Below I use it on a hang I reproduced on my Mac — a subprocess pipe
deadlock — and show what it can and can’t tell you, plus what happens when you
reach for spindump instead.
The script starts a child process, captures its stdout through a pipe, and waits for it to finish before reading:
# stuck.py -- waits for a child that is blocked writing to a full pipe
import subprocess, sys
p = subprocess.Popen(
[sys.executable, "-c", "print('x' * 200_000)"],
stdout=subprocess.PIPE,
)
p.wait() # never returns: nobody reads the pipe
print(len(p.stdout.read()))
Run it in the background so you get its PID, then look at it:
$ python3 stuck.py &
$ ps -o pid,ppid,%cpu,time -p 52520
PID PPID %CPU TIME
52520 52518 0.0 0:00.03
$ pgrep -P 52520
52522
Alive, 0% CPU, 30 ms of CPU time in total, and it has one child (52522).
Nothing is spinning; something is waiting.
sample the stuck processsample <pid> <seconds> takes a stack snapshot of every thread about once per
millisecond. Use -file so the report lands where you want it (without
-file, it printed the report to the terminal and saved a copy as
/tmp/Python_<date>_<id>.sample.txt on my Mac):
$ sample 52520 1 -file parent.txt
Sampling process 52520 for 1 second with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Sample analysis of process 52520 written to file parent.txt
$ grep -A1 "Sort by top of stack" parent.txt
Sort by top of stack, same collapsed (when >= 5):
__wait4 (in libsystem_kernel.dylib) 775
The “Sort by top of stack” section at the bottom of the report is the fastest
thing to read: it counts which function was on top of the stack in each
sample. Here all 775 samples were in __wait4 — the process is waiting
for a child to exit. The call graph above it shows how it got there (address
columns trimmed):
775 os_waitpid (in Python3) + 92
775 __wait4 (in libsystem_kernel.dylib) + 8
os_waitpid is the C function behind os.waitpid(), which Popen.wait()
calls. So: the parent is in wait(), waiting for the child. Now sample the
child:
$ sample 52522 1 -file child.txt
Sampling process 52522 for 1 second with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Sample analysis of process 52522 written to file child.txt
$ grep -A1 "Sort by top of stack" child.txt
Sort by top of stack, same collapsed (when >= 5):
write (in libsystem_kernel.dylib) 778
The child is stuck in write (its call graph runs from builtin_print down
through _io_FileIO_write to write). It is trying to print, and the write
doesn’t complete.
That’s the whole deadlock: the child waits for someone to read the pipe, and
the parent waits for the child to exit. A pipe has a fixed buffer; once the
child has filled it, its next write() blocks until the other end reads.
stuck.py doesn’t read until after wait() returns, which it never does.
On my Mac the child finished as long as it wrote at most 65,536 bytes, and hung at 65,537. This script checks a few sizes with a timeout so it can’t hang itself:
# pipesize.py -- does the child finish if nobody reads its stdout?
import subprocess, sys
for n in (16_384, 65_535, 65_536, 65_537, 200_000):
p = subprocess.Popen(
[sys.executable, "-c", f"import sys; sys.stdout.write('x' * {n})"],
stdout=subprocess.PIPE,
)
try:
p.wait(timeout=3)
print(n, "finished")
except subprocess.TimeoutExpired:
print(n, "stuck")
p.kill()
p.communicate()
$ python3 pipesize.py
16384 finished
65535 finished
65536 finished
65537 stuck
200000 stuck
Same result with Python 3.12. This is why the bug hides in testing: with small
output, wait() then read() works fine; it only hangs when the child’s output
grows past 64 KiB.
spindump is the other built-in tool people suggest for hangs (it samples user
and kernel stacks). Run as a normal user, it stopped immediately:
$ /usr/sbin/spindump 52520 1 -file spin.txt
spindump must be run as root when sampling the live system
$ echo $?
77
(It lives in /usr/sbin; a normal login shell has that on PATH, so plain
spindump gives the same message.) I didn’t run it with sudo, so I
can’t show its output here. For a process you own, sample needs no root and
was enough to find this hang.
sample can’t tell you: the Python linesample shows C frames: _PyEval_EvalFrameDefault, call_function,
os_waitpid. It doesn’t say which line of your script is waiting. For that,
let Python print its own stack on a signal with faulthandler — two lines at the
top of the script:
# stuck_fh.py -- same hang, but `kill -USR1 <pid>` prints the Python stack
import faulthandler, signal, subprocess, sys
faulthandler.register(signal.SIGUSR1)
p = subprocess.Popen(
[sys.executable, "-c", "print('x' * 200_000)"],
stdout=subprocess.PIPE,
)
p.wait() # never returns: nobody reads the pipe
print(len(p.stdout.read()))
$ python3 stuck_fh.py 2>fh.err &
$ kill -USR1 52343
$ cat fh.err
Current thread 0x00000001f399dd80 (most recent call first):
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1762 in _try_wait
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1804 in _wait
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1079 in wait
File "stuck_fh.py", line 9 in <module>
Line 9 is p.wait(). The process keeps running after the dump (SIGUSR1 doesn’t
stop it), so you can do this to a long-running service. It only works if the
handler was registered before the hang, so it’s worth adding to scripts that
run unattended.
Kill the parent by the PID you got when you started it:
$ kill 52520
Traceback (most recent call last):
File "<string>", line 1, in <module>
BrokenPipeError: [Errno 32] Broken pipe
The traceback is from the child: when the parent died, the read end of the pipe
closed, the child’s blocked write failed, and the child exited too (ps -p
found neither PID afterwards). Kill by PID rather than by process name: a
name pattern also matches every other Python process on the machine, including
services you didn’t mean to stop.
Use communicate(), which reads stdout (and stderr) while waiting for the
child to exit:
# fixed.py -- communicate() reads the pipe while waiting
import subprocess, sys
p = subprocess.Popen(
[sys.executable, "-c", "print('x' * 200_000)"],
stdout=subprocess.PIPE,
)
out, _ = p.communicate(timeout=30)
print(len(out), "exit", p.returncode)
$ python3 fixed.py
200001 exit 0
subprocess.run(..., capture_output=True) uses communicate() internally and
behaves the same way:
$ python3 -c "import subprocess,sys; r=subprocess.run([sys.executable,'-c',\"print('x' * 200_000)\"], capture_output=True, timeout=30); print(len(r.stdout), 'exit', r.returncode)"
200001 exit 0
The timeout= doesn’t fix the deadlock — communicate() does — but it turns
any future hang (a child that really never exits) into an exception instead
of a silent 0% CPU process.
sample <pid> 1 -file out.txt, then read
“Sort by top of stack” at the bottom.__wait4 on top means it’s waiting for a child process. pgrep -P <pid>
and sample the child too.write + parent in wait4 = pipe deadlock. Replace
Popen(..., stdout=PIPE) + wait() with communicate() or
subprocess.run(capture_output=True).spindump needs root on a live system; sample doesn’t for your own
processes.faulthandler on SIGUSR1 and
kill -USR1 <pid>.Tested on macOS 26.5.2 (Apple silicon) with Python 3.8.2 (/usr/bin/python3,
Command Line Tools); the pipe-size check and the fix were also run on Python
3.12.13 (Homebrew). sample and spindump are the ones shipped with that macOS.
spindump was not run as root. PIDs and the faulthandler output are from
separate runs, so the PIDs differ.
/assets/app.<hash>.js fails with net::ERR_FAILED.
In my test, offline worked once the page had been reloaded online at least
once (the “day one” step below), because the reload caches the bundle.
I reproduced this on my Mac with headless Chromium. “Offline” in this post
means I stopped the local server, so only the service worker’s caches could
answer; I didn’t test with the network actually cut (Wi-Fi off, airplane
mode). The service worker cached / at install and cached everything else
“on the way through” the first time it was requested. After a deploy the new
service worker caches the new index.html, which points at a new
hashed bundle name. Nothing has requested that file yet, so it isn’t in any
cache, and the activate step has just deleted the old cache that held the old
bundle. The cached page refers to a script that only exists on the server.
Four small files in an empty folder. build.py plays the role of a deploy: it
replaces site/ with an index.html, one content-hashed bundle and sw.js.
With --precache the bundle goes into the service worker’s precache list;
without it only / does.
# build.py — "deploy" a tiny app into site/: python3 build.py v1 [--precache]
import hashlib, json, pathlib, shutil, sys
version = sys.argv[1]
precache_bundles = "--precache" in sys.argv
site = pathlib.Path("site")
shutil.rmtree(site, ignore_errors=True) # a deploy replaces the old files
(site / "assets").mkdir(parents=True)
js = f'document.getElementById("root").textContent = "notes app {version}";\n'
name = f"app.{hashlib.sha256(js.encode()).hexdigest()[:8]}.js"
(site / "assets" / name).write_text(js)
bundles = [f"/assets/{name}"]
(site / "index.html").write_text(f"""<!doctype html>
<title>notes</title>
<div id="root"></div>
<script src="/assets/{name}"></script>
<script>navigator.serviceWorker.register("/sw.js");</script>
""")
precache = ["/"] + (bundles if precache_bundles else [])
sw = pathlib.Path("sw.template.js").read_text()
sw = sw.replace("__VERSION__", version).replace("__PRECACHE__", json.dumps(precache))
(site / "sw.js").write_text(sw)
print(version, name, "precache:", precache)
The service worker is a common cache-first pattern: precache a list at install, delete other versions’ caches at activate, and cache anything else the first time it’s fetched:
// sw.template.js
const CACHE = "app-__VERSION__";
const PRECACHE = __PRECACHE__;
self.addEventListener("install", (e) => {
e.waitUntil(
caches.open(CACHE)
.then((c) => c.addAll(PRECACHE))
.then(() => self.skipWaiting())
);
});
self.addEventListener("activate", (e) => {
e.waitUntil(
caches.keys()
.then((keys) => Promise.all(keys.filter((k) => k !== CACHE).map((k) => caches.delete(k))))
.then(() => self.clients.claim())
);
});
// cache first; anything not cached yet is fetched and cached on the way through
self.addEventListener("fetch", (e) => {
if (e.request.method !== "GET") return;
e.respondWith(
caches.match(e.request).then((hit) => hit || fetch(e.request).then((res) => {
if (res.ok) {
const copy = res.clone();
caches.open(CACHE).then((c) => c.put(e.request, copy));
}
return res;
}))
);
});
A static server on 127.0.0.1 only (service workers are allowed on
http://127.0.0.1 without HTTPS). It sends Cache-Control: no-cache so the
browser’s HTTP cache doesn’t hide the effect:
# server.py — serve site/ on 127.0.0.1:8771, always revalidate
import functools
from http.server import ThreadingHTTPServer, SimpleHTTPRequestHandler
class Handler(SimpleHTTPRequestHandler):
def end_headers(self):
self.send_header("Cache-Control", "no-cache")
super().end_headers()
def log_message(self, *args):
pass
handler = functools.partial(Handler, directory="site")
ThreadingHTTPServer(("127.0.0.1", 8771), handler).serve_forever()
And a Playwright script that opens the app in a persistent headless
Chromium profile (profile/), so the service worker and its caches survive
from one run to the next, like a real browser. It prints what’s in #root
and what’s in each cache. “Offline” here means the server is stopped, so
nothing answers on 127.0.0.1:8771 and only the service worker’s caches can
serve the page (why not Playwright’s offline mode: see below).
// visit.mjs — open the app in a persistent headless Chromium profile
// node visit.mjs online: load once, wait for the newest service worker
// node visit.mjs --reload same, then reload once
// node visit.mjs --offline one load, no waiting (stop the server first)
import { chromium } from "playwright";
const offline = process.argv.includes("--offline");
const ctx = await chromium.launchPersistentContext("profile", { headless: true });
const page = ctx.pages()[0] ?? await ctx.newPage();
page.on("requestfailed", (r) => console.log(" failed:", new URL(r.url()).pathname, r.failure().errorText));
const show = async (label) => {
const root = await page.textContent("#root").catch(() => "(no page)");
const keys = await page.evaluate(async () => {
const out = [];
for (const k of await caches.keys()) {
const reqs = await (await caches.open(k)).keys();
out.push(`${k}: ${reqs.map((r) => new URL(r.url).pathname).join(" ")}`);
}
return out;
}).catch(() => []);
console.log(`${label}: #root = ${JSON.stringify(root)}`);
for (const k of keys) console.log(" cache", k);
};
if (offline) {
await page.goto("http://127.0.0.1:8771/").catch((e) => console.log(" goto:", e.message.split("\n")[0]));
await page.waitForTimeout(500);
await show("offline");
} else {
await page.goto("http://127.0.0.1:8771/");
// wait until the newest sw.js is installed and controlling the page
await page.evaluate(async () => {
const reg = await navigator.serviceWorker.ready;
await reg.update();
while (reg.installing || reg.waiting || !navigator.serviceWorker.controller) {
await new Promise((r) => setTimeout(r, 100));
}
// give activate up to 3 s to delete the previous version's cache
for (let i = 0; i < 30 && (await caches.keys()).length > 1; i++) {
await new Promise((r) => setTimeout(r, 100));
}
});
await page.waitForTimeout(500);
await show("online");
if (process.argv.includes("--reload")) {
await page.reload();
await page.waitForTimeout(500);
await show("online, reload");
}
}
await ctx.close();
Install Playwright (npx playwright install chromium downloads the browser
if you don’t have it; I already had it), then start the server and keep its
PID — you’ll stop and start it to go “offline” and back:
$ npm init -y >/dev/null && npm i playwright && npx playwright install chromium
$ python3 server.py & SERVER=$!
Day one: deploy v1, open the app and reload once (so the bundle gets cached on the way through), then stop the server and open it again. Everything works:
$ python3 build.py v1 && node visit.mjs --reload
v1 app.bbd8455b.js precache: ['/']
online: #root = "notes app v1"
cache app-v1: /
online, reload: #root = "notes app v1"
cache app-v1: / /assets/app.bbd8455b.js
$ kill $SERVER
$ node visit.mjs --offline
offline: #root = "notes app v1"
cache app-v1: / /assets/app.bbd8455b.js
Deploy v2, open the app once while online, then go offline:
$ python3 server.py & SERVER=$!
$ python3 build.py v2 && node visit.mjs
v2 app.05452931.js precache: ['/']
online: #root = "notes app v1"
cache app-v2: /
$ kill $SERVER
$ node visit.mjs --offline
failed: /assets/app.05452931.js net::ERR_FAILED
offline: #root = ""
cache app-v2: /
That’s the bug. The online visit still showed v1 (the page was served from the
old cache before the new worker took over). The new worker cached the v2
index.html, and activate deleted app-v1 with the only bundle in it. Offline,
the v2 HTML asks for app.05452931.js, which no cache has, and #root stays
empty. In a real app that’s a white screen. I ran the same deploy sequence
again from a fresh profile: blank in 5 of 5 runs.
It’s not only deploys. On a first visit the bundle loads before the service worker exists, so it isn’t cached either. With a fresh profile, opening v1 once and then going offline was blank too (one run):
$ rm -rf profile
$ python3 server.py & SERVER=$!
$ python3 build.py v1 && node visit.mjs
v1 app.bbd8455b.js precache: ['/']
online: #root = "notes app v1"
cache app-v1: /
$ kill $SERVER
$ node visit.mjs --offline
failed: /assets/app.bbd8455b.js net::ERR_FAILED
offline: #root = ""
cache app-v1: /
setOfflineMy first version of visit.mjs used Playwright’s context.setOffline(true).
In 1 of 6 runs of the deploy sequence, the offline load worked and the
v2 bundle showed up in the cache afterwards. I didn’t find out why; my guess
is that the emulation didn’t block the service worker’s own fetch() that
time, but I haven’t confirmed it. A test that passes by luck hides exactly
this bug, so I switched to stopping the server, which gave a blank page in
7 of 7 runs of the deploy sequence (the 5 above plus two full runs of this
post). If you test offline behaviour with network emulation, check that the
result is repeatable.
With a cache-first worker like this one, the fix is to cache, at install,
the files the cached HTML will ask for, so the two arrive together. The file
names change on each build, so the list has to come from the build: here
build.py --precache writes the bundle names into sw.js. Same
steps as above, starting from a fresh profile:
$ rm -rf profile
$ python3 server.py & SERVER=$!
$ python3 build.py v1 --precache && node visit.mjs
v1 app.bbd8455b.js precache: ['/', '/assets/app.bbd8455b.js']
online: #root = "notes app v1"
cache app-v1: / /assets/app.bbd8455b.js
$ kill $SERVER
$ node visit.mjs --offline
offline: #root = "notes app v1"
cache app-v1: / /assets/app.bbd8455b.js
$ python3 server.py & SERVER=$!
$ python3 build.py v2 --precache && node visit.mjs
v2 app.05452931.js precache: ['/', '/assets/app.05452931.js']
online: #root = "notes app v1"
cache app-v2: / /assets/app.05452931.js
$ kill $SERVER
$ node visit.mjs --offline
offline: #root = "notes app v2"
cache app-v2: / /assets/app.05452931.js
The first visit now works offline, and after the deploy the offline page is v2 with its bundle — the HTML and the scripts it names are installed together.
cache.addAll() is all-or-nothing: if any URL fails, the promise rejects and
the install fails. I checked by deploying v3 with --precache and then
deleting its bundle from site/ before visiting:
$ python3 server.py & SERVER=$!
$ python3 build.py v3 --precache && rm site/assets/app.*.js && node visit.mjs
v3 app.7b8eea49.js precache: ['/', '/assets/app.7b8eea49.js']
online: #root = "notes app v2"
cache app-v2: / /assets/app.05452931.js
cache app-v3:
$ kill $SERVER
$ node visit.mjs --offline
offline: #root = "notes app v2"
cache app-v2: / /assets/app.05452931.js
cache app-v3:
The v3 worker never activated: the v2 worker stayed in control and the app
still worked offline as v2. The failed install did leave an empty app-v3
cache behind (caches.open() creates it before addAll fails). The activate
step in sw.template.js deletes every cache except its own, so the next
successful version should clean it up; I didn’t run that step. Keeping users
on the last working version is what you want from a broken deploy, instead of
installing an HTML page that points at nothing.
sw.js. Bundler plugins for Workbox or
Vite’s build.manifest can produce such a list; I didn’t test those here.cache.addAll() for the list so a missing file fails the install and
the old version stays, instead of installing half a version.setOffline), repeat the
run: in 1 of my 6 runs the offline load worked when it shouldn’t have.
Stopping the server gave the same result in 7 of 7 runs here.Cache-Control: no-cache here). Otherwise addAll may be served a stale
index.html from the HTTP cache; I set the header so I didn’t have to find
out.kill and the PID it was started with, not by
process name.Tested on macOS 26.5.2 (Apple silicon) with Python 3.8.2 (/usr/bin/python3),
Node 24.14.1 and Playwright 1.62.1 driving its bundled headless Chromium
(Chrome for Testing 151). Server bound to 127.0.0.1 only. I didn’t test
Safari/WebKit, Firefox, or an installed (home-screen) PWA; I’d expect the same
result from the same caching logic, but I haven’t run it there.
the quick brown fox jumps over the lazy dog came out as
theicox jse lg in the repro below. Nothing errors, and every save reaches
the server.
The editor listens for “document changed” events from the server (Server-Sent Events here) so that edits from another tab or device show up. When an event arrives it refetches the document and puts the result in the editor. The catch: the server sends that event for your own save too. By the time the event arrives and the refetch comes back, you’ve typed more, so the refetch hands you older text and the editor rolls back to it. Your next keystroke then saves the rolled-back text, and the server stores it.
I reproduced this on my Mac with a small server and a scripted typist, then fixed it with a per-session write sequence number.
A server with one shared document. PUT /doc saves it, GET /doc returns it,
and GET /events is an SSE stream with one event per save. The save sleeps for a
moment to stand in for a slow disk, network or sync step:
# server.py — one shared document, saved with PUT, change events over SSE
import json, queue, sys, threading, time
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
SAVE_DELAY = float(sys.argv[1]) if len(sys.argv) > 1 else 0.08 # seconds a save takes
doc = {"text": "", "seqs": {}} # seqs: last write applied per session
lock = threading.Lock()
listeners = []
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass
def do_GET(self):
if self.path == "/events":
q = queue.Queue()
with lock:
listeners.append(q)
self.send_response(200)
self.send_header("Content-Type", "text/event-stream")
self.end_headers()
try:
while True:
self.wfile.write(b"data: " + q.get().encode() + b"\n\n")
self.wfile.flush()
except OSError:
with lock:
listeners.remove(q)
return
with lock:
body = json.dumps(doc).encode()
self.reply(body)
def do_PUT(self):
msg = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
time.sleep(SAVE_DELAY)
with lock:
if msg["seq"] > doc["seqs"].get(msg["session"], 0): # drop late, older writes
doc["text"] = msg["text"]
doc["seqs"][msg["session"]] = msg["seq"]
event = json.dumps({"session": msg["session"], "seq": msg["seq"]})
for q in listeners:
q.put(event)
self.reply(b"{}")
def reply(self, body):
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
ThreadingHTTPServer(("127.0.0.1", 8767), Handler).serve_forever()
Every save carries a random session id (one per open editor) and a seq that
counts that session’s saves. The server already uses them to drop a save that
arrives after a newer one from the same session, so out-of-order requests can’t
store old text — that is not the bug here. It also records the last applied
seq per session in doc.seqs, which the fix below uses.
The client is a Node script that behaves like the editor: it presses one key every 30 ms, saves after every key, and refetches on every change event. With no flag it applies whatever the refetch returns. It also measures how long each of its own saves took to come back as an event:
// editor.mjs — types a sentence, saves after every keystroke,
// and refetches the document whenever the server says it changed.
// node editor.mjs apply every refetch (the bug)
// node editor.mjs --skip-own ignore events for my own saves
// node editor.mjs --fix ignore refetches that don't include all my saves
const BASE = "http://127.0.0.1:8767";
const MODE = process.argv[2] ?? "";
const session = Math.random().toString(36).slice(2);
const sentence = "the quick brown fox jumps over the lazy dog";
const KEY_MS = Number(process.env.KEY_MS ?? 30); // time between key presses
let text = ""; // what the user sees in the editor
let seq = 0; // saves this session has sent
let overwrites = 0; // times a refetch replaced what the editor showed
const sentAt = {}; // seq -> when that save was sent
const echoMs = []; // ms from sending a save to its event arriving
function save() {
seq += 1;
sentAt[seq] = performance.now();
fetch(`${BASE}/doc`, {
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ session, seq, text }),
});
}
async function onChange(event) {
if (event.session === session) echoMs.push(performance.now() - sentAt[event.seq]);
if (MODE === "--skip-own" && event.session === session) return;
const doc = await (await fetch(`${BASE}/doc`)).json();
if (MODE === "--fix" && (doc.seqs[session] ?? 0) < seq) return; // server hasn't got all my saves yet
if (doc.text !== text) {
overwrites += 1;
console.log(`event ${event.session === session ? "mine" : event.session} seq=${event.seq}: editor "${text}" -> "${doc.text}"`);
text = doc.text;
}
}
// listen for change events (Server-Sent Events)
const res = await fetch(`${BASE}/events`);
const decoder = new TextDecoder();
(async () => {
let buf = "";
for await (const chunk of res.body) {
buf += decoder.decode(chunk, { stream: true });
let i;
while ((i = buf.indexOf("\n\n")) >= 0) {
const line = buf.slice(0, i);
buf = buf.slice(i + 2);
onChange(JSON.parse(line.slice("data: ".length)));
}
}
})();
// the user presses one key every KEY_MS; each key goes at the end of what the editor shows
for (const key of sentence) {
text += key;
save();
await new Promise((r) => setTimeout(r, KEY_MS));
}
await new Promise((r) => setTimeout(r, 1000)); // let the last events arrive
const ms = echoMs.sort((a, b) => a - b).map(Math.round);
console.log(`final editor: "${text}"`);
console.log(`keys pressed: ${sentence.length}, overwrites: ${overwrites}, ` +
`save->event ms: min ${ms[0]} median ${ms[ms.length >> 1]} max ${ms.at(-1)}`);
process.exit(0);
Start the server, keep its PID, and type (output trimmed in the middle):
$ python3 server.py & SERVER=$!
$ node editor.mjs
event mine seq=3: editor "the qu" -> "the"
event mine seq=5: editor "theic" -> "the q"
event mine seq=4: editor "the qk " -> "theic"
event mine seq=6: editor "theicbr" -> "theic"
event mine seq=9: editor "theicow" -> "the qk"
...
event mine seq=42: editor "theicox jsee d" -> "theicox jsee do"
event mine seq=43: editor "theicox jsee do" -> "theicox jse lg"
final editor: "theicox jse lg"
keys pressed: 43, overwrites: 31, save->event ms: min 88 median 203 max 233
The first line is the bug in one step. The editor shows the qu; the event for
save 3 arrives; the refetch returns the, because that’s all the server had
stored at that moment; the editor drops back to the. The next keys are added to
the rolled-back text (theic — the qu is gone), those saves go out, their
echoes roll the editor back again, and it snowballs. Events also arrive out of
order (5 before 4): the server handles saves on separate threads, so they don’t
finish in the order they were sent.
And it isn’t only on screen — the garbled text is what got saved:
$ curl -s http://127.0.0.1:8767/doc
{"text": "theicox jse lg", "seqs": {"lpd5f2hm3hh": 43}}
Earlier runs the same morning: one ended as the qubx oe dog (25 overwrites),
and three more gave 34, 26 and 28 overwrites out of 43 keys.
Slower typing helps only once the gap between keys is longer than save + event + refetch. Three runs at each key interval, against the same server:
$ for k in 120 200 300 500; do for i in 1 2 3; do KEY_MS=$k node editor.mjs | tail -1; done; done
keys pressed: 43, overwrites: 41, save->event ms: min 84 median 222 max 241
keys pressed: 43, overwrites: 41, save->event ms: min 106 median 224 max 235
keys pressed: 43, overwrites: 43, save->event ms: min 124 median 219 max 238
keys pressed: 43, overwrites: 43, save->event ms: min 124 median 224 max 235
keys pressed: 43, overwrites: 36, save->event ms: min 96 median 210 max 234
keys pressed: 43, overwrites: 39, save->event ms: min 104 median 223 max 262
keys pressed: 43, overwrites: 0, save->event ms: min 86 median 200 max 241
keys pressed: 43, overwrites: 0, save->event ms: min 93 median 217 max 236
keys pressed: 43, overwrites: 0, save->event ms: min 91 median 214 max 235
keys pressed: 43, overwrites: 0, save->event ms: min 107 median 219 max 242
keys pressed: 43, overwrites: 0, save->event ms: min 87 median 224 max 237
keys pressed: 43, overwrites: 0, save->event ms: min 104 median 213 max 242
At 120 and 200 ms between keys almost every event rolled the editor back; at 300 and 500 ms none did. The last column says why: a save took about 85–260 ms (median a bit over 200) to come back as an event, so at 200 ms the next key was usually pressed before the refetch returned, and at 300 ms it never was.
That’s longer than the 0.08 s SAVE_DELAY. Part of it is time.sleep itself:
in the session I ran this from, five calls to time.sleep(0.08) timed with
time.perf_counter() took 0.12–0.23 s. I didn’t dig into why. The measured
save-to-event time is the number that matters, whatever its cause.
Make the save slower and the cut-off moves with it (I’ve left out the shell’s job messages):
$ kill $SERVER; python3 server.py 0.3 & SERVER=$!
$ for k in 250 300 500 700; do KEY_MS=$k node editor.mjs | tail -1; done
keys pressed: 43, overwrites: 43, save->event ms: min 309 median 443 max 457
keys pressed: 43, overwrites: 43, save->event ms: min 305 median 433 max 463
keys pressed: 43, overwrites: 0, save->event ms: min 307 median 430 max 455
keys pressed: 43, overwrites: 0, save->event ms: min 314 median 446 max 462
And with no delay at all, it never showed up:
$ kill $SERVER; python3 server.py 0 & SERVER=$!
$ for i in 1 2 3; do node editor.mjs | tail -1; done
keys pressed: 43, overwrites: 0, save->event ms: min 1 median 3 max 6
keys pressed: 43, overwrites: 0, save->event ms: min 1 median 3 max 7
keys pressed: 43, overwrites: 0, save->event ms: min 1 median 4 max 6
So a user who pauses longer than your save round trip won’t see it, and one who types faster will. On localhost with a tiny document the round trip is a few milliseconds, which is why this can pass local testing and break once there’s a slow disk, a sync step or a real network in the save path.
The client already numbers its saves (seq), and the server records the last
seq it applied for each session (doc.seqs) and returns it with the document.
So the client can tell whether a refetched document includes all of its own
saves. If it doesn’t, the editor is ahead of the server and the refetch must
not overwrite it:
if (MODE === "--fix" && (doc.seqs[session] ?? 0) < seq) return; // server hasn't got all my saves yet
seq is read when the refetch returns, not when the event arrived. If you
typed (and saved) while the refetch was in flight, the check sees the new seq
and skips that stale response too.
With the check, the same typing at 30 ms per key gave no overwrites, and neither did the slow server at 250 ms per key, which had overwritten all 43 keys before:
$ kill $SERVER; python3 server.py & SERVER=$!
$ for i in 1 2 3; do node editor.mjs --fix | tail -1; done
keys pressed: 43, overwrites: 0, save->event ms: min 96 median 223 max 233
keys pressed: 43, overwrites: 0, save->event ms: min 91 median 209 max 234
keys pressed: 43, overwrites: 0, save->event ms: min 97 median 203 max 235
$ kill $SERVER; python3 server.py 0.3 & SERVER=$!
$ KEY_MS=250 node editor.mjs --fix | tail -1
keys pressed: 43, overwrites: 0, save->event ms: min 308 median 444 max 456
Each fixed run ended with the full sentence in the editor, and on the server
(checked with GET /doc after a repeat of the slow-server run).
That’s the obvious first fix (--skip-own in the script), and on its own it
removes the echo:
$ kill $SERVER; python3 server.py & SERVER=$!
$ for i in 1 2 3; do node editor.mjs --skip-own | tail -1; done
keys pressed: 43, overwrites: 0, save->event ms: min 95 median 212 max 235
keys pressed: 43, overwrites: 0, save->event ms: min 93 median 209 max 236
keys pressed: 43, overwrites: 0, save->event ms: min 85 median 208 max 234
The difference shows up when another tab saves while you type. Its event is
not yours, so the editor refetches — and that refetch can return a document
holding an older copy of your own text, because your latest saves are still in
flight. To test it, a second “tab” (curl, session tab-b) saves four times,
about half a second apart, while the editor types:
$ kill $SERVER; python3 server.py & SERVER=$!
$ node editor.mjs --skip-own & sleep 0.25; for n in 1 2 3 4; do curl -s -o /dev/null -X PUT \
-H "Content-Type: application/json" -d '{"session":"tab-b","seq":'$n',"text":"B'$n' "}' \
http://127.0.0.1:8767/doc; sleep 0.25; done; wait $!
event tab-b seq=1: editor "the quick b" -> "B1 "
event tab-b seq=2: editor "B1 rown fox jumps over " -> "B1 rown fox jumps"
event tab-b seq=3: editor "B1 rown fox jumpsthe lazy dog" -> "B3 "
event tab-b seq=4: editor "B3 " -> "B4 "
final editor: "B4 "
keys pressed: 43, overwrites: 4, save->event ms: min 87 median 208 max 275
Look at the second line. The event is tab-b’s, but the refetch returned
B1 rown fox jumps — this editor’s own text from a few keys earlier, because
tab-b’s save 2 had been overwritten on the server by this editor’s later saves,
and its newest saves hadn’t landed yet. The editor rolled back to it and the
next keys ran together (jumpsthe). That happened in 3 of 6 runs (restarting the
server before each so tab-b starts at seq 1 again); in the other three, every
refetch happened to return tab-b’s text.
Six runs of the same test with --fix never rolled back to the editor’s own
text. The tab-b saves whose refetch came back while the editor still had saves in
flight were skipped (B1 and B2 in every run, B3 too in one), and the later
ones were shown:
$ kill $SERVER; python3 server.py & SERVER=$!
$ node editor.mjs --fix & sleep 0.25; for n in 1 2 3 4; do curl -s -o /dev/null -X PUT \
-H "Content-Type: application/json" -d '{"session":"tab-b","seq":'$n',"text":"B'$n' "}' \
http://127.0.0.1:8767/doc; sleep 0.25; done; wait $!
event tab-b seq=3: editor "the quick brown fox jumps over the lazy dog" -> "B3 "
event tab-b seq=4: editor "B3 " -> "B4 "
final editor: "B4 "
keys pressed: 43, overwrites: 2, save->event ms: min 86 median 217 max 235
That’s not a merge either: tab-b’s B1 and B2 were stored and then replaced
by this editor’s next save, and the user never saw them. Whose text should win
when two people type into the same field at once is a different problem
(locking, operational transforms, CRDTs). What the sequence check fixes is
narrower and is the bug in the title: your own edits aren’t undone by a stale
copy of themselves, whichever event triggered the refetch.
When you’re done, stop the server with the PID you saved: kill $SERVER.
doc.seqs[session] >= seq), reading seq when the response arrives.Tested on macOS 26.5.2 (Apple silicon) with Python 3.8.2 (/usr/bin/python3,
Command Line Tools), Node 24.14.1 (built-in fetch) and curl 8.7.1. Server bound
to 127.0.0.1 only. I didn’t test in a browser (EventSource) or over
WebSocket; the client logic would be the same, but I haven’t run it there.
Content-Length, sees the upload is too big, and answers
413 Payload Too Large. In curl you see the 413. But a Python client gets
BrokenPipeError: [Errno 32] Broken pipe or ConnectionResetError: [Errno 54]
Connection reset by peer, and Node’s fetch throws TypeError: fetch failed
with EPIPE or ECONNRESET — the client never sees the 413 you sent.
I reproduced this on my Mac. The server sent the 413 and closed the connection
without reading the request body. The client was still writing the body. When a socket is
closed with unread data in its receive buffer, TCP answers with a reset (RST) rather
than a normal close, and the client’s next send() or recv() fails. (I didn’t
capture packets; the client-side errors below fit that.) Clients that read the response while uploading (curl,
Chrome) pick up the 413 in time; clients that finish writing before reading
(Python’s http.client/urllib, and Node’s fetch in some runs) don’t.
A server that accepts bodies up to 1 MB. Without --drain it answers 413 straight
away; with --drain it reads and discards the body first (up to a cap — more on
that below):
# server.py
import sys
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
LIMIT = 1_000_000 # largest body we accept
MAX_DRAIN = 50_000_000 # largest body we'll read just to throw it away
DRAIN = "--drain" in sys.argv
class Handler(BaseHTTPRequestHandler):
timeout = 10 # a client that stops sending can't hold a thread forever
def do_POST(self):
length = int(self.headers.get("Content-Length", 0))
if length > LIMIT:
if DRAIN and length <= MAX_DRAIN:
self.drain(length)
self.reply(413, b"too large\n")
return
self.rfile.read(length)
self.reply(200, b"ok\n")
def drain(self, length):
# read and discard the body so the client can finish sending
while length > 0:
chunk = self.rfile.read(min(length, 65536))
if not chunk:
break
length -= len(chunk)
def reply(self, code, body):
self.send_response(code)
self.send_header("Content-Type", "text/plain")
self.send_header("Content-Length", str(len(body)))
self.send_header("Connection", "close")
self.end_headers()
self.wfile.write(body)
self.close_connection = True
ThreadingHTTPServer(("127.0.0.1", 8766), Handler).serve_forever()
A client using the standard library, posting N bytes:
# client.py
import http.client, sys
body = b"x" * int(sys.argv[1])
conn = http.client.HTTPConnection("127.0.0.1", 8766)
try:
conn.request("POST", "/", body=body, headers={"Content-Type": "application/octet-stream"})
r = conn.getresponse()
print(r.status, r.read())
except Exception as e:
print(type(e).__name__, e)
Start the server without --drain, send a small body, then a 20 MB one five times:
$ python3 server.py &
$ python3 client.py 100
200 b'ok\n'
$ for i in 1 2 3 4 5; do python3 client.py 20000000; done | sort | uniq -c
4 BrokenPipeError [Errno 32] Broken pipe
1 ConnectionResetError [Errno 54] Connection reset by peer
Not one 413. Which error you get depends on timing — whether the reset arrives while
the client is still writing (EPIPE) or when it reads (ECONNRESET). Across several
runs of five I saw both, in varying proportions (one run was five BrokenPipeErrors),
but never a 413.
urllib.request is built on http.client and fails the same way — you get a
URLError, not the HTTPError with .code == 413 your code probably handles:
$ python3 -c '
import urllib.request, urllib.error
try:
urllib.request.urlopen(urllib.request.Request("http://127.0.0.1:8766/", data=b"x" * 20_000_000))
except urllib.error.HTTPError as e:
print("HTTPError", e.code)
except Exception as e:
print(type(e).__name__, e)'
URLError <urlopen error [Errno 32] Broken pipe>
Node’s built-in fetch failed in two to four of every five tries (one run shown):
// upload.mjs
try {
const r = await fetch("http://127.0.0.1:8766/", { method: "POST", body: new Uint8Array(20_000_000) });
console.log(r.status, (await r.text()).trim());
} catch (e) { console.log(String(e), e.cause?.code); }
$ for i in 1 2 3 4 5; do node upload.mjs; done | sort | uniq -c
2 413 too large
1 TypeError: fetch failed ECONNRESET
1 TypeError: fetch failed EPIPE
1 TypeError: terminated EPIPE
The same server, the same 20 MB, with curl:
$ head -c 20000000 /dev/zero > big.bin
$ curl -s -o /dev/null -w '%{http_code}\n' --data-binary @big.bin http://127.0.0.1:8766/
413
curl -v shows why (excerpt): for a large body curl sends Expect: 100-continue
and waits before uploading, so the 413 arrives before any body is sent:
> Content-Length: 20000000
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
* HTTP 1.0, assume close after body
< HTTP/1.0 413 Request Entity Too Large
Even with -H 'Expect:' (upload immediately) curl got the 413 in five of five
tries — it watches for a response while sending. Headless Chrome (fetch from a page
on the same server) also got 413 five times out of five. So if you only test with
curl or a browser, the bug is invisible; it shows up when a script or another
service is the client.
Stop the previous server first (Ctrl-C, or kill the PID the shell printed when you
started it), then start it with --drain:
$ python3 server.py --drain &
$ for i in 1 2 3 4 5; do python3 client.py 20000000; done | sort | uniq -c
5 413 b'too large\n'
$ for i in 1 2 3 4 5; do node upload.mjs; done | sort | uniq -c
5 413 too large
and the urllib snippet above now prints HTTPError 413. Draining 20 MB over
loopback took about 50 ms for the whole client run here.
Once the server has read the whole body there is no unread data when it closes, so the close should be a normal one instead of a reset, and the client reads the 413 after it finishes writing — which is what all four clients did.
Draining means you still receive the bytes you wanted to refuse. Two guards in
server.py above:
MAX_DRAIN: above it, don’t drain — answer and close, and accept that this
client will see a reset. A client claiming a 10 GB body shouldn’t get to make you
read 10 GB. With --drain, a 60 MB body (over the 50 MB cap) gave the same
BrokenPipeError / ConnectionResetError as before.timeout on the handler: a client that sends the headers and then trickles
(or stops) can’t hold a thread forever. With a client that sent the headers for a
2 MB body plus 100 bytes and then went quiet, the server gave up and closed the
connection after 10 s (without a response — the client just sees the close).Pick MAX_DRAIN a few times above your real limit — big enough that honest clients
who are slightly over get a readable 413.
Broken pipe / EPIPE / ECONNRESET on upload, but the server
log says it sent 413: the server probably closed without reading the body (a proxy can cause the same symptom).http.client/urllib, Node fetch. curl and browsers hide the problem.Content-Length bytes.Content-Length bodies; a chunked upload needs the same
idea applied to the chunked stream.Tested on macOS 26.5.2 (Apple silicon) with Python 3.8.2 (/usr/bin/python3,
Command Line Tools) and Python 3.12.13 (Homebrew; same results with client.py),
Node 24.14.1, curl 8.7.1, Google Chrome 154.0.8037.99 (headless, fresh profile).
Server bound to 127.0.0.1 only, so all traffic was loopback; I didn’t test over a
real network, where timing — and which error you see — may differ.
TypeError: Failed to fetch (CORS error) — and the write happened anyway. Your
server logged the POST, the data is in the store, and only the response was
hidden from the page.
I reproduced this on my Mac with headless Chrome. The cause: a POST whose
Content-Type is text/plain (or application/x-www-form-urlencoded or
multipart/form-data) is a CORS simple request. The browser sends it straight
to the server without an OPTIONS preflight, and CORS is only checked
afterwards, when the page tries to read the response. Your OPTIONS handler —
however strict — never gets a say.
Work in an empty folder (I used /tmp/cors-demo). A small JSON API on port 8766 that is meant to accept writes only from
http://127.0.0.1:8765. With --strict it adds the fix shown later; ignore that
for now.
# api.py — a local JSON API on 127.0.0.1:8766 that only allows one origin
import sys
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
ALLOWED_ORIGIN = "http://127.0.0.1:8765"
STRICT = "--strict" in sys.argv
class Handler(BaseHTTPRequestHandler):
def do_OPTIONS(self):
# CORS preflight: only the allowed origin gets the green light
self.send_response(204)
if self.headers.get("Origin") == ALLOWED_ORIGIN:
self.send_header("Access-Control-Allow-Origin", ALLOWED_ORIGIN)
self.send_header("Access-Control-Allow-Methods", "POST")
self.send_header("Access-Control-Allow-Headers", "Content-Type")
self.end_headers()
def do_POST(self):
length = int(self.headers.get("Content-Length", 0))
body = self.rfile.read(length)
ctype = self.headers.get("Content-Type", "")
if STRICT and ctype.split(";")[0].strip().lower() != "application/json":
self.send_response(415)
self.send_header("Content-Length", "0")
self.end_headers()
return
print(f"WRITE from Origin={self.headers.get('Origin')} "
f"Content-Type={ctype!r} body={body!r}", flush=True)
self.send_response(200)
if self.headers.get("Origin") == ALLOWED_ORIGIN:
self.send_header("Access-Control-Allow-Origin", ALLOWED_ORIGIN)
self.send_header("Content-Length", "0")
self.end_headers()
ThreadingHTTPServer(("127.0.0.1", 8766), Handler).serve_forever()
The WRITE line stands in for “the data was saved”.
A page served from a different origin (port 8767) that sends the same JSON
body twice — once labelled application/json, once labelled text/plain. Save it
as evil/index.html (mkdir evil first):
<!-- evil/index.html — served from a different origin (port 8767) -->
<script>
const url = "http://127.0.0.1:8766/notes";
const log = (m) => document.body.insertAdjacentHTML("beforeend", `<p>${m}</p>`);
// 1. JSON content type: not a "simple" request, so the browser preflights first
fetch(url, { method: "POST",
headers: { "Content-Type": "application/json" },
body: '{"title":"from json"}' })
.then(r => log("json: " + r.status), e => log("json: " + e));
// 2. text/plain: a "simple" request, sent straight away with no preflight
fetch(url, { method: "POST",
headers: { "Content-Type": "text/plain" },
body: '{"title":"from text/plain"}' })
.then(r => log("text: " + r.status), e => log("text: " + e));
</script>
Start the API and serve the page, keeping each PID so you can stop exactly these
processes later. python3 -u keeps the server’s log unbuffered so you see it
immediately:
$ python3 -u api.py > api.log 2>&1 & API=$!
$ python3 -m http.server 8767 --bind 127.0.0.1 --directory evil > /dev/null 2>&1 & EVIL=$!
Then load the page in headless Chrome with a throwaway profile in /tmp (opening
the URL in normal Chrome and watching the console works too). On my machine
headless Chrome sometimes printed the DOM and then didn’t exit, so I run it in the
background, give it five seconds, and stop it by its PID:
$ CHROME="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
$ "$CHROME" --headless --user-data-dir=/tmp/cors-demo-profile --no-first-run \
--virtual-time-budget=3000 --dump-dom http://127.0.0.1:8767/ > dom.html 2>/dev/null & TAB=$!
$ sleep 5; kill $TAB 2>/dev/null; grep -o '<body>.*</body>' dom.html
<body><p>text: TypeError: Failed to fetch</p><p>json: TypeError: Failed to fetch</p></body>
(The two <p> lines can come out in either order.)
From the page’s point of view, both requests failed the same way. The server log tells a different story:
$ cat api.log
127.0.0.1 - - [12/Oct/2026 10:10:12] "OPTIONS /notes HTTP/1.1" 204 -
WRITE from Origin=http://127.0.0.1:8767 Content-Type='text/plain' body=b'{"title":"from text/plain"}'
127.0.0.1 - - [12/Oct/2026 10:10:12] "POST /notes HTTP/1.1" 200 -
application/json request was preflighted. The OPTIONS answer had no
Access-Control-Allow-Origin for port 8767, so Chrome never sent the POST.text/plain request had no preflight. The POST reached the handler and
the write ran. Chrome then saw no Access-Control-Allow-Origin on the 200 and
hid the response from the page. In this test, that was all CORS did for the
simple request: it hid the answer, it didn’t stop the write.The body was JSON both times. The server parsed whatever arrived; only the label differed, and the label decides whether there’s a preflight.
Stop both servers before the next step (port 8766 has to be free for the fixed server):
$ kill $API $EVIL
application/jsonapplication/json is not one of the three content types a simple request may
carry, so a browser has to preflight it — and the preflight is where your origin
check actually stops a request. Make that the only way in: reject any write whose
Content-Type isn’t application/json, before processing the request. (api.py
reads the body before the check, but on a 415 it doesn’t use or store it.)
That is the STRICT branch in api.py:
if STRICT and ctype.split(";")[0].strip().lower() != "application/json":
self.send_response(415)
It compares only the media type, so application/json; charset=utf-8 still
passes. Same test with --strict. This time I also serve a copy of the same page
from the allowed origin (port 8765), to check the legitimate client still works:
$ mkdir good && cp evil/index.html good/
$ python3 -u api.py --strict > api.log 2>&1 & API=$!
$ python3 -m http.server 8767 --bind 127.0.0.1 --directory evil > /dev/null 2>&1 & EVIL=$!
$ python3 -m http.server 8765 --bind 127.0.0.1 --directory good > /dev/null 2>&1 & GOOD=$!
Load the page from the other origin (8767), then from the allowed origin (8765):
$ rm -rf /tmp/cors-demo-profile
$ "$CHROME" --headless --user-data-dir=/tmp/cors-demo-profile --no-first-run \
--virtual-time-budget=3000 --dump-dom http://127.0.0.1:8767/ > dom.html 2>/dev/null & TAB=$!
$ sleep 5; kill $TAB 2>/dev/null; grep -o '<body>.*</body>' dom.html
<body><p>json: TypeError: Failed to fetch</p><p>text: TypeError: Failed to fetch</p></body>
$ rm -rf /tmp/cors-demo-profile
$ "$CHROME" --headless --user-data-dir=/tmp/cors-demo-profile --no-first-run \
--virtual-time-budget=3000 --dump-dom http://127.0.0.1:8765/ > dom.html 2>/dev/null & TAB=$!
$ sleep 5; kill $TAB 2>/dev/null; grep -o '<body>.*</body>' dom.html
<body><p>text: TypeError: Failed to fetch</p><p>json: 200</p></body>
$ cat api.log
127.0.0.1 - - [12/Oct/2026 10:10:20] "OPTIONS /notes HTTP/1.1" 204 -
127.0.0.1 - - [12/Oct/2026 10:10:20] "POST /notes HTTP/1.1" 415 -
127.0.0.1 - - [12/Oct/2026 10:10:25] "POST /notes HTTP/1.1" 415 -
127.0.0.1 - - [12/Oct/2026 10:10:25] "OPTIONS /notes HTTP/1.1" 204 -
WRITE from Origin=http://127.0.0.1:8765 Content-Type='application/json' body=b'{"title":"from json"}'
127.0.0.1 - - [12/Oct/2026 10:10:25] "POST /notes HTTP/1.1" 200 -
The text/plain writes now get 415 and no WRITE line, from either origin. The
only write that ran is the preflighted JSON one from the allowed origin.
For comparison, when I loaded the 8765 page against the server without
--strict, its text/plain request also wrote (and got 200) — so if your own front end sends text/plain, for example
to avoid a preflight round trip, switch it to application/json before turning
the check on.
Checking the header directly with curl (curl isn’t a browser and doesn’t do CORS; this only shows the server’s answer):
$ curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: text/plain' -d '{}' http://127.0.0.1:8766/notes
415
$ curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: application/json; charset=utf-8' -d '{}' http://127.0.0.1:8766/notes
200
Stop the three servers by the PIDs you saved, and delete the throwaway profile:
$ kill $API $EVIL $GOOD
$ rm -rf /tmp/cors-demo-profile
GET/HEAD/POST with only basic headers and a
Content-Type of text/plain, application/x-www-form-urlencoded or
multipart/form-data. These are sent without a preflight.OPTIONS handler protects nothing against those. Check the server log
for a POST with no OPTIONS before it.Content-Type: application/json (compare the media
type, ignore parameters) and answer 415 otherwise, before processing the request.application/json.Origin in the preflight; the content-type rule only makes sure the
preflight happens.Tested on macOS 26.5.2 (Apple silicon) with Google Chrome 154.0.8037.99 (headless,
fresh profile), Python 3.8.2 (/usr/bin/python3, Command Line Tools), curl 8.7.1.
All servers bound to 127.0.0.1; both origins were local ports on the same machine.
I did not test a public website calling localhost, or other browsers.
http.server.HTTPServer can stop answering without
a traceback or a log line: requests hang until the client gives up
(curl: (28) Operation timed out), and then everything works again.
I reproduced this on my Mac. HTTPServer handles one connection at a time. If a
client opens a TCP connection and doesn’t send a request on it — for example a
browser’s speculative connection (preconnect) — the server waits on that connection,
and every other request queues behind it. (I simulated the idle client with a Python
socket below; I did not reproduce it with a browser.)
A minimal server. With --threading it uses ThreadingHTTPServer, otherwise the
plain HTTPServer:
# server.py
import sys
from http.server import HTTPServer, ThreadingHTTPServer, BaseHTTPRequestHandler
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
body = b"ok\n"
self.send_response(200)
self.send_header("Content-Type", "text/plain")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
server_class = ThreadingHTTPServer if "--threading" in sys.argv else HTTPServer
server_class(("127.0.0.1", 8765), Handler).serve_forever()
And a client that simulates a preconnect: it connects and then says nothing.
# preconnect.py
# Open a TCP connection and send nothing (simulates an idle preconnect).
import socket, time
s = socket.create_connection(("127.0.0.1", 8765))
time.sleep(30)
Run the plain server, make one normal request, then open the idle connection and try again:
$ python3 server.py &
$ curl -s -m 5 -w '%{http_code} %{time_total}s\n' http://127.0.0.1:8765/
ok
200 0.002656s
$ python3 preconnect.py &
$ sleep 0.5
$ curl -s -m 5 -w '%{http_code} %{time_total}s\n' http://127.0.0.1:8765/; echo "curl exit $?"
000 5.112399s
curl exit 28
$ kill %2 # close the idle connection
$ curl -s -m 5 -w '%{http_code} %{time_total}s\n' http://127.0.0.1:8765/
ok
200 0.001716s
The second request never got an answer. As soon as the idle connection went away, the server was healthy again — so the hang can look intermittent from the outside.
While the request hangs, sample shows the server’s main thread blocked in a socket
recv (the full output is long; these are the lines around recv):
$ sample <server-pid> 1
...
776 sock_recv_into (in _socket.cpython-38-darwin.so) + 268
776 sock_call_ex (in _socket.cpython-38-darwin.so) + 368
776 sock_recv_impl (in _socket.cpython-38-darwin.so) + 36
776 __recvfrom (in libsystem_kernel.dylib) + 8
The stack doesn’t say which connection that recv is on. My reading: HTTPServer
accepts a connection and handles it to completion before calling accept() again,
and BaseHTTPRequestHandler.timeout is None by default, so “to completion” means
“until the client sends something or hangs up”. The idle connection was accepted
first, so the thread is most likely waiting for its request line, while curl waits
in the listen backlog. That fits the fact that closing the idle connection
immediately unblocked the server.
http.server.ThreadingHTTPServer (Python 3.7+) is HTTPServer with
socketserver.ThreadingMixIn: one thread per connection, so an idle connection only
blocks its own thread. It is a one-word change:
Stop the previous server and the idle client first (Ctrl-C them, or kill the PIDs
the shell printed when you started them).
$ python3 server.py --threading &
$ python3 preconnect.py &
$ sleep 0.5
$ curl -s -m 5 -w '%{http_code} %{time_total}s\n' http://127.0.0.1:8765/; echo "curl exit $?"
ok
200 0.001708s
curl exit 0
Per the Python docs its daemon_threads is True, so threads stuck on idle connections
shouldn’t keep the process alive when you stop the server (I didn’t test that part).
python3 -m http.server already uses ThreadingHTTPServer (since 3.7), so the
command-line file server doesn’t show this problem — the same test returned 200
immediately. It bites when you write HTTPServer(...) in your own code, which is
what many examples and older snippets do.
If the handler isn’t safe to run in parallel, at least stop one connection from
holding the server forever. Setting timeout on the handler applies a socket timeout
to each connection; an idle one is dropped after that many seconds:
# server_timeout.py
from http.server import HTTPServer, BaseHTTPRequestHandler
class Handler(BaseHTTPRequestHandler):
timeout = 2 # seconds to wait for a request line before dropping the connection
def do_GET(self):
body = b"ok\n"
self.send_response(200)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()
Stop the previous server and the idle client first (Ctrl-C them, or kill the PIDs
the shell printed when you started them).
$ python3 server_timeout.py &
$ python3 preconnect.py &
$ sleep 0.5
$ curl -s -m 5 -w '%{http_code} %{time_total}s\n' http://127.0.0.1:8765/; echo "curl exit $?"
ok
200 1.448683s
curl exit 0
The request now succeeds, but only after the idle connection’s timeout runs out
(the idle connection was opened about half a second before curl — the sleep 0.5
above — so the wait was ~1.5 s rather than the full 2 s).
With Python 3.12 the server’s stderr log shows the order of events — the idle
connection is dropped first, then curl’s request is served:
127.0.0.1 - - [02/Oct/2026 10:04:48] Request timed out: TimeoutError('timed out')
127.0.0.1 - - [02/Oct/2026 10:04:48] "GET / HTTP/1.1" 200 -
Every idle connection costs every other client up to timeout seconds — fine
for a script, not for a page that loads several resources. Prefer Fix 1; you can
combine both.
HTTPServer (single connection at a time).ThreadingHTTPServer — same constructor, same handler.timeout on the handler so an idle connection
can’t hold the server forever.sample <pid> showing the server’s only thread blocked in recv is a strong hint.Tested on macOS 26.5.2 (Apple silicon) with Python 3.8.2 (/usr/bin/python3,
Command Line Tools) and Python 3.12.13 (Homebrew), curl 8.7.1. Outputs above are
from 3.8.2 unless noted; the server log lines were checked on 3.12. Server bound to 127.0.0.1
only. The idle client was a Python socket, not a browser.
birthtime. Saving had one problem: a process killed halfway
through write() leaves a truncated note. On a small machine that runs out of
memory, the kernel does kill processes, and SIGKILL gives you no chance to clean up.
The textbook fix is the atomic write: write the new content to a temporary file in
the same directory, then rename() it over the original. A rename within one
directory is atomic — readers see the old file or the new file, never half of one.
It worked, and every note in the app suddenly said it was created today.
import os, time
def show(tag, p):
st = os.stat(p)
fmt = lambda t: time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(t))
print(f"{tag:22s} birth={fmt(st.st_birthtime)} mtime={fmt(st.st_mtime)}")
p = "note.md"
open(p, "w").write("v1")
old = time.time() - 86400 * 30
os.utime(p, (old, old)) # pretend the note is a month old
show("original", p)
tmp = ".note.md.tmp"
open(tmp, "w").write("v2")
os.replace(tmp, p) # atomic save
show("after temp+rename", p)
original birth=2026-09-01 09:45:23 mtime=2026-09-01 09:45:23
after temp+rename birth=2026-10-01 09:45:23 mtime=2026-10-01 09:45:23
Of course: the file at that path is now a different file — the temp file — and it was born a moment ago. The old inode, with the old birthtime, is gone.
(Notice also that setting mtime a month back in the first step pulled the
birthtime back with it. That is the behaviour we’ll use.)
On APFS, when you set a file’s modification time to something earlier than its birthtime, the birthtime moves back to match. So:
st_birthtime before replacing it.mtime to the old birthtime — birthtime follows it down.mtime back to “now” — birthtime stays where it is.def atomic_write(path: str, data: str) -> None:
born = os.stat(path).st_birthtime if os.path.exists(path) else None
tmp = os.path.join(os.path.dirname(path) or ".", "." + os.path.basename(path) + ".tmp")
with open(tmp, "w") as f:
f.write(data)
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
if born is not None:
now = os.stat(path).st_mtime
os.utime(path, (born, born)) # birthtime drops to `born`
os.utime(path, (now, now)) # mtime returns; birthtime stays
Running the same experiment with these two utime calls:
after utime(birth) birth=2026-09-01 09:45:23 mtime=2026-09-01 09:45:23
after restore mtime birth=2026-09-01 09:45:23 mtime=2026-10-01 09:45:23
Created date preserved, modified date correct, and the save is still atomic.
rename() is only atomic within one file system; keep the temp
file next to the target, not in /tmp..*.tmp files on the next start, or they pile up unnoticed.utime.utime calls are metadata changes. If
another process reacts to every change, debounce it.Tested on macOS 26.5, APFS, Python 3.12.
]]>~/Downloads. Run by hand, it took a second.
Run from a LaunchAgent at 9 a.m., it sat there for seven hours doing nothing.
No error, no CPU, no log line. The script had a 90-second retry loop, so it should have given up long before.
When a process is idle and “should” have timed out, ask the kernel where it is
blocked. macOS ships sample:
$ sample <pid> 1
...
783 __opendir2 (in libsystem_c.dylib) + 44
783 open$NOCANCEL (in libsystem_kernel.dylib) + 64
783 __open_nocancel (in libsystem_kernel.dylib) + 8
Every sample is inside opendir() → open(). My retry loop never ran a second time
because the first directory listing never returned. Python’s
pathlib.Path.glob() was simply waiting on the kernel.
~/Downloads, ~/Desktop and ~/Documents are protected by macOS privacy
controls (TCC). The first time a program you haven’t approved touches them, macOS
asks the user:
“Python” would like to access files in your Downloads folder.
When you run the script from Terminal, Terminal already has that permission, so you
never see the question. When launchd starts the script, the request is attributed
to a different process chain, the prompt is raised again — and the open() call
waits for an answer. If nobody is at the screen (that’s the point of a scheduled
job), it waits indefinitely.
Either click Allow on the prompt, or go to System Settings → Privacy & Security → Files and Folders and enable Downloads Folder for the program that appears there (here: Python).
To be sure the grant applies to the launchd context — not just to your Terminal — trigger the same access from a throwaway LaunchAgent and check its output:
<!-- ~/Library/LaunchAgents/com.example.downloads-check.plist -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
<key>Label</key><string>com.example.downloads-check</string>
<key>ProgramArguments</key><array>
<string>/bin/bash</string><string>-c</string>
<string>/path/to/venv/bin/python -c "import pathlib; print(len(list((pathlib.Path.home()/'Downloads').iterdir())), 'entries')"</string>
</array>
<key>RunAtLoad</key><true/>
<key>StandardOutPath</key><string>/tmp/downloads-check.log</string>
<key>StandardErrorPath</key><string>/tmp/downloads-check.log</string>
</dict></plist>
$ launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.example.downloads-check.plist
$ cat /tmp/downloads-check.log
5 entries
$ launchctl bootout gui/$(id -u)/com.example.downloads-check
$ rm ~/Library/LaunchAgents/com.example.downloads-check.plist
If the log stays empty and the process is alive, the prompt is waiting on screen.
Use the same interpreter path your real job uses. A virtualenv’s python is
usually a symlink to one framework build of Python, and that is the binary macOS
records the permission for.
A grant can be revoked in System Settings, and the next machine won’t have it. A scheduled job should never hang silently on a permission dialog. Probe the folder in a background thread and give up after a few seconds:
import os, pathlib, threading
DOWNLOADS = pathlib.Path.home() / "Downloads"
def folder_readable(path: pathlib.Path, timeout: float = 10) -> bool:
done = threading.Event()
def peek():
try:
next(path.iterdir(), None)
finally:
done.set()
threading.Thread(target=peek, daemon=True).start()
return done.wait(timeout)
if not folder_readable(DOWNLOADS):
print(f"Cannot open {DOWNLOADS}: a macOS privacy prompt is probably waiting. "
"Allow it, or enable it in System Settings → Privacy & Security → "
"Files and Folders.", flush=True)
os._exit(3) # the blocked thread can't be joined, so don't wait for it
Two details matter:
Event.os._exit(). It ends the process immediately without running
interpreter cleanup while another thread is still parked inside a system call —
there is nothing useful left to clean up anyway.Now the job ends in ten seconds with a message that says exactly what to click, instead of holding the morning’s work hostage until someone notices.
~/Downloads, ~/Desktop or ~/Documents needs its
own privacy grant — the one your Terminal has doesn’t count.sample <pid> before reading code.Tested on macOS 26.5 (Apple silicon), Python 3.12, a LaunchAgent in the gui/<uid>
domain.
launchd is a great scheduler and a terrible narrator. When a job fails it usually
says nothing at all. These are the four failures that cost me the most time on a
Mac mini that runs a small server and a few daily jobs.
The one command to remember for all of them:
$ launchctl print gui/$(id -u)/com.example.job # a LaunchAgent
$ sudo launchctl print system/com.example.job # a LaunchDaemon
Look at state and last exit code.
I reproduced this one just now with a two-line script:
$ cat ~/Desktop/launchd-desktop-test.sh
#!/bin/bash
echo ran-ok
$ launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.example.desktop-test.plist
$ launchctl print gui/$(id -u)/com.example.desktop-test | grep "last exit"
last exit code = 126
$ cat /tmp/desktop-test.log
/bin/bash: /Users/me/Desktop/launchd-desktop-test.sh: Operation not permitted
The same script runs fine from Terminal. ~/Desktop, ~/Documents and
~/Downloads are privacy-protected folders; Terminal has been granted access to
them, a job started by launchd has not. If you didn’t set StandardErrorPath, the
only trace is that exit code.
Interestingly, a job whose WorkingDirectory is on the Desktop but whose program
lives elsewhere ran fine in my test — it’s reading the script file that fails.
Fix: keep anything launchd runs outside the protected folders — a project
directory under your home, ~/bin, /usr/local/bin. (A different symptom of
the same protection, a job hanging while it opens ~/Downloads, is in
the previous post.)
A LaunchAgent in ~/Library/LaunchAgents is loaded into your login session
(gui/<uid>). It runs while that session exists. If the session ends — you log
out, or the GUI session crashes and the Mac sits at the login window — every agent
in it stops, and nothing is written to your job’s error log, because the job
didn’t crash. It just isn’t there.
From across the room a Mac at the login window looks exactly like a Mac that is working. I lost a home server for two days this way before noticing.
Fix: a service that must survive logouts belongs in /Library/LaunchDaemons.
You can still run it as your normal user — that’s what UserName is for — so files
it creates keep the right owner:
<key>UserName</key><string>me</string>
<key>EnvironmentVariables</key>
<dict>
<key>HOME</key><string>/Users/me</string>
<key>PATH</key><string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string>
</dict>
Daemons start with an almost empty environment, so set HOME explicitly — many
tools look for their config under $HOME and fail in confusing ways without it.
Moving to a daemon fixed uptime and broke one feature: the server shells out to a command-line tool that keeps its login token in the login keychain. From the daemon, the tool reported:
Not logged in · Please run /login
…and exited 1, while the server itself kept running happily. Setting HOME and
USER did not help.
The reason: a system daemon runs outside any login session. You can see it:
$ id -A # from Terminal
auid=501
$ id -A # from the daemon
auid=-1
The login keychain belongs to the login session, so it isn’t in the daemon’s keychain search list, no matter which user the daemon runs as.
Fix (without giving up the daemon): when that one tool is needed, hand the
work into the user’s session. A daemon running as the same uid is allowed to
bootstrap a short-lived job into gui/<uid>; there the keychain opens normally.
#!/usr/bin/env bash
# run-in-session.sh CMD ARGS... — run CMD inside the logged-in user's session.
set -euo pipefail
if [ "$(id -A | head -1)" != "auid=-1" ]; then exec "$@"; fi # already in a session
LABEL=com.example.in-session
ROOM="$(mktemp -d)"; trap 'rm -rf "$ROOM"' EXIT
cat > "$ROOM/stdin"
{ printf '#!/bin/bash\n'; printf '%q ' "$@"
printf '< %q > %q 2> %q\necho $? > %q\n' \
"$ROOM/stdin" "$ROOM/out" "$ROOM/err" "$ROOM/code"; } > "$ROOM/run"
chmod +x "$ROOM/run"
cat > "$ROOM/job.plist" <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
<key>Label</key><string>$LABEL</string>
<key>ProgramArguments</key><array><string>$ROOM/run</string></array>
<key>RunAtLoad</key><true/>
</dict></plist>
PLIST
GUI="gui/$(id -u)"
launchctl bootout "$GUI/$LABEL" 2>/dev/null || true
launchctl bootstrap "$GUI" "$ROOM/job.plist"
while [ ! -s "$ROOM/code" ]; do
launchctl print "$GUI/$LABEL" >/dev/null 2>&1 || { sleep 0.3; [ -s "$ROOM/code" ] || break; }
sleep 0.2
done
launchctl bootout "$GUI/$LABEL" 2>/dev/null || true
cat "$ROOM/out" 2>/dev/null || true
cat "$ROOM/err" >&2 2>/dev/null || true
exit "$(cat "$ROOM/code" 2>/dev/null || echo 70)"
The caller sees the same contract as the original tool: arguments in, stdin in, stdout/stderr/exit code out. If nobody is logged in, only this one feature fails — the rest of the daemon stays up, which is the trade I wanted.
Two notes:
Jobs get a minimal PATH (/usr/bin:/bin:/usr/sbin:/sbin). Homebrew’s
/opt/homebrew/bin is not in it, so ffmpeg, node or python3 from Homebrew
“aren’t installed” as far as your job is concerned.
Fix: use absolute paths in ProgramArguments, or set PATH in
EnvironmentVariables (as above). For tools whose location you discover at
install time, write the resolved path into the plist when you install the job,
rather than hoping it’s on PATH at run time.
| Symptom | Likely trap |
|---|---|
last exit code = 126, “Operation not permitted” |
script in a privacy-protected folder |
| Job simply isn’t running, no errors anywhere | LaunchAgent and the login session ended |
| A CLI says “not logged in” only under launchd | daemon outside the session, no login keychain |
| “command not found” only under launchd | minimal PATH |
Tested on macOS 26.5. Trap 1 reproduced for this post; traps 2–4 are from running a home server on the same machine.
]]>