chore(ci): re-run a test run once when a job in it times out (#96)

Kelly, 2026-10-03: a run past twice its usual time "should be verified if
it's failed and killed and/or restarted". The timeouts added in part 1
do the killing. This does the restarting, once.

A new workflow watches "tests". When a run of it ends cancelled or timed
out on its first attempt, it reads each stopped job's annotations and
re-runs the whole run only if one "exceeded the maximum execution
time". A run cancelled by hand or by a newer push is left alone, and a
second timeout stays red so a real hang reaches a person. On every other
run the job's `if` is false, so no runner starts.

Part 2 of evomedia-net/evo.testsuites#25.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

.github/workflows/rerun-timed-out.yml

@@ -0,0 +1,51 @@
+# Re-runs a test run once when one of its jobs ran out of time
+# (evo.testsuites#25, part 2).
+#
+# A timeout usually means a stuck runner or a network stall rather than a
+# broken test, so the first one gets a second chance. A second timeout stays
+# red, so a real hang still reaches a person. A run cancelled by hand, or by a
+# newer push, is left alone: only a job whose own record says it "exceeded the
+# maximum execution time" counts.
+#
+# Costs nothing on an ordinary run. The job's `if` is false for every run that
+# ended any other way, and a job skipped by its `if` never starts a runner.
+name: re-run a timed-out test run
+
+on:
+  workflow_run:
+    workflows: ["tests"]
+    types: [completed]
+
+permissions:
+  actions: write
+  checks: read
+
+jobs:
+  rerun:
+    if: >-
+      github.event.workflow_run.run_attempt == 1 &&
+      (github.event.workflow_run.conclusion == 'cancelled' ||
+       github.event.workflow_run.conclusion == 'timed_out')
+    runs-on: ubuntu-latest
+    timeout-minutes: 2
+    steps:
+      - name: Re-run it if a job ran out of time
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          RUN: ${{ github.event.workflow_run.id }}
+        run: |
+          timed_out=""
+          for job in $(gh api "repos/$REPO/actions/runs/$RUN/jobs" \
+              --jq '.jobs[] | select(.conclusion == "cancelled" or .conclusion == "timed_out") | .id'); do
+            if gh api "repos/$REPO/check-runs/$job/annotations" --jq '.[].message' \
+                | grep -q "exceeded the maximum execution time"; then
+              timed_out="$job"
+            fi
+          done
+          if [ -n "$timed_out" ]; then
+            echo "Job $timed_out ran out of time. Re-running run $RUN once."
+            gh api -X POST "repos/$REPO/actions/runs/$RUN/rerun"
+          else
+            echo "Run $RUN was cancelled, but not by a timeout. Leaving it."
+          fi

CHECKSUMS.txt

@@ -8,7 +8,7 @@ b407ef5b298cd6fe94af492e9254b4447e34333d22f8c29d3e9d4d2881651cf3  zbackup_ec2.cm
 64bc3148b09e422ffe4890339fade2354a31f2c7ba004f2728c29a0a39c32f08  zchecksums.cmd
 3e1c573e446238cae494d22d25376f278333dc25fdbc0a5abbdaa9349722b438  zchecksums.ps1
 049f8e79fc8c3d8d96ae8ccb1155d191e2dbf2753f4d4d73ceca00aa1ab0481b  zdeploy.cmd
-a55be5b5e857463fa3e321ff2a73337f8281b56f6dfd0a7e36ed1f7ba242afbc  zdeploy.ps1
+d2ead96436507c8efedc3c72045e623137a1f844940c4ff57c9c7ebe67645511  zdeploy.ps1
 30918a9260cb6220d6e2140edcf319fc0fa1e75b7fe36d0efe68b0d10cdde241  zec2.cmd
 695ec87ea6518933ee64524d369f1d0d5128bf6e54db31f555a888bdddcb9326  zec2.ps1
 ce3209ed8eaab242286e76aeb11b1249239e2804e3de456a1609cc7caa780b43  zec2_rotatekeys.cmd

zdeploy.ps1

@@ -1176,8 +1176,8 @@ function Invoke-StaticDeploy {
     param([string]$Key, $Proj)
 
     # Plain static sites - no build, no container of their own. The landing
-    # container serves them straight off disk out of /srv/$host, so shipping
-    # the files IS the deploy: there is nothing to restart afterwards.
+    # container serves them straight off disk, one directory per host, so
+    # shipping the files IS the deploy: there is nothing to restart afterwards.
     $root       = Join-Path $Proj.localRoot $Proj.siteDir
     $remotePath = $Proj.remote.path
 
@@ -1304,22 +1304,35 @@ function Invoke-ZTokensPublish {
 
 # ── Dispatch ─────────────────────────────────────────────────────────────────
 
-foreach ($key in $Projects) {
-    # 'ztokens' matches the tool it runs (ztokens.cmd / ztokens.ps1). The old
-    # singular 'ztoken' still works so existing habits and any script that
-    # already calls it keep running.
-    if ($key -in @('ztokens', 'ztoken')) { Invoke-ZTokensPublish; continue }
-    $proj = Get-ZProject -Key $key
-    Invoke-DeployGitPull -Proj $proj   # no-op unless deploy.gitPull is set
-    switch ([string]$proj.kind) {
-        "python" { Invoke-PythonDeploy -Key $key -Proj $proj -ChangeNote $Note }
-        "vite"   { Invoke-ViteDeploy   -Key $key -Proj $proj -ChangeNote $Note }
-        "nextjs" { Invoke-NextDeploy   -Key $key -Proj $proj -ChangeNote $Note }
-        "edge"   { Invoke-EdgeDeploy   -Key $key -Proj $proj }
-        "docker" { Invoke-DockerDeploy -Key $key -Proj $proj }
-        "static" { Invoke-StaticDeploy -Key $key -Proj $proj }
-        default  { throw "No deploy handler for kind '$($proj.kind)' (project '$key'). Add an Invoke-<Kind>Deploy function in zdeploy.ps1." }
+# pip, uv and docker draw progress bars with box-drawing characters: "━" is
+# the bytes E2 94 81. PowerShell 5.1 decodes a native command's stdout - ssh's,
+# here - with [Console]::OutputEncoding, which on Windows is the OEM code page
+# (437 on this machine), where those three bytes read "Γöü". Forty per bar.
+# Decode the box's output as the UTF-8 it is for the duration of the deploy,
+# and put the console back in the finally so a deploy that throws does not
+# leave the session changed.
+$prevConsoleEncoding = [Console]::OutputEncoding
+[Console]::OutputEncoding = New-Object System.Text.UTF8Encoding($false)
+try {
+    foreach ($key in $Projects) {
+        # 'ztokens' matches the tool it runs (ztokens.cmd / ztokens.ps1). The old
+        # singular 'ztoken' still works so existing habits and any script that
+        # already calls it keep running.
+        if ($key -in @('ztokens', 'ztoken')) { Invoke-ZTokensPublish; continue }
+        $proj = Get-ZProject -Key $key
+        Invoke-DeployGitPull -Proj $proj   # no-op unless deploy.gitPull is set
+        switch ([string]$proj.kind) {
+            "python" { Invoke-PythonDeploy -Key $key -Proj $proj -ChangeNote $Note }
+            "vite"   { Invoke-ViteDeploy   -Key $key -Proj $proj -ChangeNote $Note }
+            "nextjs" { Invoke-NextDeploy   -Key $key -Proj $proj -ChangeNote $Note }
+            "edge"   { Invoke-EdgeDeploy   -Key $key -Proj $proj }
+            "docker" { Invoke-DockerDeploy -Key $key -Proj $proj }
+            "static" { Invoke-StaticDeploy -Key $key -Proj $proj }
+            default  { throw "No deploy handler for kind '$($proj.kind)' (project '$key'). Add an Invoke-<Kind>Deploy function in zdeploy.ps1." }
+        }
     }
+} finally {
+    [Console]::OutputEncoding = $prevConsoleEncoding
 }
 # The timestamp goes through Stop-ZTracking as the FinalNote so it lands after
 # the tracking footer and before the trailing blank lines - the last thing on