Two things, both prompted by the same incident.
STATIC KIND
Plain static sites - no build, no container of their own; a shared web
container serves them off disk, so shipping the files IS the deploy.
Directories are staged to a sibling and swapped in with mv rather than
copied in place, because a large media file uploaded in place is served
half-written to anyone who requests it mid-copy. The swap is a rename,
so the switch is atomic.
Skips $JunkDirNames + .github + deploy.skipDirs, matching the docker
kind rather than inventing a third convention.
SANITIZATION TEST
This repo is public and must stay standalone, but the toolkit is
developed in a private checkout and copied here. Twice in three days a
wholesale copy landed carrying real project names, internal hostnames,
host disk figures, and references to scripts that exist only in the
private copy. Both times a human reading the diff caught it - the
control that fails exactly when a diff is 280 lines of good work with
three bad words buried in it.
So it is a test now. Seven rules: private project names, private product
domains, private-only script names, local drive paths, the operator home
path, the real key filename, and any IPv4 outside RFC 5737 documentation
space and the private ranges.
Two anti-vacuity guards, because a denylist that silently matches
nothing is worse than no denylist: one asserts the file scan is
non-empty, and one plants a known violation and requires the pattern to
find it.
The IP rule bounds on [\d.] rather than \d deliberately - this toolkit's
own 5-segment version (v1.0.0.0.14) contains "0.0.0.14", which a plain
digit boundary reads as an address.
Verified against the real unsanitized copy that slipped through: 3 of
the 7 rules fire, naming file and line.
Tests: 231/231 (222 + 9 new). CHECKSUMS.txt refreshed.
tests/Invoke-Coverage.ps1 runs the suite with Pester's profiler-based
coverage collector and writes both JaCoCo XML and a coverage-summary.json
in the same shape vitest and jest emit, so one reader handles every tool
in the fleet.
CodeCoverage.Path is every *.ps1 in the repo root, including the ones no
test touches. They report 0% and drag the total down, which is correct -
measuring only the already-tested files answers "how well covered is the
covered code". The same mistake in vitest form had evo.www reporting 93%.
Baseline: 222 tests pass, 7.0% of commands (219/3,125 across 24 scripts).
Pester counts commands, not statements; the collector labels it as such
rather than filing it under a unit it is not.
All 16 repos moved to the evomedia-net org and were renamed into the evo.*
namespace, so every github.com/kellymichels/<old-name> reference in source
headers, CI badges, security links and docs pointed at a redirect.
Mechanical URL-only rewrite, applied longest-name-first so smartplantehs-docs
could not be clobbered by the smartplantehs rule. Nothing else changes: no
code, no product names, no behaviour. smartplantehs -> evo.ehs here is the
REPO url only; the product rename is separate and still pending.
* fix(zdeploy): edge deploy ships asset subdirectories, not just top-level files
A project self-hosting assets in folders (fonts/, vendor/) lost them on
every deploy: docker created empty root-owned mount points and nginx
served 404s from them, so fonts fell back silently and vendored JS
never loaded. Every subdirectory except nginx-logs/ and .git/ now ships
recursively, and the ensure-dir chown is recursive so scp into
docker-created root-owned dirs cannot fail.
Closes#42
* fix(zdeploy): stop deploys hanging on ssh prompts, keep vendored archives, make unzip install idempotent
- Get-Ec2SshOpts: every deploy-path ssh/scp now carries BatchMode=yes
plus connect/keepalive timeouts. Without BatchMode ssh prompts and
waits forever, and because the deploy pipes stderr through the
pipeline the prompt never reaches the screen - the run just stops
under the last step label with no explanation.
- Archive filter exempts files under vendor/: a vendored *.tgz is a
build input, and dropping it fails a Dockerfile COPY at image build.
- unzip install checks command -v first instead of an apt round-trip
on every deploy.
Ported from the private script; three tests cover the vendor/ exemption.
* feat(zchecksums): SHA-256 manifest so a download can be verified before it's run
CHECKSUMS.txt lists a SHA-256 for every top-level .ps1 and .cmd - the files a
user actually executes. zchecksums verifies them; zchecksums -Update
regenerates after an intentional edit.
The manifest is sha256sum format, so 'sha256sum -c CHECKSUMS.txt' works on
Linux/macOS/WSL as well as the PowerShell path on Windows. Hashes are identical
on every platform because .gitattributes pins .ps1/.cmd to CRLF everywhere -
that pin is now load-bearing, so it is commented as such.
Beyond changed and missing files it also reports a script that is on disk but
NOT in the manifest, so something added outside a commit still gets noticed.
Exits non-zero on any of the three.
Honest about its limits, in the header and the README: the manifest lives in
the same repo as the code, so it is an integrity check rather than a signature.
It catches a truncated clone, a forgotten local edit, or an unlisted file - not
a compromised repo.
CHECKSUMS.txt is pinned to LF: sha256sum treats a trailing CR as part of the
filename and would report every entry as missing on Linux.
tests/Checksums.Tests.ps1 keeps it from rotting - a stale manifest is worse
than none, since it either cries wolf until people ignore it or quietly stops
covering a new script. The tests assert the format, LF endings, sort order,
full coverage of on-disk scripts, current hashes, and that zchecksums itself
exits 1 on a tampered file (proved by appending a byte and restoring it).
* feat(zversion, zrelease): toolkit versioning + downloadable release zips
Implements the versioning rule (SmartPlant's 5-segment scheme, now the global
standard; currently only sp and zscripts are on it at v1.x):
v{major}.{rc}.{beta}.{alpha}.{build}
zversion: get / bump / bump-stage / set. A stage bump zeroes every lower
segment including build. 'bump' is one per PR and one per defect fix, not per
file. Any write rewrites three things together, because they are only useful
when they agree: build-version.json (source of truth), a '# Version:' line in
all 42 script headers (a lone copied script still says which release it came
from), and CHECKSUMS.txt (stamping changes every file).
zrelease: packages the current version as releases/zscripts-<version>.zip with
a sibling .sha256, for people who want the toolkit without cloning. One hash
verifies the download; the bundled CHECKSUMS.txt verifies the extracted
contents. Refuses to overwrite an existing version's zip (released = immutable;
bump instead), and refuses to package when zchecksums fails. tests/ excluded
from the zip; releases/ never packages itself.
First release included: releases/zscripts-v1.0.0.0.0.zip (42 scripts + 7
support files) and its .sha256.
.gitattributes: releases/*.sha256 pinned LF (sha256sum treats a trailing CR as
part of the filename), releases/*.zip marked binary.
Verified end-to-end as a downloader would experience it, in WSL: sha256sum -c
on the zip passes, unzip, sha256sum -c CHECKSUMS.txt inside gives 42 OK / 0
FAILED, and the extracted zdeploy.ps1 header and build-version.json both read
v1.0.0.0.0. Double-release guard and -Verify mode exercised. Full Pester suite
219/219 (the checksum tests absorb the new files automatically).
The bash port and its bats suite are removed from the tree while they get more
testing. Everything they were referenced from is cleaned up so nothing dangles:
- README drops the 'Linux / macOS / WSL' pointer to bash/README.md (a dead
link once the folder is gone) and the aside that ZCONFIG is honoured by
both ports.
- CHANGELOG drops the two bash mentions.
- ZHelpers.ps1's ZCONFIG comment no longer cites zhelpers.sh.
- .gitattributes drops the now-dead bash/**, tests/bash/** and *.bats rules,
keeping the PowerShell CRLF rules.
No behaviour change to the PowerShell scripts; the Pester suite is untouched
and still passes 169/169.
Deliberately NOT a history rewrite: the port stays in this repo's history and
in full in the private mirror, so it can be restored with a revert when the
testing is done. Nothing here is secret - it is unfinished, not sensitive.
* test: add bats suite for the bash port + fix underscore-key guard (phase 5)
Part of #26. Closes#30.
The bash port reimplements the exclude lists, config accessors and argument
parsing, so it can drift from PowerShell independently. 50 bats tests mirror
the Pester suites assertion-for-assertion where the two are meant to agree:
z_archive_excludes (per-kind lists and the deploy-vs-backup gating), config
accessors, z_path Windows->WSL translation, json_build_label, and argument
handling (bare invocation, unknown key, 'all' expansion, --port override).
Fixes#30 along the way, because the alternative was a test enshrining the bug:
zproj_require accepted underscore comment keys. It only checked the key was
non-null, and a comment is a non-null JSON string, so 'zkill _note' sailed
through and exited 0 having done nothing - the silent-success failure mode.
Now rejects any _-prefixed key and requires the value to be a JSON object.
Both checks earn their place: the type check catches string comments, the
prefix rule catches an object-valued _template key that PowerShell refuses and
the type check alone would allow.
Documents #31 rather than fixing it: a leading dash on a project key works in
every PowerShell script but only in bash/zdeploy - the others reject -myapp as
an unknown option. Stripping it everywhere would make a mistyped flag resolve
as a project key, so the tests pin current behaviour and bash/README.md now
states the difference instead of the README's blanket claim.
Verified by mutation testing: all 9 mutations turn the suite red - removing the
python and vite backup gates, reintroducing #23 in bash, unfiltering underscore
keys in zproj_keys and zproj_require, breaking zremote_compose_dir fallback and
z_path translation, and removing zkill's all-expansion and no-args guard. Both
mutated files confirmed restored byte-for-byte.
An early run also caught a bug in the tests themselves: the membership helper
used 'grep -qx' (regex), so the needle '.env' matched 'venv' and several
'excludes .env' assertions were false passes. Now uses -qxF.
* fix(gitattributes): keep .bats files LF so 'bats tests/bash' works on a Windows checkout
The LF rule was scoped to 'bash/**', which does not match tests/bash/. With
core.autocrlf a Windows working copy got CRLF .bats files, and bats fails on
them - so the command the README documents would not run on the machine the
suite was written on without stripping \r first.
Adds tests/bash/** and *.bats to the same eol=lf rule and renormalises.
Verified by running 'bats tests/bash' with no sed preprocessing: 50/50.
Part of #26. Phase 2 asserted that Get-ArchiveExcludes returns the right list;
this asserts the archive that actually ships. A name can be on the exclude list
and still land in the zip - only opening the zip proves otherwise. Each test
builds a real temp source tree, archives it, and reads back the entries.
66 tests across: TopLevelExclude (files, directories, multiple names, absent
names), recursive junk-directory pruning at top level and any depth, junk
extensions, nested archives, OS junk filenames, script-file inclusion via
-IncludeScriptFiles, ExtraFiles (how zbackup bundles a pg_dump), zip mechanics
(overwrite, destination creation, forward-slash entry paths), and two
end-to-end shapes - a python deploy zip built from the real exclude list, and a
backup of the same tree that must retain .env and uploads.
Two behaviours are pinned deliberately:
- TopLevelExclude matches TOP-LEVEL names only, so a nested backend/.env is
NOT excluded by listing '.env'. That is current behaviour and the reason
deploys rely on server-side preservation; the test exists so changing it is
a decision rather than an accident.
- An all-junk tree throws instead of producing an empty zip - a silent empty
deploy would unpack to nothing on the server.
Verified by mutation testing, which paid for itself immediately: the first run
showed the junk-directory tests were vacuous. Inside a Where-Object, $_ rebinds
to the pipeline item and shadowed Pester's -ForEach value, so the pattern
matched nothing and the tests passed while asserting nothing. Fixed by
capturing the value first; re-run now catches all 8 mutations (disabling
TopLevelExclude, top-level and nested junk pruning, each file filter, and the
empty-archive throw). ZHelpers.ps1 restored byte-for-byte afterwards.
Part of #26. This layer has regressed more than any other - bare vs dashed
keys, 'all' expansion, and whether a bad key fails loudly or quietly selects
nothing.
42 tests over three guarantees:
- Running bare shows usage and exits non-zero, for all ten target-taking
scripts. Some of these used to mean 'do it to everything' when run with no
args, which is how an unintended full backup or deploy happens.
- An unknown key fails loudly. Exiting 0 having selected nothing is the
dangerous outcome: a typo'd key in a scheduled task looks like a
successful run that backed up nothing.
- A leading dash is stripped before the lookup, so -myapp == myapp. Asserted
via a dashed *unknown* key, so the error must name 'x' rather than '-x'.
Plus zkill's own resolution: bare key, dashed key, several keys, 'all' and
'-all' expanding to projects that have a ports.dev, edge/docker stacks skipped,
and -Port overriding the configured port.
Safety: the tests run the real scripts as child processes, so they are confined
to paths that exit before doing any work. Only zkill runs with a valid target,
because the fixture's dev ports (59990/59991) are deliberately unused and its
localRoots do not exist - it finds no listeners and kills nothing. zdeploy,
zbackup_ec2, zec2, zec2online, zrepair, zstop, zstart and zbackup are never
invoked with a real target; that is integration territory needing a disposable
server.
Each test runs against an isolated temp installation - the .ps1 files copied
beside a fixture zconfig.json - so nothing touches this repo, no real config is
read, and the suite passes on a machine that has never been configured. That
also makes it independent of the ZCONFIG seam in #27, so the PRs can merge in
any order.
Verified by mutation testing: removing zkill's all-expansion, dash tolerance
and no-args guard, making an unknown key exit 0, and letting underscore comment
keys leak in as projects each turn the suite red (2/1/2/16/4 tests). Both
mutated files confirmed restored byte-for-byte.
Part of #26. The toolkit had no automated tests at all - including for the
functions that decide what goes into a deploy zip, which is where a dev .env
reached production (#23).
Phase 1 - the seam. Get-ZConfig read a hardcoded $PSScriptRoot\zconfig.json,
so nothing config-dependent could be tested without touching the real config.
Adds Get-ZConfigPath honoring $env:ZCONFIG (the bash port has always had this,
so it also closes a parity gap) and Reset-ZConfigCache to drop the memoised
config between fixtures. Deliberately did NOT convert the exit 1 paths to
throw: that changes observed CLI output, and the pure functions don't need it.
Phase 2 - 61 tests over the functions with no side effects: Get-ArchiveExcludes
(common/python/vite/nextjs lists, deploy.exclude merging, dedupe, array shape),
Get-ZConfig / Get-ZConfigPath / Get-ZProjectKeys / Get-ZProject (dash tolerance,
underscore-key filtering, memoisation), Get-ZEdgeProject, Get-RemoteComposeDir,
Get-Ec2Target / Get-Ec2Home, Get-LabelFromBuildJsonObj, Read-JsonBuildVersion.
The suite is verified by mutation testing rather than assumed useful - six
deliberate regressions were each introduced and confirmed to turn it red,
including reintroducing the exact #23 bug and its inverse (backups silently
dropping .env/uploads, which would produce restore points that cannot restore).
Runs off a fixture config injected via ZCONFIG, so it never reads a real
zconfig.json and passes on a machine that has never been configured.
