warpline advance is the unattended entry point. It runs whatever the engine
finds due and exits with a code a scheduler can read. This page is how you put
it on a fifteen-minute tick under systemd, launchd or cron.
The tick is fixed and the scheduler is dumb on purpose. The engine decides what is due; the scheduler only decides when to ask. That is why nothing below enables a scheduler-side catch-up: the engine’s own freshness check is the catch-up, and switching on a second scheduling authority against the same fleet gets you two of them disagreeing.
Every scheduler fact on this page carries the page and section it was read from. The last section says which of those reads happened on the machine that wrote this document and which did not, and what you should re-run before trusting the ones that did not.
The order is not a preference. Each step’s failure is invisible without the one before it, which is what makes this a sequence and not a checklist.
Under every scheduler the working directory is not yours. warpline resolves its
home from WARPLINE_HOME first; failing that, from the nearest ancestor of the
working directory holding a .warpline directory; failing that, it uses
.warpline under the working directory. Under a scheduler the second and third
arms both land somewhere you did not mean.
There is a backstop, and it decides which half of that is dangerous. With no
home at the resolved path and no terminal on standard input, warpline
advance refuses to create one and exits 75 (runtime-spec.md § 11). So the
third arm does not invent an empty home under a scheduler — it resolves
.warpline under the working directory, finds nothing there, and refuses
without creating it.
What the backstop does not catch is a wrong home that happens to exist, and
that is the case to set the variable for. A wrong home with no manifests exits
1, which is visible. Every other wrong home exits 0 and reports a healthy
advance against a fleet you did not mean, which is not — whether its plugins
were still fresh, parked at a gate, or ran to completion. The last of those is
the worst case on this page: the advance did work, and it did it somewhere
else. Set the variable.
Set two more beside it. WARPLINE_TRIGGER=scheduled marks every run log this
unit writes as "trigger": "scheduled", so an advance you run by hand (unset,
or WARPLINE_TRIGGER=manual) can never be counted as a scheduled tick. warpline
cannot tell a scheduler from a person, so the unit has to say, and a value other
than scheduled or manual is refused with 1 before anything runs. Only the
warpline advance command reads this variable. A host that calls runAdvance()
as a library never sees it and passes AdvanceOptions.trigger instead, or its
run logs carry no trigger at all. If your plugins live outside the home, set
WARPLINE_PLUGINS_DIR too, and set it in the shell you run approve from as
well (runtime-spec.md § 1).
Skip this step and everything after it verifies the wrong fleet.
Schedulers do not carry your shell’s PATH, so a bare node resolves
somewhere else or nowhere.
PATH: “systemd uses
a fixed value of /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin in the
system manager. In case of the user manager, a different path may be
configured by the distribution.” (systemd.exec(5), $PATH.) None of the
usual version-manager directories is in it.systemd.service(5), command lines.)Program key must be
an absolute path. Previous versions of launchd did not enforce this
requirement but failed to run the job.” (launchd.plist(5), ProgramArguments.)SHELL is set to /bin/sh,
and LOGNAME and HOME are set from the /etc/passwd line of the crontab’s
owner” (crontab(5), DESCRIPTION) — and stops there. It says nothing about
PATH in either direction, so do not rely on one. Set PATH at the top of
the crontab, which the same page documents as an name = value line, and
use an absolute interpreter path. That instruction is right whichever way the
silence falls.Get the path once:
command -v node
warpline’s own bin enforces a Node floor of ^22.18.0 || >=23.6.0 at run time.
Pin a binary that satisfies it, and re-check the pin after any upgrade that
moves it.
Skip this step and the job fails on the first tick, in a log nobody is watching.
Now it is worth knowing what happens when two advances meet, because only now are they pointed at the same home and running the same binary. See Concurrency below.
A service and a timer, both user-level. Nothing here needs a system unit.
# ~/.config/systemd/user/warpline-advance.service
[Unit]
Description=Warpline advance
[Service]
Type=oneshot
Environment=WARPLINE_HOME=/home/operator/.warpline
Environment=WARPLINE_TRIGGER=scheduled
ExecStart=/usr/bin/node /opt/warpline/node_modules/.bin/warpline advance
# ~/.config/systemd/user/warpline-advance.timer
[Unit]
Description=Warpline advance every 15 minutes
[Timer]
OnCalendar=*:0/15
[Install]
WantedBy=timers.target
Key by key:
Type=oneshot, and it is not cosmetic. systemd.service(5), Type=:
“Behavior of oneshot is similar to exec; however, the service manager will
consider the unit up after the main process exits.” The default it replaces
reports far earlier — “simple (the default if ExecStart= is specified but
neither Type= nor BusName= are, and credentials are not used), the service
manager will consider the unit started immediately after the main service
process has been forked off (i.e. immediately after fork(), and before
various process attributes have been configured and in particular before the
new process has called execve() to invoke the actual service binary)” —
which is before advance has run a line.
Expect a oneshot service to read as dead rather than active between ticks; the
same entry says it “will never enter active unit state … it will not show up
as started afterwards, but as dead.” That is correct for a tick job. Do not add
RemainAfterExit= to make it look otherwise.Environment= is the unit’s way to set a variable for the process:
“Sets environment variables for executed processes.” (systemd.exec(5),
Environment=.)OnCalendar=*:0/15 is a repetition on the minute field. systemd.time(7),
“Calendar Events”: “Values may be suffixed with / and a repetition value,
which indicates that the value itself and the value plus all multiples of the
repetition value are matched.” The same page’s worked example is
*:2/3 → *-*-* *:02/3:00.WantedBy=timers.target is what enable acts on. systemd.unit(5),
[Install] section options: “A symbolic link is created in the .wants/ …
directory of each of the listed units when this unit is installed by
systemctl enable.”ExecStart=, for the reason in step 2.%h. A unit specifier would shorten the two absolute paths above, but
specifier semantics live in systemd.unit(5) § Specifiers, which was not read
for this document. An absolute path needs no specifier semantics and matches
what the other two units do. If you want %h, read that section first.Three settings are deliberately absent, and their defaults are the reason:
AccuracySec= — “Specify the accuracy the timer shall elapse with.
Defaults to 1min.” (systemd.timer(5).) A minute of slack on a
fifteen-minute tick is free. The same entry asks you to “make sure to set this
value as high as possible and as low as necessary”, so leave it alone.RandomizedDelaySec= — “Defaults to 0, indicating that no randomized
delay shall be applied.” (systemd.timer(5).) Nothing to unset.Persistent= — “Defaults to false”, “only has an effect on timers
configured with OnCalendar=”, and its purpose is “to catch up on missed runs
of the service when the system was powered down” (systemd.timer(5)). Leave
it at the default. That catch-up is the second scheduling authority this
design exists to avoid.This unit sets no output redirection, so the run’s output goes wherever your
manager sends it. systemd.exec(5), StandardOutput=: the setting “defaults to
the value set with DefaultStandardOutput= in systemd-system.conf(5), which
defaults to journal”. If you want a file instead, that entry lists
append:path among its values: “append:path is similar to file:path above,
but it opens the file in append mode.”
Install it:
systemctl --user daemon-reload
systemctl --user enable --now warpline-advance.timer
systemctl --user list-timers warpline-advance.timer
--user means “Talk to the service manager of the calling user, rather than the
service manager of the system” (systemctl(1), OPTIONS). daemon-reload will
“reload all unit files, and recreate the entire dependency tree”
(systemctl(1)). --now, “When used with enable … also start/stop/try-restart
the units after the specified unit file operations succeed” (systemctl(1)),
saves a separate start. list-timers will “List timer units currently in
memory, ordered by the time they elapse next” (systemctl(1)), which is how you
confirm the next tick is where you expect it.
<!-- ~/Library/LaunchAgents/dev.warpline.advance.plist -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key><string>dev.warpline.advance</string>
<key>Program</key><string>/usr/local/bin/node</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/bin/node</string>
<string>/opt/warpline/node_modules/.bin/warpline</string>
<string>advance</string>
</array>
<key>EnvironmentVariables</key>
<dict>
<key>WARPLINE_HOME</key><string>/Users/operator/.warpline</string>
<key>WARPLINE_TRIGGER</key><string>scheduled</string>
</dict>
<key>WorkingDirectory</key><string>/Users/operator</string>
<key>StartInterval</key><integer>900</integer>
<key>StandardOutPath</key><string>/Users/operator/.warpline/logs/advance.out</string>
<key>StandardErrorPath</key><string>/Users/operator/.warpline/logs/advance.err</string>
</dict>
</plist>
Key by key, all from launchd.plist(5):
StartInterval rather than StartCalendarInterval. StartInterval
“causes the job to be started every N seconds”, and it has no calendar or
timezone semantics at all, which removes the timezone question instead of
answering it. The calendar key behaves differently across sleep: “Unlike cron
which skips job invocations when the computer is asleep, launchd will start
the job the next time the computer wakes up. If multiple intervals transpire
before the computer is woken, those events will be coalesced into one event
upon wake from sleep.” Under StartInterval, “If the system is asleep during
the time of the next scheduled interval firing, that interval will be missed
due to shortcomings in kqueue(3).” A missed tick is fine here: the next one
is fifteen minutes away and the engine decides what is due.Program with an absolute path, per the quote in step 2.EnvironmentVariables — “used to specify additional environmental
variables to be set before running the job”.WorkingDirectory — “used to specify a directory to chdir(2) to before
running the job”. Set it. The page does not state a default, so do not build
on one.StandardOutPath / StandardErrorPath — each “specifies that the given
path should be mapped to” the job’s stdout or stderr, and “If the file does
not exist, it will be created …”. The page says nothing about the directory,
so create it yourself; the install steps below do.Two keys are deliberately absent. The keep-alive key is documented as being “used to control whether your job is to be kept continuously running”, and its “use … implicitly implies” the run-at-load key, “causing launchd to speculatively launch the job”. The run-at-load key carries its own warning: “This key should be avoided, as speculative job launches have an adverse effect on system-boot and user-login scenarios.” A continuously-running advance is a daemon, and warpline does not ship one.
One footnote so nobody blames warpline while testing: launchd throttles
respawns. “The value is in seconds, and by default, jobs will not be spawned
more than once every 10 seconds.” (launchd.plist(5), ThrottleInterval.) At
900 seconds it never bites. Tune the interval down to seconds and it will.
Install it:
mkdir -p ~/.warpline/logs
chmod 644 ~/Library/LaunchAgents/dev.warpline.advance.plist
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/dev.warpline.advance.plist
Remove it:
launchctl bootout gui/$(id -u)/dev.warpline.advance
The bootstrap | bootout entry in launchctl(1) takes a “domain-target
[service-path service-path2 …] | service-target”, and the target grammar is in
that page’s DESCRIPTION: gui/<uid>/[service-name] “targets the domain based on
which user it is associated with”. Its own worked example: “when referring to a
service with the identifier com.apple.example loaded into the GUI domain of a
user with UID 501, domain-target is gui/501/, service-name is
com.apple.example, and service-target is gui/501/com.apple.example.” Note
the page writes the domain target with a trailing slash; the service name is the
optional part of the specifier.
The older pair of subcommands is still documented rather than removed, which is
why a recipe written from memory keeps reaching for them and why the mistake
stays invisible. launchctl(1) prints, on its load | unload entry,
“Recommended alternative subcommands: bootstrap | bootout | enable | disable”.
Use the ones above.
WARPLINE_HOME=/home/operator/.warpline
WARPLINE_TRIGGER=scheduled
PATH=/usr/local/bin:/usr/bin:/bin
*/15 * * * * /usr/bin/node /opt/warpline/node_modules/.bin/warpline advance >> /home/operator/.warpline/logs/advance.log 2>&1
Create the log directory first, or the redirection fails on the first tick:
mkdir -p ~/.warpline/logs
*/15 is a step after an asterisk, which crontab(5) permits directly:
“Steps are also permitted after an asterisk, so if you want to say every two
hours'', just use*/2’’.”cron(8) examines cron entries once every minute” (crontab(5),
DESCRIPTION), and “The cron utility then wakes up every minute, examining all
stored crontabs” (cron(8), DESCRIPTION).crontab(5) documents
as “An active line in a crontab will be either an environment setting or a
cron command. An environment setting is of the form, name = value”.crontab(5) BUGS warns
only that “jobs scheduled during the rollback or advance will be affected”. A
*/15 job is not scheduled at an hour, so the worst case is one extra tick or
one missing one, and the engine’s freshness check absorbs both.On Linux your distribution ships cronie or Debian’s variant rather than the
Vixie cron those macOS pages describe. The two facts this entry rests on hold
there as well: cronie’s own crontab(5) source at tag cronie-1.7.2 carries
“Step values are also permitted after an asterisk, so if specifying a job to be
run every two hours, you can use “*/2””, and the same enumeration of SHELL,
LOGNAME and HOME with no mention of PATH anywhere in the page. Re-read it
on your own box anyway; see the last section.
Pick one. They carry no order among themselves, and nothing in this document sequences them, because there is nothing to sequence: each is a complete way to call the same verb every fifteen minutes.
Two of them installed against one home is not twice the coverage. It is the collision the next section is about.
The run lock is not what stops a scheduler starting a second copy of a job that is still running. Two of the three already do that themselves, and their own documentation says so:
systemd.timer(5), DESCRIPTION: “Note that
in case the unit to activate is already active at the time the timer elapses
it is not restarted, but simply left running. There is no concept of spawning
new service instances in this case.”StartInterval. launchd.plist(5):
“If the job is running during an interval firing, that interval firing will
likewise be missed.”crontab(5) nor cron(8)
makes any statement about skipping a firing while a previous instance of the
same job is running, and neither does cronie’s crontab(5). Silence is not a
guarantee. Treat cron as able to overlap.So the lock is load-bearing in exactly two places. It is a cron operator’s only
protection, and it is everybody’s protection against a manual warpline advance
racing a scheduled one, because nothing in any scheduler knows about a human at
a terminal. Two schedulers installed against one home collide the same way.
When two advances do meet, the second exits 75, names the holder, and says
that nothing ran. runtime-spec.md § 12 is the whole of it, including when a
lock left behind by a dead process heals on the next tick (the lock and this
machine carry the same known host identifier), and when it waits out the two
hours from its last heartbeat, or from when it was taken if it carries none it
can use, instead.
review_gate before the first tickreview_gate defaults to true, and on that default every plugin declaring
autonomy_level: autonomous is treated as supervised. The plugin still runs —
this gate decides what happens to the RESULT, not whether the handler is
invoked — and then that result parks at an approval gate waiting for a human.
The advance reports partial, the plugin’s state in the --json document is
gated, and the exit code is 0. A plugin whose result is failed is not
parked. Its state reads failed, and the advance exits 1, just as it does
with review_gate off.
That is the behaviour for a plugin declaring no side effects. A plugin whose
manifest declares any is stopped by a second and separate gate, before its
handler is invoked at all, and it reports skipped rather than gated. No
review_gate setting reaches that one: turning review_gate off does not
approve a side effect, and runtime-spec.md § 9 is the gate’s specification.
So a fresh install walked through exactly as this page describes parks everything on its first tick:
{"run_id":"...","status":"partial","gated":1,"pending_gates":1,
"failed":0,"refused":0,"pruned":0,"exit_code":0,"refused_plugins":[],
"audit":{"seq":2,"bytes":913,"segments":1,"checkpoint_seq":2,"indeterminate":[]},
"plugins":[{"name":"metrics-rollup","state":"gated"}]}
rc=0
Every later tick waits on the same gates. The plugin is not due again yet, so it
reads skipped, status reads complete and gated reads 0, because
gated counts what this tick parked. The dead-man file’s pending_gates stays
1 until somebody answers, and it is the field to watch.
audit is the audit store as this tick’s Checkpoint left it (runtime-spec.md
§ 14). audit.indeterminate is the field to watch for fires with no outcome:
each entry is a side-effecting fire whose process died, or whose outcome could
not be recorded, after its intent was written. Warpline does not hold the
plugin on its account. Whether the effect happened is for you to check. Once
you know, answer it with warpline resolve --intent <seq> --shipped or
--not-shipped, and the next tick no longer lists it. A content fire whose
approval still reads indeterminate is the exception. Answer it with
warpline resolve <plugin> --shipped <effect-id> or `–not-shipped