Clockwork

Clockwork is the scheduler of a Korall Engineering™ node. It keeps a book of stored schedules. Each schedule has a time rule and a batch job type. At each time of the rule, clockwork submits one batch job, and the batch system does the work.

Clockwork remembers what it fired, what it missed and how each run ended. So it can wait for the last run before it starts the next. After the node was off, it knows which runs it missed, and it starts them at once.

Every node has clockwork: Hub, Agent and Remote.

The schedule

A schedule also keeps its state: the last time that it dealt with, the next time, the firings that wait, the batch job that is busy, and its last 200 runs.

The two syntaxes

Cron

Five fields: minute, hour, day of month, month and day of the week.

Each field takes *, a number, a range a-b, a list a,b,c, and a step */n or a-b/n. A name can be in any case.

When the day of month and the day of the week both start with something other than *, a day matches when either one matches. When one of them starts with *, both must match. This is the rule of Vixie cron.

The short forms:

Every

every(N).unit, then optionally .at(HH:MM) and .jitter(S).

The singular forms (minute, day) are the same units. .at() is only for days and weeks. .jitter(S) sets the jitter, from 0 to 3600 seconds. A rule that sets .jitter() needs the jitter field of the schedule at 0.

Refused rules

Clockwork refuses a rule that it cannot fire, and the refusal names the field and the reason. The editor, the OPAL tasks and the book use the one parser.

Time, zones and summer time

Clockwork evaluates a cron rule and an .at() rule in the zone of the schedule. It evaluates an elapsed rule in UTC. It stores every time in UTC.

These rules are the rules of Vixie cron.

A nominal time is the time that the rule names. An actual time is the time when clockwork submitted the job. The two differ by the jitter, by a wait for the last run, and by a catch-up. Clockwork counts nominal times.

A firing

Each firing becomes one batch job:

reason is scheduled, catch-up or manual. A job of a catch-up carries its own nominal time, so a job that does the work of one period can use it.

Clockwork submits through submitOrGet(). If the queue has a job with the same correlation key, it returns that job, so no second job starts for one nominal time. That holds after a crash between the submit and the save of the book.

Overlap

A run is busy from its submit until its batch job is done, failed, stopped or canceled.

Under wait-for-last, one scheduled firing waits at most. A newer one takes its place, and the history records the older one as coalesced. The firings of a catch-up and of a run now keep their place.

A waiting firing keeps the command and the arguments that the schedule had when the firing became due. An edit of the schedule changes the later firings.

A job that no worker runs stays pending in the queue. Its run stays busy, so the next firing waits or is skipped. The Batch screen says why no worker takes the job.

Catch-up

When the node was off, or the computer slept, or the clock jumped forward by more than one period, clockwork finds the runs that it missed.

The runs of a catch-up go through the overlap policy, so under wait-for-last they run one after another. They take no jitter.

Example: a schedule fires every day at 02:00. The node is off from the 13th at 20:00 to the 17th at 09:00. At the start, clockwork submits the run of the 14th. When it ends, it submits the run of the 15th, then the 16th, then the 17th. The next firing is the 18th at 02:00.

A clock that goes back fires nothing again. The next firing is the first nominal time after the last one that clockwork dealt with.

A disabled schedule counts no missed run. When it is enabled again, it starts with its next time.

The book and the lock

The book is the file clockwork.json in the base folder of the node.

• Each write is the whole file, so a crash leaves the old file or the new one.
• A book of a newer schema version is refused. Nothing fires until a clockwork that reads that version runs, and the file is not changed.
• A book that does not parse is moved aside as clockwork.json.broken-<time>, and the book starts empty.

Two nodes can share one base folder. The file clockwork.lock in the folder decides which node fires. The other node shows the schedules read-only, names the node that fires them, and tries the lock again every 60 seconds. When the first node ends, the second one takes over.

The node switch

The setting clockwork.enabled, "Run stored schedules", is in the node schema. Off: no schedule fires, and the time that it is off counts no missed run.

The OPAL tasks

clockwork.put takes the fields of the schedule by name. A refused rule answers with the field and the reason of the parser:

{"task": "clockwork.put", "args": {"name": "Nightly audit", "rule": "0 2 * * *", "command": "trove.recurring.run", "arguments": {"spec": "wfc7wjvxtnw"}}}

Run outcomes