mesh - Clockwork Remote Orchestration Framework


Clockwork is a system configuration management system designed to securely enforce policies on one or more host systems. It can ensure that files have the prescribed attributes (owner, group, permissions, etc.) and content. It can maintain a specific set of installed packages, and even make sure that system user accounts exist.

With more than a handful of systems, passive orchestration (via policies and cogd runs) starts to lose its luster. This is where Mesh comes in. Mesh lets you run queries in parallel across all connected nodes in the environment, with full-blown access control and user-specified filtering.

It's difficult to describe just how useful Mesh is, so let's start with some examples.

Want to find out what what version of Clockwork is installed?

$ cw show version
Password: 0 2.3.0 0 2.3.0 0 2.3.0 0 2.3.0 0 2.3.0 0 2.3.0 0 2.3.0 0 2.3.0 0 2.3.0 0 2.3.0

What if you only care about the web boxes?

$ cw show version -w sys.fqdn=/^web/
Password: 0 2.3.0 0 2.3.0 0 2.3.0 0 2.3.0 0 2.3.0 0 2.3.0

We've got lots of plans for Mesh, including the ability to query resource states (packages, files, users, services, etc.), manage resources actively (install a package, restart a service) and execute commands.


This section desribes the commands that you can run, either via cw-mesh(1) or dedicated command aliases like cw-ping(1) and cw-show(1).


Similar to the network utility of the same name, except without the round-trip timings.

show version

Reports the version of Clockwork on each node.


At its core, Mesh is a Pendulum code broadcaster with a back-channel, or control channel, for result submission. cogd(1) nodes subscribe to the broadcast channel and read COMMAND messages off the wire. They then look at the embedded filters to determine if they should handle the request or opt out.

Optout results in an OPTOUT message being sent via the control channel. This message contains the serial number of the original COMMAND message, so that meshd(1) can relay that result back to the original requester.

If the node decides to act on the COMMAND, it will check the user and group information supplied with the request against its own access control list. This ACL is managed directly by the Clockwork policy (see clockwork(7) for more information). If the user/group/command combination is allowed by the ACL, the Pendulum code in the payload of the COMMAND message is executed. The return code and output (stdout only) is then submitted back across the control channel as a RESULT message.

There are a few key takeaways here:

First, the meshd daemon does not know anything about the population of subscribed cogd nodes. They do not register explicitly, and it does not keep track of who has subscribed or dropped off. This is by design, and it allows the whole system to be more tolerant of real-world issues like network congestion, host crashes, etc.

Secondly, Mesh clients will receive a response from every subscribed node (barring unforeseen issues during the execution of Pendulum code), whether they participated in the query or not. You won't normally see these since the cw-mesh utility filters them out before printing results. See cw-mesh, specifically the --optouts flag.

Finally, authentication is done at meshd; subscribed nodes are only responsible for authorization. Put another way, cogd nodes implicitly trust that meshd is being truthful about who requested the command.


meshd supports password authentication, which it delegates to PAM for credentials verification, and key-based authentication.

In key-based authentication, the user provides proof that she has access to a private key (usually ~/clockwork/mesh.key) by signing a small random message and sending that with the request, alongside her public key (in base-16 encoding). meshd then verifies the signature (to prove that the correct private key was used) and checks to see if the public key has been authorized for the requesting user. At no time is the private key ever sent across the wire. Since the entire exchange is itself encrypted, replay attacks are minimizied.


Clockwork was designed and written by James Hunt.

The Clockwork website is licensed under the Creative Commons Attribution-NoDerivs 3.0 United States License