Skip to Content

The portal

One WebSocket for every operation, calls and streams at once, on rookeryd and on a Rookery server.

The portal is a WebSocket at /v1/portal. A client opens it once and sends operations as JSON frames, each with a number of its own, so many calls and streams share the connection. Rookery in server mode reaches the server's hosts this way.

Opening it

ProgramWhere
rookerydGET /v1/portal on /run/rookery/control.sock for host level, or on agent.sock for agent level
rookery serveGET /v1/portal on its unix socket, or over TCP with the token in Authorization: Bearer. A browser, which cannot set that header, gives the token as the Sec-WebSocket-Protocol value

A call

The client numbers each request with id, an integer below 2^32 that it does not reuse while that request or its stream is open. The answer comes back under the same number, in any order:

→ {"id": 7, "op": "sessions.start", "args": {"wait": "30s"}, "data": {"kind": "process", "command": "uname -a"}}
← {"id": 7, "result": {"id": "k3v9qz2m8w1x", "exit_code": 0, "stdout_b64": "TGludXggc3R1ZGlvIC4uLgo="}}
FieldHoldsOver HTTP it is
idThe request's numbernone
opThe operation's namethe method and path
argsThe operation's argumentsthe path, the query and some headers
dataThe operation's inputthe body
hostOn rookery serve only: the host whose operation this isthe /v1/hosts/NAME prefix

An error comes back instead of a result, with the same fields as an HTTP error (Errors):

← {"id": 8, "error": {"code": "not_found", "message": "No session nothing", "status": 404}}

An unknown operation is 404, and a missing argument is 400.

A host's operation through rookery serve

On the portal of rookery serve, host names the host and the rest of the frame is the host's operation, exactly as rookeryd takes it:

→ {"id": 9, "host": "studio", "op": "actions.run", "args": {"action": "status"}}
← {"id": 9, "result": {"id": "a81kq0wd3m5c"}}

rookery serve refuses the controller's own secrets action, because the secrets stay on the machine that runs it.

A stream

The reply to a streaming operation is {"open": true}. The server then sends frames under the same number, as sub, until either side ends it:

→ {"id": 10, "op": "sessions.attach", "args": {"id": "k3v9qz2m8w1x", "since": 0}}
← {"id": 10, "result": {"open": true}}
← binary: 00 00 00 0a, then the session's bytes
→ binary: 00 00 00 0a, then typed bytes
→ {"sub": 10, "data": {"resize": {"cols": 120, "rows": 40}}}
← {"sub": 10, "data": {"exited": {"exit_code": 0}}}
← {"sub": 10, "end": true}
FrameDirectionMeans
{"sub": N, "data": …}bothA JSON frame of the stream
binarybothN as a big-endian 32-bit integer, then the stream's bytes
{"sub": N, "end": true}bothThe last frame. From the server it may include error

The frames inside data and the bytes after the four-byte prefix are those of the stream's own WebSocket, listed on each stream's page. Calls go on beside open streams.

A window on a stream's bytes

When nobody reads a stream, its bytes would fill the other side's memory or hold up the frames of the other streams. A client asks for a window when it opens the stream to prevent this:

→ {"id": 12, "op": "ports.connect", "args": {"port": 8080}, "window": 262144}
← {"id": 12, "result": {"open": true, "window": 262144}}
→ binary: 00 00 00 0c, then up to 262144 bytes
← {"sub": 12, "credit": 131072}
RuleMeans
window in the requestHow many bytes of binary frames each side may send before the other side credits them. The server takes 16 KB to 4 MB
window in the answerThe server keeps to that window. Without it, no window applies to the stream
{"sub": N, "credit": BYTES}Either side has taken so many bytes, and the other may send that many more

A side may send one frame beyond the window, so a large frame isn't held forever. JSON frames don't count against it. Each side says what it took once it has taken half the window. Rookery's client asks for a window on every stream it opens, and a client that asks for none gets the streams as before.

Bytes

Bytes in input or a result are {"base64": "…"}, and a result adds content_type. The portal takes up to 8 MB of bytes per operation, and a larger result is a 413 error. Use HTTP for larger transfers, such as big files.

→ {"id": 11, "op": "files.write", "args": {"path": "/srv/agent/workspaces/notes.txt"}, "data": {"base64": "aGVsbG8K"}}
← {"id": 11, "result": {"ok": true}}

Limits

The server keeps a fixed amount of memory for each portal, so a caller that sends without reading can't fill it. rookeryd and rookery serve keep to the same limits:

LimitValuePast it
One messageThe 8 MB of an operation as base64, and 1 MB moreThe portal closes
The name of an operation128 bytesA 400 error that doesn't repeat the name
An error's message2 KBThe message is cut short
Calls in flight256A 429 error with the code limit
Streams open256, such as the connections of a forwarded portA 429 error with the code limit
Calls, their answers until they are sent, and input waiting for its stream64 MBA call is refused with a 429 error with the code limit, and a stream ends with that error in its last frame
The portal's own frames waiting to be sent, such as errors and credits64 KBThe server reads nothing more from the caller until they are sent

A caller that keeps to its windows and reads what comes never reaches these limits.

Rookery's client holds up to 16 MB of a stream that it has not read. A server that sends more ends that stream with an error.

When to use HTTP instead

  • One-off calls from a shell, with curl.
  • Transfers of files and screenshots larger than 8 MB.
  • Operations the reference marks as not on the portal, which answer 501 there.

Clients

Rookery's own client is rookery_api::portal in crates/rookery-api/src/portal.rs. The tests the_portal_carries_calls_and_streams_together for rookeryd and the_portal_reaches_serve_and_its_hosts for rookery serve show calls, streams, bytes and errors in use.

Edit on GitHub

Last updated on

On this page