The portal
One WebSocket for every operation, calls and streams at once, on rookeryd and on a Rookery server.
The portal is a WebSocket at /v1/portal. A client opens it once and sends operations as JSON frames, each with a number of its own, so many calls and streams share the connection. Rookery in server mode reaches the server's hosts this way.
Opening it
| Program | Where |
|---|---|
| rookeryd | GET /v1/portal on /run/rookery/control.sock for host level, or on agent.sock for agent level |
rookery serve | GET /v1/portal on its unix socket, or over TCP with the token in Authorization: Bearer. A browser, which cannot set that header, gives the token as the Sec-WebSocket-Protocol value |
A call
The client numbers each request with id, an integer below 2^32 that it does not reuse while that request or its stream is open. The answer comes back under the same number, in any order:
→ {"id": 7, "op": "sessions.start", "args": {"wait": "30s"}, "data": {"kind": "process", "command": "uname -a"}}
← {"id": 7, "result": {"id": "k3v9qz2m8w1x", "exit_code": 0, "stdout_b64": "TGludXggc3R1ZGlvIC4uLgo="}}| Field | Holds | Over HTTP it is |
|---|---|---|
id | The request's number | none |
op | The operation's name | the method and path |
args | The operation's arguments | the path, the query and some headers |
data | The operation's input | the body |
host | On rookery serve only: the host whose operation this is | the /v1/hosts/NAME prefix |
An error comes back instead of a result, with the same fields as an HTTP error (Errors):
← {"id": 8, "error": {"code": "not_found", "message": "No session nothing", "status": 404}}An unknown operation is 404, and a missing argument is 400.
A host's operation through rookery serve
On the portal of rookery serve, host names the host and the rest of the frame is the host's operation, exactly as rookeryd takes it:
→ {"id": 9, "host": "studio", "op": "actions.run", "args": {"action": "status"}}
← {"id": 9, "result": {"id": "a81kq0wd3m5c"}}rookery serve refuses the controller's own secrets action, because the secrets stay on the machine that runs it.
A stream
The reply to a streaming operation is {"open": true}. The server then sends frames under the same number, as sub, until either side ends it:
→ {"id": 10, "op": "sessions.attach", "args": {"id": "k3v9qz2m8w1x", "since": 0}}
← {"id": 10, "result": {"open": true}}
← binary: 00 00 00 0a, then the session's bytes
→ binary: 00 00 00 0a, then typed bytes
→ {"sub": 10, "data": {"resize": {"cols": 120, "rows": 40}}}
← {"sub": 10, "data": {"exited": {"exit_code": 0}}}
← {"sub": 10, "end": true}| Frame | Direction | Means |
|---|---|---|
{"sub": N, "data": …} | both | A JSON frame of the stream |
| binary | both | N as a big-endian 32-bit integer, then the stream's bytes |
{"sub": N, "end": true} | both | The last frame. From the server it may include error |
The frames inside data and the bytes after the four-byte prefix are those of the stream's own WebSocket, listed on each stream's page. Calls go on beside open streams.
A window on a stream's bytes
When nobody reads a stream, its bytes would fill the other side's memory or hold up the frames of the other streams. A client asks for a window when it opens the stream to prevent this:
→ {"id": 12, "op": "ports.connect", "args": {"port": 8080}, "window": 262144}
← {"id": 12, "result": {"open": true, "window": 262144}}
→ binary: 00 00 00 0c, then up to 262144 bytes
← {"sub": 12, "credit": 131072}| Rule | Means |
|---|---|
window in the request | How many bytes of binary frames each side may send before the other side credits them. The server takes 16 KB to 4 MB |
window in the answer | The server keeps to that window. Without it, no window applies to the stream |
{"sub": N, "credit": BYTES} | Either side has taken so many bytes, and the other may send that many more |
A side may send one frame beyond the window, so a large frame isn't held forever. JSON frames don't count against it. Each side says what it took once it has taken half the window. Rookery's client asks for a window on every stream it opens, and a client that asks for none gets the streams as before.
Bytes
Bytes in input or a result are {"base64": "…"}, and a result adds content_type. The portal takes up to 8 MB of bytes per operation, and a larger result is a 413 error. Use HTTP for larger transfers, such as big files.
→ {"id": 11, "op": "files.write", "args": {"path": "/srv/agent/workspaces/notes.txt"}, "data": {"base64": "aGVsbG8K"}}
← {"id": 11, "result": {"ok": true}}Limits
The server keeps a fixed amount of memory for each portal, so a caller that sends without reading can't fill it. rookeryd and rookery serve keep to the same limits:
| Limit | Value | Past it |
|---|---|---|
| One message | The 8 MB of an operation as base64, and 1 MB more | The portal closes |
| The name of an operation | 128 bytes | A 400 error that doesn't repeat the name |
| An error's message | 2 KB | The message is cut short |
| Calls in flight | 256 | A 429 error with the code limit |
| Streams open | 256, such as the connections of a forwarded port | A 429 error with the code limit |
| Calls, their answers until they are sent, and input waiting for its stream | 64 MB | A call is refused with a 429 error with the code limit, and a stream ends with that error in its last frame |
| The portal's own frames waiting to be sent, such as errors and credits | 64 KB | The server reads nothing more from the caller until they are sent |
A caller that keeps to its windows and reads what comes never reaches these limits.
Rookery's client holds up to 16 MB of a stream that it has not read. A server that sends more ends that stream with an error.
When to use HTTP instead
- One-off calls from a shell, with
curl. - Transfers of files and screenshots larger than 8 MB.
- Operations the reference marks as not on the portal, which answer
501there.
Clients
Rookery's own client is rookery_api::portal in crates/rookery-api/src/portal.rs. The tests the_portal_carries_calls_and_streams_together for rookeryd and the_portal_reaches_serve_and_its_hosts for rookery serve show calls, streams, bytes and errors in use.
Last updated on