Skip to Content

Streams

Frames, resuming, reconnecting and the events stream of the WebSockets in rookeryd and a Rookery server.

Anything that keeps producing output is a streaming operation, such as sessions.attach, logs.follow or channel.listen. A client opens one as a subscription on the portal, next to other calls and streams, or as a WebSocket of its own at the operation's path. The frames are the same either way. This page covers what all streams have in common. The API reference lists each stream's arguments and frames.

Opening a stream

On the portal, send the operation as a request and the stream's frames follow under its number (A stream). On its own, a stream is a WebSocket opened at the path of the thing streamed. A session's output is at /v1/sessions/{id}, the channel at /v1/channel, and a service's serving connection at /v1/services/{service}/serve. Open it with a GET and the usual upgrade headers, on the same unix socket as the plain requests.

rookery serve passes a host's streams through under /v1/hosts/{name}/controller, so the session above is at /v1/hosts/{name}/controller/sessions/{id} there, with the same frames. Over TCP the token from serve.token goes in an Authorization: Bearer header. A browser cannot set a header on a WebSocket, so the token is also accepted as the Sec-WebSocket-Protocol value.

What a client has to say back goes over the same socket. A client never needs a second request to type into a session or to resize it.

Text and binary frames

Text frames are JSON, one message each. Binary frames hold only session bytes (and, on the workstation, a job's output bytes). Everything else is a text frame.

Bytes that appear inside JSON are base64, in a field whose name ends in _b64.

The session stream

A session has two kinds, and the binary frames differ.

A terminal. Binary frames are the terminal's bytes in both directions. The server sends what the program wrote. The client sends what was typed. The attach add-on of xterm.js expects exactly this. A browser terminal connects directly.

A process. Its streams are separate, so the first byte of every binary frame says which one:

First byteDirectionMeaning
0client to serverstandard input
1server to clientstandard output
2server to clientstandard error

The rest of the frame is the bytes.

Text frames are the same for both kinds:

FrameFromMeaning
{"resize":{"cols":120,"rows":40}}clienta terminal's new size
{"signal":"INT"}clientINT, TERM, HUP or KILL to the process group
{"eof":true}clientclose a process's standard input
{"exited":{"exit_code":0}}serverthe command ended; this is the last frame
{"dropped":{"bytes":65536}}serverthis viewer was too slow and missed output

Any number of viewers can be connected to one session. A terminal has the size that the last viewer to resize it asked for.

The stream of a port

ports.connect is a TCP connection to a port on the host's loopback, 127.0.0.1. It is at /v1/ports/{port}, and through rookery serve at /v1/hosts/{name}/controller/ports/{port}. rookeryd connects to the port before the handshake, so a port that nothing listens on gives the error 503 not_served. It needs the role operate and the host level.

Binary frames are the connection's bytes in both directions. Either side ends its direction with a text frame:

FrameFromMeaning
{"eof":true}clientThe client sends no more. The program on the port reads the end, and what it sends still comes back
{"eof":true}serverThe program closed its side. Bytes sent towards it still arrive

The stream ends once both sides have sent {"eof":true}. It ends at once when the client closes the WebSocket or the connection to the port breaks. A stream that closes before its {"eof":true} has lost its connection. rookery_api::tunnel reads and writes such a stream like a socket.

The stream of an address on a host

endpoints.connect of rookeryd is a TCP connection to an address on one of the host's own networks, such as a container on a bridge. It is at /v1/endpoints/{address}/{port}, and its frames are those of the stream of a port. Only rookery serve opens it, for an endpoint of a resource on a node. It does not pass it through under /v1/hosts/{name}/controller, and a portal refuses it.

rookeryd resolves the address once, checks every address it has and connects to the one it checked, all before the handshake:

ErrorWhen
403 forbiddenThe address is a loopback, link-local or metadata address, or on none of the host's networks
502 unreachable, 504 timeoutThe name does not resolve, or not in time
503 not_served, 504 timeoutThe address refuses the connection, or doesn't answer in time

A network of the host is the subnet of an address of one of its interfaces that are up, without the loopback. A prefix shorter than /8 for IPv4 or /16 for IPv6 doesn't count.

The stream of an endpoint

resources.connect of rookery serve is a TCP connection to an endpoint of a resource, one that its provisioner reported (Endpoints). It is at /v1/resources/{id}/endpoints/{endpoint}, where endpoint is the endpoint's name or its port. Its frames are those of the stream of a port.

The server reads the endpoints from the resource's record, then resolves the address, checks it and connects, all before the handshake. It needs the role operate. What it can refuse:

ErrorWhen
404 not_foundNo such resource, or its provisioner did not report that endpoint
400 bad_requestThe endpoint is UDP
403 forbiddenThe address is one Rookery does not dial from the server
503 not_served, 504 timeoutThe endpoint refuses the connection, or doesn't answer in time
502 unreachableThe provisioner is on a node that cannot be reached
501 not_availableThe provisioner is on a node whose controller is older than endpoints.connect

Who dials the endpoint:

Where the provisioner isWho dials
On a nodeThe node's controller: ports.connect for 127.0.0.1, endpoints.connect for any other address
Anywhere elseThe server itself

Portal::connect_endpoint and rookery_api::tunnel::endpoint_tunnel open one.

Resuming

A stream that can be resumed takes since= and starts there. The page of each stream says whether it can. What since counts depends on the stream:

Streamsince is
a sessiona byte offset from the start of the session's output
the channela message id; messages after it come first
the logsa journal cursor from an earlier entry, or a time that journalctl understands
a joba byte offset into the job's output

Without since, a terminal starts with a replay of its current screen. A process starts at the beginning of its output.

Reconnecting

A socket ends when the network does, when rookeryd restarts, or when the client closes it. To carry on where it stopped:

  1. Count the bytes of a session as they arrive, or remember the id of the last channel message, or the cursor of the last log entry.
  2. Open the stream again with since= set to that value.
  3. Wait a little longer after each failure, and start over from a short wait after a connection works.

A session itself does not depend on the connection. It keeps running while nobody is connected, and its output is kept in a transcript of up to 16 MB. If rookeryd itself restarts, its sessions end. They are listed as ended with lost: true, and a stream of one of them can still be read from its transcript.

box serve does the reconnecting for a service, with a backoff, so a service written in shell needs nothing for it.

A slow listener is dropped

A client that cannot keep up does not slow the others down:

  • On a session, the stream carries on, the viewer misses output, and the server sends a dropped frame with how much it missed. Read the missing range with GET /v1/sessions/{id}/output?since=OFFSET, or reconnect with since=.
  • On the channel, a listener that falls 1,000 messages behind is closed. Open it again with since set to the last id it saw.
  • On events.follow, inbox.follow and kv.follow of rookery serve, a follower that is too slow gets {"dropped":{"events":12}} and the stream goes on. The number counts all the server's events it missed, also those it does not follow. Read the current state again with inbox.list, kv.list or jobs.list.

Events on your machine

rookery serve has one stream, events.follow, that follows several things at once. On the portal it is one subscription; on its own it is the WebSocket at:

GET /v1/events

The first frame the client sends says what to follow. It is sent once, first:

{ "hosts": ["studio"], "jobs": true, "kv": ["ui"], "topics": ["myplugin/"], "changes": ["hosts", "resources"] }
FieldFollows
hosts, topicsChannel messages, by host and by topic prefix. All hosts when hosts is empty
jobsJob starts and ends
kvChanges in these key-value namespaces
changesChanges to the saved hosts and to resources

Each frame the server sends names where it came from in source:

{ "source": "kv", "namespace": "ui", "key": "layout", "revision": 4, "value": {} }
sourceThe frame also has
channelThe host and the message
jobThe job
kvThe namespace, the key, and the value and revision, or removed
hostThe host and the change: added, removed, renamed or changed. A rename has the old name in from
resourceThe resource id, its provisioner and kind, and the change: created, removed or changed

A host or resource frame only names what changed. Read the rest with hosts.get or resources.get. A host changes when its profile is saved, as by hosts.save or an action, and a resource when its record does, as when a job creates it or its state is checked. A change that another Rookery makes in the store is told within a few seconds.

{ "source": "host", "host": "forge", "change": "renamed", "from": "studio" }

Events are not resumable. After a reconnect, read what was missed with inbox.list, kv.list, hosts.list and resources.list, then follow again. inbox.follow and kv.follow follow one of them alone.

Edit on GitHub

Last updated on

On this page