Streaming

Set "stream": true and the reply arrives as server-sent events in OpenAI's chat.completion.chunk format. The first chunk carries the role with empty content, then come the pieces of text, then a chunk with a finish_reason, and the stream ends with data: [DONE]. When an idle host has the model loaded, the first text usually shows up in under 5 seconds.

curl -N https://ballast.outerscope.xyz/v1/chat/completions \
  -H "Authorization: Bearer $BALLAST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3.1-8b-instruct", "stream": true, "messages": [{"role": "user", "content": "Count to five."}]}'

import os
from openai import OpenAI

client = OpenAI(base_url="https://ballast.outerscope.xyz/v1", api_key=os.environ["BALLAST_API_KEY"])

stream = client.chat.completions.create(
    model="llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "Count to five."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://ballast.outerscope.xyz/v1", apiKey: process.env.BALLAST_API_KEY });

const stream = await client.chat.completions.create({
  model: "llama-3.1-8b-instruct",
  messages: [{ role: "user", content: "Count to five." }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

Token counts while streaming

Add "stream_options": {"include_usage": true} and the stream gets one more chunk before [DONE]. Its choices list is empty and its usage field holds the prompt, completion and total token counts.

If a host drops

Before the first token, Ballast retries on another host, up to 3 tries and never the same host twice. You see one unbroken reply and you pay once. After the first token has reached you, the stream cannot move to a different host. If the host dies then, the stream ends with finish_reason: "error" and you pay only for the tokens that arrived.

Time and length

A request runs for at most 120 seconds. max_tokens is capped at 4,096. A reply that hits either one stops there and comes back with finish_reason: "length", and you are billed for the tokens produced.