5. OpenAI-compatible server

stokastisk.serving runs the bundled vllm serve as a supervised child process with the T4 defaults, and offers small client helpers. Use it from a notebook cell or script.

from stokastisk import serving

with serving.ServerProcess("Qwen/Qwen2.5-0.5B-Instruct") as srv:
    srv.wait_ready()
    print(serving.chat(srv.base_url, srv.served_model_name, [{"role": "user", "content": "Hi"}]))
    print("".join(serving.stream_chat(srv.base_url, srv.served_model_name,
                                      [{"role": "user", "content": "Count to five."}])))

Leaving the with block stops the server (it is never started implicitly or left running).

ServerProcess(model, config=None, *, ...)

Argument Default Meaning
model — model id or path
config StokastiskConfig() T4 defaults; replica_parallel is refused
host, port 127.0.0.1, 8000 find_free_port() gives a free one
served_model_name model name clients send
quantization None refused if the method needs a GPU newer than SM75
extra_args () additional vllm serve arguments
cuda_visible_devices None e.g. "0"
log_path temp file server output; kept after stop()

Always passed: --attention-backend TRITON_ATTN, FP16, pinned xgrammar.

Member Purpose
start() / wait_ready(timeout=600) / start_and_wait() start, poll /health until 200
health() True if /health answers now
restart() graceful stop then fresh start on the same port
stop(grace=30) SIGINT → SIGTERM → SIGKILL on the process group; returns the exit code
base_url, pid, running, command, log_path, served_model_name state
log_tail(max_lines) last lines of the server log
shutdown_report which signals were needed, whether children survived

If start-up fails, ServerStartupError carries the exit code, the exact command, the log path and the last log lines.

Client helpers

Function Returns
chat(base_url, model, messages, *, max_tokens=64, temperature=0.0, timeout=120) assistant text
stream_chat(...) generator of text deltas; a truncated stream raises ServerHTTPError
list_models(base_url) /v1/models response
fetch_metrics(base_url) Prometheus /metrics parsed to {sample: value}

You can also call the server with any OpenAI client using base_url=f"{srv.base_url}/v1" and any API key.

Utilities

build_serve_command(model, config, ...) returns the exact argv (pure function, nothing started). find_free_port(), port_is_free(host, port), check_server_requirements() (lists missing serving packages), validate_model_id, validate_host, validate_port.

Next: Using both GPUs.