API reference
Every Python module of the stokastisk package and its public names. Generated from the source by scripts/release/build_docs_site.py.
Names exported at the top level (import stokastisk) are in stokastisk.__all__; submodule names are imported as shown, e.g. from stokastisk.serving import ServerProcess.
stokastisk
Stokastisk: dual-Tesla-T4 (SM75, CUDA 13.0) SDK around the bundled vLLM 0.25.1 engine.
stokastisk.benchmarks
Reproducible dual-GPU benchmarks (import lazily: they need vLLM / CUDA only when run).
stokastisk.benchmarks.baseline
Shared engine benchmark: identical workload for every strategy, ALL runs recorded (median/p95/min/max, never only the best).
| Name | Summary |
|---|---|
class Workload |
|
Workload.prompts(self) |
|
def measure(llm, workload: Workload, sampler_interval: float=0.5) |
Time workload on an already-loaded engine. Warm-up runs are discarded; every measured repeat is kept. |
def run_engine(config, model: str, workloads: 'Workload | list[Workload]', *, label: str | None=None, sampler_interval: float=0.5) |
Load ONE in-process stokastisk.LLM and measure every workload on it. Call from a __main__-guarded script (vLLM spawns). |
stokastisk.benchmarks.communication
Transfer-size sweep: host-staged (torch / native C++) vs NCCL send/recv. NCCL runs inside a 2-rank torchrun job.
| Name | Summary |
|---|---|
def bench_host_staged(src: int=0, dst: int=1, *, sizes=SIZES, iters: int=20, warmup: int=5, chunk_bytes: int=4 << 20, n_buffers: int=2, native=None) |
Verifies the payload (byte-exact) on every size, then times iters transfers. native = NativeHostStaged class to use C++ instead. |
def bench_nccl(rank: int, *, sizes=SIZES, iters: int=20, warmup: int=5) |
Inside an initialised 2-rank NCCL group. Each rank returns its own view; rank 1 (receiver) completion time is the transfer latency. |
def compare(host_staged: dict, nccl: dict) |
Per size: ratio of median latencies and which transport wins. Decision rule: host-staged is integrated nowhere unless it wins. |
stokastisk.benchmarks.gpu_balance
Compute / communication / idle accounting and imbalance for replica runs.
| Name | Summary |
|---|---|
def from_replica_record(rec: dict[str, Any], workload: str) |
stokastisk.benchmarks.pipeline_parallel
PP=2 (experimental). Stage-level compute/communication times are NOT exposed by vLLM's public API: they are reported as NOT_MEASURED
| Name | Summary |
|---|---|
def config_for(**kw) |
|
def run(model: str, workloads, **kw) |
stokastisk.benchmarks.replica_parallel
Independent TP=1 replicas vs one engine. Offline batch generation reports BATCH completion latency (individual request completion
| Name | Summary |
|---|---|
def run(model: str, workloads, devices=(0, 1), **kw) |
stokastisk.benchmarks.report
Combine benchmark records into one JSON + a markdown table. Missing strategies are listed as NOT_RUN, never omitted silently.
| Name | Summary |
|---|---|
def comparison_table(records: dict[str, dict[str, Any]], workload: str) |
|
def relative_to(records: dict[str, dict[str, Any]], workload: str, base: str='tp1') |
stokastisk.benchmarks.tensor_parallel
TP=1 vs TP=2 on an identical workload.
| Name | Summary |
|---|---|
def config_for(tp: int, **kw) |
|
def run(model: str, workloads, tp: int, **kw) |
stokastisk.communication
EXPERIMENTAL communication layer: host-staged transport, NCCL timing helpers, transport registry. Not used by the vLLM engine.
stokastisk.communication.cuda_transport
EXPERIMENTAL host-staged GPU->GPU transfer for platforms without CUDA peer access.
| Name | Summary |
|---|---|
class HostStagedTransport |
|
HostStagedTransport.init(self, src_device: int, dst_device: int, *, chunk_bytes: int=4 << 20, n_buffers: int=2) |
|
HostStagedTransport.transfer(self, src_tensor, dst_tensor) |
Copy src_tensor (on src device) into dst_tensor (on dst device); returns timing. Blocks until complete. |
stokastisk.communication.native
Optional ctypes loader for native/host_staged_xfer.cpp (C++ / CUDA Runtime). Built on demand with nvcc; absence is not an error.
| Name | Summary |
|---|---|
def build_native(src: str | os.PathLike, out_dir: str | os.PathLike, arch: str='sm_75') |
|
def load_native(path: str | os.PathLike | None=None) |
|
class NativeHostStaged |
|
NativeHostStaged.init(self, src_device: int, dst_device: int, *, chunk_bytes: int=4 << 20, n_buffers: int=2, lib=None) |
|
NativeHostStaged.transfer(self, src_tensor, dst_tensor) |
|
NativeHostStaged.close(self) |
stokastisk.communication.nccl
NCCL timing helpers for a torchrun/torch.distributed process group (2 ranks). NCCL is the reference transport; it is never replaced.
| Name | Summary |
|---|---|
def time_p2p(nbytes: int, rank: int, *, iters: int=20, warmup: int=5) |
rank 0 sends nbytes to rank 1 (NCCL send/recv). Returns per-iteration seconds measured on this rank (barrier-aligned). |
def time_allreduce(nbytes: int, *, iters: int=20, warmup: int=5) |
stokastisk.communication.pinned_memory
A small ring of page-locked host buffers (cudaHostAlloc via PyTorch's pinned allocator).
| Name | Summary |
|---|---|
class PinnedBufferRing |
|
PinnedBufferRing.init(self, n_buffers: int, nbytes: int) |
|
PinnedBufferRing.host_bytes(self) |
stokastisk.communication.synchronization
Buffer ownership tracking for the host-staged pipeline.
| Name | Summary |
|---|---|
class BufferTicket |
|
BufferTicket.init(self) |
stokastisk.communication.transport_registry
Registry of known transports with an honest status. Nothing is selected automatically.
| Name | Summary |
|---|---|
def list_transports() |
stokastisk.compat
Static Tesla T4 / SM75 / TRITON_ATTN compatibility inspection of the bundled vLLM 0.25.1.
| Name | Summary |
|---|---|
def feature_matrix() |
A copy of the feature table (safe to mutate). |
def get_feature(name: str) |
|
def check_feature(name: str) |
Return the feature record, or raise if the feature cannot work on this build. UNKNOWN_REQUIRES_EVIDENCE is returned, not raised: it is untested, not forbidden. |
def check_dtype(dtype: str | None, capability: tuple[int, int]=SM75) |
Reject an explicit bfloat16 below SM80 with the same reasoning as vllm/platforms/cuda.py, but before an engine is started. |
def check_quantization(method: str | None, capability: tuple[int, int]=SM75) |
Compare a vLLM quantization method with the minimum capability declared in the source. Unknown methods are reported UNKNOWN_REQUIRES_EVIDENCE, never assumed to work. |
def read_registry(package_root: str | pathlib.Path | None=None) |
architecture name -> (module, class) from vllm/model_executor/models/registry.py, read with ast (no import). |
def uses_removed_backend(module_file: pathlib.Path, root: pathlib.Path) |
True if the architecture's OWN module imports names from the Flash-family placeholder module at import time, i.e. the model is built on a removed backend. |
def find_missing_imports(module_file: pathlib.Path, root: pathlib.Path, *, max_depth: int=6) |
Chains of unconditional vllm imports from module_file that end in a module absent from the shipped package. |
def check_model(model: str | None=None, *, architectures: Iterable[str] | None=None, package_root: str | pathlib.Path | None=None, capability: tuple[int, int]=SM75, dtype: str | None='half', quantization: str | None=None, mode: str | None=None) |
Static, OFFLINE pre-flight for a model on the T4 build (nothing is downloaded and no GPU is touched). |
stokastisk.config
Typed configuration with T4-safe defaults and environment overrides.
| Name | Summary |
|---|---|
class StokastiskConfig |
|
StokastiskConfig.strategy(self) |
Effective strategy. None keeps the baseline meaning: TP=2 -> tensor_parallel, otherwise single_gpu (PP>1 -> pipeline_parallel). |
StokastiskConfig.world_size(self) |
|
StokastiskConfig.is_experimental(self) |
|
StokastiskConfig.validate(self) |
|
StokastiskConfig.from_env(cls, prefix: str='STOKASTISK_') |
|
StokastiskConfig.apply_environment(self) |
Export NCCL settings (never overriding values the user already set). |
StokastiskConfig.engine_kwargs(self) |
|
StokastiskConfig.as_dict(self) |
stokastisk.cuda
Optional CUDA stream/event/memory helpers (PyTorch-backed). Not used by the vLLM engine path.
stokastisk.cuda.device
Thin, optional CUDA device helpers on top of PyTorch. They NEVER touch vLLM's internal streams or contexts.
| Name | Summary |
|---|---|
def device_count() |
|
def on_device(index: int) |
|
def synchronize(index: int | None=None) |
stokastisk.cuda.events
Event helpers: dependency edges between streams (also across devices) and GPU-side timing.
| Name | Summary |
|---|---|
def new_event(*, timing: bool=False) |
|
def record(event, stream=None) |
|
def make_wait(stream, event) |
cudaStreamWaitEvent: stream will not run later work until event completes (event may belong to another device). |
def elapsed_ms(start, end) |
stokastisk.cuda.memory
Memory helpers: free/total per device, pinned allocation, timed async copies.
| Name | Summary |
|---|---|
def mem_info(device: int) |
|
def pinned_empty(nbytes: int) |
|
def timed_copy(dst, src, stream) |
Async copy on stream timed with CUDA events on the stream's device. |
stokastisk.cuda.streams
Stream helpers. Create your own streams; do not enqueue onto vLLM's streams from outside vLLM.
| Name | Summary |
|---|---|
def create_stream(device: int, *, priority: int=0) |
|
def stream_synchronize(stream) |
stokastisk.dependencies
Feature -> third-party import names. The single registry that the SDK (serving, engine), the Kaggle installer and the unit tests share.
| Name | Summary |
|---|---|
def missing(names: Iterable[str]) |
Import names that cannot be found (find_spec returning None, or raising for a half-installed namespace). |
def required_imports(*profiles: str) |
Locked + baseline import names for the given profiles (order preserved, no duplicates). |
def missing_for(*profiles: str) |
stokastisk.diagnostics
Runtime diagnostics. torch is imported lazily so this module works on CPU-only hosts.
| Name | Summary |
|---|---|
def discover_gpus() |
|
def peer_access_matrix() |
|
def check_native_extensions() |
Import every bundled extension; value is 'ok' or the error text. |
def flash_attn_native_loaded() |
FlashAttention native modules present in this process (expected: none). |
def prohibited_report() |
Detect FlashAttention / FlashInfer / FlashMLA distributions, importable modules and shipped files (expected: all empty). |
def validate_environment(*, require_gpus: int=2, strict_torch: bool=True) |
Validate the runtime against the SDK build target (CUDA 13.0 / SM75 / torch 2.11 cu130). |
def report() |
Non-raising environment report (safe on CPU hosts). |
stokastisk.engine
LLM facade over the bundled vLLM engine (Tesla T4 / SM75 / TRITON_ATTN / fp16).
| Name | Summary |
|---|---|
class LLM |
Thin wrapper: applies the T4 defaults (NCCL configured not to depend on peer access, TRITON_ATTN, pinned structured-output backend), then delegates to vllm.LLM. |
LLM.init(self, model: str, *, config: StokastiskConfig | None=None, **kwargs: Any) |
|
LLM.selected_attention_backend(self) |
Backend recorded in the live engine's VllmConfig (config-level evidence; the worker's own log line and the Triton cache are the kernel-level evidence). |
LLM.verify_attention_backend(self) |
|
LLM.structured_output_backend(self) |
The structured-output backend recorded in the live engine's VllmConfig (pinned to 'xgrammar'; never 'auto'). |
LLM.verify_structured_output_backend(self) |
|
LLM.engine(self) |
|
LLM.closed(self) |
True once :meth:close has run, or when the engine failed to start. |
LLM.generate(self, prompts, sampling_params=None, *, timeout: float | None=None, **kwargs) |
Pass-through to vllm.LLM.generate. An empty prompt list returns [] without touching the engine; non-str or empty prompts raise TypeError / |
LLM.chat(self, messages, sampling_params=None, *, timeout: float | None=None, **kwargs) |
Pass-through to vllm.LLM.chat (the chat template is applied by vLLM). messages is a list of {"role", "content"} dicts or a list of such lists. |
LLM.complete(self, prompt: str, *, max_tokens: int=64, temperature: float=0.0, timeout: float | None=None) |
Generate one completion for prompt and return its text. |
LLM.generate_structured(self, prompts, schema, *, max_tokens: int=64, temperature: float=0.0, timeout: float | None=None) |
Generate JSON constrained to schema by the engine's pinned structured-output backend, then validate every output against the schema. |
LLM.close(self) |
Shut the engine (and, for TP>1, its worker processes) down explicitly, then release cached CUDA memory. Idempotent. |
stokastisk.errors
Stokastisk exception hierarchy.
| Name | Summary |
|---|---|
class StokastiskError |
Base class for all SDK errors. |
class EnvironmentValidationError |
The runtime does not match the SDK build target. |
class GPUDiscoveryError |
Required GPUs are missing or have the wrong architecture. |
class EngineError |
vLLM engine failed to initialise or generate. |
class ConfigurationError |
Invalid Stokastisk configuration (e.g. a FlashAttention backend was requested). |
class AttentionBackendError |
The engine did not end up with the requested Triton attention backend. |
class EngineClosedError |
The engine was closed (explicitly, or because it never started); create a new stokastisk.LLM. |
class EngineTimeoutError |
A generation call did not finish within the caller's timeout. The request may still be running inside the engine. |
class EngineCloseError |
The engine shut down with an error. The SDK still released everything it could; __cause__ carries the original failure. |
class StructuredOutputError |
A structured-output request failed, or its output violated the schema that was requested. |
class MissingDependencyError |
A third-party package needed by the requested feature is not importable. missing lists the import names; feature names the feature. |
MissingDependencyError.init(self, feature: str, missing, hint: str='') |
|
class ServerError |
Base class for failures of the OpenAI-compatible server process. |
class ServerStartupError |
The server process exited, or did not become ready, before wait_ready returned. |
ServerStartupError.init(self, reason: str, *, exit_code, command, log_path, log_excerpt: str, timeout=None, waited=None) |
|
class ServerHTTPError |
The server answered an API request with an error status or an unparsable body. status is the HTTP status (None for a transport failure). |
ServerHTTPError.init(self, message: str, *, status=None, body: str='') |
|
class ServerShutdownError |
The server process could not be stopped cleanly (it had to be killed, or children survived). |
class UnsupportedParallelismError |
The requested TP/PP/replica combination is not valid for the two-T4 target. |
class InsufficientMemoryError |
The model (weights + KV cache) does not fit the requested GPU layout. |
class TopologyError |
GPU topology could not be determined or does not match the requested layout. |
class ReplicaError |
A replica worker failed to start, died, or returned an error. |
class StokastiskCompatibilityError |
A model, dtype, quantization method or vLLM feature cannot work on the Tesla T4 / SM75 / TRITON_ATTN build. |
stokastisk.limits
Hard resource limits of the Kaggle target and helpers that enforce / report them.
| Name | Summary |
|---|---|
class KaggleLimits |
|
def available_cpus() |
CPUs this process may really use (affinity-aware; equals nproc), never more than the Kaggle budget of 4 for planning. |
def cpu_budget() |
|
def threads_per_worker(tensor_parallel_size: int) |
Split the CPU budget across the TP worker processes (4 cores, TP=2 -> 2 threads each) so CPU-side libraries never oversubscribe. |
def cpu_thread_env(tensor_parallel_size: int) |
|
def system_ram_gib() |
|
def dir_size_bytes(path: str | os.PathLike) |
|
def working_dir_status(path: str | None=None) |
Usage of the (quota-limited) output directory against the 20 GiB Kaggle limit. |
def check_resources() |
Compare the host with the Kaggle limits. Never raises. Statuses: PASS (within the limit) / WARNING (host differs from the Kaggle design). |
stokastisk.nccl
NCCL validation helpers (launch with python -m torch.distributed.run --standalone --nproc_per_node=2).
| Name | Summary |
|---|---|
def run_collective_selftest(expected_world: int=2) |
Run all-reduce / all-gather / reduce-scatter / broadcast / barrier over the torchrun world. Never raises. |
stokastisk.parallel
Application-level parallelism: independent GPU replicas, token-aware scheduling, rule-based recommendations.
stokastisk.parallel.memory
Weight / KV-cache memory estimation from a model's config.json (an ESTIMATE, labelled as such everywhere it is shown).
| Name | Summary |
|---|---|
class ModelSpec |
|
ModelSpec.head_dim_(self) |
|
ModelSpec.params(self) |
|
ModelSpec.weight_mib(self) |
|
ModelSpec.kv_mib(self, tokens: int) |
|
ModelSpec.unsupported_reason(self) |
|
ModelSpec.to_dict(self) |
|
ModelSpec.from_hf_config(cls, name: str, cfg: dict[str, Any], **kw) |
|
def load_model_spec(model: 'str | ModelSpec', *, local_files_only: bool=False) |
ModelSpec from a ModelSpec, a local model directory, or a Hub repo id (downloads ONLY config.json). |
stokastisk.parallel.queue
Request records and a deterministic FIFO queue.
| Name | Summary |
|---|---|
def estimate_tokens(text: str) |
Cheap, deterministic input-token estimate (~4 characters per token). Replace with a tokenizer when exactness matters. |
class Request |
|
Request.cost(self) |
Token-aware work estimate: prompt tokens (prefill) + requested output tokens (decode). |
class RequestQueue |
|
RequestQueue.init(self, maxsize: int | None=None) |
|
RequestQueue.put(self, r: Request, *, front: bool=False) |
|
RequestQueue.get(self) |
|
RequestQueue.pending_tokens(self) |
stokastisk.parallel.recommender
Rule-based, deterministic parallelism recommendation. It only RECOMMENDS: the caller decides (Recommendation.to_config()).
| Name | Summary |
|---|---|
class Recommendation |
|
Recommendation.to_dict(self) |
|
Recommendation.to_config(self, **kw) |
A validated StokastiskConfig for this recommendation. Raises for non-runnable outcomes. |
def default_recommendation(topology) |
Model-independent summary used by Topology.recommended_parallelism. |
def recommend_parallelism(model, topology=None, workload: str='latency', *, max_model_len: int=4096, gpu_memory_utilization: float=0.88, min_concurrent_seqs: int=1) |
stokastisk.parallel.replica
ReplicaPool: independent TP=1 vLLM engines, one per GPU, behind a deterministic token-aware scheduler.
| Name | Summary |
|---|---|
class ReplicaPool |
|
ReplicaPool.init(self, model: str, devices: list[int] | tuple[int, ...]=(0, 1), *, config: StokastiskConfig | None=None, ready_timeout: float=900.0, max_attempts: int=2, engine_kwargs: dict[str, Any] | None=None) |
|
ReplicaPool.restart_dead(self, timeout: float=900.0) |
Restart every worker that has exited. Never called implicitly. |
ReplicaPool.generate(self, prompts: list[str], *, max_tokens: int=64, temperature: float=0.0, timeout: float=1800.0) |
|
ReplicaPool.close(self, timeout: float=60.0) |
Stop every worker. Always completes the whole cleanup (shutdown message, join, terminate, kill, queue close); then, if any worker had to be terminated, exited |
stokastisk.parallel.routing
Weighted least-loaded routing (deterministic; ties go to the lowest worker id).
| Name | Summary |
|---|---|
class WorkerLoad |
|
WorkerLoad.available(self) |
|
def effective_speed(w: WorkerLoad, reference_speed: float | None=None) |
Speed used for routing. An observed speed is tokens/s; the default 1.0 is unit-less, so an unobserved worker is |
def projected_load(w: WorkerLoad, extra_tokens: int, reference_speed: float | None=None) |
Estimated time-to-drain (arbitrary units): queued tokens + the new work, divided by the worker's speed. |
def choose_worker(workers: list[WorkerLoad], cost_tokens: int) |
stokastisk.parallel.scheduler
Deterministic, token-aware, weighted least-loaded scheduler. Pure Python: no CUDA, no processes (the pool owns those).
| Name | Summary |
|---|---|
class Decision |
|
class Scheduler |
|
Scheduler.init(self, workers: list[WorkerLoad], *, max_attempts: int=2, speed_alpha: float=0.3) |
|
Scheduler.assign(self, req: Request) |
|
Scheduler.assign_batch(self, reqs: list[Request], *, longest_first: bool=True) |
Greedy LPT (longest-processing-time-first) placement; deterministic. Returns worker id -> requests; unplaceable ones under key -1. |
Scheduler.complete(self, req_id: int, *, seconds: float | None=None) |
|
Scheduler.mark_dead(self, worker_id: int) |
Mark a worker dead and return its in-flight requests that may still be retried elsewhere. |
Scheduler.revive(self, worker_id: int) |
stokastisk.parallel.worker
One replica worker process: one physical GPU, one TP=1 vLLM engine, one full copy of the weights.
| Name | Summary |
|---|---|
def gpu_identity(physical_index: int) |
uuid + memory.used of a PHYSICAL GPU via nvidia-smi (independent of CUDA_VISIBLE_DEVICES, no CUDA context). |
def worker_main(worker_id: int, device: int, model: str, config_dict: dict, engine_kwargs: dict, in_q, out_q) |
stokastisk.prohibited
Policy and scanners for the Stokastisk "no FlashAttention / FlashInfer / FlashMLA" rule.
| Name | Summary |
|---|---|
def is_prohibited_path(path: str) |
True if any component of path (a /-separated path inside a wheel or tree) names a Flash-family module or package. |
def is_binary(path: str) |
|
def native_markers(data: bytes) |
Flash-family markers present in a native object's bytes. |
def scan_members(names: Iterable[str], read) |
Scan a set of archive/tree members. read(name) -> bytes is only called for files whose content must be inspected. |
def scan_wheel(path: str | pathlib.Path) |
|
def scan_tree(root: str | pathlib.Path) |
|
def flash_in_maps(lines: Iterable[str]) |
Flash-family shared objects in the lines of /proc/<pid>/maps (path column only). |
def loaded_flash_libraries(pid: int | str='self') |
Flash-family shared objects mapped by process pid (empty list = none; unreadable maps raise OSError, never silently pass). |
stokastisk.serving
OpenAI-compatible serving for the T4 build: launch the bundled vllm serve with the Stokastisk T4 defaults, wait for /health, read /metrics, stream tokens.
| Name | Summary |
|---|---|
def check_server_requirements(names: Sequence[str] | None=None) |
Return the import names that cannot be found (empty list = nothing missing). Uses find_spec only (no import side effects). |
def validate_model_id(model: Any) |
A Hugging Face id (org/name) or a path. Rejects empty strings, whitespace/control characters and anything that vLLM's argument parser would read as an option. |
def validate_host(host: Any) |
|
def validate_port(port: Any) |
|
def port_is_free(host: str, port: int) |
True if nothing is listening on (host, port) and the address can be bound. |
def find_free_port(host: str='127.0.0.1') |
|
def build_serve_command(model: str, config: StokastiskConfig | None=None, *, host: str='127.0.0.1', port: int=8000, served_model_name: str | None=None, quantization: str | None=None, extra_args: Sequence[str]=(), python: str | None=None) |
The exact argv that ServerProcess runs. Pure function: no process, no import of vllm. |
class ServerProcess |
Supervise one vllm serve child. Use as a context manager. Never started implicitly, never left running by with. |
ServerProcess.init(self, model: str, config: StokastiskConfig | None=None, *, host: str='127.0.0.1', port: int=8000, served_model_name: str | None=None, quantization: str | None=None, extra_args: Sequence[str]=(), cuda_visible_devices: str | None=None, log_path: str | None=None, python: str | None=None) |
|
ServerProcess.base_url(self) |
|
ServerProcess.pid(self) |
|
ServerProcess.running(self) |
|
ServerProcess.log_tail(self, max_lines: int=40) |
|
ServerProcess.start(self) |
|
ServerProcess.health(self, timeout: float=5.0) |
True if GET /health answers 200 right now. |
ServerProcess.wait_ready(self, timeout: float=600.0, interval: float=2.0) |
Poll GET /health until HTTP 200. Raises :class:ServerStartupError if the child exits first or the timeout passes (with the exit code, the command, |
ServerProcess.start_and_wait(self, timeout: float=600.0) |
|
ServerProcess.restart(self, timeout: float=600.0) |
Stop the server (graceful first) and start a fresh one on the same port; returns once it is healthy. |
ServerProcess.stop(self, grace: float=30.0) |
SIGINT the process group (vLLM's own graceful shutdown), then SIGTERM, then SIGKILL; then make sure no process of the group survives. |
def chat(base_url: str, model: str, messages: list[dict[str, str]], *, max_tokens: int=64, temperature: float=0.0, timeout: float=120.0) |
Non-streaming /v1/chat/completions; returns the assistant text. |
def stream_chat(base_url: str, model: str, messages: list[dict[str, str]], *, max_tokens: int=64, temperature: float=0.0, timeout: float=120.0) |
Yield content deltas from /v1/chat/completions with stream=true (server-sent events: data: {...} lines, terminated by data: [DONE]). |
def list_models(base_url: str, timeout: float=10.0) |
|
def fetch_metrics(base_url: str, timeout: float=10.0) |
Parse the Prometheus text at /metrics into {sample_name{labels}: value} (comment lines skipped; non-numeric samples ignored). |
stokastisk.structured
Structured (JSON-schema) output helpers: pure validation that needs no GPU, plus the sampling-parameter builder.
| Name | Summary |
|---|---|
def check_schema(schema: Any) |
Return schema (a dict, or a JSON string parsed to one) if it is a valid JSON Schema, else raise :class:StructuredOutputError. |
def validate_json_output(text: str, schema: Any) |
Parse text as JSON and validate it against schema. Returns the parsed object; raises :class:StructuredOutputError (never returns invalid data). |
def json_schema_params(schema: Any, *, max_tokens: int=64, temperature: float=0.0) |
vllm.SamplingParams that constrain the output to schema. The backend is the engine's (pinned) one; a request cannot choose another. |
stokastisk.telemetry
Optional GPU / communication / worker telemetry. Every probe degrades to an explicit 'unavailable'.
stokastisk.telemetry.communication_profiler
Wall-clock + CUDA-event timing for communication/compute segments. Segments are recorded; nothing is inferred.
| Name | Summary |
|---|---|
class SegmentTimer |
|
SegmentTimer.init(self) |
|
SegmentTimer.wall(self, name: str) |
|
SegmentTimer.cuda(self, name: str, stream=None) |
GPU-side duration via CUDA events (synchronises the stream when the block exits). |
SegmentTimer.totals(self) |
|
SegmentTimer.report(self) |
stokastisk.telemetry.cuda_profiler
Nsight availability probe. Nsight is never required: an unusable profiler is an explicit SKIPPED.
| Name | Summary |
|---|---|
def nsight_status() |
stokastisk.telemetry.metrics
Pure statistics + latency helpers (no GPU needed).
| Name | Summary |
|---|---|
def percentile(values: Sequence[float], q: float) |
Linear-interpolated percentile (q in [0,100]); raises on empty input. |
def summarize(values: Sequence[float]) |
min / median / p95 / max / mean over ALL runs (never only the best one). |
def imbalance_percent(t0: float, t1: float) |
abs(T0 - T1) / max(T0, T1) * 100. 0 = perfectly balanced; 0.0 when both are 0. |
def throughput(tokens: int, seconds: float) |
|
def token_latencies(first_token_s: float | None, total_s: float, n_tokens: int) |
TTFT and mean inter-token latency from a single request's timing. |
stokastisk.telemetry.nvml
GPU sampling: NVML if present, else nvidia-smi, else an explicit 'unavailable'. Counters Kaggle does not expose stay None.
| Name | Summary |
|---|---|
def sample_once() |
|
def pcie_throughput_kib_s() |
PCIe RX/TX in KiB/s per GPU via NVML, or None when NVML / the counter is unavailable. |
class GpuSampler |
Background sampler: with GpuSampler(0.5) as s: ...; s.summary(). Overhead = one nvidia-smi call per interval. |
GpuSampler.init(self, interval: float=0.5) |
|
GpuSampler.summary(self) |
stokastisk.telemetry.report
Local JSON evidence + optional OpenTelemetry spans. All metrics are available as JSON without any network telemetry.
| Name | Summary |
|---|---|
def write_json(path: str | pathlib.Path, obj: Any) |
|
def span(name: str, **attrs: Any) |
OpenTelemetry span if opentelemetry is installed AND a provider is configured; otherwise a no-op. Never required. |
def provenance() |
Versions + identity every benchmark records. |
stokastisk.telemetry.worker_metrics
Per-worker accounting used by the balance benchmark: compute vs communication vs idle, and queue wait.
| Name | Summary |
|---|---|
class WorkerMetrics |
|
WorkerMetrics.idle_s(self) |
|
WorkerMetrics.to_dict(self) |
|
def balance_report(w0: WorkerMetrics, w1: WorkerMetrics) |
Imbalance on compute time and on total busy time, reported separately (utilisation alone does not prove balance). |
stokastisk.topology
GPU topology discovery (torch / NVML / nvidia-smi / sysfs; all but torch optional).
stokastisk.topology.discovery
discover_topology(): torch -> NVML -> nvidia-smi -> sysfs, every layer optional. Nothing is hardcoded to Kaggle's PHB.
| Name | Summary |
|---|---|
def discover_topology(*, with_recommendation: bool=True) |
stokastisk.topology.gpu_capabilities
Basic GPU properties from PyTorch (the primary provider). cuda-python is an optional, never-required refinement.
| Name | Summary |
|---|---|
def torch_devices() |
|
def cuda_python_version() |
stokastisk.topology.models
Plain, JSON-serialisable topology records. A value that could not be measured is None — never invented.
| Name | Summary |
|---|---|
class DeviceInfo |
|
class PeerAccess |
CUDA peer access per ordered pair. matrix[src][dst] is torch.cuda.can_device_access_peer(src, dst) (None = not measured). |
PeerAccess.can_access(self, src: int, dst: int) |
|
PeerAccess.all_pairs_available(self) |
|
PeerAccess.any_pair_available(self) |
|
class Topology |
|
Topology.gpu_count(self) |
|
Topology.pair_link(self, a: int, b: int) |
|
Topology.to_dict(self) |
stokastisk.topology.nvml
Optional NVML (pynvml / nvidia-ml-py) and nvidia-smi helpers. Neither is required: every function degrades to None.
| Name | Summary |
|---|---|
def nvml_devices() |
Per-GPU dict from NVML, or None when NVML is unusable. |
def smi_query(fields: str) |
Rows of nvidia-smi --query-gpu=<fields>, or None when nvidia-smi is missing/fails. |
def smi_topo_matrix() |
stokastisk.topology.pcie
PCIe / link classification from nvidia-smi topo -m text, and PCIe link speed from sysfs.
| Name | Summary |
|---|---|
def classify_cell(cell: str) |
|
def parse_topo_matrix(text: str) |
{"0-1": "PHB"} (lower index first). Unknown / unparsable cells become UNKNOWN; an unparsable matrix yields {}. |
def pcie_sysfs(pci_bus_id: str | None) |
Current/max PCIe generation + width from /sys (no root needed). Missing files -> missing keys. |
stokastisk.topology.peer_access
CUDA peer access, measured with torch.cuda.can_device_access_peer in BOTH directions. Never forced, never assumed.
| Name | Summary |
|---|---|
def measure_peer_access(device_count: int | None=None) |