Feature matrix and model evidence

Generated from stokastisk.compat (FEATURES, MODEL_EVIDENCE) by scripts/release/build_docs_site.py; do not edit by hand. These are the records the installed SDK reports through stokastisk.feature_matrix() and stokastisk.check_model().

Evidence classes. kaggle: executed on Kaggle 2 × T4 (the notebooks column says which: 15 = acceptance suite on build 3, 16/18/19 = end-user notebooks, 01–14 = earlier notebooks). kaggle-kernel: a Triton kernel compiled and launched, output not asserted. cpu-tests: CPU unit tests only. static: source inspection only. Builds 4 and 5 were not executed on Kaggle; their inference code is byte-identical to build 3.

Features

Feature Status Evidence Notebooks Note
offline_generation SUPPORTED kaggle 15,16,18,19 vllm.LLM generate() in FP16, enforce_eager, TRITON_ATTN. Exact models run: Qwen/Qwen2.5-0.5B-Instruct (TP=1; TP=2, PP=2 and 2 replicas in the notebook 15 parallel stage), Qwen/Qwen2.5-3B-Instruct (TP=2, PP=2) and Qwen/Qwen2.5-Coder-7B-Instruct (TP=2, PP=2); max_model_len 2048. See MODEL_EVIDENCE; other checkpoints are not covered
batch_generation SUPPORTED kaggle 09,11,14 batches of 1..64 requests, 64 output tokens
triton_attention SUPPORTED kaggle 05,13 TRITON_ATTN selected and confirmed in the live VllmConfig; kernel_unified_attention compiled; no Flash library mapped in any process
tensor_parallel_2 SUPPORTED kaggle 05,06,13,15,18,19 both GPUs hold a shard; slower than TP=1 for a model that fits one T4 (about 0.55x decode, earlier notebooks); generated with Qwen2.5-0.5B (nb 15), Qwen2.5-3B (nb 18) and Qwen2.5-Coder-7B (nb 19)
pipeline_parallel_2 SUPPORTED_WITH_CONDITIONS kaggle 08,13,15,18,19 experimental in this SDK; both GPUs hold stages; about 0.6x of TP=1 decode for a model that fits one T4 (earlier notebooks); generated with Qwen2.5-0.5B (nb 15), Qwen2.5-3B (nb 18) and Qwen2.5-Coder-7B (nb 19)
replica_parallel SUPPORTED_WITH_CONDITIONS kaggle 09,14,15 SDK feature (ReplicaPool): one TP=1 engine per GPU, both receive work; no throughput gain over one engine up to batch 64; also the notebook 15 parallel stage (two workers on two GPUs, both received work, clean close)
gpu_telemetry SUPPORTED kaggle 09,11,14 nvidia-smi sampling during benchmarks
topology_discovery SUPPORTED kaggle 01,07 PHB; CUDA peer access measured per session (true in one Kaggle VM, false in another)
nccl_collectives SUPPORTED kaggle 04 all_reduce/all_gather/reduce_scatter/broadcast on both ranks
sampling_temperature_top_k_top_p SUPPORTED_WITH_CONDITIONS kaggle-kernel — kernels compiled during runs; only greedy (temperature 0) output was asserted
logprobs SUPPORTED_WITH_CONDITIONS kaggle-kernel — logprob / prompt-logprob kernels compiled; output not asserted
penalties_min_p_bad_words_logit_bias SUPPORTED_WITH_CONDITIONS kaggle-kernel — kernels compiled; output not asserted
chat_generation SUPPORTED kaggle 15,16,18,19 LLM.chat and LLM.complete returned text on 2 x T4: notebook 15 engine stage (build 3), the notebook 16 quickstart, notebook 18 (0.5B TP=1; 3B TP=2 and PP=2) and notebook 19 (Coder-7B TP=2 and PP=2)
openai_server SUPPORTED_WITH_CONDITIONS kaggle 15 the real vllm serve launched by stokastisk.serving.ServerProcess was started and exercised on 2 x T4 (notebook 15, server stage, 21/21: /health, /v1/models, chat completions, SSE streaming, HTTP error mapping, structured output over HTTP, /metrics, restart on the same port, graceful SIGINT shutdown, GPU memory released). Conditions: exercised with Qwen/Qwen2.5-0.5B-Instruct, TP=1 only; Kaggle's fastapi 0.141.1 is newer than upstream vLLM's <0.137 cap; the SageMaker /ping and /invocations routes were NOT exercised and are not claimed. Calling upstream vllm serve directly is unsupported (see direct_upstream_vllm)
streaming SUPPORTED kaggle 15 stream_chat: SSE through the real server yielded several non-empty deltas and ended with [DONE] (notebook 15, server stage, Qwen2.5-0.5B, TP=1); the truncated-stream error path is CPU-tested against a fake server only
prometheus_metrics SUPPORTED_WITH_CONDITIONS kaggle 15 GET /metrics on the running server exposed vllm: samples (notebook 15, server stage). Only the presence of those samples was asserted, not their values; the fetch_metrics text parser is CPU-tested
structured_outputs SUPPORTED_WITH_CONDITIONS kaggle 15,16,18 JSON-schema output with the pinned xgrammar 0.2.3 backend (never 'auto'): notebook 15 engine stage (every generate_structured output satisfied the bounded schema; not-JSON, wrong type, missing key, extra key and truncated outputs were each detected; a schema xgrammar does not support was rejected with no fallback; xgrammar's native library was mapped and outlines_core's was not), the server stage (structured output over HTTP) and notebooks 16 and 18 (generate_structured). Conditions: bound open-ended schema fields (a grammar cannot make a model stop); acceptance model Qwen2.5-0.5B
direct_upstream_vllm UNSUPPORTED_ENTRY_POINT static — calling vllm.LLM or vllm serve directly bypasses the SDK's dtype, NCCL, structured-output and backend checks and is NOT a supported Stokastisk entry point (use stokastisk.LLM / ServerProcess / ReplicaPool). The Flash-free tree makes upstream auto-selection resolve to TRITON_ATTN on SM75 (CPU-evaluated), and FlashAttention/FlashInfer/FlashMLA are refused with ProhibitedBackendError, but only the SDK paths are verified
prefix_caching UNKNOWN_REQUIRES_EVIDENCE static — no run asserts it
chunked_prefill UNKNOWN_REQUIRES_EVIDENCE static — no run asserts it
cuda_graphs UNKNOWN_REQUIRES_EVIDENCE static — every Kaggle run used enforce_eager=True (the SDK default); graph capture on T4 is unverified
custom_all_reduce UNKNOWN_REQUIRES_EVIDENCE static — disabled by the SDK default; needs CUDA peer access, which differs between Kaggle VMs
lora UNKNOWN_REQUIRES_EVIDENCE static — present in the source; no run
multimodal UNKNOWN_REQUIRES_EVIDENCE static — present in the source; no run
embedding_pooling_reranking UNKNOWN_REQUIRES_EVIDENCE static — present in the source; no run
speculative_decoding UNKNOWN_REQUIRES_EVIDENCE static — present in the source; no run
mla_models_deepseek UNKNOWN_REQUIRES_EVIDENCE static — FlashMLA/FlashInfer-MLA backends are excluded; other MLA backends were not run
worker_failure_recovery SUPPORTED_WITH_CONDITIONS cpu-tests — ReplicaPool restarts/retries covered by CPU tests with fake workers only
bfloat16 NOT_SUPPORTED_SM75 static — vllm/platforms/cuda.py rejects bfloat16 below compute capability 8.0; use dtype='half'
flash_attention EXCLUDED_FLASH_DEPENDENCY static — FlashAttention is not shipped and not allowed
flashinfer EXCLUDED_FLASH_DEPENDENCY static — FlashInfer attention, sampler and all-reduce fusion are not shipped
flashmla EXCLUDED_FLASH_DEPENDENCY static — FlashMLA is not shipped
allreduce_rms_fusion EXCLUDED_FLASH_DEPENDENCY static — needs FlashInfer comm; pass_manager raises a clear error if requested
fp8_mxfp4_nvfp4_native NOT_SUPPORTED_SM75 static — native FP8/FP4 kernels need SM89/SM90/SM100 (see check_quantization for the per-method minimum declared in the source)
expert_parallel_all2all NOT_SUPPORTED_SM75 static — DeepEP / NIXL / FlashInfer all2all backends are not shipped or need newer GPUs
qwen_omni_models NEEDS_ADAPTATION static — qwen2_5_omni_thinker was removed from the T4 wheel by the module-level flash-import rule (its flash_attn import is guarded); restoring it needs a new vLLM wheel
l4_sm89 NOT_IMPLEMENTED static — planned only; no SM89 binaries, no support claim

Models that were run

Model id Architecture Dtype Max tested max_model_len Modes Evidence Limitations
Qwen/Qwen2.5-0.5B-Instruct Qwen2ForCausalLM half 2048 TP=1, TP=2, PP=2, replicas=2 notebook 15 (build 3 acceptance): engine 44/44 and server 21/21 at TP=1; parallel 23/23 with TP=2, PP=2 and two replicas; notebook 16 (quickstart script, TP=1); notebook 18 (plain cells, TP=1) FP16, eager execution, max_model_len 2048 only; TP=2, PP=2 and replicas are slower than TP=1 for a model this small
Qwen/Qwen2.5-3B-Instruct Qwen2ForCausalLM half 2048 TP=2, PP=2 notebook 18 (build 3): completion and chat with TP=2, and with PP=2 TP=1 was not tested for this model; free memory during inference: about 1.09 GiB per GPU at TP=2, 1.96 and 1.48 GiB at PP=2 (gpu_memory_utilization 0.88); PP=2 is experimental
Qwen/Qwen2.5-Coder-7B-Instruct Qwen2ForCausalLM half 2048 TP=2, PP=2 notebook 19 (build 3): original unquantized checkpoint (BF16 weights cast to FP16); one Python-coding chat answer with TP=2 and one with PP=2 does not fit one T4 in FP16, so TP=1 is not an option; memory is tight: about 1.34 GiB free per GPU during inference at TP=2, 2.42 and 1.93 GiB at PP=2 (max_model_len 2048, gpu_memory_utilization 0.88); a single prompt per mode; this is not a throughput measurement and does not show that other 7B models fit; PP=2 is experimental

Any other checkpoint — including other Qwen2ForCausalLM models — is UNKNOWN_REQUIRES_EVIDENCE in check_model(); see Checks and diagnostics.