6. Using both GPUs

The two T4s share a PCIe host bridge (no NVLink). For a model that fits one T4, TP=1 is fastest. Use two GPUs when the model does not fit one.

Strategy Config Use for Status
Single GPU StokastiskConfig() models up to about 3B in FP16 Kaggle-tested
Tensor parallel tensor_parallel_size=2 a model too large for one T4 (e.g. 7B) Kaggle-tested
Pipeline parallel pipeline_parallel_size=2 same, layer-split Kaggle-tested, experimental
Replicas ReplicaPool(model, devices=[0, 1]) throughput: two independent copies Kaggle-tested

TP and PP are alternatives, not a combination: TP=2 × PP=2 would need four GPUs and raises UnsupportedParallelismError.

Tensor parallel (7B example)

cfg = StokastiskConfig(tensor_parallel_size=2, max_model_len=2048, gpu_memory_utilization=0.88)
with LLM("Qwen/Qwen2.5-Coder-7B-Instruct", config=cfg) as llm:
    out = llm.chat([{"role": "user", "content": "Write a Python function that parses JSON Lines."}])
    print(out[0].outputs[0].text)

Between experiments, close the previous engine first (the with block does this) so GPU memory is free.

Memory is tight for a 7B model in FP16. In the Kaggle run (max_model_len=2048, gpu_memory_utilization=0.88, one prompt) about 1.34 GiB per GPU was free during TP=2 and 2.42 / 1.93 GiB during PP=2. That is one measurement of one model, not a promise that other 7B models, longer contexts or larger batches fit.

Pipeline parallel

cfg = StokastiskConfig(pipeline_parallel_size=2, max_model_len=2048)

A warning says it is experimental and slower than TP=1 for models that fit one T4. NCCL may print an "unbatched P2P op" warning; it is harmless.

ReplicaPool

Two independent TP=1 engines, one per GPU, behind a token-aware scheduler. Each GPU holds a full copy of the weights.

from stokastisk import ReplicaPool

pool = ReplicaPool("Qwen/Qwen2.5-0.5B-Instruct", devices=[0, 1],
                   config=StokastiskConfig(max_model_len=2048))
try:
    texts = pool.generate(["Hello", "Bonjour", "Hola", "Ciao"], max_tokens=16)
    print(texts, pool.last_stats["output_tok_per_s"])
finally:
    pool.close()
Member Purpose
ReplicaPool(model, devices=(0, 1), *, config=None, ready_timeout=900, max_attempts=2, engine_kwargs=None) spawns one worker per device; config must be TP=1/PP=1
generate(prompts, *, max_tokens=64, temperature=0.0, timeout=1800) list of texts in prompt order
restart_dead() restarts exited workers (never done implicitly)
close(timeout=60) stops all workers, raises ReplicaError if any had to be killed
info, last_stats, close_report, config state; last_stats has wall time, tokens/s, per-GPU busy time

Failure recovery has been exercised with fake workers on CPU only, not on real GPUs.

Topology

topo = stokastisk.discover_topology()
print(topo.gpu_count, topo.pair_link(0, 1), topo.peer_access.all_pairs_available)
print(topo.to_dict())

Topology fields include devices (name, UUID, compute capability, VRAM, PCIe generation/width), peer_access, link_class (NVLINK, PIX, PXB, PHB, NODE, SYS, UNKNOWN), nccl_available, cuda_runtime_version, driver_version, cpu_count, RAM, notes, recommended_parallelism. A value that could not be measured is None, never invented. Peer access varies between Kaggle VMs, so the SDK does not rely on it.

Recommender

rec = stokastisk.recommend_parallelism("Qwen/Qwen2.5-Coder-7B-Instruct", workload="latency", max_model_len=2048)
print(rec.strategy, rec.reason, rec.warnings, rec.estimates)
cfg = rec.to_config()                 # a validated StokastiskConfig; raises if nothing can run

workload is "latency" or "throughput". Strategies include single_gpu, tensor_parallel, pipeline_parallel, replica_parallel, and the non-runnable insufficient_memory, unsupported_architecture, no_gpu. Memory numbers are estimates from the model's config.json. It only recommends; you decide.

Next: Checks and diagnostics.