· 1 min read
Fixing a GPT-OSS Triton shared-memory error on an RTX 5090
A version-specific workaround for a Triton kernel that requested more shared memory than the GPU allowed.
Loading post…
While running the original Triton backend for gpt-oss-20b on an RTX 5090, model
warm-up failed before the first prompt:
triton.runtime.errors.OutOfResources: out of resource: shared memory,
Required: 131072, Hardware limit: 101376.
Reducing block sizes or num_stages may help.
The important numbers are per-block shared memory: the generated kernel requested 131,072 bytes, while that launch could use 101,376 bytes. This was not ordinary model VRAM exhaustion, so changing PyTorch's allocator or closing another application did not address the failing kernel.
Triton's pipeline stages and tile sizes affect the shared memory reserved by a program. I applied tighter constraints before the model's warm-up call:
from triton_kernels.matmul_ogs_details.opt_flags import (
update_opt_flags_constraints,
)
update_opt_flags_constraints(
{
"num_stages": 2,
"block_m": 64,
"block_k": 64,
}
)
Reducing num_stages was the key change. Smaller blocks provided additional margin.
The tradeoff is that a configuration which fits is not necessarily the fastest one,
so this is a compatibility workaround rather than a universal performance setting.
The complete change is preserved in my RTX 5090 fix commit. It imports the constraint helper and applies the values immediately before warm-up.
If the project was installed from a working checkout, I reinstalled it in editable mode so Python loaded the modified source:
python -m pip install -e '.[triton]'
python -m gpt_oss.chat gpt-oss-20b/original/
When debugging this kind of error, I now check three things in order:
gpt-oss-20b is an open-weight mixture-of-experts model intended for local or
specialized use; its current specifications are on the
official model page.
Comments
View on GitHub