Fixing a GPT-OSS Triton shared-memory error on an RTX 5090

While running the original Triton backend for gpt-oss-20b on an RTX 5090, model warm-up failed before the first prompt:

triton.runtime.errors.OutOfResources: out of resource: shared memory,
Required: 131072, Hardware limit: 101376.
Reducing block sizes or num_stages may help.

The important numbers are per-block shared memory: the generated kernel requested 131,072 bytes, while that launch could use 101,376 bytes. This was not ordinary model VRAM exhaustion, so changing PyTorch's allocator or closing another application did not address the failing kernel.

Constrain the kernel configuration

Triton's pipeline stages and tile sizes affect the shared memory reserved by a program. I applied tighter constraints before the model's warm-up call:

from triton_kernels.matmul_ogs_details.opt_flags import (
    update_opt_flags_constraints,
)

update_opt_flags_constraints(
    {
        "num_stages": 2,
        "block_m": 64,
        "block_k": 64,
    }
)

Reducing num_stages was the key change. Smaller blocks provided additional margin. The tradeoff is that a configuration which fits is not necessarily the fastest one, so this is a compatibility workaround rather than a universal performance setting.

The complete change is preserved in my RTX 5090 fix commit. It imports the constraint helper and applies the values immediately before warm-up.

Reinstall the edited package

If the project was installed from a working checkout, I reinstalled it in editable mode so Python loaded the modified source:

python -m pip install -e '.[triton]'
python -m gpt_oss.chat gpt-oss-20b/original/

When debugging this kind of error, I now check three things in order:

Confirm that the traceback points to Triton shared memory, not CUDA device memory.
Record the exact GPT-OSS, Triton, PyTorch, driver, and GPU versions.
Change one launch constraint at a time and benchmark after the kernel compiles.

gpt-oss-20b is an open-weight mixture-of-experts model intended for local or specialized use; its current specifications are on the official model page.

Loading post…

Constrain the kernel configuration

Triton's pipeline stages and tile sizes affect the shared memory reserved by a program. I applied tighter constraints before the model's warm-up call:

from triton_kernels.matmul_ogs_details.opt_flags import ( update_opt_flags_constraints, ) update_opt_flags_constraints( { "num_stages": 2, "block_m": 64, "block_k": 64, } )

The complete change is preserved in my RTX 5090 fix commit. It imports the constraint helper and applies the values immediately before warm-up.

Reinstall the edited package

If the project was installed from a working checkout, I reinstalled it in editable mode so Python loaded the modified source:

python -m pip install -e '.[triton]' python -m gpt_oss.chat gpt-oss-20b/original/

When debugging this kind of error, I now check three things in order:

Confirm that the traceback points to Triton shared memory, not CUDA device memory.

Record the exact GPT-OSS, Triton, PyTorch, driver, and GPU versions.

Change one launch constraint at a time and benchmark after the kernel compiles.

gpt-oss-20b is an open-weight mixture-of-experts model intended for local or specialized use; its current specifications are on the official model page.

Constrain the kernel configuration#

Reinstall the edited package#

Comments

Constrain the kernel configuration#

Reinstall the edited package#

Constrain the kernel configuration

Reinstall the edited package

Constrain the kernel configuration

Reinstall the edited package