gpt-oss is not working with flash-attention #42736
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: gpt_oss |
vasqu
reviewed
Dec 9, 2025
Contributor
vasqu
left a comment
vasqu
left a comment
There was a problem hiding this comment.
Added some comments, I have some concerns on the check + somehow there are a lot of changes in the modeling that shouldn't be there (e.g. __ changes to _)
| def __init__(self, config: GptOssConfig): | ||
| super().__init__(config) | ||
|
|
||
| if config._attn_implementation in ["flash_attention_2", "flash_attention_3"]: |
Contributor
There was a problem hiding this comment.
2 things that might not work as expected
- This only checks during init, but we can also update afterwards with e.g.
set_attn_implementation. We probably cannot avoid checking during runtime (e,g, forward). - There are more FA implementations especially with the kernels ones. Afaik, we only have the vllm kernel working so we should check along
if "flash" in config._attn_implementation and config._attn_implementation != "vllm-kernel": raise ValueError(...)(don't remember the name of the kernel).
Contributor
There was a problem hiding this comment.
This is weird, there is a lot of changes like __init__ -> _init_ which shouldn't happen. Maybe some ruff version mismatch? It's weird
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When initializing
gpt-ossmodel withattn_implementation="flash_attention_2"or"flash_attention_3"would result in silent failures and garbage generation output as reported in #42533.gpt-ossmodels rely on attention sinks which are not yet implemented for theflash_attentionas suggested the safest path is to strictly block unsupported attention backends rather than failing silently or assuming a fallback.@vasqu can you see if this is what you had in mind?