fix: align RotaryEmbedding with Qwen2Moe pattern for transformers compat

by kashif HF Staff - opened Mar 15

base: refs/heads/main

←

from: refs/pr/4

Discussion Files changed

+37

-32

fix: align RotaryEmbedding with Qwen2Moe pattern for transformers compatdfa9ac6c

kashif

Mar 15

Replace custom rope_type handling with the standard Qwen2Moe pattern: use compute_default_rope_parameters for "default" type, only look up ROPE_INIT_FUNCTIONS for non-default types. Also adds partial_rotary_factor support.

fix: call super()._init_weights() to match Qwen2Moe convention for transformers v577298929

fix: align _init_weights with Qwen2Moe using nn.init APIc555c2f1

Use create_bidirectional_mask for backend-agnostic attention mask handlingd8a5a92f

lccurious

inclusionAI org 2 days ago

LGTM. Standardizing the RoPE implementation and adding partial_rotary_factor support makes total sense. Approved for merge. Thanks for the work!

lccurious changed pull request status to merged 2 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment