mistralai
/

Mixtral-8x7B-v0.1

Mixture of Experts

Model card Files Files and versions

Question about sliding window

#23

by Guanzheng - opened Dec 20, 2023

•

edited Dec 20, 2023

According to the config.json, there is no sliding window in Mixtral? Does it mean Mixtral was trained on 32k full context length?

Hi @Guanzheng
I think so yes! Perhaps @TimeRobber can confirm

Thank you!

Guanzheng changed discussion status to closed Dec 21, 2023

There's no sliding window on Mixtral

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment