[brucethemoose2]

Its presumably llama just like Mistral.

Everything open source is llama now. Facebook all but standardized the architecture.

I dunno about the moe. Is there existing transformers code for that part? It kinda looks like there is based on the config.

[jasonjmcghee]

Mistral is not llama architecture.

https://github.com/mistralai/mistral-src

[brucethemoose2]

Its basically llama architecture, all but drop in compatible with llama runtimes.

[refulgentis]

Because it's JSON? :)