[joshlk]

GPT-NeoX is an example project that is using deepspeed and Zero-3 offloading. The wider project intend to train a GPT-3 sized model and release it freely to the world.

https://github.com/EleutherAI/gpt-neox

[ma2rten]

It seems like Zero-3 doesn't work for them:

https://github.com/EleutherAI/gpt-neox/issues/171

[stellaathena]

Hi! I’m the one who wrote this code. My ZeRO-3 implementation is currently not working, but I’ve spoken with DeepSpeed devs and they’ve explained to me what I’ve been doing wrong. I haven’t had time to implement the fix but I don’t see any reason to assume it won’t work.

https://github.com/microsoft/DeepSpeed/issues/846

Also, the specific problem described in that Issue was due to a bug I found in DeepSpeed that has since been corrected.

[joshlk]

Looks like they got it working recently https://github.com/EleutherAI/gpt-neox/pull/178

[dqpb]

Did you even read through the issue? I don't see anything that indicates it won't work.

[ma2rten]

Yes, I did. The last comment is a traceback and an explanation what would have to be done to fix it.

[minimaxir]

Your comment implied it's not possible at all for them to use it, not that it's currently not working.

[ma2rten]

I guess an ambiguity in the English language is at fault here. I meant it literally doesn't work for them, and not the idiom.