Hacker News new | ask | show | jobs
by soraki_soladead 746 days ago
Fwiw, that's SwiGLU in #3 above. Swi = Swish = silu. GLU is gated linear unit; the gate construction you describe.