Hacker News new | ask | show | jobs
by jasonjmcghee 988 days ago
> GPT-4 boasts 100 trillion parameters

Source? I’ve never heard this.

I’ve heard 1 trillion and that it’s a 8 x 175B ensemble.

3 comments

The entire article looks like an amalgam of stitched together pieces taken from different sources without much care.
Almost like an AI written article
We have many existing sources on LLMs. I referenced a couple of them that I find great. Repeating the same content doesn't offer much value. :)

And I wrote the entire content myself.

My observation from this is that the process by which humans assemble and summarize information, at least at a somewhat high level, is pretty darn close to the way that LLMs do it. I think that falls apart when you want to talk about deeper learning, drawing inferences and so on but if you are just trying to pull together an executive summary on XYZ, an LLM with some fact checking gets you pretty far IMHO.
Up until recently I was editor at an AI company, and parts of this read exactly like some of the outputs I'd get out of GPT-4/ChatGPT Plus.

Many of the linked references are too recent to be in the training corpus (Llama 2, for example), so unless there's some web-search component to this it looks like an LLM wrote the first draft, and a human went through to edit, add links, and populate with images, etc.

OpenAI hasn't confirmed it yet, so as of now, there are no reliable sources to rely on for this information. I just removed that part.
There are some people close to the company who claim 8 x 220B MOE.
I believe it's actually 8 x 220B. Just want to make it clear it's not simply a MOE GPT-3.5.