[patelajay285]

We found the same result a few years ago in our ICLR paper: https://arxiv.org/pdf/2209.14500

We found Google's T5 models which were released in 2019, pre-GPT-3, were "secretly" capable of in-context learning with a simple inference technique.

Given they use a bidirectional MLM (Masked Language Modeling) objective, it wasn't obvious how to do it, but MLM objectives are known to produce better language representations than causal (next token prediction) objectives. We were able to outperform much larger sized GPT-3 models or get very close to their performance with far smaller T5 models.

[cscurmudgeon]

Are there any intrinsic dis/advantages of bidirectional models over causal models for in-context learning? It seems that unidirectional model just have been explored and worked on more.

[patelajay285]

When you train bidirectionally only, you don't get a generative model, that would be the downside. However, you can train on a mixture of causal and bidirectional objectives as some LLM pre-training has done. As far as I am aware, there are no downsides of that, but it is not more common simply because the standard practice has been to train causal only and there just isn't enough funding/attention to go into experimenting on every axis of pre-training (which can be very expensive).

[namibj]

No, you can generate with them using diffusion.

[zxexz]

Yep. That technique works very well. Surprised that it’s not more widely used.

[byefruit]

This is very interesting. Have you got any references describing this approach?

[namibj]

I'll try to remember to search some out that I read back when I did a literature review of the subject (probably was 11 months ago).

[mycall]

Isn't Q* (or Quiet-STaR) a causal and bidirectional objective learning system?

[toxik]

From that paper it seems the sampling method (SAP) is also slower, so that it beats larger models seems expected.

[patelajay285]

It's not at all expected. T5 models are not generative models by default and they were not thought to be able to perform generation, let alone in-context learning. Remember these models were released before any of the existing LLMs and in-context learning/prompting as a technique became popularized with GPT-3.

While the technique requires multiple samples to coax generations from this particular model, other LLM training schemes have incorporated both unidirectional and bidirectional objectives in their training now. However, this exploration hasn't been fully resolved as most models are still trained only on the causal objective by standard practice. There's still a a lot of exploration that can be done on pre-training objectives.

[authorfly]

You are right, but it's a little misleading (as it sounds like is the usefulness of your work nowadays) - Comparisons on language modelling prowess of BERT/T5 being compared to the default, non-instruct GPT-3 or OPT-3 isn't really that useful if done by size, because in practice we don't use 1.3B generative models, and more importantly, because focusing on default decoding generation without an instruct/PPO step is not how these models are used practically. The instruct models blow this performance out of the water, but instruct plus better-performance-at-size for GPT models completely shows the dominance of decoder-only architectures in my opinion for now.

I think you have to consider that in 2020/2021 many PhDs and Professors attempted to shift grant funded research with BERT and T5 to explore how they could compete with GPT-3 or to express other properties of it that supposedley outdid GPT-3. Very few (besides sentence transformers) succeeded. It's not like this is an unexplored niche. A lot of people in denial were trying to keep on with BERT research for a while despite the fact their work was essentially made obsolete by GPT-3.

(and notably Table 1 and Figure 4 are cherrypicking the smallest size with the largest gaps in task difference, and a size we know decoding is not performative at - 1.3B param mark - the characteristics and conclusions the authors come to (wow, BERT is trained on less data but does better!) obviously can't be made at larger sizes because the actual GPT models become much larger)