[galaxytachyon]

I remember there is a study about the alignment cost. Basically the more restrictions and limit you put on a model, the worse its general performance becomes. Things like a ban on violence, race, or any other sensitive topics effectively throttle or change how the model "reason" or connect information within its network of parameters and result in degraded capacity.

I wonder if this is the reason behind all of this.

Edit: the study: https://arxiv.org/pdf/2308.13449.pdf

[RationPhantoms]

How much of it is OpenAI/Microsoft curtailing the compute being used to generate responses?

[practice9]

The accuracy loss is more consistent with some kind of quantization of the model(-s) behind the scenes than the alignment gone wrong. Quantization to serve more users faster, on same amount or less of compute.

[arrowsmith]

Sorry, what does quantization mean here?

[iamjackg]

Reducing the precision of the weights from high precision floating points to either lower precision floats or even integers. You'd think it would greatly reduce the performance of a model, but in most cases the decline in quality is extremely tolerable compared to the reduction in memory/processing requirements.

[mlboss]

It means using less number of bits to store float values. This reduces the memory/compute requirement at the cost of making model less precise.

[imdsm]

Reducing the precision of the parameters — result being less memory intensive

[mov_eax_ecx]

How can i locate this study?. I think you are misrepresenting something.

In the gpt4 paper they specifically address this, and find that "Averaged across all exams, the base model achieves a score of 73.7% while the RLHF model achieves a score of 74.0%, suggesting that post-training does not substantially alter base model capability."

[nicce]

The problem with these studies is that we really still don’t know. Nobody can replicate the papers of OpenAI.

[galaxytachyon]

Found it, it is a pretty recent paper.

https://arxiv.org/pdf/2308.13449.pdf

[adamsb6]

Given the homogeneity of responses on taboo subjects, there's probably something exogenous to the model at work.

[dalore]

It feels the same thing happens with humans.