[lmeyerov]

It seems the quality goes up & cost goes down significantly with Colossal AI's recent push: https://medium.com/@yangyou_berkeley/colossalchat-an-open-so...

Their writeup makes it sounds like, net, 2X+ over Alpaca, and that's an early run

The browser side is interesting too. Browser JS VMs have a memory cap of 1GB, so that may ultimately be the bottleneck here...

[lmeyerov]

Interesting, since I looked last year, Chrome has started raising the caps internally on buffer allocation to potentially 16GB: https://chromium.googlesource.com/chromium/src/+/2bf3e35d7a4...

Last time I tried on a few engines, it was just 1-2GB for typed arrays, which are essentially the backing structure for this kind of work. Be interesting to try again..

For our product, we actually want to dump 10GB+ on to the WebGL side, which may or may not get mirrored on the CPU side. Not sure if additional limits there on the software side. And after that, consumer devices often have another 10GB+ CPU RAM free, which we'd also like to use for our more limited non-GPU stuff :)

[jesse__]

I thought the memory limit (in V8 at least) was 2GB due to the GC not wanting to pass 64 bit pointers around, and using the high bit of a 32-bit offset for .. something I now forget ..?

Do you have a source showing a JS runtime with a 1GB limit?

[jesse__]

UPDATE: After a nominal amount of googling around it appears valid sizes have increased on 64-bit systems to a maximum of 8GB, and stayed at 2GB on 32-bit systems, for FF at least. I guess it's probably 'implementation defined'

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

[SebJansen]

does the 1gb limit extend to wasm?

[jesse__]

WASM is specified to have 32-bit pointers, which is 4GB. AFAIK browser implementations respect that (when I did some nominal testing a couple years ago)