[Delitio]

Is there any source which explains what billion of parameters actually are?

In my mind a parameter is: language, dialect, perhaps context parameters (food, dinner, lunch, travel) and if we than talk about language and audio perhaps sound waves, gender.

Or are context parameters which gives you insight? Like a billion of parameters are literally something like travel=false, travel-europe=true people speaking=e, age, height,

[dotnet00]

Parameters are just floating point numbers, at most they can be seen as degrees of freedom or kind of like the order of a polynomial used in curve fitting.

They're too abstract to assign much meaning to individual parameters, as our understanding of why their values are exactly the way they are is extremely limited.

[brrrrrm]

A good visual introduction to neural networks can be found here: https://playground.tensorflow.org

A parameter is a "weight" in this case (the lines drawn from neuron to neuron). The neurons are effectively runtime values or "activations." Parameters (weights) are updated during training and then set as constant during "inference" (also called "prediction").

There's unfortunately a ton of jargon and different groups use different words almost exclusively.

[matt123456789]

A parameter is a scalar value, most of which are in the attention matrices and feedforward matrices, you also hear these called “weights”. Any intro to DL course will cover these in detail. I recommend started with Andrew Ng’s Coursera class on Intro to Machine Learning, although there may be better ones out there now.

[Delitio]

Input parameter vs. weights then?

I see tx

[lostmsu]

These networks (text models) usually have around a few thousand inputs.

[jefft255]

The parameters are the number of weights in a neural network, in this case.

[nl]

It's rare a single parameter maps to a human understandable concept. Occasionally someone finds one that does map fairly well, for example this case back in 2017: https://openai.com/blog/unsupervised-sentiment-neuron/#senti...