[asdff]

Eventually the generation and evaluation will be quite fast where testing a million alternatives will be viable. Impressive you suggest that there might be a million alternatives but it would be faster to just read the code and settle on one. How might that be determined? Did the author who wrote the standard library really come up with the best way when writing those functions? Or did they come up with something that seemed alright to ship relative to other ideas people came up with?

I think we need to think outside the box here and realize ideas can be generated, evaluated, and settled upon far faster than any human operates. The idea of doing what a trillion humans evaluating different functions can do is actually realistic with the path of our present technology. We are at the cusp of some very remarkable times, even more remarkable than the innovations of the past 200 years, should we make progress on this effort.

[jimbokun]

This comment strikes me as not having a good intuition for how fast the space of possible programs can grow.

[asdff]

You don't think the space of possible problems can be parsed with increased compute?

[xg15]

Not for all problems, definitly not. As an example of extremely fast-growing problem spaces, look at the Busy Beaver functions:

https://en.wikipedia.org/wiki/Busy_beaver

[lefra]

A million alternatives is peanuts. Restricting the search space to text files with 37 possible symbols (letters, numbers, space), a million different files can be generated with just 4 symbols.

A trillion is 8 symbols. You still haven't reached the end of your first import statement.

I just took a random source file on my computer. It has about 8000 characters. The number of possible files with 8000 characters has 12500 digits.

At this point, restricting the search space to syntactically valid programs (how do you even randomly generate that?) won't make a difference.

[johndough]

> restricting the search space to syntactically valid programs (how do you even randomly generate that?)

By using a grammar. Here is an example on how to only generate valid JSON with llama.cpp: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...

> A trillion is 8 symbols. You still haven't reached the end of your first import statement.

Since LLMs use tokens from a vocabulary instead of characters, the number is likely somewhere in the lower billions for the first import statement.

But of course, LLMs do not sample from a uniform random distribution, so there are even fewer likely possibilities.

[stavros]

If this were viable, we'd all be running Haiku ten times in the time it took to run Opus once, but nobody does.

[asdff]

We don't have the compute for this today. We will in several centuries, if compute growth continues.

[hiimkeks]

We definitely would need centuries for this, because Moore's law has been dead for a while, while the number of possible programs grows exponentially in it's length.

But I hope we have more efficient ways to do this in a century.