[WorldMaker]

It should be clear that brute-forcing a passphrase is much more complicated than a PIN number, in the best case for you (worse case for an attacker).

Just because your passphrase may essentially be a PIN look up into a lookup table doesn't mean the attacker knows that or has access to the same lookup table.

You can also add additional entropy via punctuation or casing.

The point to a random passphrase is to try to avoid "human" mistakes like over favoring a subset of words, and rerolling words you don't like potentially makes your collection of passphrases more susceptible to analysis or social engineering (word association) attacks.

Like I said, it's generally better to pick a word-list you are comfortable with all the possible words than to subset a word-list you aren't entirely comfortable with.

The goal of something like Diceware is to be easy for humans to memorize but also still true random (see: xkcd's battery horse comic). If you don't need to memorize it, then yes, why not entirely generate a random sequence of letters/numbers/symbols/emoji.