Actually AES, unlike more ad-hoc block ciphers, is based on the theory of finite fields, including GF(8) that is used for its non-linear component.
The theory of finite fields is based on the theory of prime numbers, because the finite fields are sets of residues modulo a prime number or modulo a power of a prime number.
The theory of finite fields is involved in the design of many other block cipher functions or secure hash functions and also in the design of the most important message-authentication methods, like GCM, which is used to authenticate this HTML page on the HN site.
So prime numbers are important in most cryptographic applications, not only in asymmetric cryptography, like Diffie-Hellman or RSA. Prime numbers are used in one way or another for the transmission of any HTTPS data packet, not only in the key establishment phase of a TLS connection.
> The theory of finite fields is based on the theory of prime numbers, because the finite fields are sets of residues modulo a prime number or modulo a power of a prime number.
It is note quite correct that the finite field of order p^k is the set of residues modulo p^k when k > 1. Instead this field is obtained as a splitting field of the field of order p (which is the set of residues mod p).
Prime numbers and elliptic curves are much more connected than one might expect. Each elliptic curve generates a function similar to zeta function, and there is a version of a Riemann hypothesis for elliptic curves https://m.youtube.com/@PeakMathLandscape
ECC is pretty closely related to the study of prime numbers. It might not be built directly on the difficulty of factoring, but the theory of how to construct curves, how to use them, what's expected to be secure etc goes pretty deep.
With asymmetric crypto, you must also send by courier the root certificates (downloading Chrome or Firefox just fulfills the role of a courier that is not very trustworthy).
There exists absolutely no method of secure communication that does not depend on a piece of information that is transmitted separately, through a presumed trustworthy courier. All the existing methods only attempt to minimize the amount of information that must be sent through the secure courier.
With symmetric crypto without digital signatures but with some kind of Diffie-Hellman, you must send by courier only a pre-shared key that is used only for computing message-authentication codes that are used only in the couple of packets used in a key-exchange algorithm, when establishing a secure connection.
Using only symmetric crypto, secure communication can be performed in pretty much the same way as with asymmetric crypto, by generating fresh random session keys for every connection.
The only difference is that the key exchange packets are authenticated with a MAC using a pre-shared key, instead of being authenticated with digital signatures and a chain of certificates going to trusted root certificates.
If for some weird reason one would not want to use a Diffie-Hellman variant (e.g. with elliptic curves) to protect the session keys, one could use another pre-shared key only for encrypting the key-exchange packets.
There are only two advantages for asymmetric crypto, when used for secure communication connections.
The first is provided by Diffie-Hellman in any of its variants, which ensures perfect forward secrecy, i.e. even knowing all the content of some sessions, including their secret keys, that does not allow the decryption of other sessions. Without Diffie-Hellman, if the pre-shared encryption key that is used to protect the key exchange packets is captured, all recorded sessions could be decrypted. This can be only partially avoided by changing that key frequently, which would prevent the decryption of past sessions, but not the decryption of future sessions.
The second advantage is provided by the authentication of the key exchange with digital signatures instead of MACs based on pre-shared keys, which is the possibility of half authentication, where the server is authenticated based on the certificates provided by it, but the client is not authenticated, which is the most frequent kind of secure communication used on the Internet.
For communication inside a closed environment, i.e. a private network, using key exchange authentication based on pre-shared keys (but with elliptic-curve Diffie-Hellman for protecting the session keys) can be simpler, faster and more secure than using digital signatures and certificates.
While in the beginning I have used your metaphor about sending a pre-shared key or the root certificates by courier, the normal mode of transferring pre-shared authentication keys is by initial physical pairing (e.g. cable connection) of the devices that must be able of communicating securely between themselves.
(Not to say that the study of prime numbers isn't hugely important to most of cryptography)