[mrshu]

Though not exactly what you are after, Contextual Document Embeddings (https://huggingface.co/jxm/cde-small-v1), which generate embeddings based on "surrounding context" might be of some interest.

With 281M params it's also relatively small (at least for an embedding model) so one can play with it relatively easily.