you'd need to train a bidirectional adapter for every person instead of (N-1)^2 adapters for each pair of people
you'd need to train a bidirectional adapter for every person instead of (N-1)^2 adapters for each pair of people