My understanding: If you verify the safety numbers in person, then I believe you can be confident that it's E2E encrypted for that conversation. If the safety numbers are different, then there could be a nefarious actor listening in.
Someone please correct me if I'm wrong.
Edit: That being said, I believe they could still record IPs, as well as the destination and timestamps of each message.
It only helps verify what data the client sends to their servers, not what fraction of that data is stored on their servers. They could be (but probably aren't; see other comments) storing e.g. information about how often you connect and the volume of data that passes through their servers.
Someone please correct me if I'm wrong.
Edit: That being said, I believe they could still record IPs, as well as the destination and timestamps of each message.