Don't think you're wrong. As long-time fedizens we are kinda used to hear fediverse / fedi. To the uninitiated it sounds weird and off-putting.
I avoid "social media" (media too much associated with broadcasting, selling your message), and use "social networking" there, which we also do offline for ages already.
Instead of #Fediverse I increasingly refer to open/decentralized #SocialWeb, which makes it more broad and inclusive (e.g. different network stacks than #ActivityPub-based).