The whole point of the fediverse in general is that there is no single “we”. There are those ActivityPub servers that one federates with and those ActivityPub servers that one does not federate with. Lack of consensus is built into the technology by design.
but I think it’s still important that everyone at least thinks about what they want the fediverse to grow into
To have wants about the social constructs that sit atop the technology is to misunderstand the technology. The technology enables your enemies/people you don’t like/etc. to communicate and benefit in all the ways you do. By design. You cannot exclude from the fediverse, you can only exclude from your server. If you have wants about excluding then you’ve misunderstood the technology.