@z428 look, if you want all the contacts just to be able to answer each other or see each others answers, you have to tell every server and other user who is allowed and should see the answer of every other participant.
So you have to technically group and share this "virtual group with all participants" to everybody else in the group. Otherwise nobody knows who to send replies, except to you.
Even this might be possible in other protocols, you have to trust every other involved node, that it respects your intention to privacy and keep the closed list that you set initially.