My recollection is that you can’t directly influence the names that appear yourself, unless you choose to deprovision those people; then they will be removed from your workspace and from the lists.
See more here on how/when they appear in the lists:
See here for how to deprovision users in a workspace:
See here for some potential things to do beforehand: