Currently that is not really possible as Megan mentioned.
Here is an existing feedback request thread I recommend upvoting.
You could for example create a separate project. Then in the relevant tasks, where you need a guest user to share some feedback or they need to provide something, add a subtask and only assign that to the guest user and multihome these tasks only to the special project.
And here is a nice overview into guest visibility: Who sees whom, a guide to guest visibility with Batman & Robin