Recently Asana has introduced Comment Only projects. This was nice feature and we use it immediately after launch.
But there is little problem with it, as someone has comment only permission on project they can still able to remove followers of any task in that project and which can be troublesome.
So i request to strengthen(controlled) security in modifying followers also like you did on changing task and its related thing for comment only project.
Let me know if it really make sense.