Custom field values, like anything else in the task, is visible if the task is visible.
I too wish there was a way to have some task information project-related and visible only if you have visibility into that project, but nothing like that exists.
They are hacks but two workarounds exist:
One would be to add a link to a task in a private (to some) project as I mentioned here:
That link will appear as Private Link to those w/o access. For the rest, it could contain the set of “private” custom fields.
Another approach: Define private (to some) projects for each binary piece of data you want to privately attach to a task so that it’s seen by some people but not others. For example, instead of a custom field like Client Is In Arrears? (true/false), create a private project called Client Is In Arrears. Multi-home the task to that project if the client is in arrears; in the task, if it’s multi-homed there, those with access to that project will see that project link, anyone else will not see any such link. You could multi-home to multiple ones of these, though this could be more tedius than the first suggestion.
Hope one helps,