When you say are the fields linked to the custom fields in this specific project too?
The fields on the form are linked to the custom field saved in the library
I’m afraid the above is the issue. You can’t connect form fields to project-level custom fields defined in a portfolio. You can only connect form fields to task-level custom fields defined in the project.
…unless the task is converted to a project and added to a portfolio with the same fields (that are added to the field library), then the fields in the portfolio will get populated with the same values as the converted task.