Asana AI Models: Same Prompt -> Different Output -> Always failing (Bug?)

English Forum Ask the Community
, , ,
1

Hey everybody,

I’m trying to validate, if there is a wrong format of a file attachment in a task using AI-Studio.

If there is just one of X attachments in the wrong format, ai should put the task in the column “Blacklist”.

I tried Claude Sonnet 4 and GPT. The output is always wrong and completely different. Tasks became always whitelisted, even there is a blacklisted format inside.

The task has two different attachments:

  • 1 right format (.pdf)
  • 1 wrong format (.exe)

Claude Sonnet 4:

  • Inspects task, finds one whitelisted format, ignores blacklisted attachment and puts task into column “Whitelist”. Done

GPT:

  • Doesn’t find any attachments

Now I’m trying to get the list “AddAttachmentStory” / “Activity Stories” and validate that one.

Does anybody run into the same problem?

2

Hi Eike,

I never tried that use-case before.

But from what I read, my guess would be that:

  • With the Claude Sonnet, you should be able to reach a better outcome improving your prompt
  • With GPT, that’s probably a technical limitation that can’t read certain attachments

Just a guess, not sure if that helps :person_shrugging:‍

Remarks/Suggestions:

  • Ask your rule to list all the attachments in a comment, to make sure they are all taken into accounts
  • Create new task for new tests, as the Activities can mess up with your tests.
3

Hi Arthur,

thank you for your reply.

I’ve already asked my rule to list all the attachments.
Only the allowed format gets listed, and it ignores the other banned attachment completely.

It seems like, there is an order dependency. And if the rule hits the first condition right, it stops.

4

Yes, rules in Asana work in descending order - if the task meets the requirements of the first option in the condition it will trigger the appropriate action without checking if the task meets any other conditions.

5

I’ve tested both orders.

  • Bad, Good
  • Good, Bad

Output → always good.

The comment in the task says: “Tested all attachments, all safe.”
The rule only lists the allowed format.

6

Are you using AI conditions? Or just the guidance?

A screenshot of the rule would help.