Asana AI Models: Same Prompt -> Different Output -> Always failing (Bug?)

English Forum AI Studio
, , ,
1

Hey everybody,

I’m trying to validate, if there is a wrong format of a file attachment in a task using AI-Studio.

If there is just one of X attachments in the wrong format, ai should put the task in the column “Blacklist”.

I tried Claude Sonnet 4 and GPT. The output is always wrong and completely different. Tasks became always whitelisted, even there is a blacklisted format inside.

The task has two different attachments:

  • 1 right format (.pdf)
  • 1 wrong format (.exe)

Claude Sonnet 4:

  • Inspects task, finds one whitelisted format, ignores blacklisted attachment and puts task into column “Whitelist”. Done

GPT:

  • Doesn’t find any attachments

Now I’m trying to get the list “AddAttachmentStory” / “Activity Stories” and validate that one.

Does anybody run into the same problem?

2

Hi Eike,

I never tried that use-case before.

But from what I read, my guess would be that:

  • With the Claude Sonnet, you should be able to reach a better outcome improving your prompt
  • With GPT, that’s probably a technical limitation that can’t read certain attachments

Just a guess, not sure if that helps :person_shrugging:‍

Remarks/Suggestions:

  • Ask your rule to list all the attachments in a comment, to make sure they are all taken into accounts
  • Create new task for new tests, as the Activities can mess up with your tests.
3

Hi Arthur,

thank you for your reply.

I’ve already asked my rule to list all the attachments.
Only the allowed format gets listed, and it ignores the other banned attachment completely.

It seems like, there is an order dependency. And if the rule hits the first condition right, it stops.

4

Yes, rules in Asana work in descending order - if the task meets the requirements of the first option in the condition it will trigger the appropriate action without checking if the task meets any other conditions.

5

I’ve tested both orders.

  • Bad, Good
  • Good, Bad

Output → always good.

The comment in the task says: “Tested all attachments, all safe.”
The rule only lists the allowed format.

6

Are you using AI conditions? Or just the guidance?

A screenshot of the rule would help.

7

Im using AI-Studio.

Thanks for ur help

Here is my prompt:

image
image1860×899 45.3 KB

English translation:

Context:You are responsible for ensuring that no infected files enter our network via task attachments. To do this, you check all file attachments and appendices of newly added tasks. This includes tasks created directly in the project as well as tasks received via a form. As this is very important, you perform the check twice in a row to be absolutely sure. Only then do you make a decision.

Instructions:Check all files attached to the task (task description, file attachment, comments). Some files/attachments may take a while to load. Therefore, wait 30 seconds before performing the check.If ANY file attachment has an unauthorized format, move the task directly to the “Blacklist” column.Unpermitted formats for attachments are:.exe.bat.cmd.js.vbs.docm.xlsm.zip.rar.msi.ps1

Only if NONE of the attachments have an unauthorized data format and a second check has been performed:→ Move the task to the “Whitelist” column.

If there are no attachments and a second check has been performed:→ Move the task to the “No attachment” column.

9

I’ll add that as a bug and get back to you when we have more info.

1 Like
10

Hi @Eike_F,

Apologies for the late response.

Can you also screenshot the logic of the rule?

11

image
image1913×1024 60.1 KB

12

Hi @Eike_F

AI Studio guidance can only to instruct AI on how to execute the conditions and rule actions that are added to the rule. It cannot influcence what is not there.

You can’t instruct the rule to do things differently to when the trigger happens. If the rule is triggered, it will run.

Multiple conditions (either in the same or different branches) is the only way to duplicate checks. But that’d be a last resort, as I think there are many ways to improve the accuracy of the rule without duplication.

A “When: Attachment is added” seems a better check, as you cannot instruct a delay into a rule, as it is not part of rule functionality.

It seems AI Studio is able do determine file extentions:

image
image670×227 14.2 KB

I’m assuming this structure will make it more accurate:

image
image1267×655 23.4 KB

I reccommend this certification training to improve your understanding of AI studio: AI Studio Foundations Skill Badge

PS: I’m guessing if you could do this with a script (or two) it would have absolute accuracy, and not use AI studio credits. See @Arthur_BEGOU Use Script Actions to Make the Most of your AI Studio credits (I don’t have experience with scripts yet, but it sounds like something that should be possible)

1 Like