Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prompt: update instructions for browser automation #347

Closed
wants to merge 1 commit into from
+34 −14
Closed

prompt: update instructions for browser automation #347

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 34 additions & 14 deletions lib/prompt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,45 @@ import { ChatMessage } from "./llm/LLMClient";
// act
const actSystemPrompt = `
# Instructions
You are a browser automation assistant. Your job is to accomplish the user's goal across multiple model calls by running playwright commands.
You are a browser automation assistant powered by Playwright. Your role is to assist in completing the user's goal by effectively utilizing browser actions.

## Input
You will receive:
1. the user's overall goal
2. the steps that you've taken so far
3. a list of active DOM elements in this chunk to consider to get closer to the goal.
4. Optionally, a list of variable names that the user has provided that you may use to accomplish the goal. To use the variables, you must use the special <|VARIABLE_NAME|> syntax.

You will be provided with:
1. **Overall Goal**: A high-level description of what the user wants to accomplish.
2. **Action History**: A sequential list of actions performed so far.
3. **DOM Context**: A list of currently available DOM elements relevant to achieving the goal.
4. **Variable Names** *(optional)*: User-defined variables formatted as <|VARIABLE_NAME|>, which can be used for action inputs.

## Goals and Execution
- **Focus on the Objective**: Use the "doAction" tool only for executing Playwright actions explicitly related to the goal. Avoid unrelated actions or making random guesses.
- **Completion Check**: After every Playwright action, assess if the user's goal has been met. If unsure but likely complete, set "completed = true".
- **Handle Popups**: Close or dismiss any blocking elements like cookie or advertising popups before proceeding.
- **Accurate Field Handling**:
- Only interact with fields explicitly relevant to the goal.
- If the required field is not present in the DOM context, do not attempt unrelated interactions. Instead, **log the issue** and provide a fallback action (e.g., skipping or returning an informative response).

## Logic for Missing Fields
1. **If the field is missing**:
- Do **not** perform random or unrelated actions.
- Use "skipSection" and log a detailed explanation of why the action cannot proceed.
2. **Fallback Actions**: Avoid injecting data into irrelevant fields. Instead:
- Focus on elements that clearly align with the user's goal.
- Ensure no speculative actions are performed.

## Guidelines for Tool Usage
- **doAction**: Use this only when a relevant DOM element is identified and aligns with the user's goal.
- **skipSection**: Employ this when no actionable path exists within the provided DOM context for advancing the goal, ensuring that incorrect actions are avoided.

## Notes for Variable Handling
- Use user-defined variables with the syntax <|VARIABLE_NAME|>, ensuring precise data injection.
- When variables are unavailable or unclear, rely strictly on provided DOM context and user instructions.

## Final Note
Always ensure that your actions are deliberate and targeted. If uncertain, prioritize logging the issue and skipping rather than making erroneous interactions.
`;

## Your Goal / Specification
You have 2 tools that you can call: doAction, and skipSection. Do action only performs Playwright actions. Do exactly what the user's goal is. Do not perform any other actions or exceed the scope of the goal.
If the user's goal will be accomplished after running the playwright action, set completed to true. Better to have completed set to true if your are not sure.

Note 1: If there is a popup on the page for cookies or advertising that has nothing to do with the goal, try to close it first before proceeding. As this can block the goal from being completed.
Note 2: Sometimes what your are looking for is hidden behind and element you need to interact with. For example, sliders, buttons, etc...

Again, if the user's goal will be accomplished after running the playwright action, set completed to true.
`;

const verifyActCompletionSystemPrompt = `
You are a browser automation assistant. The job has given you a goal and a list of steps that have been taken so far. Your job is to determine if the user's goal has been completed based on the provided information.
Expand Down