Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] #75

Open
gitkitian opened this issue Jul 5, 2024 · 4 comments
Open

[BUG] #75

gitkitian opened this issue Jul 5, 2024 · 4 comments
Labels
bug Something isn't working triaged

Comments

@gitkitian
Copy link

Title

openai.BadRequestError: Error code: 400 - {'error': {'code': 'invalid_type', 'param': 'messages[0].content', 'message': "Invalid type for 'messages[0].content': expected one of a string or array of objects, but got an object instead.", 'type': 'invalid_request_error'}}

Description

(Agent-E) D:\Platform\Agent-E>python -m ae.main

Creating AutogenWrapper with ['user', 'planner_agent', 'browser_nav_agent', 'browser_nav_executor'] and 1000 rounds.
INFO:ae.utils.logger:Using model qwen2-72b-instruct for AutoGen from the environment variables.
Created browser_nav_executor_agent: <ae.utils.autogen_sequential_function_call.UserProxyAgent_SequentialFunctionExecution object at 0x000002DE7D476750>
INFO:ae.utils.logger:User preferences loaded from: D:\Platform\Agent-E\ae\user_preferences\user_preferences.txt
INFO:ae.utils.logger:User preferences loaded from: D:\Platform\Agent-E\ae\user_preferences\user_preferences.txt
INFO:ae.utils.logger:User dir: C:\Users\xxhua\AppData\Local\Google\Chrome\User Data\Default
INFO:ae.utils.logger:Received command: open https://github.com/ and search agent-e
INFO:ae.utils.logger:Processing command: open https://github.com/ and search agent-e
INFO:ae.utils.logger:Prompt for command: Execute the user task "open https://github.com/ and search agent-e" Current Page: https://www.baidu.com/
user (to planner_agent):

Execute the user task "open https://github.com/ and search agent-e" Current Page: https://www.baidu.com/


planner_agent (to user):

{"plan": "1. Navigate to https://github.com/.
2. Confirm that you are on the github homepage.
3. Use the search bar on github to search for agent-e.",
"next_step": "Navigate to https://github.com/.",
"terminate":"no"}


WARNING:ae.utils.logger:LLM response was not properly formed JSON. Will try to use it as is. LLM response: "{"plan": "1. Navigate to https://github.com/.
2. Confirm that you are on the github homepage.
3. Use the search bar on github to search for agent-e.",
"next_step": "Navigate to https://github.com/.",
"terminate":"no"}". Error: Invalid control character at: line 1 column 46 (char 45)
WARNING:ae.utils.logger:LLM response was not properly formed JSON. Will try to use it as is. LLM response: "{"plan": "1. Navigate to https://github.com/.
2. Confirm that you are on the github homepage.
3. Use the search bar on github to search for agent-e.",
"next_step": "Navigate to https://github.com/.",
"terminate":"no"}". Error: Invalid control character at: line 1 column 46 (char 45)
Adding system message: "Plan:\n\n 1. Navigate to https://github.com/.2. Confirm that you are on the github homepage.3. Use the search bar on github to search for agent-e."
Adding system message: "Next step: Navigate to https://github.com/."
WARNING:ae.utils.logger:LLM response was not properly formed JSON. Will try to use it as is. LLM response: "{"plan": "1. Navigate to https://github.com/.
2. Confirm that you are on the github homepage.
3. Use the search bar on github to search for agent-e.",
"next_step": "Navigate to https://github.com/.",
"terminate":"no"}". Error: Invalid control character at: line 1 column 46 (char 45)


Starting a new chat....


browser_nav_executor (to browser_navigation_agent):

: Navigate to https://github.com/., Current Page: https://www.baidu.com/,


USING AUTO REPLY...
browser_navigation_agent (to browser_nav_executor):

***** Suggested tool call (): openurl *****
Arguments:
{"url": "https://github.com/"}



EXECUTING ASYNC FUNCTION openurl...
INFO:ae.utils.logger:Opening URL: https://github.com/
WARNING:ae.utils.logger:Initial navigation to https://github.com/ failed: BindingCall.call: Timeout 3000ms exceeded.
Call log:
navigating to "https://github.com/", waiting until "load"
. Will try to continue anyway.
Traceback (most recent call last):
File "D:\Platform\Agent-E\ae\core\skills\open_url.py", line 31, in openurl
await page.goto(url, timeout=timeout*1000) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\playwright\async_api_generated.py", line 8657, in goto
await self._impl_obj.goto(
File "D:\Platform\Agent-E.venv\Lib\site-packages\playwright_impl_page.py", line 519, in goto
return await self._main_frame.goto(**locals_to_params(locals()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\playwright_impl_frame.py", line 145, in goto
await self._channel.send("goto", locals_to_params(locals()))
File "D:\Platform\Agent-E.venv\Lib\site-packages\playwright_impl_connection.py", line 59, in send
return await self._connection.wrap_api_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\playwright_impl_connection.py", line 514, in wrap_api_call
raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None
playwright._impl._errors.TimeoutError: BindingCall.call: Timeout 3000ms exceeded.
Call log:
navigating to "https://github.com/", waiting until "load"

Adding system message: "Opened URL: https://github.com/"
browser_nav_executor (to browser_navigation_agent):

browser_nav_executor (to browser_navigation_agent):

***** Response from calling tool () *****
Page loaded: https://github.com/, Title: GitHub: Let’s build from here · GitHub



USING AUTO REPLY...
ERROR:ae.utils.logger:Unable to process command: "open https://github.com/ and search agent-e". Error code: 400 - {'error': {'code': 'invalid_type', 'param': 'messages[0].content', 'message': "Invalid type for 'messages[0].content': expected one of a string or array of objects, but got an object instead.", 'type': 'invalid_request_error'}}
Traceback (most recent call last):
File "D:\Platform\Agent-E\ae\core\autogen_wrapper.py", line 350, in process_command
result=await self.agents_map["user"].a_initiate_chat( # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 1063, in a_initiate_chat
msg2send = await self.a_generate_reply(messages=self.chat_messages[recipient], sender=recipient)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 2023, in a_generate_reply
final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 397, in _summary_from_nested_chats
res = initiate_chats(chat_to_run)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\agentchat\chat.py", line 200, in initiate_chats
chat_res = sender.initiate_chat(**chat_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 1000, in initiate_chat
self.send(msg2send, recipient, request_reply=True, silent=silent)
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 645, in send
recipient.receive(message, self, request_reply, silent)
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 808, in receive
reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 1949, in generate_reply
final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 1315, in generate_oai_reply
extracted_response = self._generate_oai_reply_from_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 1334, in _generate_oai_reply_from_client
response = llm_client.create(
^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\oai\client.py", line 638, in create
response = client.create(params)
^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\autogen\oai\client.py", line 285, in create
response = completions.create(**params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\openai_utils_utils.py", line 275, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\openai\resources\chat\completions.py", line 663, in create
return self._post(
^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\openai_base_client.py", line 1200, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\openai_base_client.py", line 889, in request
return self._request(
^^^^^^^^^^^^^^
File "D:\Platform\Agent-E.venv\Lib\site-packages\openai_base_client.py", line 980, in _request
raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'code': 'invalid_type', 'param': 'messages[0].content', 'message': "Invalid type for 'messages[0].content': expected one of a string or array of objects, but got an object instead.", 'type': 'invalid_request_error'}}
INFO:ae.utils.logger:Command "open https://github.com/ and search agent-e" took: 8.29 seconds.

@gitkitian gitkitian added the bug Something isn't working label Jul 5, 2024
@teaxio
Copy link
Collaborator

teaxio commented Jul 5, 2024

@gitkitian this is a very interesting error as it worked for the first few steps of the plan then something happened seemingly in relation to conversation management in openai library. Is there a way for me to try this LLM? I tried hosting it on my mac, but I do not have enough memory.
Does Qwen2 support function calling capability? I am not seeing concrete evidence one way or another.

@teaxio
Copy link
Collaborator

teaxio commented Jul 5, 2024

In the absence of an access point that has 70B, I tried this with qwen2 7B running locally and I get as far as github is opened but nothing else. The key issue is that the model does not adhere to instructions and does not return the termination string requested. So we just sit there and do nothing.

image

If I try this in openai gpt4, I get something like this:

Page loaded: https://github.com/, Title: GitHub: Let’s build from here · GitHub

browser_navigation_agent (to browser_nav_executor):

The page has been successfully navigated to GitHub.

##TERMINATE TASK##

Notice ##TERMINATE TASK## at the end. That is what the code is looking for as a termination signal. I tried 5 iterations on the prompt including asking in the user command to add the termination string, but the model still never output it.

At this point we do not have a fix quite yet. If you have ideas on a different way to determine termination please share or make a PR. Meanwhile we are going to think about this.

@gitkitian
Copy link
Author

gitkitian commented Jul 6, 2024

Thanks for your attention and response. I am using the latest version of agent-e. You can reproduce this bug with the following LLM parameters configuration.
AUTOGEN_MODEL_NAME=qwen2-72b-instruct
AUTOGEN_MODEL_API_KEY=redacted for privacy
AUTOGEN_MODEL_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1

@teaxio
Copy link
Collaborator

teaxio commented Jul 6, 2024

@gitkitian I removed your key. You can't put an API key like this in a public forum. It would have been better to share it in a direct message in Discord. I tried what you gave and I get an error from that endpoint. I do not know much about that service so I can't debug it. I suspect some parameters are missing. Here is the error:

httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
DEBUG:openai._base_client:Not retrying
DEBUG:openai._base_client:Re-raising status error
ERROR:ae.utils.logger:Unable to process command: "go to github  and search agent-e". Error code: 400 - {'error': {'code': 'invalid_parameter_error', 'param': None, 'message': 'field required: payload.parameters.tools.0.type & field required: payload.parameters.tools.1.type & field required: payload.parameters.tools.2.type & field required: payload.parameters.tools.3.type & field required: payload.parameters.tools.4.type & field required: payload.parameters.tools.5.type & field required: payload.parameters.tools.6.type & field required: payload.parameters.tools.7.type', 'type': 'invalid_request_error'}, 'id': 'chatcmpl-215d076a-0cac-9d8c-aa23-1ebb0464bb6f'}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triaged
Projects
None yet
Development

No branches or pull requests

2 participants