The addition of auth callbacks with OpenAPI plugins seems promising but I am confused on how to use it

[ColtonBehannon]

My guess is that via the auth_callback in OpenAPIFunctionExecutionParameters I could pass header args and query or path_params that are specific to that OpenAPI function. The benefit here would be that if we use automatic function calling it would then already know what headers are required for that function. However, I am having some difficulty in the execution of this. Could someone provide a suggestion on how to inject a standard header with an API key into an OpenAPI call? Where in the workflow should I do this?

[moonbox3]

@ColtonBehannon, have you seen this kernel example that shows an example of setting up the auth callback? It doesn't need to be as complex as this one, but you can tailor the callback to grab your API key from wherever you need to load it.

https://github.com/microsoft/semantic-kernel/blob/9b8a218f5b6df9dcc72d69d989aff9b904cdecf0/python/samples/kernel-syntax-examples/openai_plugin_azure_key_vault.py#L154C13-L154C71

[ColtonBehannon]

Though parts of that example are useful for what I'm trying to do it becomes obscure when I layer them together. The big picture is as follows.

Goal:

Have 3-5 plugins available that the agent can call as needed that are all specified via OpenAPI specs.

Example Output 1:

User: "Get the latest financial news from Microsoft and summarize your findings."
Agent: "[PLUGIN-FUNCTION CALLS: YahooFinance.EarningsCallNotes, query_params['ticker'] = 'MSFT']
Microsoft (MSFT) is performing well thanks to their recent influx of AI partnerships and investments [1]."

or

Example Output 2:

User: "Check John Doe's calendar for free time Monday morning and schedule a 45-minute Microsoft Team's meeting when he is available."
Agent: "[PLUGIN-FUNCTION CALLS: MSGraphs.findMeetingTimes,
'attendees': [
'emailAddress': {
'address": '{user-mail}',
'name": 'John Doe'
],
'timeslots': [
{
'start': {
'dateTime': '2024-04-15T07:00:00',
'timeZone': 'Central Standard Time'
},
"end": {
'dateTime': '2024-04-15T10:00:00',
'timeZone': 'Central Standard Time'
}
}
];

MSGraphs.events,
{
'subject': 'My event',
'start': {
'dateTime': '2024-04-15T08:00:00',
'timeZone': 'CST'
},
'end': {
'dateTime': '2024-04-15T08:45:00',
'timeZone': 'CST'
}
}

Your meeting with John Doe is scheduled 4-15 from 8:00AM to 8:45AM."

This is functionality I can achieve with other tools like Chatbot-UI and I'm sure I can do with SK but am struggling to piece together. Here is an example of my current struggle with supplying parameters for function calling with inspiration taken from the auth_callback tests in test_sk_openapi.py.

import semantic_kernel as sk
from semantic_kernel.functions.kernel_arguments import KernelArguments
from semantic_kernel.connectors.openapi_plugin.openapi_function_execution_parameters import (
    OpenAPIFunctionExecutionParameters,
)

if __name__ == "__main__":
    """Client"""
    kernel = sk.Kernel()

    async def my_auth_callback(**kwargs):
        return {"Cache-Control": "no-cache", "Api-Key": "*****"}

    kernel_openapi_exec_settings = OpenAPIFunctionExecutionParameters(auth_callback=my_auth_callback)
    openapi_plugin = kernel.import_plugin_from_openapi(
        plugin_name="openApiPlugin",
        openapi_document_path="./OpenAPI_spec.yaml",
        execution_settings=kernel_openapi_exec_settings,
    )

    arguments = {
        "request_body": "{}",
        "path_params": "{}",
        "query_params": '{"limit": "5", "text": "Microsoft", "format": "json"}',
        "headers": '{}',
    }

    kernel_arguments = KernelArguments(**arguments)

    print(openapi_plugin["GetEarningsCallNotes"])

    result = await (
        # Call the function defined in OpenAPI_spec.yaml
        kernel.invoke(openapi_plugin["GetEarningsCallNotes"], arguments=kernel_arguments)
        )

When this plugin is invoked the API call is successfully made with the correct headers from the callback but the initial arguments like query_params are ignored. The reason I tried to structure it like this is that so that when auto_invoke_kernel_functions is applied with a planner then the LLM could figure out the correct query_params and the headers would already be supplied on the backend. I realize this is more primitive for authentication than your example but first I would just like the LLM to be able to make API calls on its own.

To apply this to the example you shared (with some minor changes for OpenAPI), if I were to use:

    plugin = await kernel.import_plugin_from_openapi(
        plugin_name="ExamplePlugin",
        openapi_document_path=openapi_spec,
        execution_parameters=OpenAPIFunctionExecutionParameters(
            http_client=http_client,
            auth_callback=authentication_provider.authenticate_request,
            server_url_override=endpoint,
            enable_dynamic_payload=True,
        ),
    )

How could I write it so that I could hardcode the headers (for now) while still allowing for dynamic query_params input from the LLM?

I realize that was a lot but any suggestions would be greatly appreciated. I'm also happy to clarify anything that may be unclear.

tldr:

I just want an agent to be able to correctly utilize auto_invoke_kernel_functions that were specified via OpenAPI spec.

[moonbox3]

Hi @ColtonBehannon, there's a lot to unpack. I'll do my best to help, but if I miss something, please let me know.

1. You bring up a few different concepts -- both function calling and then I see you mentioned a planner. In the code you provided. Is this example not working? You're explicitly calling the GetEarningsCallNotes function as part of that OpenAPI plugin, and suppling the explicity kernel arguments. I want to make sure this example is indeed processing properly (it's similar to either the Klarna or AKV example we have, and your example should be working).

result = await (
        # Call the function defined in OpenAPI_spec.yaml
        kernel.invoke(openapi_plugin["GetEarningsCallNotes"], arguments=kernel_arguments)
        )

2. In terms of using auto function calling, and having the LLM call the function with the correct function arguments: the reason the query parameters may be getting ignored is due to the following code. Based on the OpenAPI spec, only those parameters that are part of the allowed query parameters will be filled. In the kernel syntax example, this sample OpenAPI spec has one query parameter q. Therefore if the model tries to pass back irrelevant parameters, they won't be included as part of the query params. I am wondering if this is happening for you?
3. If you want to have the function use specific kernel arguments you supply, and at the sametime, have the LLM generate dynamic parameters that are used, the model would need some information about which parameters it's generating. The only thing the model will know is the name of the parameter that it should include as part of auto function calling. I think as part of your function prompt, you would need to be more specific about what the model is allowed to include and how that data should be formatted.

@SergeyMenshykh, you have a lot more experience working with OpenAPI plugins. Do you have any thoughts around what Colton is trying to achieve? Thank you. If it's simply a feature gap between SK Python and dotnet, that would be the best -- we can pull in any update as needed.

[ColtonBehannon]

Thanks for the reply @moonbox3, the first example I provided was me trying the auth_callback with an explicit function call just to test it. In the final solution this would be done by the agent in the automatic function calling.

1. After another review this example is indeed working, my apologies. I had done some experimenting on a fork and forgot to swap back.

2. This could very well be the issue if not at least related. I recall that in previous attempts I have done the LLM has made up query_params. For example, if I wanted it to search for earnings call notes about Microsoft it would use "company": "Microsoft" when that parameter does not exist, and it should instead use "text": "Microsoft".

After a little more digging I am seeing that error is coming from line 353 of openapi_manager.py and gives the following error which is seemingly indicating that it is trying to parse an empty string.

Error occurred while invoking function GetEarningsCallNotes: Expecting value: line 1 column 1 (char 0)
Error occurred while invoking function openApiPlugin-GetEarningsCallNotes
Traceback (most recent call last):
File "/home/cbeh/Projects/semantic-kernel/python/semantic_kernel/functions/kernel_function.py", line 188, in invoke
return await self._invoke_internal(kernel, arguments)
File "/home/cbeh/Projects/semantic-kernel/python/semantic_kernel/functions/kernel_function_from_method.py", line 102, in _invoke_internal
result = await result
File "/home/cbeh/Projects/semantic-kernel/python/semantic_kernel/connectors/openapi_plugin/openapi_manager.py", line 352, in run_openapi_operation
json.loads(path_params)
File "/home/cbeh/miniconda3/envs/semantic-kernel/lib/python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/home/cbeh/miniconda3/envs/semantic-kernel/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/cbeh/miniconda3/envs/semantic-kernel/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This is the code leading to this error:

async def chat() -> bool:
    try:
        user_input = input("User:> ")
    except KeyboardInterrupt:
        print("\n\nExiting chat...")
        return False
    except EOFError:
        print("\n\nExiting chat...")
        return False

    if user_input == "exit":
        print("\n\nExiting chat...")
        return False
    chat_history.add_user_message(user_input)
    arguments = KernelArguments(system_message=system_message, user_input=user_input, chat_history=chat_history)

    stream = True
    if stream:
        answer = kernel.invoke_stream(
            chat_function,
            arguments=arguments,
        )
        print("bot:> ", end="")
        async for message in answer:
            print(str(message[0]), end="")
        print("\n")
        return True
    answer = await kernel.invoke(
        chat_function,
        arguments=arguments,
    )
    print(f"bot:> {answer}")
    chat_history.add_assistant_message(str(answer))
    return True


async def main() -> None:
    chatting = True
    while chatting:
        chatting = await chat()


# if __name__ == "__main__":
#     asyncio.run(main())
await main()

The plugin-function being called is the same as shared earlier:

    async def my_auth_callback(**kwargs):
        return {"Cache-Control": "no-cache", "Api-Key": "*****"}

    kernel_openapi_exec_settings = OpenAPIFunctionExecutionParameters(auth_callback=my_auth_callback)
    openapi_plugin = kernel.import_plugin_from_openapi(
        plugin_name="openApiPlugin",
        openapi_document_path="./OpenAPI_spec.yaml",
        execution_settings=kernel_openapi_exec_settings,
    )

I query it with "Get earnings call notes about {company_here}" and get the error above. I have also tried "Get earnings call notes about {company_here}. If using a function call be sure to use 'text': '{company_here}' as a query parameter." in hopes of mimicking a bit of what you suggested in your 3. response but have received the same error.

Any follow up suggestions or tips for debugging? It would be useful to see what the LLM is passing as query_params but I cannot figure that out if it's possible.

[moonbox3]

A couple of thoughts:

• You can configure a logger so you can capture the logger.info() calls. Before the function call is made, we log the function arguments as shown here. Here's an example on how to configure logging.
• I see you're using the Anaconda package manager. If you are able to use Poetry, you'd be able to step through the code and see the arguments. If that's off the table, then configuring the logger should be fine.
• You can try to configure one specific system message as part of your chat history (or a separate user message) with specific instructions on how the query parameters should be formed. One thought around this, though: you're asking a non-deterministic model to try and form a more deterministic query parameter, in a particular form that a function (unrelated to the model expects). Even with a very low temperature, or one at 0.0, you still may not get the deterministic parameter form you're looking for. This comment is something to think about as you develop your application.

I can do some more digging on my end about how to better handle this.

[ColtonBehannon]

I've never actually looked into poetry but will give it a shot and will keep the logger in mind as well! I'll also change the prompt to be extremely explicit just to see if I can get it to overcome the nondeterminism. Thanks again!

[ColtonBehannon]

Update: Following the advice from @moonbox3 I enabled the logger which was a massive help for identifying the args being used with the function call. The model kept wanting to use an empty string for the value of path_params which triggers

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

from this line. By including in the system prompt that it should not specify empty strings as args but instead use {} it is now working.

[moonbox3]

Great to hear, thanks for the update, @ColtonBehannon.