Support for custom LLMs in python

[sjadlakha]

When will semantic kernel for python support custom LLM connections for eg. connecting to a locally trained LLM?

[moonbox3]

@sjadlakha thanks for your question. We have an ollama connector in Python right now, that allows for one to load a local LLM. Will this work for your use-case?

[sjadlakha]

@moonbox3 That helped but my model is hosted centrally for a couple of applications and I can access it through a Kerberos authenticated https endpoint. I was wondering if there is anything that I can use to connect semantic kernel to this LLM

[moonbox3]

@juharris is one able to use ollama in this type of scenario? If not, I'll have to think some more about how this may be done with SK. (Asking you, Justin, as I have seen you contribute on ollama and you may know :)).

[juharris]

The Ollama connector is mainly for using models hosted by Ollama running locally or elsewhere. I think you can either:

• Support the Ollama API spec and use the Ollama connectors/services in Python: https://github.com/ollama/ollama/blob/main/docs/api.md
• Make custom connectors and copy the Ollama ones as simple examples

[sjadlakha]

Thanks @juharris , will making custom connectors based on the ollama connectors work with planners and other components of sk without making a custom implemention of those components?

[sjadlakha]

@juharris , @moonbox3 what do you suggest?

[moonbox3]

@eavanvalkenburg thoughts here?

[eavanvalkenburg]

As long as you create a class that implements ChatCompletionClientBase you can do whatever stuff in there you need to make it work, and indeed Ollama is a good example because it does not rely on a package currently, just RESTfull api calls.

[sjadlakha]

Awesome, that helps .
Thanks 👍