LocalAI

note

If your tasking AI server is locally deployed with Docker, and the target model is also running in your local environment, LOCALAI_HOST should start with http://host.docker.internal:port instead of http://localhost:port. Replace port with your actual port number.

LocalAI is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing.

Requisites

To integrate a model running on LocalAI to TaskingAI, you need to have a valid LocalAI service first. To get started, please visit LocalAI's website, or follow the simple instructions in the Quick Start.

Required credentials:

• LOCALAI_HOST: Your LocalAI host URL.

Supported Models:

Wildcard

• Model schema id: localai/wildcard

Quick Start

Deploy LocalAI service to your local environment

1. Download and Install Docker.
2. Start the service with docker command:

   docker run -ti -p 8090:8090 --gpus all localai/localai:v2.9.0-cublas-cuda11-core <model_name>
   In this way, a model will be running on your localhost at 8090 port. You can access the model by sending a POST request to http://localhost:8090. For a detailed model list, please check LocalAI Models.

Integrate Ollama to TaskingAI

Now that you have a running Ollama service with your desired model, you can integrate it to TaskingAI by creating a new model with the following steps:

1. Visit your local TaskingAI service page. By default, it is running at http://localhost:8090.
2. Login and navigate to Model management page.
3. Start creating a new model by clicking the Create Model button.
4. Select LocalAI as the provider, and wildcard as model.
5. Use Ollama service's address http://localhost:8090 as LOCALAI_HOST