Skip to main content

Installation and Setup

Running through Docker​

A simple way to initiate self-hosted TaskingAI-Inference service is through Docker.

Prerequisites​

  • Docker and Docker Compose installed on your machine.
  • Git installed for cloning the repository.
  • Curl or other HTTP client installed for testing and debugging.

First, pull the latest Docker image taskingai/taskingai-inference:latest from Docker Hub (or the appropriate registry) by executing the following command in your terminal:

docker pull taskingai/taskingai-inference:latest

Then, you can run a new container using the following command:

docker run -d -p 8000:8000 taskingai/taskingai-inference:latest

Running through Source Code​

First download TaskingAI-Inference GitHub repository with:

git clone https://github.com/TaskingAI/Universal-LLM-API.git
cd Universal-LLM-API

Install the required Python packages by running the following command:

pip install -r requirements.txt

Then, you can run the service using the following command:

PYTHONPATH=$(pwd) python app/main.py

Your TaskingAI-Inference is now running on http://localhost:8000 🚀

Once deployed, use tools like Postman, cURL, or any HTTP client that allows you to send requests to your local TaskingAI-Inference service for testing and debugging.

Example Usage​

Here's an example of how to request a chat completion inference from OpenAI's GPT-4 model using TaskingAI-Inference.

curl "http://localhost:8000/v1/chat_completion" \
-X POST \
-H "Content-Type: application/json" \
-d '{
    "model_schema_id": "openai/gpt-4",
    "credentials": {
      "OPENAI_API_KEY": "$OPENAI_API_KEY"
    },
    "stream": false,
    "configs": {
      "temperature": 0.8
    },
    "messages": [
        {
            "role": "user",
            "content": "What is machine learning?"
        }
    ]
}'

Note that $OPENAI_API_KEY is an environment variable that stores your OpenAI API key. You can replace it with your actual API key.