Chat Completion

The Chat Completion API allows you to generate text using powerful language models.

---

Overview

TaskingAI's Inference API stands out for its unified approach, enabling users to access and utilize a wide range of mainstream large language models from various providers through a single, streamlined interface. This unique feature significantly reduces the learning curve, as it allows users to interact with different models, each with varying parameter configurations, without the need to understand the intricacies of each individual provider's API.

Prerequisites

Before using the API, make sure you already have an active TASKINGAI_API_KEY from your project.

Examples

Single-round Chat Completion

Here is a simple example of how to create a chat_completion task using taskingai in your Python code. Note that the system messages is an optional prompt that asks the assistant to perform a specific task or play a specific role during the following conversation. Only one system message is allowed in messages.

model_id = "YOUR_MODEL_ID"
chat_completion_result = taskingai.inference.chat_completion(
    model_id=model_id,
    messages=[
        {"role": "system", "content": "You are a health advisor providing nutritional advice. You should always reply with a professional, kind, and patient tone."},
        {"role": "user", "content": "How much sugar can a woman take in a day?"},
    ]
)

Please replace YOUR_MODEL_ID with the actual chat completion model ID in your TaskingAI project.

Multi-round Chat Completion

After the chat is initialized, you can continue to add messages to the messages list to continue the chat. For example:

# multi round chat completion
chat_completion_result = taskingai.inference.chat_completion(
    model_id=model_id,
    messages=[
        {"role": "system", "content": "You are a health advisor providing nutritional advice. You should always reply with a professional, kind, and patient tone."},
        {"role": "user", "content": "How much sugar can a woman take in a day?"},
        {"role": "assistant", "content": "The AHA suggests a added-sugar limit of no more than 100 calories per day (about 6 teaspoons or 24 grams) for woman."},
        {"role": "user", "content": "What about man?"},
    ]
)

Chat Completion in Stream Mode

You can also use the chat completion API in stream mode by specifying the stream parameter to True. In the stream mode, the API response is streamed as a series of chunks. This feature is particularly useful for scenarios where the output is expected to be lengthy or generated incrementally over time.

In the following example, the chat_completion function is called with stream=True. The user requests the assistant to count from 1 to 100, adding a new line after every 10 numbers.

chat_completion_result = taskingai.inference.chat_completion(
    model_id=model_id,
    messages=[
        {"role": "user", "content": "Please count from 1 to 20"},   
    ],
    stream=True
)

for item in chat_completion_result:
    if hasattr(item, "delta"):
        # print each message chunk content
        print(item.delta)
    else:
        print(f"Finished with reason: {item.finish_reason}")

We can expect to get an output like this:

1
2
3
...
20
Finished with reason: stop

Since the output is generated in stream mode, the numbers are displayed one by one, not all at once. This simulates a real-time generation of the output.

The streamed response allows users to see each part of the output as soon as it is generated, rather than waiting for the entire completion to be finished.

Function Calls

The chat completion API also supports function calls, a feature enabling the model to create a JSON object with arguments for specific functions based on the prompt's context. You can define these functions in your code and pass them to the API. The model, upon deciding if a function should be called, generates the appropriate JSON for execution, allowing for greater control and interactivity in chat-based applications.

The following example demonstrates how to define a function and pass it to the chat completion API.

Define a Function

function_sum = {
    "name": "sum_a_and_b",
    "description": "Sum up a and b and return the result",
    "parameters": {
        "type": "object",
        "properties": {
            "a": {
                "type": "integer",
                "description": "The first number"
            },
            "b": {
                "type": "integer",
                "description": "The second number"
            }
        },
        "required": ["a", "b"]
    },
}

function_multiply = {
    "name": "multiply_a_and_b",
    "description": "Multiply a and b and return the result",
    "parameters": {
        "type": "object",
        "properties": {
            "a": {
                "type": "integer",
                "description": "The first number"
            },
            "b": {
                "type": "integer",
                "description": "The second number"
            }
        },
        "required": ["a", "b"]
    },
}

This segment defines two functions. One named sum_a_and_b which sums up two integers a and b, and another one named multiply_a_and_b which multiplies two integers a and b. parameters is a dictionary describing the inputs that the function accepts. Both a and b are required and must be integers. You can check this to learn more about the function parameters schema.

Chat Completion with Function Calls

chat_completion_result = taskingai.inference.chat_completion(
    model_id=model_id,
    messages=[
        {"role": "user", "content": "What is the result of 112 plus 22?"},
    ],
    functions=[function_sum, function_multiply]
)
function_calls_assistant_message = chat_completion_result.message
print(function_calls_assistant_message)

The functions argument includes the previously defined sum_a_and_b and multiply_a_and_b function, indicating that those functions may be used during the conversation.

The result assistant message, namely chat_completion_result.message, is an AssistantMessage object. And when its attribute function_calls is not an empty list, it's indicating that the assistant wants to call a function before generating next response.

Get the Function Call Result Locally

def sum_a_and_b(a, b):
    return a + b

def multiply_a_and_b(a, b):
    return a * b

# the function information
func_name = function_calls_assistant_message.function_calls[0].name
arguments = function_calls_assistant_message.function_calls[0].arguments
function_call_id = function_calls_assistant_message.function_calls[0].id

# Use the function name to get the function object and call it with arguments
# The result should be 134
function_calls_result = locals()[func_name](**arguments)
print("The function_calls_result: ", function_calls_result)

We first define a Python function sum_a_and_b to perform the addition operation. Then extract the arguments and get the function call result by executing the function.

Chat Completion with the Function Result

chat_completion_result = taskingai.inference.chat_completion(
    model_id=model_id,
    messages=[
        {"role": "user", "content": "What is the result of 112 plus 22?"},
        function_calls_assistant_message,
        {"role": "function", "content": str(function_calls_result), "id": function_call_id},
    ],
    functions=[function_sum, function_multiply],
)

This part of the code simulates a continuation of the chat, incorporating the result of the function call into the conversation. The user and assistant messages represent previous messages in the chat session, followed by a FunctionMessage which is a special type of message that includes the name of the function and its result.

As a consequence, the expected result is a generated assistant message, describing the output of the executed function call.