Other Settings and Configs

Besides the fields mentioned above, there are other settings and configs that you can pass to the model (if they accepts such configuration).

Stream

stream is a boolean value indicating whether the model should return the results immediately chunk by chunk, or wait until the model finishes the completion job and return the result as a whole.

Configs

configs is a dictionary containing other configurations that you want to pass to the model. Since different model has different configurations, TaskingAI-Inference only supports five common configurations: temperature, top_p, top_k, max_tokens and stop. These configurations are supported by most of the models, but not all of them. And they will only take effect if the model supports them.