OpenAI compatibility

scx inference APIs are designed to be compliant with OpenAI client libraries to simplify the adoption of our inference technologies to enhance your AI applications.

Download the library

Run the command below to download the library.

1pip install openai
2

1pip install openai
2

Use scx APIs with OpenAI client libraries

Configuring your OpenAI client libraries to use scx inference APIs is as simple as setting two values: the base_url and your api_key, as shown below.

Note

Don’t have a scx API key? Get yours from the API keys and URLs page.

1from openai import OpenAI
2
3client = OpenAI(
4    base_url="https://api.scx.ai/v1", 
5    api_key="your-scx-api-key"
6)
7

1from openai import OpenAI
2
3client = OpenAI(
4    base_url="https://api.scx.ai/v1", 
5    api_key="your-scx-api-key"
6)
7

Now you can make an API request to a model and choose how to receive your output.

Non-streaming example

The following code demonstrates using the OpenAI Python client for non-streaming completions.

1completion = client.chat.completions.create(
2  model="Meta-Llama-3.1-8B-Instruct",
3  messages = [
4      {"role": "system", "content": "Answer the question in a couple sentences."},
5      {"role": "user", "content": "Share a happy story with me"}
6    ]
7)
8
9print(completion.choices[0].message)
10

1completion = client.chat.completions.create(
2  model="Meta-Llama-3.1-8B-Instruct",
3  messages = [
4      {"role": "system", "content": "Answer the question in a couple sentences."},
5      {"role": "user", "content": "Share a happy story with me"}
6    ]
7)
8
9print(completion.choices[0].message)
10

Streaming example

The following code demonstrates using the OpenAI Python client for streaming completions.

1completion = client.chat.completions.create(
2  model="Meta-Llama-3.1-8B-Instruct",
3  messages = [
4      {"role": "system", "content": "Answer the question in a couple sentences."},
5      {"role": "user", "content": "Share a happy story with me"}
6    ],
7  stream= True
8)
9
10for chunk in completion:
11  print(chunk.choices[0].delta.content)
12

1completion = client.chat.completions.create(
2  model="Meta-Llama-3.1-8B-Instruct",
3  messages = [
4      {"role": "system", "content": "Answer the question in a couple sentences."},
5      {"role": "user", "content": "Share a happy story with me"}
6    ],
7  stream= True
8)
9
10for chunk in completion:
11  print(chunk.choices[0].delta.content)
12

Note

In streaming mode, the API returns chunks that contain multiple tokens. When calculating metrics like tokens per second or time per output token, ensure that you account for all tokens in each chunk.

Currently unsupported OpenAI features

The following features are not yet supported and will be ignored:

logprobs
top_logprobs
n
presence_penalty
frequency_penalty
logit_bias
seed

Feature differences

temperature: The scx API supports a value between 0 and 1 whereas OpenAI supports values between 0 and 2.

scx API features not supported by OpenAI clients

The scx API supports the top_k parameter, which is not supported by the OpenAI client libraries.