Send requests to vLLM workers

This guide covers different methods for sending requests to vLLM workers on Runpod, including code examples and best practices for Runpod’s native API format. Use this guide to effectively integrate LLMs into your applications while maintaining control over performance and cost.

Requirements

• You’ve created a Runpod account.
• You’ve created a Runpod API key.
• You’ve installed Python.
• (For gated models) You’ve created a Hugging Face access token.

Many of the code samples below will require you to input your endpoint ID. You can find your endpoint ID on the endpoint details page:

Runpod API requests

Runpod’s native API provides additional flexibility and control over your requests. These requests follow Runpod’s standard endpoint operations format.

Python Example

Replace [RUNPOD_API_KEY] with your Runpod API key.

import requests

url = "https://api.runpod.ai/v2/<endpoint_id>/run"
headers = {"Authorization": "Bearer [RUNPOD_API_KEY]", "Content-Type": "application/json"}

data = {
    "input": {
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Write a short poem."}
        ],
        "sampling_params": {"temperature": 0.7, "max_tokens": 100}
    }
}

response = requests.post(url, headers=headers, json=data)
print(response.json())

cURL Example

Run the following command in your local terminal, replacing [RUNPOD_API_KEY] with your Runpod API key and [RUNPOD_ENDPOINT_ID] with your vLLM endpoint ID.

curl -X POST "https://api.runpod.ai/v2/[RUNPOD_ENDPOINT_ID]/run" \
     -H "Authorization: Bearer [RUNPOD_API_KEY]" \
     -H "Content-Type: application/json" \
     -d '{
           "input": {
             "prompt": "Write a haiku about nature.",
             "sampling_params": {"temperature": 0.8, "max_tokens": 50}
           }
         }'

Request formats

vLLM workers accept two primary input formats:

Messages format (for chat models)

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me about the solar system."}
  ]
}

Prompt format (for text completion)

{
  "prompt": "Summarize the following text: Climate change is a global challenge that affects..."
}

Request input parameters

vLLM workers support various parameters to control generation behavior. Here are some commonly used parameters:

Parameter	Type	Description
temperature	float	Controls randomness (0.0-1.0)
max_tokens	int	Maximum number of tokens to generate
top_p	float	Nucleus sampling parameter (0.0-1.0)
top_k	int	Limits consideration to top k tokens
stop	string or array	Sequence(s) at which to stop generation
repetition_penalty	float	Penalizes repetition (1.0 = no penalty)
presence_penalty	float	Penalizes new tokens already in text
frequency_penalty	float	Penalizes token frequency
min_p	float	Minimum probability threshold relative to most likely token
best_of	int	Number of completions to generate server-side
use_beam_search	boolean	Whether to use beam search instead of sampling

You can find a complete list of request input parameters on the GitHub README.

Error handling

When working with vLLM workers, it’s crucial to implement proper error handling to address potential issues such as network timeouts, rate limiting, worker initialization delays, and model loading errors.

Here is an example error handling implementation:

import requests
import time
import backoff  # pip install backoff

@backoff.on_exception(backoff.expo, requests.exceptions.RequestException, max_tries=5)
def send_request(url, headers, payload):
    response = requests.post(url, headers=headers, json=payload)
    response.raise_for_status()  # Raises an exception for 4XX/5XX responses
    return response.json()

try:
    result = send_request(url, headers, payload)
    print(f"Success: {result}")
except requests.exceptions.HTTPError as e:
    if e.response.status_code == 429:
        print("Rate limit exceeded. Try again later.")
    elif e.response.status_code == 500:
        print("Server error. The model may be having trouble loading.")
    else:
        print(f"HTTP error: {e}")
except requests.exceptions.ConnectionError:
    print("Connection error. Check your network and endpoint ID.")
except requests.exceptions.Timeout:
    print("Request timed out. The model may be processing a large batch.")
except Exception as e:
    print(f"An unexpected error occurred: {e}")

Best practices

Here are some best practices to keep in mind when creating your requests:

1. Use appropriate timeouts: Set timeouts based on your model size and complexity.
2. Implement retry logic: Add exponential backoff for failed requests.
3. Optimize batch size: Adjust request frequency based on model inference speed.
4. Monitor response times: Track performance to identify optimization opportunities.
5. Use streaming for long responses: Improve user experience for lengthy content generation.
6. Cache frequent requests: Reduce redundant API calls for common queries.
7. Handle rate limits: Implement queuing for high-volume applications.

Next steps

• Send requests using the OpenAI-compatible API.
• Learn how to use Serverless endpoint operations.