With OpenAI’s batch API, you can process large volumes of data asynchronously. As a result, you’ll get lower costs, higher rate limits, and a guaranteed 24-hour response time. Velvet's proxy unlocks additional data so you can see exactly what's happening inside your batch job.
Batching is useful when you’re automating an asynchronous task. It makes sense to use this API for features that don’t require an immediate response from the API. OpenAI’s’ Batch API enables you to package requests into a single file, then retrieve the collected results after the batch job is complete.
Benefits of OpenAI’s Batch API:
Read OpenAI’s batch API docs to learn more about implementation and how it works. Velvet logs every request inside the file, so you'll have full observability.
The Batch API works similarly to OpenAI’s synchronous APIs, but you pass through a list of strings instead of just one string. It’s useful for tasks that can run in the background - and you’ll reduce costs in the process.
Use batching to automate non-immediate tasks, like these.
One of our customers, Find AI, uses the Batch API to automate many tasks outside of immediate customer queries.
“OpenAI batches save lots of money, but observability is a challenge. Their APIs say that results come back within 24 hours, but we had no idea until using Velvet that our average response time was more like 3 hours. Batch prompts are difficult to debug, because it's a completely different flow than Chat. With Velvet, we can pull exact prompts that we used for batches, which makes it easier to debug and tweak.” - Find AI CTO
Velvet supports logging for the OpenAI batch API. When you send a batch request, we deconstruct the file sent in each request and response. As a result, you’ll have queryable data on costs, errors, and performance.
Read our batch jobs documentation to learn more.
Continuously test AI features in production, set alerts to take action.
Test models, settings, and metrics against historical request logs.
Use our data copilot to query your AI request logs with SQL.