asyncopenai-concurrency-httpx-pool

# AsyncOpenAI Concurrency: the Hidden httpx Pool Cap ## Problem Async batch scorers typically gate concurrency with `asyncio.Semaphore(N)`. Raising N above ~100 silently does nothing: the OpenAI SDK's default httpx transport caps the connection pool at `max_connections=100`, so excess tasks queue inside httpx instead of reaching the provider. The semaphore looks like the throttle but is not the binding constraint — there is no error, just a throughput ceiling. ## Context / Trigger Conditions - `asyncio.Semaphore(N)` with N > 100 around `client.chat.completions.create` shows the same throughput as N = 100 - Client constructed as `AsyncOpenAI(api_key=..., base_url=...)` with no `http_client` argument (the default transport) - Provider is known to allow high concurrency (DeepSeek v4-flash: ~2500) - Symptom check: requests-in-flight measured at the server never exceeds ~100 ## Solution Size the httpx pool to the semaphore when constructing the client: ```python import httpx from openai import AsyncOpenAI CONCURRENCY = 2000 client = AsyncOpenAI( api_key=..., base_url="https://api.deepseek.com", http_client=httpx.AsyncClient(limits=httpx.Limits( max_connections=CONCURRENCY, max_keepalive_connections=CONCURRENCY))) sem = asyncio.Semaphore(CONCURRENCY) ``` Both edits are required; either alone caps the other. Keep the per-request retry loop — at high concurrency transient failures are more likely, and the retry envelope is what turns them into non...