Azure OpenAI from Python in November 2024: Keyless Auth, Deployments, Structured Outputs and Failure Handling

On this page

Azure OpenAI Service offers OpenAI’s models inside an Azure subscription, with Azure identity, networking, quotas and regional deployment options. From Python it is reached through the same openai package used for OpenAI’s own API, with an AzureOpenAI client. The differences are small in code and large in operation: models are called through named deployments, every request carries an API version, authentication can and should use Microsoft Entra ID instead of keys, and the deployment type decides where prompts are processed.

This article sets out how to call Azure OpenAI from Python as of November 2024: client setup with keyless authentication, API versions, deployments and deployment types, structured outputs, and handling of throttling, content filtering, refusals and truncation. The sample code is a small, tested client module that does all of these.

Client, API version and authentication

Python app using the AzureOpenAI client with a pinned API version, getting tokens from Entra ID, calling a named deployment on the Azure OpenAI resource; and the standard, data zone, global and provisioned deployment types
Code calls a deployment, not a model. The deployment fixes the model version, the quota and where requests are processed.

Every Azure OpenAI request names an API version. Microsoft’s documentation lists 2024-10-21 as the latest GA release, replacing the previous GA version 2024-06-01, with preview versions alongside it for newer features. Pin a GA version in configuration and move deliberately, because preview versions are retired on a schedule.

For authentication, use Entra ID rather than API keys. azure-identity‘s get_bearer_token_provider wraps a credential such as DefaultAzureCredential and returns a function the client calls for a fresh token, scoped to https://cognitiveservices.azure.com/.default. In Azure the credential resolves to the app’s managed identity; on a developer machine, to the developer’s login. Grant the identity the Cognitive Services OpenAI User role on the resource, and there is no secret to store, rotate or leak.

%%{init: {'theme': 'base', 'themeVariables': {'signalColor': '#0073BB', 'signalTextColor': '#0F172A', 'actorBkg': '#FFF3E0', 'actorBorder': '#FF9900', 'actorTextColor': '#232F3E', 'noteBkgColor': '#FFF8E7', 'noteBorderColor': '#FF9900', 'noteTextColor': '#232F3E', 'lineColor': '#232F3E'}}}%%
sequenceDiagram
    participant App as Python app
    participant Entra as Entra ID
    participant AOAI as Azure OpenAI
    App->>Entra: Token for cognitiveservices scope
    Entra-->>App: Bearer token, about one hour
    App->>AOAI: POST deployments/invoices-gpt4o/chat/completions
    Note over AOAI: Content filter on prompt,<br/>quota check per deployment
    AOAI-->>App: Completion, or 429 with retry-after
    Note over App: Token provider refreshes<br/>before expiry

Deployments and where data is processed

Code passes the deployment name where OpenAI’s API takes a model name. Each deployment fixes a model and version, a tokens-per-minute quota and a deployment type. The type decides where requests are processed: Standard deployments in the resource’s region; Global deployments in any Azure region, with the highest default quota; and since 1 November 2024, Data Zone deployments, which Microsoft made available for both Standard and Provisioned offerings and which keep processing within a Microsoft-defined zone such as the EU or the US. Provisioned deployments reserve throughput in provisioned throughput units for predictable latency.

⚖️
RESIDENCY IS A DEPLOYMENT DECISION

Data stored at rest stays in the resource’s geography, but processing location follows the deployment type. If prompts may contain personal or regulated data that must stay in the EU, use Standard deployments in an EU region or Data Zone EU deployments, not Global, and record the choice in your data processing records. Use separate deployments for workloads with different rules.

Structured outputs

Asking a model for JSON and parsing whatever comes back fails in production. Structured outputs send a JSON schema with the request and constrain the model to it; with the openai SDK the schema can come from a Pydantic model, and client.beta.chat.completions.parse returns a validated object. The GA API version 2024-10-21 supports structured outputs on model versions that offer them, such as GPT-4o 2024-08-06.

The sample builds the client and extracts an invoice into a Pydantic model, returning a clear status for each way the call can go wrong:

Python95 lines
"""Calling Azure OpenAI from Python with keyless auth and structured outputs.

- `make_client`: an `AzureOpenAI` client that authenticates with Microsoft
  Entra ID (managed identity in Azure, your developer login locally) through
  `azure-identity`, pins the GA API version, and sets timeouts and retries.
- `extract_invoice`: structured outputs. The response is validated against a
  Pydantic model; refusals, content-filter blocks and truncated answers are
  reported as distinct outcomes instead of exceptions deep in parsing.

Environment:
    AZURE_OPENAI_ENDPOINT    https://<resource>.openai.azure.com
    AZURE_OPENAI_DEPLOYMENT  the deployment name (not the model name)
    AZURE_OPENAI_API_VERSION optional, defaults to the GA version 2024-10-21
"""
from __future__ import annotations

import os
from dataclasses import dataclass
from typing import Callable

from openai import AzureOpenAI, BadRequestError, ContentFilterFinishReasonError, LengthFinishReasonError
from pydantic import BaseModel, Field

GA_API_VERSION = "2024-10-21"
SCOPE = "https://cognitiveservices.azure.com/.default"


def make_client(endpoint: str | None = None, api_version: str | None = None,
                token_provider: Callable[[], str] | None = None, **kwargs) -> AzureOpenAI:
    """Build a client that uses Entra ID tokens; no API key is read or stored."""
    endpoint = endpoint or os.environ["AZURE_OPENAI_ENDPOINT"]
    if token_provider is None:
        from azure.identity import DefaultAzureCredential, get_bearer_token_provider
        token_provider = get_bearer_token_provider(DefaultAzureCredential(), SCOPE)
    return AzureOpenAI(
        azure_endpoint=endpoint,
        api_version=api_version or os.environ.get("AZURE_OPENAI_API_VERSION", GA_API_VERSION),
        azure_ad_token_provider=token_provider,
        timeout=kwargs.pop("timeout", 60.0),
        max_retries=kwargs.pop("max_retries", 3),      # 429 and 5xx, honouring retry-after
        **kwargs,
    )


class LineItem(BaseModel):
    description: str
    quantity: float
    unit_price: float


class Invoice(BaseModel):
    supplier: str
    invoice_number: str
    currency: str = Field(description="ISO 4217 code, for example EUR")
    total: float
    lines: list[LineItem]


@dataclass
class Outcome:
    status: str                    # "ok", "refused", "filtered", "truncated"
    invoice: Invoice | None = None
    detail: str = ""
    prompt_tokens: int = 0
    completion_tokens: int = 0


SYSTEM = ("Extract the invoice fields from the user's text. Use only values present in the text. "
          "If a value is missing, use an empty string or 0.")


def extract_invoice(client: AzureOpenAI, deployment: str, text: str) -> Outcome:
    try:
        resp = client.beta.chat.completions.parse(
            model=deployment,                          # Azure: the deployment name
            messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}],
            response_format=Invoice,
            temperature=0,
            max_tokens=800,
        )
    except LengthFinishReasonError:
        # The SDK will not parse a cut-off JSON answer
        return Outcome("truncated", detail="raise max_tokens or shorten the input")
    except ContentFilterFinishReasonError:
        return Outcome("filtered", detail="completion blocked by content filtering")
    except BadRequestError as exc:
        body = exc.body if isinstance(exc.body, dict) else {}
        if body.get("code") == "content_filter":
            return Outcome("filtered", detail="prompt blocked by content filtering")
        raise
    choice = resp.choices[0]
    usage = (resp.usage.prompt_tokens, resp.usage.completion_tokens) if resp.usage else (0, 0)
    if choice.message.refusal:
        return Outcome("refused", detail=choice.message.refusal, prompt_tokens=usage[0], completion_tokens=usage[1])
    return Outcome("ok", invoice=choice.message.parsed, prompt_tokens=usage[0], completion_tokens=usage[1])

Failure handling

Throttling, content filtering, refusals and truncated answers compared by signal, SDK behaviour, how the sample reports them and the fix
Four outcomes, four different responses. Treating them all as exceptions hides which one is happening.
  • Throttling. When a deployment exceeds its quota, Azure OpenAI returns HTTP 429 with a retry-after header. The SDK retries with backoff (max_retries, default 2; the sample uses 3). Persistent 429s mean the deployment needs more quota, or load should be spread across deployments or regions.
  • Content filtering. Azure applies content filters to prompts and completions. A blocked prompt returns HTTP 400 with the code content_filter; a blocked completion ends with that finish reason. Log these separately from errors, because the fix is a review of the input or the filter configuration, not a retry.
  • Refusals. With structured outputs, a model that declines returns a refusal message instead of the schema. Route these to a person or a fallback.
  • Truncation. If the answer hits max_tokens, the JSON is incomplete and the SDK refuses to parse it. Raise the limit or send less input.

The tests run the client against a mocked endpoint and check the request itself: the deployment path, the API version, a bearer token and no API key, and a strict schema in response_format.

Python74 lines
import json

import httpx
import pytest
from openai import BadRequestError

import aoai

INVOICE = {"supplier": "Contoso Ltd", "invoice_number": "INV-1042", "currency": "EUR", "total": 240.0,
           "lines": [{"description": "Licences", "quantity": 4, "unit_price": 60.0}]}


def client_with(handler, seen):
    def wrapped(request: httpx.Request) -> httpx.Response:
        seen.append(request)
        return handler(request)
    return aoai.make_client(endpoint="https://contoso.openai.azure.com", token_provider=lambda: "entra-token",
                            max_retries=0, http_client=httpx.Client(transport=httpx.MockTransport(wrapped)))


def completion(content=None, finish="stop", refusal=None):
    return httpx.Response(200, json={
        "id": "c1", "object": "chat.completion", "created": 1731800000, "model": "gpt-4o-2024-08-06",
        "choices": [{"index": 0, "finish_reason": finish,
                     "message": {"role": "assistant", "content": content, "refusal": refusal}}],
        "usage": {"prompt_tokens": 210, "completion_tokens": 64, "total_tokens": 274}})


def test_request_uses_deployment_path_ga_version_bearer_token_and_schema():
    seen = []
    c = client_with(lambda r: completion(json.dumps(INVOICE)), seen)
    out = aoai.extract_invoice(c, "invoices-gpt4o", "Invoice INV-1042 from Contoso Ltd ...")
    req = seen[0]
    assert req.url.path == "/openai/deployments/invoices-gpt4o/chat/completions"
    assert req.url.params["api-version"] == "2024-10-21"
    assert req.headers["authorization"] == "Bearer entra-token"
    assert "api-key" not in req.headers
    body = json.loads(req.content)
    assert body["response_format"]["type"] == "json_schema"
    assert body["response_format"]["json_schema"]["strict"] is True
    assert out.status == "ok" and out.invoice.total == 240.0 and out.invoice.lines[0].quantity == 4
    assert (out.prompt_tokens, out.completion_tokens) == (210, 64)


def test_refusal_is_reported():
    out = aoai.extract_invoice(client_with(lambda r: completion(None, refusal="I can't help with that."), []),
                               "d", "text")
    assert out.status == "refused" and "can't" in out.detail


def test_completion_content_filter_is_reported():
    out = aoai.extract_invoice(client_with(lambda r: completion(None, finish="content_filter"), []), "d", "text")
    assert out.status == "filtered"


def test_prompt_content_filter_400_is_reported():
    err = {"error": {"message": "The response was filtered", "type": None, "param": "prompt",
                     "code": "content_filter", "status": 400}}
    out = aoai.extract_invoice(client_with(lambda r: httpx.Response(400, json=err), []), "d", "text")
    assert out.status == "filtered"


def test_other_400_errors_are_raised():
    err = {"error": {"message": "bad", "type": "invalid_request_error", "code": "invalid_value"}}
    with pytest.raises(BadRequestError):
        aoai.extract_invoice(client_with(lambda r: httpx.Response(400, json=err), []), "d", "text")


def test_truncated_answer_is_reported_without_parsing():
    seen = []
    def handler(r):
        return completion('{"supplier": "Cont', finish="length")
    out = aoai.extract_invoice(client_with(handler, seen), "d", "text")
    assert out.status == "truncated"
💡
DISABLE KEYS ONCE TOKENS WORK

After every caller uses Entra ID, turn off key-based authentication on the resource (the disableLocalAuth property) where your policy allows. It removes the risk of a leaked key and makes the Azure activity and sign-in logs the single record of who called the service.

⚠️
PIN MODEL VERSIONS IN DEPLOYMENTS

A deployment can follow a model’s default version and be upgraded automatically. For workloads validated against a specific behaviour, pin the model version on the deployment and plan upgrades before the version’s retirement date, re-running your evaluation first.

📝
HOW THIS WAS CHECKED

aoai.py was tested with Python 3.12, openai==1.54.4, azure-identity==1.19.0, pydantic==2.9.2, httpx==0.27.2 and pytest==8.3.3: six tests against a mocked endpoint cover the request path, API version, bearer token, strict schema, parsing, refusals, content filtering on prompt and completion, other errors and truncation. No Azure resource was called. API versions and Data Zone availability are from Microsoft’s documentation and announcement.

Key takeaways

  • Use the openai package’s AzureOpenAI client, pin the GA API version (2024-10-21 as of November 2024) and set timeouts and retries.
  • Authenticate with Entra ID through a token provider and managed identity; then disable key authentication.
  • Call deployments, not models; the deployment type (Standard, Data Zone, Global, Provisioned) decides where prompts are processed.
  • Use structured outputs with a Pydantic model instead of parsing free-form JSON.
  • Handle throttling, content filtering, refusals and truncation as distinct outcomes.

How are you choosing between Standard, Data Zone and Global deployments for your workloads? Share your reasoning in the comments.

Glossary

Deployment
A named instance of a model version in an Azure OpenAI resource, with its own quota and type.
API version
The dated version of the Azure OpenAI REST API sent with every request.
Entra ID
Microsoft’s identity service, formerly Azure Active Directory.
Managed identity
An Azure-managed identity for a workload, used to get tokens without secrets.
Data Zone deployment
A deployment type that processes requests within a Microsoft-defined zone such as the EU.
Structured outputs
Constraining a model’s response to a supplied JSON schema.
PTU
Provisioned throughput unit: reserved model processing capacity.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! 👋🏻 I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud 📌 Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.