Azure OpenAI and Vertex AI Behind One Interface: Where the Abstraction Leaks

Outcome model for a two-provider chat port mapped from Azure OpenAI and Vertex AI signals
On this page

Most teams that run on two clouds end up wanting one interface for chat completion: a message list goes in, text or a tool call or a JSON object comes out, and Azure OpenAI and Gemini on Vertex AI become configuration. The interface is easy to write. What it hides is the problem, because the two services agree on the shape of a request and disagree on what happens at the edges: which schemas are honoured, how a blocked request is reported, what a 429 means, and where a setting lives.

This article goes through those edges, from the providers’ own documentation, and proposes a design for an abstraction that admits them instead of hiding them. It is design guidance. Nothing in it was run against either cloud.

ℹ️
TEST SCOPE

This is a design article. Every statement about Azure OpenAI or Vertex AI comes from the documentation pages listed under References, as published when they were read. No request was sent to either service, no latency or cost was measured, and the design has not been built or tested. Treat it as a checklist to verify against your own deployments.

The problem

A provider abstraction is usually judged on the happy path: same prompt, similar answer, one code path. The cost shows up elsewhere. A JSON schema that is enforced on one side and partly ignored on the other produces records that pass on Azure and fail validation on Vertex, or the reverse. A safety block that arrives as an exception from one provider and as a normal response with an empty body from the other ends up retried by one code path and stored as an empty answer by the other. A retry policy tuned for one provider’s throttling misbehaves on the other’s.

None of this is a defect in either service. They are different products with different contracts. The defect is an interface that promises they are the same.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
    APP["Application code"] --> PORT["Chat port:
messages, tools, schema"] PORT --> AZ["Azure OpenAI adapter"] PORT --> VX["Vertex AI adapter"] AZ --> L1["Schema dialect"] VX --> L1 AZ --> L2["Safety outcome"] VX --> L2 AZ --> L3["Throttling meaning"] VX --> L3 AZ --> L4["Where settings live"] VX --> L4 L1 --> RISK["Same call, different guarantees"] L2 --> RISK L3 --> RISK L4 --> RISK style PORT fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74 style RISK fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B style L1 fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95 style L2 fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95 style L3 fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95 style L4 fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95

Structured output

Both services can constrain a response to a JSON schema. Both accept a subset of JSON Schema, and the subsets point in different directions.

AspectAzure OpenAIVertex AI (Gemini)
How it is requestedresponse_format with a JSON schema in Chat Completions; text.format in ResponsesresponseMimeType: application/json plus responseSchema
Required fieldsEvery property must be listed in required; optional fields are written as a union with nullFields are optional by default; the model may skip a field not marked required
Extra propertiesadditionalProperties: false on every objectNot a listed field
String constraintsminLength, maxLength, pattern, format not supportedformat supported for date, date-time, duration and time
Number and array constraintsminimum, maximum, minItems, maxItems and others not supportedminimum, maximum, minItems, maxItems supported
Unsupported keywordsListed as unsupportedIgnored; the request still succeeds
Size limitsUp to 100 object properties and five levels of nestingNo fixed numbers; a complex schema can fail with HTTP 400 InvalidArgument
Key orderFollows the order in the schemaNot strictly followed; propertyOrdering sets it
Interaction with toolsNot supported with parallel function calls; set parallel_tool_calls to falseNot stated on the page read
Token accountingNot stated on the page readThe schema counts toward the input token limit

Three of those rows cause most of the trouble.

  • Optionality is inverted. On Azure, a field the model may leave out has to be written as required and nullable. On Vertex, a field you need has to be marked required or it may not appear. A schema written for one and sent unchanged to the other changes meaning, not just syntax.
  • Constraints are enforced on one side only. A maxItems or a minimum shapes the output on Vertex and is not supported on Azure. A pattern is supported on neither. Any constraint that matters has to be checked after the response arrives, whichever provider produced it.
  • Failure is silent on one side and loud on the other. Vertex ignores fields it does not support, so a request succeeds with a weaker guarantee than the schema suggests. Azure documents the same keywords as unsupported. A schema that grows over time can cross Vertex’s complexity limit and start returning 400 for a request that worked last month.

Refusals are a further case. Azure’s documentation shows a separate refusal field on the message in Chat Completions, and a content item of type refusal in Responses, to be checked before the parsed object is read. A port that returns “the parsed object” with no room for “the model declined” loses that case.

Content safety

Both providers screen prompts and responses for harmful content. They differ in what they screen for, where the thresholds are set, and above all in how a block reaches the caller.

AspectAzure OpenAIVertex AI (Gemini)
Configurable categoriesHate and fairness, sexual, violence, self-harmHate speech, harassment, sexually explicit, dangerous content
LevelsSeverity: safe, low, medium, highProbability and severity, each discretized to negligible, low, medium, high; severity method is the default
Where thresholds are setA content filter configuration attached to the deploymentsafetySettings on each request
Turning filtering offOnly for customers approved through a review form; annotate-only also needs approvalBLOCK_NONE and OFF thresholds in the request
Filters that cannot be configuredNot covered on the page readFilters for child sexual abuse material and sensitive personal data
Blocked promptHTTP 400, error code content_filter, param promptA response with promptFeedback.blockReason set
Blocked responseHTTP 200, finish_reason: content_filter, content withheldfinishReason of SAFETY, RECITATION, SPII or PROHIBITED_CONTENT; candidate content cleared
Filtering unavailableHTTP 200, unfiltered, with a content_filter_error inside content_filter_resultsNot covered on the page read

The last three rows are the abstraction leak. A blocked prompt on Azure is an error response, and most HTTP clients and SDKs turn it into an exception. A blocked prompt on Vertex is a successful response with a field set. A blocked completion is a normal response on both, with different field names and different values. If the port maps only exceptions to errors, an Azure prompt block becomes a failure that a retry policy will retry, and a Vertex prompt block becomes an empty answer.

The Azure case of filtering being unavailable deserves its own line in any design. The documentation says the request completes without filtering and reports it inside the response. A port that drops content_filter_results cannot tell a screened answer from an unscreened one.

⚠️
CLINICAL TEXT

Both sets of categories were written for general content. Clinical notes discuss self-harm, violence and the body as a matter of course, and a filter that is right for a consumer chat can block a discharge summary. Whether a block is a safety outcome or a false positive is a clinical governance question, and the abstraction should make every block visible and countable rather than turn it into a generic error.

Throttling and capacity

Both services answer HTTP 429 when they cannot take a request. The status code is the same and the meaning is not.

AspectAzure OpenAI (standard deployments)Vertex AI (Gemini, pay as you go)
Unit of capacityTokens per minute assigned to a deployment, from a per-region, per-model subscription quotaShared capacity; no reserved throughput
Request limitRequests per minute set in proportion to tokens per minute, evaluated over windows of typically 1 or 10 secondsNot a separate fixed limit on the page read
What a 429 meansThe deployment’s limit was reached, or the system is under high demandTemporary contention for a shared resource, not a fixed quota
What the response carriesx-ratelimit-* headers on every response; retry-after-ms on a 429Not stated on the page read
Documented remediesHonour retry-after-ms with jitter; more quota; provisioned deploymentsExponential backoff; the global endpoint; smoother traffic; Provisioned Throughput
Counted toward the limitActual input and output tokens; failed requests count tooNot stated on the page read

A single retry policy over both is either too eager for one or too patient for the other. Azure tells the caller how long to wait, and a caller that ignores retry-after-ms and retries on its own schedule spends requests that count against the same limit. Vertex gives no fixed limit to plan against, and the documented answer to repeated 429s is a different endpoint or a different purchasing model, not a longer wait.

Retries also interact with agents. A model call retried after a partial stream, or after a tool call it requested has already run, is the duplicate-effect problem from the crash and resume article in another form. The retry belongs below the agent loop, on whole model calls, never on a step whose tools have acted.

Endpoints and authentication

  • Model identity. On Azure the request names a deployment, which carries the model, its version, its quota and its content filter. On Vertex the request names a model in a project and a location. The same logical “model” in configuration maps to an operational object on one side and a catalogue entry on the other.
  • Credentials. Vertex’s OpenAI-compatible endpoint takes a Google Cloud access token as the API key, and its documentation notes that access tokens live for one hour by default. A client built once at start-up with a token as its key fails an hour later unless the token is refreshed.
  • Location. Azure quota is per region and per model. Vertex recommends the global endpoint for capacity. Where data residency matters, which endpoint a request is allowed to use is a policy decision that the abstraction must not make implicitly.

Vertex also offers a Chat Completions endpoint that the OpenAI libraries can call, under /projects/{project}/locations/{location}/endpoints/openapi. It is tempting to treat that as the abstraction. It removes the wire-format difference and keeps every other difference in this article: Gemini settings still have to travel in extra_body, the safety outcomes are still Gemini’s, and the throttling is still Vertex’s. It is a reasonable adapter. It is not a reason to skip the design below.

Proposed design

The aim is not to make the providers identical. It is to make the differences explicit in one place, so that application code handles each outcome once and the operations team can see which provider produced it.

Outcome model for a two-provider chat port: completed, refused, blocked on input, blocked on output, unscreened, truncated, throttled and invalid request, each mapped from Azure OpenAI and Vertex AI signals
One outcome type, mapped from each provider’s signals.

An outcome type, not an exception. Every call returns one of a closed set of outcomes. Exceptions are kept for transport failures and bugs.

OutcomeAzure OpenAI signalVertex AI signalTypical handling
Completedfinish_reason stop or tool callsfinishReason STOPValidate, then use
Truncatedfinish_reason lengthfinishReason MAX_TOKENSDo not parse as complete JSON
Refusedrefusal on the message or a refusal output itemNot documented as a separate signal on the pages readShow or route; do not retry unchanged
Blocked on inputHTTP 400 with code content_filterpromptFeedback.blockReasonCount, review, never retry unchanged
Blocked on outputfinish_reason content_filterfinishReason SAFETY, RECITATION, SPII or PROHIBITED_CONTENTCount by reason, review
Unscreenedcontent_filter_error in the resultsNot documentedDecide by policy: accept, hold or reject
ThrottledHTTP 429 with retry-after-msHTTP 429Provider-specific retry, then fail over or shed
Invalid requestOther HTTP 400HTTP 400, including schema complexityFix the request; alert

A capability profile per adapter. Each adapter declares what it can do: which schema keywords it enforces, whether structured output and parallel tool calls can be combined, where safety thresholds are set, and whether the outcome set above is fully reported. Application code that needs a guarantee asks for it, and the port refuses a request it cannot honour rather than quietly sending a weaker one.

A schema compiler, not a schema. The application owns one schema in full JSON Schema. Each adapter compiles it to its provider’s dialect and reports what was lost: constraints removed for Azure, fields Vertex will ignore, optional fields rewritten as nullable. The full schema is then validated against every response. The model’s constrained output is a strong hint; the validator is the contract.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
    S["Application schema
full JSON Schema"] --> CA["Compile for Azure:
all required, nullable optionals,
drop unsupported keywords"] S --> CV["Compile for Vertex:
mark required fields,
keep supported constraints"] CA --> RA["Report: what was dropped"] CV --> RV["Report: what will be ignored"] RA --> CALL["Provider call"] RV --> CALL CALL --> VAL{"Validate the response
against the full schema"} VAL -- "valid" --> USE["Use the result"] VAL -- "invalid" --> REJ["Outcome: invalid output,
logged with provider"] style S fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74 style VAL fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95 style USE fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202 style REJ fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B

Safety settings as named policies. Configuration names a policy, such as “standard” or “clinical documentation”. The Azure adapter maps it to the content filter configuration that must be attached to the deployment, and checks at start-up that it is. The Vertex adapter maps it to safetySettings sent on every request. The policy name goes into the log next to every block.

Retry inside the adapter. The Azure adapter honours retry-after-ms with jitter. The Vertex adapter uses exponential backoff. Neither retries a block, a refusal or an invalid request. Fail-over to the other provider happens above the adapters, only for calls whose schema and policy the other adapter can honour.

Where each leak is handled: schema dialect in the compiler, safety signals in the outcome mapping, throttling in the adapter retry policy, settings location in named policies, credentials and location in adapter configuration
Each difference has one owner in the design.

Testing the design

Because this article ran nothing, the design comes with the tests that would have to pass before relying on it. Most can run without either cloud.

  • Compiler tests. For each schema in use, the compiled form for each provider, and the report of what was lost, checked in as a snapshot. A change to the application schema that makes a provider drop a constraint shows up in review.
  • Outcome mapping tests. Recorded or hand-written responses for each row of the outcome table, per provider, run through the adapter. These need no network.
  • Contract tests against each cloud. A small scheduled suite that sends a prompt each provider should block, a schema near Vertex’s complexity limit, and a burst that should be throttled, and checks the adapter reports the right outcome. This is the only part that needs credentials, and the only part that notices when a provider changes behaviour.
  • Counters per outcome and provider. In production, a rising count of “blocked on output” on one provider after a model version change is the signal that matters, and it is invisible if blocks are folded into errors.

Cost

No costs were measured. Two structural points follow from the documentation. On Vertex the response schema counts toward input tokens, so a large schema is paid for on every call. On Azure, failed requests count toward the per-minute limit, so a retry policy that ignores retry-after-ms reduces the capacity it is trying to wait for. The abstraction itself adds a validation pass per response, which is small next to a model call.

Production considerations

  • Log the provider, the deployment or model, the policy name and the outcome on every call. Without them, a difference in behaviour between clouds cannot be told from a difference in input.
  • Treat “unscreened” as a decision. For regulated content, holding an Azure response whose filter did not run may be the right default.
  • Do not fail over on blocks. Sending a prompt that one provider blocked to the other provider until something answers is a policy bypass, whatever the intent.
  • Pin model versions and rerun contract tests on change. Schema support, thresholds and defaults are properties of models as much as of services.
  • Keep data residency out of fail-over logic. Which regions and endpoints a workload may use is fixed by policy, and the router only chooses among those.

Decision framework

Your situationApproachWhy
One provider in production, the other a possibilitySingle adapter, outcome type and schema validation now; second adapter laterThe outcome type and validator pay off even with one provider
Both providers serve the same featureFull design: compiler, outcome mapping, named policies, per-adapter retryEvery leak in this article reaches users
Fail-over for availabilityOnly between adapters whose capability profiles both satisfy the callA fail-over to a weaker guarantee is a silent downgrade
Structured output with strict constraintsValidate every response against the full schema, on both providersNeither dialect enforces everything
Clinical or other sensitive textNamed safety policies, every block counted and reviewableFilters tuned for general content will block legitimate text
Considering the OpenAI-compatible Vertex endpointUse it as the Vertex adapter’s transport if convenientIt unifies the wire format, not the behaviour

Key takeaways

  • Azure OpenAI and Vertex AI agree on the shape of a chat request and differ on schema support, safety reporting, throttling and where settings live.
  • Structured output optionality is inverted between them, and each enforces constraints the other does not. Validate every response against the full schema.
  • A blocked prompt is an HTTP 400 on Azure and a successful response with a field set on Vertex. Model outcomes as a closed type, not as exceptions.
  • A 429 means a deployment limit on Azure and shared contention on Vertex. Retry inside each adapter, by that provider’s rules.
  • The abstraction is worth having when it names the differences. One that hides them moves the failures into production.

If you run the same workload on both clouds and have found a difference this article does not cover, leave a comment.

πŸ“
HOW THIS WAS CHECKED

This is a design article and nothing in it was run. The statements about each service were taken from the provider documentation pages listed below: Azure OpenAI structured outputs, content filtering, and quota and rate limits; Vertex AI structured output, safety and content filters, OpenAI compatibility, and standard pay-as-you-go capacity. Where a page did not cover a point, the tables say so instead of filling the gap. No request was sent to Azure OpenAI or Vertex AI, no sample was built, and no latency, cost or block rate was measured. Provider behaviour changes with model versions and service updates; check each point against the current documentation and your own deployments before relying on it.

Glossary

  • Abstraction leakβ€” A difference between implementations that callers of a shared interface still have to know about.
  • Adapterβ€” The component that translates the shared interface to one provider’s API.
  • Capability profileβ€” A declaration of what an adapter can guarantee, checked before a call is sent.
  • Deploymentβ€” In Azure OpenAI, a named instance of a model with its own quota and content filter.
  • Structured outputβ€” A response constrained to a supplied JSON schema.
  • Content filterβ€” Azure’s classification and blocking of prompts and responses by harm category and severity.
  • Safety settingsβ€” Vertex AI’s per-request thresholds for blocking by harm category.
  • Refusalβ€” A model’s explicit decision not to answer, reported separately from the content.
  • TPM and RPMβ€” Tokens per minute and requests per minute, the units of Azure OpenAI quota.
  • Provisioned throughputβ€” Capacity reserved in advance, offered by both providers, as opposed to shared pay-as-you-go capacity.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! πŸ‘‹πŸ» I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud πŸ“Œ Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.