LLM gateway vs MCP gateway

An LLM gateway routes requests to language models, while an MCP gateway routes requests to external tools. The two sit on different paths in an agent architecture and can be used together. Cloudflare AI Gateway is an example of the model-request layer; AIPx supplies connected-account tools and does not include model inference.

What is an LLM gateway?

An inference gateway sits between an application and model providers. Its useful capabilities can include request routing, usage visibility, caching and rate controls. The application still owns its task logic, and the model provider still supplies the inference service.

For example, your application can send a generation request through a gateway that records its latency and provider outcome. If the application later executes a tool, that separate service call may never pass through the inference gateway. The model’s suggestion to call a tool is part of the model response; the actual tool execution belongs to another path.

Cloudflare describes its AI Gateway as a layer for managing and observing AI requests, with features including analytics, caching and rate limiting. Its official overview describes the supported providers and features. Do not infer a gateway’s exact capabilities from the generic category name.

An LLM gateway is not necessarily the agent host. It need not decide when a task is complete, hold every account credential or execute business operations. Its visibility is strongest over the requests actually routed through it.

What is an MCP gateway?

An MCP gateway presents external tools through a shared MCP endpoint and can add connection, authentication and access management. It sits between the host’s MCP client and the upstream systems. A model provider does not have to know the details of every service connection behind it.

AIPx connects supported accounts once and exposes permitted tools through agent-specific URLs. A coding host can use those tools for tracker context or production error evidence, then make its model calls through a separate provider or inference gateway.

An MCP connection does not include a model subscription. The gateway can deliver a tool result without choosing which model generates the next response. The MCP gateway guide explains account and tool views, and what an MCP server is explains the capability provider.

LLM gateway vs MCP gateway: feature table

The table compares responsibility boundaries, not a promise that every product in each category has every feature.

Question	LLM gateway path	MCP gateway path
What request is routed?	A model inference request	A server discovery or tool request
Which downstream service is called?	A model provider	A connected tool server or app API
What credential is relevant?	Provider or gateway inference authorization	Client authorization and upstream account access
What result returns?	Generated output or model events	Tool capabilities or operation results
Where is token usage visible?	When measured on routed model calls	Not automatically visible from tools alone
Where is account selection visible?	Not necessarily part of the inference path	In the tool connection and operation context
What does access policy govern?	Model request and provider usage	Reachable tools, accounts and operations
Does it run the whole agent?	Not inherently	Not inherently

A model gateway can offer additional policy features, and an MCP platform can offer broader orchestration. Evaluate each documented implementation rather than turning this table into a yes/no vendor ranking.

The distinction becomes useful when troubleshooting. A successful model request followed by a failed Jira call is not necessarily an inference problem. A tool call returning quickly does not prove the model generation is fast. Measure the stages separately before changing providers or permissions.

Cloudflare AI Gateway example

Suppose an application handles issue triage. It sends the user’s task and available tool definitions to a model. The model proposes reading a ticket. The application validates that proposal and calls the relevant MCP endpoint, then includes the returned ticket data in another model request.

The model requests can pass through Cloudflare AI Gateway. The tracker calls can pass through AIPx. The host application links the task, request and tool operation; neither gateway automatically possesses the other’s full context.

Stage	Responsible component	Evidence to inspect
Receive the task	Host application	The authorized user request
Send generation request	Model client through inference path	Provider outcome and latency
Propose tool arguments	Model response handled by host	Chosen operation and structured inputs
Validate and execute	Host plus tool gateway/provider	Account access and operation result
Continue the task	Host makes another model request	Whether the result satisfies the request

A model response can propose an unauthorized action even when inference succeeds. A tool can return irrelevant content even when its HTTP call succeeds. The host must preserve task authority and evaluate outcomes; forwarding traffic alone does not establish correct agent behavior.

The MCP architecture article shows the host, client and server boundaries. The MCP introduction explains why a common tool protocol can sit alongside any model provider.

Cloudflare integration is a separate concept

Connecting Cloudflare tools through AIPx gives an agent access to supported Cloudflare account operations. That is not the same action as configuring an application to send inference through Cloudflare AI Gateway.

One is a tool connection for account context or administration. The other is a path for model requests. The shared vendor name does not make their credentials, permissions or billing identical. Review the exact product and endpoint before following a setup example.

Cloudflare AI Gateway pricing

Cloudflare’s core AI Gateway capabilities, including dashboard analytics, caching and rate limiting, are available free. Model inference remains separately billable by the selected provider or applicable service. Additional features and log storage have their own pricing rules.

A material date boundary applies to logs. Customers creating their first gateway on or after September 24, 2026 follow Workers Logs pricing and retention; earlier customers use the documented legacy log rules. Do not apply an old universal free-log quota to a new account. Check the current pricing documentation for your account and features.

AIPx pricing covers another layer: Free includes three connected accounts and one agent, while Pro costs $3 per month for unlimited accounts and agents plus Pro integrations. It does not pay for a model subscription or token usage, and upstream API restrictions remain in force.

Cost layer	Example owner	Question to verify
Model inference	Selected model provider	What generation usage is billed?
Inference gateway extras	Gateway operator	Which logs or optional features incur charges?
Tool gateway	AIPx or another integration operator	What accounts, agents or calls are included?
Connected service	Tracker, database or app provider	Does API access require a plan or usage payment?
Application hosting	Your runtime	What execution and storage costs remain?

This breakdown prevents a common budgeting mistake: calling the system free because one forwarding layer has a free core. The task can still incur inference, connected-app and application costs.

When an agent needs both gateways

Both can be useful when you need model-request visibility and a shared account-tool boundary. A host you operate can route inference through one layer and connect its MCP client through another. Use correlation identifiers in your own orchestration when you need to connect the stages.

You may need only the tool layer if an existing client already handles model access and you simply want it to use your accounts. A developer using a coding application with its own subscription should not assume they can or need to reconfigure every model call through an inference gateway.

You may need only the inference layer if your application has a few direct API handlers and no need to share tools across MCP hosts. Keeping those handlers deliberate and well validated can be simpler than adding another protocol endpoint.

Avoid overlapping assumptions about policy

A model-request guardrail does not automatically enforce provider account privileges. A tool rule does not automatically inspect the full conversation or prevent every unsafe generation. Decide which layer enforces each requirement and verify that the relevant request actually crosses it.

In AIPx, Hide is enforced server-side and removes the tool from an agent’s access. Allow and Confirm are execution hints; Confirm requests client-side review, with actual behavior dependent on the client. It should not be treated as a universal pre-execution gateway gate.

A valid tool response can contain instructions that conflict with the user’s task. Keep it as external data, not fresh authority to perform additional operations. Narrow available tools and validate business effects in the executor regardless of which model route generated the proposal.

Diagnose with separate stages

If a task is slow, distinguish time spent generating, discovering tools, authenticating, waiting on the provider and generating the final response. Adding an inference gateway will not by itself speed up an upstream API. Changing the MCP endpoint will not fix an overlong generation request.

If a task fails, identify whether the failure was a model provider error, invalid tool input, denied account access, provider error or an unsatisfactory answer. A single “agent failed” metric hides the decisions needed to repair it.

AIPx adds another hosted operator and depends on network and upstream availability. Use the gateway layers whose concrete controls and visibility solve your problem, and keep model and tool costs separate. Connect free.