Unit 6.5: AI Inference: Managed Model Hub

Introduction

FinCorp wants an internal assistant that answers staff questions from its own policy and product documentation. The architectural decision is not which model is cleverest; it is where the inference runs and who carries the operational weight. For a German financial-services firm under GDPR and BSI obligations, the default should be the option that keeps data in-country, adds no infrastructure to operate, and bills only for what is consumed. That option is the managed AI Model Hub. This unit establishes why managed inference is the enterprise default, when a dedicated GPU server earns its keep instead, and how FinCorp builds retrieval-augmented generation from platform primitives. It ends by lifting a working inference call so the API shape is concrete.

1. Managed Inference as the Enterprise Default

AI Model Hub is an inference service: it serves pre-trained models behind an API so you implement AI features without provisioning or maintaining the underlying hardware. For an architect the appeal is that it removes an entire operational surface, GPU drivers, model loading, capacity headroom, patching, from the design. Four properties make it the default choice for a regulated workload.

OpenAI-compatible API. The hub exposes two API surfaces: a native IONOS API and an OpenAI-compatible API that mirrors the OpenAI request and response structure. The OpenAI-compatible base URL is https://openai.inference.de-txl.ionos.com/v1, and it serves the familiar routes: list models, POST /v1/chat/completions for text, POST /v1/embeddings for vectors, and POST /v1/images/generations for images. The architectural consequence is portability. Any client, library, or framework already written against the OpenAI API can be pointed at the hub by changing the base URL and the token, so FinCorp's choice does not lock its application code to one vendor.

Per-token pricing. Billing is per million tokens, split between input and output, and varies by model. The following figures are the documented rates for two representative text models:

Model	Model identifier	Input (EUR / 1M tokens)	Output (EUR / 1M tokens)	Context window
Llama 3.3 70B	`meta-llama/Llama-3.3-70B-Instruct`	0.65	0.65	128,000 tokens
GPT-OSS 120B	`openai/gpt-oss-120b`	0.15	0.65	128,000 tokens

As the table shows, the cost lever is token volume, not a provisioned instance. There is no GPU sitting idle between requests and no commitment to size up front. For a bursty internal assistant whose load follows the working day, paying per token is structurally cheaper than reserving accelerator hardware that is busy a few hours out of twenty-four.

Stateless. The service discards prompts and outputs at the end of each session. Interactions are not logged, not recorded, and not reused for model training, and customer data is not used for training under any circumstance. Statelessness is a compliance property, not just an efficiency one: there is no retained-content store for FinCorp's data-protection officer to reason about, and no inference-side data lifecycle to govern.

Processed in-country. All processing and inference occur exclusively in Germany, in the Berlin data center. AI Model Hub services, including the inference endpoints, are hosted in ISO 27001-certified German data centers. For FinCorp this closes the sovereignty question that drove the platform choice in the first place: the model call never leaves German jurisdiction. Scope this claim precisely. The ISO 27001 certification attaches to the German data centers hosting the hub; it is not a platform-wide statement, and the deeper sovereignty-boundary and EU-AI-Act role analysis belongs to Unit 6.6.

Two operational boundaries belong in the design. The per-service uptime SLA is 99.9%, and its scope is the AI Model Hub API endpoints only, not per-model availability or inference quality. The general API carries a base rate limit of 5 requests per second with a burst ceiling of 10; exceeding it returns HTTP 429 Too Many Requests. A production client must therefore implement backoff and retry rather than assume unlimited throughput, and a high-fan-out workload needs a request-pacing layer in front of the hub.

2. When Self-Hosted GPU Serving Is Justified

The managed hub serves a fixed catalogue of platform-hosted models. When a customer needs to deploy its own pre-trained model, fine-tune on proprietary data, or run a model the catalogue does not offer, IONOS positions dedicated GPU-enabled Compute Engine servers instead. These give root access to enterprise-class GPU hardware on a pay-as-you-go basis, so the workload runs on infrastructure the customer controls.

The trade-off is ownership of everything the managed service was hiding. The honest framing for an architect is this: the moment you self-host, the inference SLA becomes yours, not the hub's. A single GPU server is single-tenant dedicated hardware with no live migration, so a host fault or maintenance event interrupts serving unless you have built redundancy yourself. Achieving the kind of availability the managed endpoint provides means running at least two serving nodes behind a Layer 4 load balancer, plus your own model-loading, health-checking, autoscaling, and patching, but the default per-contract quota is only one H200-S instance (larger templates or additional instances require a support-ticket resource-limit increase before deployment) and availability-zone placement is fixed to Auto, so you cannot explicitly force the two nodes into separate zones. Predictable cost is the upside the documentation calls out, a fixed rate for sustained, always-on serving rather than variable per-token billing, so self-hosting tends to win only for steady-state, high-utilization workloads or for models the hub does not serve.

Keep two sizing questions separate. Inference sizing is driven by concurrency, latency targets, and context-window memory, and for steady high load a dedicated GPU can be cheaper than per-token billing. Training and fine-tuning sizing is a different and heavier regime: it needs sustained multi-GPU compute against proprietary datasets, and the platform path for that is the GPU server, not the inference hub. Do not size an inference deployment as though it were a training cluster, and do not assume an inference-sized GPU will fine-tune a large model in reasonable time. FinCorp's assistant is pure inference with modest, bursty load, which is exactly the profile the managed hub serves best, so FinCorp stays on the hub and does not stand up GPU servers.

3. Customer-Built Retrieval-Augmented Generation

FinCorp's assistant must answer from FinCorp's own documents, not from the model's general training. The pattern for that is retrieval-augmented generation (RAG): combine the model's language ability with relevant passages retrieved from a knowledge base at query time. The hub provides the model-side pieces; the architect composes the storage-side pieces from platform primitives.

Build RAG as customer-owned infrastructure with three platform components:

Embeddings from the hub. Convert each document chunk into a dense vector with POST /v1/embeddings, using a hub embedding model such as BAAI/bge-m3 (documented at 0.02 EUR per million tokens). The same endpoint embeds the user's query at retrieval time so query and corpus live in the same vector space.
Vectors in Managed PostgreSQL. Store the embeddings in a managed PostgreSQL instance with the pgvector extension. This gives you a relational database you already know how to operate, back up, and place on the private data tier, with vector similarity search as a first-class query. It composes cleanly with everything Module 5 established about the relational tier.
Corpus in Object Storage. Keep the source documents in Object Storage as the system of record, using its lifecycle and object-lock features for retention. The vector store holds embeddings and references; the authoritative text stays in the S3-compatible bucket.

At query time the flow is: embed the question, run a similarity search in PostgreSQL to retrieve the closest chunks, then pass those chunks as context to POST /v1/chat/completions. Nothing in this loop requires a managed vector store, and you keep full control of where the vectors and corpus live.

This is a deliberate architectural choice. AI Model Hub historically offered a managed document-collections feature, a built-in vector store with chunking and a native semantic-query endpoint, with a default chromadb backend and an optional pgvector backend. That managed vector-store feature is being retired and must not be designed in as a go-forward option. Build RAG instead on the durable primitives above: hub embeddings, your own pgvector on Managed PostgreSQL, and your corpus in Object Storage. The result is identical in capability, sits entirely on services FinCorp already operates, and carries no deprecation risk.

DCD Implementation Walkthrough

This unit's build is deliberately light because AI Model Hub is consumed through its API rather than provisioned as Data Center Designer infrastructure. The goal is to test a model and lift a working OpenAI-compatible call so the API shape is concrete and FinCorp's application team can wire it into the assistant. The only prerequisite is access to the contract's token management.

Build goal: Test a model and lift a working OpenAI-compatible call.

Steps:

Generate an API token. In the DCD, open Access Management (token management) and create a new API token for the AI Model Hub. The hub authenticates with a Bearer token, and the token is a JSON Web Token (JWT) with an expiration date. Copy the token immediately, because it is shown once.
Hold the token in the expected environment variable. The hub how-tos assume the token is held in the IONOS_API_TOKEN environment variable. Set it in your shell rather than pasting the literal token into scripts or source control.
List the available models. Confirm connectivity and discover model identifiers by calling the OpenAI-compatible models route at the base URL https://openai.inference.de-txl.ionos.com/v1. A successful response proves the token, endpoint, and region are correct before you send any prompt.
Send a chat completion. Call POST /v1/chat/completions against the base URL with a model identifier from step 3 and a messages array. This is the core inference call FinCorp's assistant will make.
Lift the working call into the application. Once the call returns, hand it to the application team as the canonical client configuration: base URL, model identifier, and the Bearer-token convention. Any OpenAI-compatible library now works by setting only the base URL and the token.

A minimal chat-completion call illustrates the OpenAI-compatible shape. The architectural point is the request contract, not the script: changing only the base URL and token repoints any OpenAI client at the in-country hub.

curl https://openai.inference.de-txl.ionos.com/v1/chat/completions \
  -H "Authorization: Bearer $IONOS_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.3-70B-Instruct",
    "messages": [{"role": "user", "content": "Summarise our refund policy."}]
  }'

Common mistakes:

Pasting an expired or wrong-type token. The Bearer token is a JWT with an exp claim. Authentication failures are usually an expired token, regenerate it and re-set IONOS_API_TOKEN, not a malformed request.
Hardcoding the token into scripts or repositories. The token is shown once and grants paid inference access. Hold it in IONOS_API_TOKEN, keep it out of source control, and rotate it on the usual schedule.
Pointing the client at the wrong base URL. The OpenAI-compatible base is https://openai.inference.de-txl.ionos.com/v1. A client configured for the generic OpenAI host will fail; only the base URL and token should change when migrating an OpenAI client.
Assuming unlimited throughput. The base rate limit is 5 requests per second, burst 10, and the service returns HTTP 429 when exceeded. Build backoff and retry into the client and pace high-fan-out workloads; do not treat the endpoint as elastic.
Designing on the managed document-collections vector store. It is being retired. Build RAG on hub embeddings plus your own pgvector on Managed PostgreSQL plus Object Storage instead.
Reading the 99.9% SLA as a quality or per-model guarantee. It covers the API endpoints only, not the availability of any specific model or the quality of inference.

Summary

For a regulated enterprise, managed inference on AI Model Hub is the default: an OpenAI-compatible API, per-token pricing with no idle GPU to fund, a stateless service that retains nothing, and processing confined to German data centers. Self-hosted GPU serving on dedicated hardware is justified only when you need a model the hub does not serve, you are fine-tuning, or steady high-utilization load makes fixed-rate compute cheaper, and it makes the inference SLA and redundancy your responsibility. FinCorp's assistant fits the managed profile, so it consumes the hub directly and builds retrieval-augmented generation from hub embeddings, pgvector on Managed PostgreSQL, and a corpus in Object Storage, sidestepping the deprecating managed vector store entirely.

Key Points:

The OpenAI-compatible base URL is https://openai.inference.de-txl.ionos.com/v1; repointing an OpenAI client needs only the base URL and a Bearer token.
Pricing is per million tokens and varies by model; there is no provisioned instance, so token volume is the cost lever.
The hub is stateless and processes exclusively in Germany; data is never used for training. The 99.9% SLA covers API endpoints only.
Self-host on GPU servers only for unsupported models, fine-tuning, or steady high-utilization load, accepting that you then own the SLA and redundancy.
Build RAG as customer-owned: hub embeddings, vectors in pgvector on Managed PostgreSQL, corpus in Object Storage. Do not adopt the deprecating managed vector store.

Important Terminology:

OpenAI-compatible API: an API surface that mirrors OpenAI's request and response structure, letting existing OpenAI clients work by changing only the base URL and token.
Per-token pricing: billing measured per million input and output tokens rather than per provisioned instance.
Retrieval-augmented generation (RAG): combining a language model with passages retrieved from a knowledge base at query time so answers are grounded in the customer's own documents.
pgvector: a PostgreSQL extension that stores embedding vectors and supports similarity search, enabling a customer-built vector store on Managed PostgreSQL.

Learning Objectives