Unit 4.2: Object Storage Integration

Introduction

TaskBoard lets users attach files to tasks: screenshots, PDFs, design mockups, the occasional multi-gigabyte video. You do not want those bytes flowing through your API process. They consume memory, block worker threads, and turn a stateless API into a bottleneck. The answer is IONOS Cloud Object Storage with presigned URLs: the browser talks to Object Storage directly, and your API only ever handles small metadata records.

In this unit you wire the TaskBoard API to Object Storage through the S3-compatible API using boto3. The single most important thing to internalize up front: Object Storage does not use your IONOS Cloud bearer token. It authenticates with a separate Access Key and Secret Key pair, exactly like AWS S3. You will set up the client, generate presigned upload and download URLs, handle large files with multipart upload, and apply lifecycle rules so abandoned uploads do not accumulate cost forever.

1. Authenticating and Connecting with boto3

Object Storage exposes an AWS S3 API v2 surface, so the standard AWS SDKs work against it. What changes is the endpoint and the credentials. You must pass an explicit endpoint_url (the SDK defaults to AWS, not IONOS) and an Access Key + Secret Key pair. These keys are not your Cloud API bearer token and not an IAM role. They are generated separately and authenticate every S3 request via AWS Signature.

Object Storage authenticates users with a pair of keys: an Access Key and a Secret Key. A key must be generated manually, either in the DCD under Object Storage Credentials and Management or through the Object Storage Management API. The current Access Key length is 92 characters and the Secret Key length is 64 characters, so do not hardcode field-width assumptions from older 20/40-character formats. Each user can hold up to 5 access keys.

1.1 Building the S3 Client

Before you write any client code, pin your boto3 version: boto3 1.36.0 and later fail every PutObject call against IONOS Object Storage with InvalidTrailer ("Invalid trailing header names in x-amz-trailer") because IONOS does not support the AWS trailing-checksum extension that boto3 shipped starting in 1.36.0. Install boto3<=1.35.99, or if you need a newer boto3, set the environment variables AWS_REQUEST_CHECKSUM_CALCULATION=when_required and AWS_RESPONSE_CHECKSUM_VALIDATION=when_required before every upload path in this unit (presigned PUT, upload_file, upload_fileobj, multipart) will work.

Pick the endpoint for the region your bucket lives in. The endpoint hostname encodes the region, and the SDK derives the signing region from it, so you can leave the region_name minimal and let the endpoint drive routing.

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url="https://s3.eu-central-1.ionoscloud.com",  # Frankfurt (de)
    aws_access_key_id="YOUR_ACCESS_KEY",       # 92-char Object Storage key
    aws_secret_access_key="YOUR_SECRET_KEY",   # 64-char Object Storage secret
    region_name="eu-central-1",
    config=Config(signature_version="s3v4"),
)

# Smoke test: list buckets you own
for b in s3.list_buckets()["Buckets"]:
    print(b["Name"], b["CreationDate"])

Never inline credentials in source. Pull them from environment variables or a secret manager. In TaskBoard, the Access Key and Secret Key come out of Terraform state (the ionoscloud_s3_key resource from Unit 2.4) and are injected as Kubernetes secrets.

import os

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["OBJSTORAGE_ENDPOINT"],
    aws_access_key_id=os.environ["OBJSTORAGE_ACCESS_KEY"],
    aws_secret_access_key=os.environ["OBJSTORAGE_SECRET_KEY"],
    region_name=os.environ.get("OBJSTORAGE_REGION", "eu-central-1"),
    config=Config(signature_version="s3v4"),
)

1.2 Choosing the Right Regional Endpoint

The endpoint is not interchangeable. A bucket exists in exactly one region, and you must address it through that region's endpoint. Object Storage is available in Berlin, Frankfurt, Logroño, and Lenexa. The following table maps data centers to their regions and S3 endpoints:

Choose a region close to your application and users to reduce latency, and a geographically separate region for backups so a local outage does not take your primary data with it. Note the two bucket types: user-owned buckets live in de, eu-central-2, and eu-south-2, while contract-owned buckets live in eu-central-4, eu-central-3, and us-central-1. The limits differ by type: a user can hold up to 500 user-owned buckets and up to 1000 contract-owned buckets. The single-request upload ceiling is 4.65 GB for user-owned buckets and 5 GB for contract-owned buckets, which is the practical trigger for multipart upload covered in Section 3.

2. Bucket Operations and Presigned URLs

With a client in hand, the everyday operations are standard S3: create a bucket, list objects, set ACLs, and generate presigned URLs. For TaskBoard, presigned URLs are the centerpiece. They let the browser upload and download attachments directly against Object Storage so your API never streams the bytes.

By default, objects are private and only the bucket owner can access them. A presigned URL grants time-bound access to a single object without changing the object's permissions and without sharing your keys. The owner signs the URL, hands it to the client, and the client uses it until it expires.

2.1 Creating Buckets and Listing Objects

Bucket names are globally unique across all of Object Storage, must be between 3 and 63 characters, and follow DNS naming rules. Object keys can be up to 1024 characters.

bucket = "taskboard-attachments-prod"

# Create the bucket (idempotent-ish: catch the "already owned" case)
try:
    s3.create_bucket(Bucket=bucket)
except s3.exceptions.BucketAlreadyOwnedByYou:
    pass

# List objects under a prefix, paginated (collections can be huge)
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=bucket, Prefix="task-42/"):
    for obj in page.get("Contents", []):
        print(obj["Key"], obj["Size"])

Always paginate listings. A single bucket can hold up to 200,000,000 objects in the unversioned case, so a non-paginated list will silently truncate at the first page.

2.2 Presigned Upload and Download URLs

Generate a presigned PUT URL so the browser uploads the file directly. Your API returns the URL and the object key; it never sees the file content.

def presign_upload(key: str, content_type: str, expires: int = 900) -> str:
    return s3.generate_presigned_url(
        "put_object",
        Params={"Bucket": bucket, "Key": key, "ContentType": content_type},
        ExpiresIn=expires,  # seconds; keep short for uploads
    )

def presign_download(key: str, expires: int = 300) -> str:
    return s3.generate_presigned_url(
        "get_object",
        Params={"Bucket": bucket, "Key": key},
        ExpiresIn=expires,
    )

The browser then uploads with a plain HTTP request to the signed URL. The Content-Type must match what was signed, or the signature check fails:

await fetch(presignedUrl, {
  method: "PUT",
  headers: { "Content-Type": file.type },
  body: file,
});

Presigned URLs are ideal for letting other users upload objects directly to your bucket without ever handing them your Access Key and Secret Key. They are also the right tool for sharing a private object with someone who has no IONOS account: the URL works for the configured window, then expires. Both Signature v2 and v4 are supported; use v4.

3. File Upload and Download Patterns

Small attachments are a single put_object. Large attachments need multipart upload, both to beat the single-request size ceiling and to upload parts in parallel for speed. An individual object can be up to 5 TB (5,497,558,138,880 bytes).

3.1 Multipart Upload for Large Files

The single-request upload limit is 4.65 GB (user-owned) or 5 GB (contract-owned). Beyond that, you must use the multipart upload API. boto3's high-level upload_file and upload_fileobj handle multipart automatically once a file crosses the configured threshold, which is the production-grade path.

from boto3.s3.transfer import TransferConfig

# Switch to multipart above 100 MB, 8 MB parts, up to 4 parallel uploads
transfer_cfg = TransferConfig(
    multipart_threshold=100 * 1024 * 1024,
    multipart_chunksize=8 * 1024 * 1024,
    max_concurrency=4,
)

s3.upload_file(
    Filename="/tmp/build-artifact.zip",
    Bucket=bucket,
    Key="task-42/build-artifact.zip",
    Config=transfer_cfg,
    ExtraArgs={"ContentType": "application/zip"},
)

Multipart upload breaks a large object into smaller parts and uploads them in parallel, maximizing throughput. It is available through the API and SDKs, not through the DCD web interface. If a multipart upload is abandoned, the uploaded parts linger and accrue storage cost until you clean them up, which is exactly what the lifecycle rule in Section 4 handles.

3.2 Streaming Downloads and Content-Type

For downloads, stream the body rather than loading the whole object into memory. The response body is a file-like stream you can iterate in chunks.

resp = s3.get_object(Bucket=bucket, Key="task-42/report.pdf")
content_type = resp["ContentType"]  # set at upload time

with open("/tmp/report.pdf", "wb") as f:
    for chunk in resp["Body"].iter_chunks(chunk_size=1024 * 1024):
        f.write(chunk)

Set ContentType explicitly on upload. If you skip it, downloads default to a generic binary type and browsers will offer a download prompt instead of rendering the file inline.

4. Lifecycle Policies and Cost Control

Object Storage charges for what you store, so orphaned objects and abandoned multipart uploads are a slow cost leak. Lifecycle rules let you expire objects automatically and clean up incomplete uploads. You configure them through the S3 API.

A lifecycle configuration supports these actions: expire current versions, permanently delete noncurrent versions, delete expired object delete markers, and delete incomplete multipart uploads. You can set up to 1000 rules in a configuration. Each rule applies to a different object prefix, and you cannot set more than one rule for the same prefix.

4.1 Expiring Objects and Cleaning Up Multipart Uploads

This configuration expires temporary uploads after 7 days and aborts incomplete multipart uploads after 1 day:

s3.put_bucket_lifecycle_configuration(
    Bucket=bucket,
    LifecycleConfiguration={
        "Rules": [
            {
                "ID": "expire-temp-uploads",
                "Filter": {"Prefix": "tmp/"},
                "Status": "Enabled",
                "Expiration": {"Days": 7},
            },
            {
                "ID": "abort-incomplete-multipart",
                "Filter": {"Prefix": ""},
                "Status": "Enabled",
                "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 1},
            },
        ]
    },
)

Object Storage currently supports only the STANDARD storage class, so you cannot use lifecycle rules to transition objects to a colder tier. The only available actions are expiration and cleanup. If a bucket uses Object Lock, noncurrent versions cannot be deleted before their retention period completes.

5. S3 Compatibility: What Is and Is Not Supported

Object Storage has one of the highest levels of S3 API support, but it is not 100% AWS parity. Verify a feature before you depend on it rather than assuming it behaves exactly like AWS S3. The following table summarizes key feature support:

A few constraints to code around. Object Copy does not support cross-region copies, so a "move bucket to another region" operation is a download-then-upload, not a server-side copy. Object Lock (GOVERNANCE and COMPLIANCE modes) can only be enabled at bucket creation time, never retrofitted onto an existing bucket. Bucket Policy uses standard JSON with a Version of 2012-10-17. Encryption in transit is TLS 1.2 or 1.3.

5.1 CORS for Browser Uploads

Because TaskBoard's browser uploads directly to Object Storage via presigned URLs, the bucket needs a CORS configuration that allows PUT from your web origin. Without it, the browser blocks the cross-origin request before it ever reaches Object Storage.

s3.put_bucket_cors(
    Bucket=bucket,
    CORSConfiguration={
        "CORSRules": [
            {
                "AllowedOrigins": ["https://app.taskboard.example"],
                "AllowedMethods": ["PUT", "GET"],
                "AllowedHeaders": ["*"],
                "MaxAgeSeconds": 3000,
            }
        ]
    },
)

The authoritative S3 API reference is at https://api.ionos.com/docs/s3/v2/, and the key/bucket management API is documented at https://api.ionos.com/docs/s3-management/v1/. When in doubt about a feature, check those before writing code that depends on it.

API Reference Quick Card

Key S3 operations for Object Storage (via boto3 or signed HTTP):

Endpoint: https://s3.<region>.ionoscloud.com (e.g. s3.eu-central-1.ionoscloud.com) Authentication: AWS Signature v4 with Access Key + Secret Key (NOT a bearer token)

Code Lab

Objective: Upload and retrieve a TaskBoard attachment via boto3 using presigned URLs and multipart upload, confirming Access Key + Secret Key auth.

Prerequisites:

• IONOS Cloud account with an Object Storage Access Key + Secret Key generated
• Python 3.9+ with a compatible boto3 installed (pip install "boto3<=1.35.99"); boto3 1.36.0 and later fail every PutObject call against IONOS Object Storage with InvalidTrailer because IONOS does not support the AWS trailing-checksum extension introduced in that release, so pin the version or set AWS_REQUEST_CHECKSUM_CALCULATION=when_required and AWS_RESPONSE_CHECKSUM_VALIDATION=when_required if you must use a newer boto3
• A region endpoint chosen (this lab uses Frankfurt de / eu-central-1)

Step 1: Export credentials

export OBJSTORAGE_ENDPOINT="https://s3.eu-central-1.ionoscloud.com"
export OBJSTORAGE_ACCESS_KEY="<your-92-char-access-key>"
export OBJSTORAGE_SECRET_KEY="<your-64-char-secret-key>"
export OBJSTORAGE_REGION="eu-central-1"

Expected output:

(no output; variables set)

Step 2: Create the client and a bucket

import os, boto3
from botocore.config import Config

s3 = boto3.client("s3",
    endpoint_url=os.environ["OBJSTORAGE_ENDPOINT"],
    aws_access_key_id=os.environ["OBJSTORAGE_ACCESS_KEY"],
    aws_secret_access_key=os.environ["OBJSTORAGE_SECRET_KEY"],
    region_name=os.environ["OBJSTORAGE_REGION"],
    config=Config(signature_version="s3v4"))

bucket = "taskboard-lab-<your-initials>"
s3.create_bucket(Bucket=bucket)
print("created", bucket)

Expected output:

created taskboard-lab-ct

Step 3: Generate a presigned upload URL

url = s3.generate_presigned_url("put_object",
    Params={"Bucket": bucket, "Key": "task-1/note.txt", "ContentType": "text/plain"},
    ExpiresIn=900)
print(url)

Expected output:

https://s3.eu-central-1.ionoscloud.com/taskboard-lab-ct/task-1/note.txt?X-Amz-Algorithm=...

Step 4: Upload through the presigned URL with curl

echo "hello taskboard" > note.txt
curl -X PUT -H "Content-Type: text/plain" --upload-file note.txt "<PASTE_PRESIGNED_URL>"

Expected output:

(HTTP 200, empty body)

Step 5: Multipart upload a large file

from boto3.s3.transfer import TransferConfig
cfg = TransferConfig(multipart_threshold=5*1024*1024, multipart_chunksize=5*1024*1024)
s3.upload_file("bigfile.bin", bucket, "task-1/bigfile.bin", Config=cfg,
               ExtraArgs={"ContentType": "application/octet-stream"})
print("uploaded large file")

Expected output:

uploaded large file

Step 6: Download via a presigned GET URL

get_url = s3.generate_presigned_url("get_object",
    Params={"Bucket": bucket, "Key": "task-1/note.txt"}, ExpiresIn=300)
import urllib.request
print(urllib.request.urlopen(get_url).read().decode())

Expected output:

hello taskboard

Step 7: Add a lifecycle rule to clean up incomplete uploads

s3.put_bucket_lifecycle_configuration(Bucket=bucket,
    LifecycleConfiguration={"Rules": [{
        "ID": "abort-mpu", "Filter": {"Prefix": ""}, "Status": "Enabled",
        "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 1}}]})
print("lifecycle set")

Expected output:

lifecycle set

Validation Checklist:

• [ ] Bucket created and visible in s3.list_buckets()
• [ ] File uploaded and downloaded via presigned URLs
• [ ] Multipart upload of a large file succeeded
• [ ] Lifecycle rule applied without error

Cleanup:

for obj in s3.list_objects_v2(Bucket=bucket).get("Contents", []):
    s3.delete_object(Bucket=bucket, Key=obj["Key"])
s3.delete_bucket(Bucket=bucket)
print("cleaned up")

Common Pitfalls

Developer mistakes to avoid with Object Storage integration:

1. Trying to authenticate with your Cloud API bearer token

   • Problem: Every S3 request returns 403 SignatureDoesNotMatch or InvalidAccessKeyId.
   • Why it happens: Object Storage uses AWS Signature with an Access Key + Secret Key pair, not the IONOS Cloud bearer token used by the rest of the API and not IAM roles. They are entirely separate credential systems.
   • Fix: Generate a dedicated Object Storage key (DCD or Management API) and pass it as aws_access_key_id / aws_secret_access_key. The current keys are 92 and 64 characters; reject older 20/40-char assumptions.

2. Omitting the endpoint_url and hitting AWS instead

   • Problem: Requests time out, fail DNS, or land on real AWS S3.
   • Why it happens: boto3 defaults to AWS endpoints. Without endpoint_url, the SDK never talks to IONOS.
   • Fix: Always pass the region-specific endpoint, e.g. endpoint_url="https://s3.eu-central-1.ionoscloud.com", and match region_name to that endpoint's region.

3. Content-Type mismatch breaking presigned uploads

   • Problem: The browser PUT to a presigned URL returns 403 SignatureDoesNotMatch.
   • Why it happens: The Content-Type was included when signing the URL but the client sent a different (or no) Content-Type header. Signed headers must match exactly.
   • Fix: Sign with the exact content type and send the identical header from the client, or omit ContentType from Params entirely so it is not part of the signature.

Summary

You can now integrate IONOS Cloud Object Storage into application code through its S3-compatible API. You configure a boto3 client with the correct regional endpoint and Access Key + Secret Key auth, move file bytes off your API path with presigned upload and download URLs, handle large files with multipart upload, stream downloads, and apply lifecycle rules to keep storage cost under control. The TaskBoard attachment flow now lets browsers upload directly to Object Storage while the API only records metadata.

Key Points:

• Object Storage authenticates with a 92-character Access Key and a 64-character Secret Key via AWS Signature v4, never with the Cloud API bearer token or IAM roles
• Always pass an explicit endpoint_url matching the bucket's region; bucket names are globally unique and 3 to 63 characters
• Presigned URLs grant time-bound, credential-free access so clients upload and download directly against Object Storage
• Use multipart upload for files above the single-request limit (4.65 GB user-owned, 5 GB contract-owned); objects can reach 5 TB
• Object Storage has high S3 compatibility but not full parity: cross-region copy is unsupported, Object Lock is creation-time only, and only the STANDARD storage class exists

Important Terminology:

• Access Key / Secret Key: The S3 credential pair (92 and 64 characters) that authenticates every Object Storage request via AWS Signature.
• Presigned URL: A time-limited, signed URL that grants access to a single object without sharing keys or changing permissions; the basis for direct browser uploads and downloads.
• Multipart upload: Splitting a large object into parts uploaded in parallel; required above the single-request size limit and cleaned up via lifecycle rules.
• Lifecycle rule: A bucket configuration (up to 1000 rules) that expires objects and aborts incomplete multipart uploads automatically.
• Bucket type: User-owned versus contract-owned buckets, which differ in region availability, per-user bucket limits, and single-request upload ceilings.

Next Steps

Continue Learning: Unit 4.3: Event Streaming Integration

Related Topics:

• Unit 2.4: Storage Provisioning as Code
• Unit 4.1: Database and Caching Integration