S3-Compatible Object Endpoint

BifrostQL can expose the binary contents of your file columns through an S3-compatible HTTP endpoint, so any tool that speaks the AWS S3 REST API with Signature Version 4 — the AWS CLI, rclone, an SDK — can list, download, and (optionally) upload those blobs as S3 objects. It is a protocol adapter: the wire is S3, but every read and write still executes through the same transformer pipeline as GraphQL, so tenant isolation, soft-delete, policy read guards, audit stamping, and field encryption are enforced on the wire.

This is a deliberately narrow, read-first front door. It is not an S3 server: it maps a small object-operation surface onto the file columns of your relational tables and answers everything outside that surface with a clean, honest error. Writes exist but are off by default (see Optional writes).

Enabling the endpoint

Register the endpoint with AddS3Endpoint on your BifrostQL setup, then mount the middleware with UseBifrostS3. Both the endpoint and its writes are opt-in and default off, mirroring the pgwire and RESP adapters. Registering an IS3AccessKeyStore is a hard requirement, resolved fail-fast at pipeline build — an enabled endpoint with no identity source throws at startup, never at first request, and enabling the endpoint logs a startup warning because exposing an S3 front door is a posture change worth surfacing.

builder.Services.AddBifrostQL(o => o
    // ... your database + module setup ...
    .AddS3Endpoint(s3 =>
    {
        s3.Enabled = true;                 // opt-in; default false
        s3.Region = "us-east-1";           // the SigV4 credential-scope region requests must match
        s3.PathPrefix = "/s3";             // where the endpoint listens; default "/s3"
        // s3.EnableWrites = true;         // opt-in write surface (PutObject/DeleteObject/CopyObject); default off
        // s3.ContinuationTokenSecret = "…"; // stable HMAC secret so list pages survive restarts / a scaled fleet
    }));

// REQUIRED — the identity source SigV4 authenticates against. There is no default
// registration and no fallback identity: an unknown access key fails closed.
builder.Services.AddSingleton<IS3AccessKeyStore, MyAccessKeyStore>();

var app = builder.Build();
app.UseBifrostS3();   // no-op when the endpoint was not enabled, so it is safe to call unconditionally

S3Options

Option	Default	Meaning
Enabled	false	Master gate. Off by default; the endpoint serves nothing until a deployment opts in.
EnableWrites	false	Gate for PutObject/DeleteObject/CopyObject. Off by default: every write is a clean, authenticated 501 NotImplemented that builds no intent and cannot be probed. Enabling it also enables the underlying FileObjectSeamOptions.EnableWrites, which logs its own startup warning.
Region	us-east-1	The AWS region a request’s SigV4 credential scope must match. Any other region is rejected.
PathPrefix	/s3	Path prefix the endpoint listens under.
Endpoint	null	Which registered BifrostQL endpoint’s cached model/connection to read against; null selects the single registered endpoint.
MaxClockSkew	15 min	Maximum tolerated skew between a signed request’s timestamp and server time (header and presigned).
MaxPresignedExpiry	7 days	Maximum X-Amz-Expires a presigned URL may declare. A longer expiry is rejected before the signature is even checked.
MaxBodyBytes	64 MiB	Maximum request body accepted, enforced from Content-Length before any buffering.
MaxHeaderBytes	16 KiB	Maximum total request-header size, enforced before the body is read.
MaxUrlLength	8 KiB	Maximum raw request URL (path + query) length.
MaxMetadataBytes	2 KiB	Maximum summed size of x-amz-meta-* user metadata on a PutObject. Exceeding it is rejected before any write.
DefaultMaxKeys	1000	ListObjectsV2 page size when the request omits max-keys.
MaxKeysLimit	1000	Hard cap applied to any client-requested max-keys.
MaxListMaterialize	10000	Maximum candidate rows a single list scan may materialize. A bucket with more addressable objects fails fast rather than returning a silently truncated (lying) listing.
ContinuationTokenSecret	null	HMAC secret binding and integrity-protecting continuation tokens. When set, tokens survive restarts and resolve across a scaled fleet; when unset, a per-instance random key is used, so tokens live only for the process’s lifetime. Must never be a value a client could learn — it is the only thing preventing a forged resume position.

Bucket, table, and object-key mapping

The mapping between S3 coordinates and Bifrost coordinates is deterministic and injective — no filesystem root is scanned, no bucket is invented.

A bucket is a table with a file column. A table is exposed as a bucket when it has at least one column carrying the file metadata tag (the same FileColumnConfig blob the GraphQL upload path uses) and its name is a legal S3 bucket name once lowercased. The bucket name is the table’s database name, lowercased — lowercasing is the only normalization applied. Case-only collisions (two tables differing only by case) are detected, not collapsed: the bucket is reported ambiguous and the request fails rather than silently exposing one of the tables. A table whose lowercased name is not a legal bucket name (3–63 chars, lowercase alphanumeric / . / -, not IP-shaped) is simply not addressable over S3 — it is honestly absent, never rewritten.

An object key is {column}/{pk0}/{pk1}/… — the file column’s name followed by the row’s primary-key components in the table’s declared key order. This is composite-primary-key safe: every key component is present, in order; there is never a first-column guess. BigInt keys render as decimal strings, so no precision is lost. Each component is percent-encoded over the unreserved set [A-Za-z0-9_-], which is injective and can never emit /, ., or an empty segment — so two distinct rows can never collapse onto one key, and no component can ever emerge as a ./.. traversal segment (see Security).

Where the object bytes physically live is a separate concern: a file column’s blob is stored through its configured storage provider (the storage metadata blob — provider:local, provider:s3, prefix, region, etc.). The S3 endpoint’s bucket is the table; do not confuse it with a backing cloud-storage bucket.

Access-key management

SigV4 proves a caller knows the secret behind an access-key id; BifrostQL resolves that id through the IS3AccessKeyStore you register. There is no built-in store — a deployment that enables the endpoint must supply one.

public interface IS3AccessKeyStore
{
    // Returns the key for accessKeyId, or null when no such key exists.
    // Never returns a fallback / ambient identity.
    Task<S3AccessKey?> FindAsync(string accessKeyId, CancellationToken cancellationToken);
}

public sealed record S3AccessKey(
    string AccessKeyId,
    string SecretAccessKey,     // the shared secret; proven via the HMAC chain, never sent on the wire
    ClaimsPrincipal Principal,  // the candidate identity this key maps to
    bool Enabled);

Hard rules the store must honor, all fail-closed:

• An unknown access key resolves to null — authentication fails. Never hand back an ambient or anonymous identity to stand in for a failed lookup.
• A disabled key (Enabled == false) must fail exactly as an unknown key does. Never distinguish “disabled” from “unknown” in the response — doing so is an enumeration oracle.
• The Principal is a candidate only. It is still projected through the shared IBifrostAuthContextFactory (the same identity seam every transport gate uses), which rejects a subject-less principal or a token from an issuer this deployment has no claim mapper for. The store never gets the final say on identity.

Client configuration

The examples below use placeholder credentials — never commit real secrets. Match region to your S3Options.Region and point the client at your endpoint’s PathPrefix with path-style addressing.

AWS CLI

# Placeholder credentials — replace with a key pair your IS3AccessKeyStore knows.
aws configure set aws_access_key_id     AKIAEXAMPLE0000000000
aws configure set aws_secret_access_key wJalrXUtnFEMIexampleKEYdonotuseREAL00000
aws configure set region                us-east-1

BUCKET=s3://orders     # a lowercased table name that has a file column

# list (ListObjectsV2)
aws --endpoint-url http://localhost:5000/s3 s3 ls $BUCKET/

# download (GetObject)  — key is  {column}/{pk}
aws --endpoint-url http://localhost:5000/s3 s3 cp $BUCKET/invoice_pdf/42 ./invoice-42.pdf

# server-side copy (CopyObject, opt-in write)
aws --endpoint-url http://localhost:5000/s3 s3 cp $BUCKET/invoice_pdf/42 $BUCKET/invoice_pdf/43

# upload (PutObject, opt-in write) — see the note below on chunked signing
aws --endpoint-url http://localhost:5000/s3 s3 cp ./invoice-44.pdf $BUCKET/invoice_pdf/44

# delete (DeleteObject, opt-in write)
aws --endpoint-url http://localhost:5000/s3 s3 rm $BUCKET/invoice_pdf/44

Uploads and chunked signing. The AWS CLI’s default PutObject streams the body with x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD (chunked signing), which this endpoint answers with a clean 501 NotImplemented — it only accepts a single-part body (a concrete payload hash, or UNSIGNED-PAYLOAD). Reads (ls, cp download, server-side copy) work with the default configuration; for uploads, use a client that sends a non-chunked body — rclone (below) does so by default.

rclone

# rclone.conf — placeholder credentials; do not commit real ones.
[bifrost]
type              = s3
provider          = Other
access_key_id     = AKIAEXAMPLE0000000000
secret_access_key = wJalrXUtnFEMIexampleKEYdonotuseREAL00000
region            = us-east-1
endpoint          = http://localhost:5000/s3
force_path_style  = true

# list
rclone lsf bifrost:orders/

# download (copy object out)
rclone copyto bifrost:orders/invoice_pdf/42 ./invoice-42.pdf

# upload (opt-in write)
rclone copyto ./invoice-44.pdf bifrost:orders/invoice_pdf/44

# server-side copy (opt-in write)
rclone copyto bifrost:orders/invoice_pdf/42 bifrost:orders/invoice_pdf/43

# delete (opt-in write)
rclone delete bifrost:orders/invoice_pdf/44

Supported operations

Operation	HTTP	Notes
ListBuckets	GET /s3/	Every table the caller can read that has a file column and a legal bucket name.
ListObjectsV2	GET /s3/{bucket}	Prefix, delimiter, max-keys, and continuation token. Legacy ListObjects v1 is a non-goal.
GetObject	GET /s3/{bucket}/{key}	Supports conditional headers (RFC 7232) and a single Range.
HeadObject	HEAD /s3/{bucket}/{key}	Same metadata as GetObject, no body.
PutObject	PUT /s3/{bucket}/{key}	Opt-in (EnableWrites). Single-part only.
CopyObject	PUT + x-amz-copy-source	Opt-in. Server-side copy; COPY/REPLACE metadata directive.
DeleteObject	DELETE /s3/{bucket}/{key}	Opt-in. Idempotent.

Everything outside this set — multipart, versioning, lifecycle, ACL/policy, bucket create/delete, and (when writes are off) every mutation — answers a clean, authenticated 501 NotImplemented. Auth is always enforced first, so an unsupported operation never reveals whether an object exists.

SigV4 modes and limits

• Two authentication modes: header authorization (Authorization: AWS4-HMAC-SHA256 …) and presigned query (X-Amz-Algorithm=AWS4-HMAC-SHA256 in the URL).
• Presigned URLs are GET/HEAD only. A presigned PUT/POST/DELETE is rejected with AccessDenied before the signature is checked — presigned writes are a non-goal, and because a presigned URL binds its method into the signature, allowing one would let a signed write authenticate on its own.
• Credential-scope region must equal S3Options.Region; any other region is rejected.
• Clock skew beyond MaxClockSkew (default 15 min) is rejected, on both the header and presigned paths.
• Presigned expiry (X-Amz-Expires) beyond MaxPresignedExpiry (default 7 days) is rejected before the signature is verified.
• SigV2 is not supported. Chunked STREAMING-AWS4-HMAC-SHA256-PAYLOAD uploads answer 501; a PutObject must send a single-part body — a concrete x-amz-content-sha256 (integrity-checked against the received bytes) or UNSIGNED-PAYLOAD (opts out of the body check; the signature still authenticated the request).

ETag semantics

An object’s ETag is the single-part MD5 the mutation pipeline persisted at write time, returned quoted ("…"). It is not a multipart composite ETag (no -N suffix), because multipart upload is a non-goal.

Range and conditional requests

Conditional preconditions (RFC 7232) are applied before any range handling. Only a single byte range is honored:

• A single satisfiable range → 206 Partial Content with Content-Range.
• An unsatisfiable range → 416.
• A multi-range request, or malformed/overflowing range syntax, is ignored and the whole object is served (200) — never a multipart/byteranges response, which is a non-goal. Accept-Ranges: bytes is always advertised.

Continuation tokens

ListObjectsV2 pages with an opaque, HMAC-integrity-protected continuation token bound to the bucket, prefix, delimiter, page size, and caller identity — a token minted for another bucket/prefix/identity, or tampered with, is rejected. With ContinuationTokenSecret set, tokens survive process restarts and resolve across a scaled fleet; unset, a per-instance random key is used, so an in-flight token is invalidated by a restart. Set the secret for any multi-instance or long-lived-pagination deployment. max-keys is clamped to MaxKeysLimit (default 1000), and a scan that would materialize more than MaxListMaterialize (default 10000) candidate rows fails fast rather than lying by truncation.

User metadata

PutObject persists x-amz-meta-* headers as user metadata, echoed back on GetObject/HeadObject. Every key and value must be printable US-ASCII (0x20–0x7E) — no control characters (CR/LF injection) and no non-ASCII that cannot survive the header round-trip — and the summed size is capped at MaxMetadataBytes (default 2 KiB). On CopyObject, the metadata directive is COPY (default: inherit the source’s content type and user metadata) or REPLACE (take them from the copy request). A self-copy (source == destination) is legal only under REPLACE.

Optional writes

PutObject, CopyObject, and DeleteObject are served only when EnableWrites is true. This is a dangerous opt-in capability, so it is fail-closed by construction: with the gate off, the write-method check is the first thing the handler does — before arity parsing, model lookup, or intent construction — so a disabled surface builds zero intent and cannot be probed for behavior. Enabling it flips the underlying FileObjectSeamOptions.EnableWrites, whose constructor logs the startup warning.

Security guarantees

Every object operation is an ordinary read or write through the shared pipeline — the S3 wire is a codec, never a second data path. Concretely:

• Row resolution before provider access. An object key is resolved to its owning row through the authorized read path first; only then is the storage provider touched. A caller who cannot read the row never reaches the blob.
• Writes route exclusively through the mutation pipeline. PutObject and DeleteObject go through IMutationIntentExecutor — never direct SQL, never a direct provider delete, and the adapter builds no predicate of its own. The pipeline applies tenant scoping, audit-actor stamping, soft-delete rewrite, field-encryption-on-write, and CDC/history hooks. DeleteObject routes the removal the operation actually means: it clears the row’s file pointer via the pipeline (an update), then reclaims the blob — it does not delete the row.
• Tenant isolation is structural. The adapter supplies only the object’s address (bucket + key → column + primary key) plus the session’s user context; the pipeline narrows scope from the identity. An out-of-tenant key therefore matches zero rows — “caller A cannot touch caller B’s object” holds because the predicate comes from the pipeline, not because the adapter remembered to filter.
• Path hardening by construction. The key encoder percent-encodes over [A-Za-z0-9_-], so a key can never carry a /, a ./.. segment, or an empty component — directory traversal is impossible by construction, not by a filter. On write, the object’s storage key is a fresh random key decoupled from the caller-derived address (the row’s pointer binds address → storage key), so a write’s compensating rollback can only ever remove what that same call created — a read-visible-but-write-denied caller can never overwrite or destroy another object’s bytes.
• Non-enumerating errors. A missing object, an object the caller may not read, a cross-tenant or forged address, and a malformed key all answer the same NoSuchKey (404) — never AccessDenied, which would confirm existence. On the write path, an internal PutObject failure maps to a sanitized 500; a Bifrost-internal exception message is never forwarded onto the wire (full detail is logged server-side only). DeleteObject is idempotent: a missing, forged, or scoped-away address all answer 204.

Deferred features and non-goals

The following are explicit non-goals and will answer 501 NotImplemented (or AccessDenied where noted) rather than partially work:

• Multipart upload (uploads/uploadId/partNumber, UploadPartCopy).
• Object versioning, lifecycle rules, replication.
• ACL / bucket-policy APIs, website hosting, bucket create/delete.
• Presigned writes (presigned PUT/POST/DELETE) — rejected with AccessDenied.
• SigV2 and chunked STREAMING-AWS4-HMAC-SHA256-PAYLOAD uploads.
• Legacy ListObjects v1 and multipart/byteranges multi-range responses.

For the architectural reasoning behind the one-pipeline / one-identity design that this endpoint inherits, see Protocol Adapters: One Pipeline, Many Front Doors.