On this page
Amazon S3 Tables launched at re:Invent on 3 December 2024 as a third kind of S3 bucket: the table bucket, which stores Apache Iceberg tables and maintains them for you. It went straight to general availability in three US Regions and reached thirty Regions by May 2025. The pitch was that teams running Iceberg on S3 could stop writing compaction and snapshot-expiry jobs and stop running a catalog, because the bucket does both.
Fourteen months later the product is noticeably different from the launch version. It gained an Iceberg REST endpoint, sort and z-order compaction, a 90% cut in compaction processing prices, Intelligent-Tiering, replication and Iceberg V3 deletion vectors. This article looks at it as of February 2026: what a table bucket is, what AWS maintains and what can go wrong, how engines reach the tables and which access controls apply on each path, what it costs, and when Iceberg in an ordinary bucket is still the better choice. The sample sets up tables and maintenance with boto3 and is tested against the service’s own API model.
What a table bucket is

A table bucket holds tables, grouped into namespaces, and every table is an Iceberg table. Each table gets its own warehouse location, an S3 prefix where engines write Parquet, Avro or ORC data files and Iceberg metadata. Engines use ordinary S3 calls for that, but only a subset: GET, HEAD, PUT and multipart uploads. There is no listing and no deleting objects; S3 removes files itself as part of maintenance. Permissions live in their own IAM namespace, s3tables, so s3:* policies do not grant access to tables.
In February 2026 the default quotas are 10 table buckets per Region per account and 10,000 tables per bucket, both adjustable. Names of namespaces, tables and columns should be lower case: upper-case names are invisible to the Glue Data Catalog and so to Athena and Redshift.
An existing general purpose bucket cannot become a table bucket, and Iceberg tables already in one cannot be registered in place. Moving a table to S3 Tables means rewriting its data into a new table. Plan table buckets as new resources, usually one per domain or environment, because encryption, maintenance defaults, policies and replication are set per bucket.
What changed in fourteen months
| Date | Change |
|---|---|
| 3 Dec 2024 | Launch in three US Regions: table buckets, automatic compaction, snapshot management and unreferenced file removal; Glue Data Catalog integration in preview |
| 30 Jan 2025 | Tables per bucket raised to 10,000 |
| 13 Mar 2025 | Iceberg REST catalog APIs on the S3 Tables endpoint; integration with AWS analytics services, through SageMaker Lakehouse, generally available |
| 24 Jun 2025 | Sort and z-order compaction |
| 1 Jul 2025 | Compaction prices cut: 50% per object, 90% per GB for binpack, 80% per GB for sort and z-order |
| 26 Nov 2025 | Iceberg V3 deletion vectors and row lineage |
| 2 Dec 2025 | Intelligent-Tiering storage class; replication across Regions and accounts |
The pattern is that S3 Tables has been closing the gaps that kept teams on self-managed Iceberg. Replication in particular removes the need for custom cross-Region copy jobs, which had to copy data and metadata in the right order.
Creating tables and setting maintenance
You can create tables from an engine, with CREATE TABLE in Athena or Spark, or with the S3 Tables API. The API route is useful for platform teams that create tables from code, review them in pull requests and apply the same maintenance settings everywhere. create_table accepts the Iceberg schema, partition spec and sort order. Maintenance is a separate call per setting, not a field on create_table:
"""Create Amazon S3 Tables resources idempotently and set table maintenance.
The functions take a boto3 ``s3tables`` client so they can be tested with
botocore's Stubber and reused from scripts, Lambda functions or CI jobs.
Defaults follow the S3 Tables documentation as of February 2026:
compaction target file size 64-512 MB (default 512), snapshot retention
default 1 snapshot / 120 hours, unreferenced file removal 3 + 10 days.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass, field
from typing import Any
from botocore.exceptions import ClientError
log = logging.getLogger(__name__)
COMPACTION_STRATEGIES = {"auto", "binpack", "sort", "z-order"}
RISKY_PROPERTIES = (
"history.expire.max-snapshot-age-ms",
"history.expire.min-snapshots-to-keep",
)
@dataclass(frozen=True)
class Column:
id: int
name: str
type: str # Iceberg type string, e.g. "long", "decimal(18,2)", "timestamptz"
required: bool = False
@dataclass(frozen=True)
class TableSpec:
namespace: str
name: str
columns: list[Column]
partition: list[tuple[str, str]] = field(default_factory=list) # (column, transform)
sort: list[str] = field(default_factory=list) # columns, ascending
storage_class: str = "STANDARD"
def __post_init__(self) -> None:
for label in (self.namespace, self.name, *(c.name for c in self.columns)):
if label != label.lower():
# Upper-case names are invisible to Glue Data Catalog and Athena.
raise ValueError(f"use lower-case names for S3 Tables: {label!r}")
ids = [c.id for c in self.columns]
if len(ids) != len(set(ids)):
raise ValueError("column ids must be unique")
def _id(self, column: str) -> int:
for c in self.columns:
if c.name == column:
return c.id
raise ValueError(f"unknown column {column!r}")
def create_table_kwargs(self, bucket_arn: str) -> dict[str, Any]:
iceberg: dict[str, Any] = {
"schema": {"fields": [
{"id": c.id, "name": c.name, "type": c.type, "required": c.required}
for c in self.columns]},
}
if self.partition:
iceberg["partitionSpec"] = {"fields": [
{"sourceId": self._id(col), "transform": t, "name": f"{col}_{t.split('[')[0]}"}
for col, t in self.partition]}
if self.sort:
iceberg["writeOrder"] = {"orderId": 1, "fields": [
{"sourceId": self._id(col), "transform": "identity",
"direction": "asc", "nullOrder": "nulls-last"} for col in self.sort]}
return {
"tableBucketARN": bucket_arn,
"namespace": self.namespace,
"name": self.name,
"format": "ICEBERG",
"metadata": {"iceberg": iceberg},
"storageClassConfiguration": {"storageClass": self.storage_class},
}
def _is(err: ClientError, code: str) -> bool:
return err.response.get("Error", {}).get("Code") == code
def ensure_table_bucket(client: Any, name: str, kms_key_arn: str | None = None) -> str:
"""Create the table bucket, or return the ARN of the existing one."""
kwargs: dict[str, Any] = {"name": name}
if kms_key_arn:
kwargs["encryptionConfiguration"] = {"sseAlgorithm": "aws:kms", "kmsKeyArn": kms_key_arn}
try:
return client.create_table_bucket(**kwargs)["arn"]
except ClientError as err:
if not _is(err, "ConflictException"):
raise
for page in client.get_paginator("list_table_buckets").paginate(prefix=name):
for bucket in page["tableBuckets"]:
if bucket["name"] == name:
return bucket["arn"]
raise RuntimeError(f"table bucket {name} exists but is not visible to this caller")
def ensure_namespace(client: Any, bucket_arn: str, namespace: str) -> None:
try:
client.create_namespace(tableBucketARN=bucket_arn, namespace=[namespace])
except ClientError as err:
if not _is(err, "ConflictException"):
raise
log.info("namespace %s already exists", namespace)
def ensure_table(client: Any, bucket_arn: str, spec: TableSpec) -> str:
"""Create the table with its schema, partitioning and sort order; return its ARN."""
try:
return client.create_table(**spec.create_table_kwargs(bucket_arn))["tableARN"]
except ClientError as err:
if not _is(err, "ConflictException"):
raise
return client.get_table(tableBucketARN=bucket_arn, namespace=spec.namespace,
name=spec.name)["tableARN"]
def configure_maintenance(client: Any, bucket_arn: str, spec: TableSpec, *,
target_file_mb: int = 512, strategy: str = "auto",
min_snapshots: int = 1, max_snapshot_age_hours: int = 120) -> None:
"""Set compaction and snapshot management for one table."""
if not 64 <= target_file_mb <= 512:
raise ValueError("targetFileSizeMB must be between 64 and 512")
if strategy not in COMPACTION_STRATEGIES:
raise ValueError(f"strategy must be one of {sorted(COMPACTION_STRATEGIES)}")
if strategy in {"sort", "z-order"} and not spec.sort:
raise ValueError(f"{strategy} compaction needs a sort order on the table")
where = {"tableBucketARN": bucket_arn, "namespace": spec.namespace, "name": spec.name}
client.put_table_maintenance_configuration(
**where, type="icebergCompaction",
value={"status": "enabled", "settings": {"icebergCompaction": {
"targetFileSizeMB": target_file_mb, "strategy": strategy}}})
client.put_table_maintenance_configuration(
**where, type="icebergSnapshotManagement",
value={"status": "enabled", "settings": {"icebergSnapshotManagement": {
"minSnapshotsToKeep": min_snapshots,
"maxSnapshotAgeHours": max_snapshot_age_hours}}})
def configure_unreferenced_file_removal(client: Any, bucket_arn: str, *,
unreferenced_days: int = 3,
noncurrent_days: int = 10) -> None:
"""Bucket-level setting. Noncurrent files are deleted permanently after noncurrent_days."""
client.put_table_bucket_maintenance_configuration(
tableBucketARN=bucket_arn, type="icebergUnreferencedFileRemoval",
value={"status": "enabled", "settings": {"icebergUnreferencedFileRemoval": {
"unreferencedDays": unreferenced_days, "nonCurrentDays": noncurrent_days}}})
def snapshot_management_blockers(properties: dict[str, str], refs: dict[str, Any]) -> list[str]:
"""Return reasons S3 Tables snapshot management would fail for a table.
``properties`` and ``refs`` come from the table's metadata.json. Any tag or
branch other than ``main``, or either history.expire property, makes the
managed snapshot expiry fail for the whole table.
"""
problems = [f"table property {p} is set" for p in RISKY_PROPERTIES if p in properties]
problems += [f"{ref['type']} {name!r} exists" for name, ref in refs.items() if name != "main"]
return problems
def pyiceberg_properties(region: str, account_id: str, bucket: str) -> dict[str, str]:
"""Catalog properties for PyIceberg against the S3 Tables Iceberg REST endpoint."""
return {
"type": "rest",
"warehouse": f"arn:aws:s3tables:{region}:{account_id}:bucket/{bucket}",
"uri": f"https://s3tables.{region}.amazonaws.com/iceberg",
"rest.sigv4-enabled": "true",
"rest.signing-name": "s3tables",
"rest.signing-region": region,
}
if __name__ == "__main__":
import boto3
logging.basicConfig(level=logging.INFO)
s3tables = boto3.client("s3tables")
payments = TableSpec(
namespace="finance",
name="payments",
columns=[
Column(1, "payment_id", "string", required=True),
Column(2, "account_id", "string", required=True),
Column(3, "amount", "decimal(18,2)", required=True),
Column(4, "currency", "string"),
Column(5, "status", "string"),
Column(6, "created_at", "timestamptz", required=True),
],
partition=[("created_at", "day")],
sort=["account_id"],
)
arn = ensure_table_bucket(s3tables, "finance-lakehouse")
ensure_namespace(s3tables, arn, payments.namespace)
print(ensure_table(s3tables, arn, payments))
configure_maintenance(s3tables, arn, payments, target_file_mb=256)
configure_unreferenced_file_removal(s3tables, arn)Three settings matter in practice. Compaction targets 512 MB files by default and accepts 64 to 512 MB; smaller targets suit tables that are read with selective filters soon after they are written. The auto strategy uses sort compaction when the table has a sort order and binpack otherwise; sort and z-order cost more per GB processed and need a sort order on the table. Snapshot management keeps at least one snapshot and expires snapshots older than 120 hours by default, which is also your time-travel window. Unreferenced file removal is set per bucket: files no table snapshot references are marked noncurrent after 3 days and deleted after another 10.
Managed snapshot expiry fails for the whole table, and no snapshots expire, if the table has any tag or branch other than main, or if history.expire.max-snapshot-age-ms or history.expire.min-snapshots-to-keep is set as a table property, whatever its value. Tables migrated from self-managed Iceberg often carry those properties, and audit tags are common. Metadata then grows until queries slow down. Check with GetTableMaintenanceJobStatus; the sample’s snapshot_management_blockers checks a table’s metadata before you migrate it.
Two ways in, two security models
Engines reach S3 Tables in one of two ways, and the difference matters more than any other design decision.
The S3 Tables Iceberg REST endpoint, https://s3tables.<region>.amazonaws.com/iceberg, speaks the standard Iceberg REST catalog protocol with SigV4 signing (service name s3tables). Spark, PyIceberg, Trino and other Iceberg clients connect to it with the table bucket ARN as the warehouse. Each request is authorised by the matching s3tables IAM action, so access is granted per bucket, namespace or table. There are no column or row filters on this path. AWS positions it for basic read and write access to a single table bucket and for partner or custom catalogs. It does not support CREATE TABLE AS SELECT, views or multi-level namespaces, and dropping a table requires purge.
The Glue Data Catalog integration adds a federated catalog called s3tablescatalog to the Data Catalog, with each table bucket as a child catalog, each namespace as a database and each table as a table. In February 2026 the integration also registers the buckets with Lake Formation, so Lake Formation grants control access, including column-level and row-level permissions. Athena, Redshift, EMR and QuickSight use this path. In Athena a table is addressed as "s3tablescatalog/finance-lakehouse".finance.payments, with the quotes because of the slash. Other engines can use the Glue Iceberg REST endpoint to get the same governance.
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#FFF3E0', 'primaryTextColor': '#232F3E', 'primaryBorderColor': '#FF9900', 'lineColor': '#232F3E', 'clusterBkg': '#F8FAFC', 'clusterBorder': '#CBD5E1', 'edgeLabelBackground': '#FFFFFF', 'nodeTextColor': '#232F3E'}}}%%
flowchart TD
A["Who needs to read<br/>the table?"] --> B{"Restricted columns<br/>or rows?"}
B -- "Yes" --> C["Glue Data Catalog path<br/>with Lake Formation grants"]
B -- "No" --> D{"AWS analytics service<br/>or Iceberg client?"}
D -- "Athena, Redshift,<br/>EMR, QuickSight" --> C
D -- "Spark, PyIceberg,<br/>Trino" --> E{"Single table bucket,<br/>whole-table access?"}
E -- "Yes" --> F["S3 Tables Iceberg<br/>REST endpoint with IAM"]
E -- "No" --> G["Glue Iceberg<br/>REST endpoint"]
F --> H["Grant s3tables actions<br/>to service roles only"]
classDef aws fill:#FFF3E0,stroke:#FF9900,stroke-width:2px,color:#232F3E
classDef db fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74
classDef warn fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B
class A,D,E aws
class C,G db
class F,H warn
class B awsLake Formation column and row permissions protect regulated fields such as health identifiers or card data only for principals that come through the Data Catalog path. A principal with s3tables:GetTableData on the table can read every column through the S3 Tables REST endpoint or straight from the warehouse location. Keep those permissions to the service roles that write or compact data, deny them to analysts in IAM or service control policies, and encrypt the bucket with a customer-managed KMS key whose policy names the same roles.
Cost
S3 Tables adds charges that ordinary S3 does not have. Storage costs more than S3 Standard, there is a monitoring charge per object for maintenance, and compaction is charged per object and per GB it processes. The sample’s cost model reproduces AWS’s published example, a 1 TB table with 100 MB objects in US West (Oregon), after the July 2025 compaction price cut:
| Charge | Rate | Example quantity | Monthly cost |
|---|---|---|---|
| Storage | $0.0265 per GB-month (first 50 TB) | 1,024 GB | $27.14 |
| PUT requests | $0.005 per 1,000 | 30,000 | $0.15 |
| GET requests | $0.0004 per 1,000 | 500,000 | $0.20 |
| Object monitoring | $0.025 per 1,000 objects | 10,486 objects | $0.26 |
| Compaction, objects | $0.002 per 1,000 objects | 30,000 objects | $0.06 |
| Compaction, data (binpack) | $0.005 per GB | 146.48 GB | $0.73 |
| Total | $28.54 |
Worked cell: 1,024 GB × $0.0265 = $27.14. In this example maintenance is about 4% of the bill and storage is the rest, so the storage rate matters more than the compaction rate. Tables with frequent small writes compact more data and have more objects to monitor, and sort or z-order compaction costs more per GB. Run the model with your own object counts and write volume. The alternative is not free either: Glue Data Catalog optimizers for Iceberg tables in general purpose buckets cost $0.44 per DPU-hour, and self-managed compaction costs the EMR or Glue compute you run plus the engineering time to keep it working. Intelligent-Tiering, available since December 2025, lowers the storage rate for data that is not read for 30 days or more.
"""Monthly cost estimate for one S3 table, using S3 Tables price dimensions.
Rates default to AWS's published S3 Tables example for US West (Oregon) after
the July 2025 compaction price cut; pass a Rates object for other Regions.
Binpack compaction rates only: sort and z-order cost more per GB.
"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass(frozen=True)
class Rates:
storage_gb_month: float = 0.0265 # first 50 TB, Standard
put_per_1000: float = 0.005
get_per_1000: float = 0.0004
monitoring_per_1000_objects: float = 0.025
compaction_per_1000_objects: float = 0.002
compaction_per_gb: float = 0.005
@dataclass(frozen=True)
class Workload:
storage_gb: float
objects: int
puts: int
gets: int
compaction_objects: int
compaction_gb: float
def monthly_cost(w: Workload, r: Rates = Rates()) -> dict[str, float]:
lines = {
"storage": w.storage_gb * r.storage_gb_month,
"put_requests": w.puts / 1000 * r.put_per_1000,
"get_requests": w.gets / 1000 * r.get_per_1000,
"monitoring": w.objects / 1000 * r.monitoring_per_1000_objects,
"compaction_objects": w.compaction_objects / 1000 * r.compaction_per_1000_objects,
"compaction_data": w.compaction_gb * r.compaction_per_gb,
}
lines = {k: round(v, 2) for k, v in lines.items()}
lines["total"] = round(sum(lines.values()), 2)
return linesWhen Iceberg in a general purpose bucket still wins

S3 Tables is not the only managed option. Since 2024 the Glue Data Catalog can compact, expire snapshots and delete orphan files for Iceberg tables in ordinary buckets, and since June 2025 it supports sort and z-order too. Keep data in general purpose buckets when one of these applies:
- Delta Lake or Hudi. Table buckets hold Iceberg only. Converting is a data rewrite and a change for every job that writes the table.
- Tools that need the full S3 API. Anything that lists, copies or deletes data files directly, such as backup tools, S3 Batch Operations jobs or lifecycle rules, does not work against a table bucket.
- An established Lake Formation estate. If tables are already governed in the Data Catalog and compacted by Glue optimizers, moving them buys little and costs a full rewrite.
- Many buckets per Region. With 10 table buckets per Region by default, designs that want one bucket per team or tenant need a quota increase or a different layout.
Choose S3 Tables for new Iceberg tables, for teams that do not want to run maintenance at all, and for workloads with heavy small-file churn such as streaming ingestion, where managed compaction pays for itself fastest.
Migrating an existing Iceberg table
There is no in-place conversion, so migration is a copy. Create the target table with the same schema, partition spec and sort order, using the API as in the sample or CREATE TABLE through the Data Catalog path, then copy the data with Spark or with Athena INSERT INTO ... SELECT. Athena can also do it in one step with CREATE TABLE AS SELECT on the Data Catalog path; the S3 Tables REST endpoint cannot. For large tables, copy historical partitions first, run the writers against both tables for a period, compare row counts and aggregates per partition, then switch readers. Remove history.expire.* properties and unneeded tags before the copy so managed snapshot expiry works from day one. Snapshot history does not carry over, so keep the old table for as long as you need its time travel.
Before moving shared tables, set up the integration and test every consumer: each Athena workgroup, Redshift cluster, EMR role and cross-account principal. Lake Formation grants on S3 Tables use the account-qualified catalog name, such as 111122223333:s3tablescatalog/finance-lakehouse, and an Athena workgroup with SSE-KMS or CSE-KMS result encryption cannot run INSERT, UPDATE, DELETE or MERGE against S3 Tables.
s3tables_setup.py and cost_model.py were tested with Python 3.13, boto3==1.42.50 and pytest==9.0.2: fourteen tests run every request through botocore’s Stubber, which validates parameters and responses against the S3 Tables API model in that release, and one test confirms the model rejects a maintenanceConfig argument to create_table. The cost test reproduces AWS’s published example to the cent. Nothing was run against a live AWS account. Dates, quotas and rates are from AWS announcements, the S3 Tables user guide and the S3 pricing page.
Key takeaways
- S3 Tables stores Iceberg tables in a table bucket and runs compaction, snapshot expiry and unreferenced file removal for you, with defaults you can tune per table and per bucket.
- Create tables with their schema, partitioning and sort order, then set maintenance with separate API calls; keep names lower case.
- Managed snapshot expiry stops for a table that has tags, extra branches or
history.expire.*properties. Check for them before migrating. - Lake Formation column and row permissions apply on the Glue Data Catalog path only. Keep
s3tablesdata permissions to service roles. - Storage dominates the bill for most tables; compaction became much cheaper in July 2025.
- Stay on general purpose buckets for Delta Lake or Hudi, tools that need the full S3 API, or an Iceberg estate already maintained by Glue optimizers.
Are you running S3 Tables in production, and which access path do your engines use? Share it in the comments.
The complete sample for this article is on GitHub: nithinmohantk/blog-samples-working-repo/s3tables-starter. Clone it with git clone https://github.com/nithinmohantk/blog-samples-working-repo.git and open the s3tables-starter folder. Its README says how to run it and what has been verified.
Glossary
- Table bucket
- An S3 bucket type that stores Apache Iceberg tables as subresources and maintains them.
- Apache Iceberg
- An open table format that adds snapshots, schema evolution and ACID commits to data files in object storage.
- Compaction
- Rewriting many small data files into fewer large ones, optionally sorted or z-ordered.
- Snapshot
- A version of an Iceberg table; expiring old snapshots limits time travel and lets old files be removed.
- Iceberg REST catalog
- A standard HTTP API through which engines create, find and commit Iceberg tables.
- s3tablescatalog
- The federated Glue Data Catalog catalog that exposes table buckets to AWS analytics services.
- Lake Formation
- The AWS service that grants database, table, column and row permissions on Data Catalog resources.
- Z-order
- A layout that interleaves several columns’ values so filters on any of them skip more files.
References & Further Reading
- → Announcing Amazon S3 Tables— Launch announcement, 3 December 2024
- → Working with Amazon S3 Tables and table buckets— User guide overview
- → Creating an Amazon S3 table— Schema and partition spec at creation
- → Maintenance for tables— Compaction strategies, snapshot management and failure conditions
- → Maintenance for table buckets— Unreferenced file removal defaults
- → Accessing tables using the S3 Tables Iceberg REST endpoint— Configuration, considerations and limitations
- → Integrating S3 Tables with AWS analytics services— The s3tablescatalog federated catalog
- → S3 Tables integration with Data Catalog and Lake Formation— Coarse- and fine-grained access control
- → Querying S3 Tables with Athena— Catalog naming and limitations
- → S3 Tables quotas— Buckets, namespaces and tables
- → Amazon S3 Tables reduce compaction costs by up to 90%— July 2025 price change
- → Replication support and Intelligent-Tiering for Amazon S3 Tables— December 2025 launches
- → Amazon S3 pricing— S3 Tables rates and worked example
- → AWS Glue pricing— Iceberg table optimization at $0.44 per DPU-hour
- → Sample code for this article— Table setup, maintenance, cost model, tests
Discover more from C4: Container, Code, Cloud & Context
Subscribe to get the latest posts sent to your email.