On May 23, 2026, InfoQ published Google Cloud Introduces Cross-Engine Iceberg Support in BigQuery. With BigQuery launching a public preview of its serverless REST catalog for Apache Iceberg, users can now share the exact same tables with Spark, Flink, Trino, and Presto under an identical schema and identical transaction boundaries. This marks a turning point that resolves the long-standing dichotomy of "BigQuery proprietary data versus open lakehouses," unifying them into a world where "all engines, including BigQuery, read and write to the same Iceberg tables."
From the perspective of supporting data platforms for mid-sized enterprises through custom development, this means a solution has arrived that reduces vendor lock-in while leveraging existing BigQuery assets to solve typical challenges like "data locked inside BigQuery" and "data fragmented across multiple warehouses." Following the local lakehouse for SMBs covered in DuckLake 1.0 SMB Data Lakehouse Development for Clients and the dbt governance covered in Monzo dbt Governance Data Mesh Development for Clients, this article organizes architecture development for enterprise multi-engine interoperability.
Why "BigQuery × Iceberg" is a Watershed
| Dimension | Traditional Architecture (BigQuery Proprietary + Separate Lake) | BigQuery × Iceberg Integration |
|---|---|---|
| Data Ownership Model | BigQuery Native | Open Format (Parquet + Iceberg) |
| Query Engines | BigQuery only | Cross-engine across BigQuery / Spark / Flink / Trino |
| Duplicate Storage | Duplicated across DWH + Data Lake | Single physical store |
| ETL Duplication | Two separate pipelines for DWH / Lake | Unified into one |
| Transaction Boundaries | Engine-level | Unified across engines via Iceberg snapshots |
| Vendor lock-in | High (difficult migration) | Low (standardized catalog + storage) |
| Cost structure | DWH storage + compute | Standalone storage + billing per engine |
In other words, BigQuery × Iceberg is an option that structurally reduces vendor lock-in by allowing organizations to "open up data ownership while maintaining BigQuery's ease of use."
Three structural changes beneficial to custom development projects
Structure 1: From "Dual ETL Hell" to a "Single Iceberg Table"
Many mid-sized companies have run duplicate, dual-track operations with ETL for the DWH (BigQuery) and ETL for the data lake (GCS Parquet). With cross-engine Iceberg support, these can be consolidated into unified ETL where "writing once allows all engines to read," structurally cutting pipeline maintenance overhead in half. This step brings the pipeline governance via dbt covered in Monzo dbt Governance Data Mesh Development for Clients to unification at the physical data layer as well.
Structure 2: From "DWH Lock-In" to "Data Portability"
Against the issue where data is effectively locked in regardless of choosing BigQuery, Snowflake, Redshift, or Databricks, Iceberg offers the choice to switch engines later while keeping data stored in GCS, S3, or Azure. This structurally alters long-term TCO and vendor bargaining power, granting mid-sized companies a dramatic advantage in multi-year contract negotiations.
Structure 3: From "Split Spark/Flink and DWH Teams" to "Collaboration on the Same Data"
Many organizations have had data engineering teams (Spark/Flink) and analytics teams (BigQuery/Looker) working across separate copies of data. Cross-engine Iceberg enables "both teams to read and write to the same tables," dramatically accelerating collaboration velocity with workflows like "streaming processing results reflecting immediately in BI" and "missing data identified in BI being promptly remediated via Spark."
5 Phases of the "BigQuery × Iceberg Lakehouse Platform" Provided via Custom Development
Phase 1: Current state assessment (3 weeks)
- Existing DWH / data lake inventory (BigQuery / Snowflake / GCS / S3)
- ETL pipeline inventory (dbt / Airflow / Dataflow)
- Estimation of data duplication rate / dual-storage costs
- Query engine usage audit (BigQuery / Spark / Trino)
- Prioritization of candidate tables for Iceberg migration
Phase 2: Architecture design (2–3 weeks)
- Iceberg catalog design (comparison of BigQuery REST catalog / Polaris / Nessie)
- Storage architecture design (GCS bucket structure + partitioning strategy)
- Schema evolution policy
- Access control (IAM + data masking)
- Migration priorities + parallel operation period
Phase 3: PoC build (3–4 weeks)
- Iceberg migration for 5–10 representative tables
- Read/write validation from BigQuery / Spark / Trino
- Performance benchmarking (query latency / costs)
- Snapshot / Time Travel operational verification
- Preparation of migration runbooks
Phase 4: Production deployment (4–8 weeks)
- Phased migration by domain (e.g., Marketing → Sales → Finance)
- Operational rules during the dual-storage period
- Migrating dbt / Airflow pipelines to Iceberg
- Rollout of access controls + data masking
- Training for BI teams and analysts
Phase 5: Monthly operational reviews (ongoing)
- Tracking storage cost and query cost trends
- Review and refinement of partitioning strategies
- Schema evolution auditing
- Evaluation of new use case additions
- Tracking BigQuery and Iceberg version updates
Standard technology stack set for custom development
| Layer | Recommended technology | Alternative |
|---|---|---|
| Table Format | Apache Iceberg | Delta Lake / Hudi |
| Catalog | BigQuery REST Catalog / Polaris | Hive Metastore / Unity Catalog |
| Storage | GCS (region/class selection) | S3 / Azure ADLS |
| DWH Engine | BigQuery | Snowflake (Iceberg integration) |
| Batch Processing | Spark on Dataproc / Glue | Databricks |
| Streaming | Flink / Dataflow | Kafka Streams |
| Orchestration | Airflow / Dagster | Argo Workflows |
| dbt / Semantic Layer | dbt Cloud / dbt Core | Cube / MetricFlow |
| BI | Looker / Tableau / Metabase | Power BI / Superset |
Which projects need this and which do not
| Projects requiring this | Projects not requiring this |
|---|---|
| Duplicate data storage across BigQuery + GCS | Self-contained within BigQuery alone |
| Spark / Flink teams and DWH teams are divided | Handled entirely within a single team |
| Annual DWH costs exceeding 30 million yen | On the scale of several hundred thousand yen per month |
| Vendor lock-in viewed as an executive challenge | Committed to a single-cloud strategy |
| Considering future hybrid use with Snowflake / Databricks | Fixed to a single engine |
Six clauses to include in client contracts
| Clause | Details | What the client should verify |
|---|---|---|
| Data Ownership | Ownership of GCS buckets / catalog | Handover upon contract completion |
| Migration scope | Number of tables / domains | Business impact level |
| Parallel operation period | Dual-storage period + cutover date | Rollback plan |
| Query Performance SLA | p95 latency targets | Operational requirement |
| Cost caps | Monthly caps for storage and queries | Budget control |
| Handover Upon Project Completion | Iceberg metadata + dbt + documentation | Internal operational continuity |
Client-Side ROI Estimate (Assuming 48 Million Yen Annual BigQuery Costs / 12 Million Yen Separate GCS Storage)
| Item | Existing (DWH + Data Lake Dual Storage) | BigQuery × Iceberg Integration | Difference |
|---|---|---|---|
| Duplicate Storage Costs | 12 million yen / year | 2 million JPY / year | -10 million yen |
| Dual-Pipeline ETL Maintenance Hours | 1,800 h / year | 700 h / year | -1,100h |
| Proprietary DWH Costs (Queries) | 48 million yen / year | 30 million yen / year | -18 million yen |
| Data Team Collaboration Hours | 900 h / year | 300 h / year | -600h |
| Scope for Vendor Renegotiation | Weak | High | Improved future contracts |
| Annual benefit | — | — | Equivalent to ~37 million yen + strategic flexibility |
Even calculated at an hourly rate of 8,000 yen, net annual savings exceeding 31 million yen can be expected. However, the costs required for migration implementation and operations will vary significantly depending on the number of target tables, existing ETL complexity, and the duration of parallel operations. When assessing the payback period, please calculate an individualized estimate matching this projected value against your organization's migration scale.
Five common pitfalls
Pitfall 1: Attempting an All-at-Once Migration of All Tables
Attempting to "turn everything into Iceberg at once" broadens the production impact scope too much, causing incident rates to spike. Proceed with a phased migration divided by domain and priority over a 3- to 6-month plan.
Pitfall 2: Putting Off Partitioning Strategy
In Iceberg, getting partition design wrong results in the same issues as the Hive era, causing query costs to skyrocket. Always finalize partition keys and bucket counts during the migration design phase.
Pitfall 3: Reusing dbt Models Without Redesigning Them
If you simply redirect existing dbt models to point directly to Iceberg tables, materialization strategies will fail to match engine characteristics, degrading performance. Plan a redesign on a per-dbt-model basis.
Pitfall 4: Prolonging the Dual-Storage Period
Adopting an approach of "keeping native BigQuery for the time being just in case" leads to doubled costs and consistency breakdown incidents. Include a contractual constraint capping parallel operations at a maximum of 3 months.
Pitfall 5: Neglecting the Catalog
Treating Iceberg as merely a table format and putting off catalog selection leads to a dead end in schema evolution, access control, and multi-engine consistency. Treat the catalog as an executive-level decision from the outset.
90-day action plan
| Week | Action |
|---|---|
| Week 1〜3 | DWH / lake audit + ETL audit |
| Week 4〜6 | Architecture design + catalog selection |
| Week 7〜10 | PoC implementation + migration of 5–10 tables |
| Week 11 | Testing Iceberg integration for dbt pipelines |
| Week 12 | Phase 1 domain migration (Marketing, etc.) |
| Week 13 | Launch of monthly operational review meetings |
Conclusion — The Era of Getting Both "BigQuery Usability + Open Format Freedom"
With the arrival of BigQuery × Iceberg, we have entered an era where organizations can simultaneously achieve "BigQuery's operational simplicity" and "open format portability." From the standpoint of supporting data platforms for mid-sized enterprises through custom development, "BigQuery × Iceberg Lakehouse Platforms"—integrating Iceberg catalog design, phased migration, dbt redesign, and monthly operations—will become a major new service.
Data infrastructure challenges such as "storing duplicate data across BigQuery and GCS," "considering hybrid use of Snowflake / Databricks," and "wanting to structurally reduce vendor lock-in" will vary greatly in approach depending on the number of target tables and existing ETL configurations. We provide customized quotes for Iceberg catalog design and phased migration approaches after assessing your current situation. Please feel free to reach out via our contact form.
