Building Data Infrastructure for SMBs with DuckLake 1.0 — A New Model for Custom Data Lakehouses 2026

In May 2026, InfoQ published DuckLake 1.0: Data Lake Format with SQL Catalog Metadata, marking the official general availability of the "SQL catalog × Parquet" data lake format championed by the DuckDB team. Amid the rivalry among Iceberg, Delta Lake, and Hudi, DuckLake is drawing attention as a "just right data infrastructure" for SMBs thanks to its pragmatic design choice to "manage catalogs directly in SQL."

At our company, cases where we build small-scale data lakes as "an intermediate layer between BI dashboards and business systems" in custom development are surging. In this article, we outline design guidelines for adopting DuckLake 1.0 in custom development and the migration steps from existing stacks.

Why SMBs need "yet another" data lake

Data infrastructure for SMBs has long been a choice between the following two options:

Between the two, the need to handle "hundreds of gigabytes to several terabytes of semi-structured logs" is growing rapidly. The causes are clear: the volume of data generated and analyzed by AI has exploded, and the practice of accumulating data dumped daily from SaaS as CSV / JSON has become standard.

Here, DuckLake 1.0 has stepped into the gap with a minimal footprint of "S3 / R2 + Parquet + SQL catalog." Amid the trend toward "AI-native data lakehouses" discussed in Integrated Data Infrastructure Powered by Google Agentic Data Cloud, this is positioned as a pragmatic stepping stone for SMBs before jumping straight into Google-scale implementations.

The characteristics of DuckLake 1.0: Comparison with Iceberg and Delta

Here is a comparison of data lake format options from the perspective of adopting them in custom development.

The biggest highlight of DuckLake is that "there is no need to run a catalog server as a separate process." Iceberg requires running AWS Glue or Polaris separately, which posed a psychological and operational hurdle for SMBs. DuckLake "only requires standing up a single PostgreSQL instance," dramatically lowering operational overhead.

Three architectural patterns to adopt in custom development

Here are the DuckLake architectures categorized by project scale that we have standardized at our company.

Pattern 1: Single-node model (up to 500 GB)

[Source: SaaS / 業務システム]
  └ 日次バッチで Parquet を S3/R2 に出力

[Catalog: PostgreSQL（既存の業務 DB に同居）]

[Engine: DuckDB（BI サーバー上）]
  ├ Metabase / Superset で接続
  └ 月次レポートは Notebook で生成

The minimal setup. Its strength lies in being deployable with "virtually zero" additional infrastructure, fitting SMB requirements to "start small first."

Pattern 2: Distributed-read model (500 GB to several TB)

[Source: 各種 → S3/R2（Parquet）]

[Catalog: PostgreSQL（専用 RDS / Cloud SQL）]

[Engine: DuckDB on 複数ノード]
  ├ ETL: バッチワーカー（Cloud Run / ECS）
  ├ BI: 分析ノード（独立 VM）
  └ 業務 API: Read Replica 経由

Writes are consolidated on the ETL worker side, while reads are distributed across multiple nodes. Because DuckLake's catalog guarantees transactions, conflicts where "one side reads while the other writes" are handled safely.

Pattern 3: Phased migration model (from existing BigQuery / Snowflake)

[既存 DWH: BigQuery / Snowflake]
  └ 1 年以上のコールドデータを Parquet で S3/R2 に export

[新規層: DuckLake]
  └ コールドデータの長期保管 + 分析用クエリ層
  └ ホットデータは引き続き既存 DWH

[BI 層]
  └ Metabase / Looker Studio で両方を参照

This pattern is suitable for requirements like "cooling data older than the last three months to lower DWH costs." Because cold data can be shifted incrementally to DuckLake while keeping hot data in the existing DWH, there is no need for an all-at-once migration. Cost-reduction benefits vary significantly depending on data volume, query frequency, and the existing DWH's pricing model, so we calculate estimates using real data during the Step 2 PoC (described below).

Caveats when adopting in custom development: Areas where "DuckLake alone is not self-contained"

DuckLake is powerful, but using it directly for enterprise use cases requires supplementing it with peripheral tooling.

In particular, permission management is a common stumbling block in custom development; requirements like "giving client executives access to all data while field staff see only their department" cannot be met by DuckLake alone. Always insert a wrapper that operates via "extracted views and business APIs." This aligns with the "business data permission separation" design covered in Internal AI Assistants for SMBs, and the principle remains the same even when DuckLake sits at the data tier.

Migration steps from existing stacks: Six steps

Here are our standard migration steps when handling custom projects.

The key is to unconditionally allocate one month for parallel operations, during which query result diff testing is conducted across all reports. This aligns with the philosophy of "running regression tests on both old and new" discussed in Vitest 4.1 AI Agent Reporter, serving as insurance to prevent clients from claiming that "the numbers changed" after migration.

Four common pitfalls

Finally, here are common pitfalls frequently encountered when using DuckLake in custom development.

Pitfall 1: Misestimating catalog DB capacity

If metadata volume on PostgreSQL is not calculated straightforwardly as "tables × files × columns," it can end up 5 to 10 times larger than expected. Establish an operational practice of forecasting six months ahead during initial setup to estimate capacity.

Pitfall 2: Parquet files that are too small

Running ETL jobs in fine intervals such as every minute leads to massive numbers of tiny Parquet files, slowing down queries. Incorporate a compaction process that merges small files on a daily or hourly basis.

Pitfall 3: BI tools lacking native connector support

While Metabase and Superset offer native DuckDB support, tools like Looker Studio must connect via Trino or DuckDB APIs. Always validate connectors in the early stages of a project.

Pitfall 4: Client requests for additional data sources

Requests like "please add Salesforce and kintone too" often arise post-release. Avoid disputes by structuring additional source integrations on a unit basis, such as "a set price per data stream," and making the conditions for incurring additional costs transparent during the contract phase.

Summary: The third option of "just right" data infrastructure

DuckLake 1.0 offers an exceptionally lightweight operational alternative for SMBs facing the dilemma where "BigQuery and Snowflake are too heavy, yet PostgreSQL has reached its limits." The pragmatic choice to avoid operating a separate catalog server delivers substantial implementation cost savings in custom development.

If you are interested in "centralizing CSV files scattered across your company" or "reassessing your DWH costs," please feel free to reach out via our contact form. Because the optimal architecture and effort depend on data volume, update frequency, and your existing stack, we provide individualized quotes once we review your requirements.

Sources

• DuckLake 1.0: Data Lake Format with SQL Catalog Metadata（InfoQ）
• DuckLake official website
• DuckDB — In-Process Analytical Database
• Integrated Data Infrastructure Powered by Google Agentic Data Cloud (GH Media)
• Internal AI Assistants for SMBs (GH Media)