Choosing Your Integration Architecture in Microsoft Fabric
A client recently asked me what seemed straightforward: “What’s the best way to get our Shopify data into Fabric?” Three years ago, I would have given them a single answer. Today, I walked them through two viable options, each with different architectural implications.
This shift reflects the maturation of Microsoft Fabric’s ecosystem. We’ve moved past limited integration paths into a landscape where multiple architectural patterns achieve the same outcome.
Let’s take Delta tables in Fabric as an example. Delta tables are a type of data storage format designed for big data analytics. They store data in an optimized, columnar format and support advanced capabilities such as ACID transactions, scalable metadata handling, and efficient querying.
Delta tables enable users to track and manage changes over time, ensuring reliable and consistent access to up-to-date information. In the context of data pipelines, they serve as the final destination for transformed and integrated data, allowing for fast, reliable analytics and reporting.
The question is no longer “Can we get this data into Delta tables?” but rather “Which path to Delta tables aligns with our operational priorities?”
Two Paths, Different Trade-offs
Direct-Write Integration
Tools like Fivetran replicate data directly to Delta tables in your Lakehouse. You configure the source, select your data, and the tool handles conversion to Delta format. After the initial seed completes, subsequent syncs happen quickly. Your data flows from source to table with minimal intermediate steps.
The appeal is clear: fewer moving parts means faster time-to-query and simpler operations. Your team monitors one integration rather than multiple conversion stages. The trade-off lies in visibility: the transformation from source to Delta happens inside the tool’s processing without exposing intermediate states.
Staged Architecture via OneLake Mirroring
CData takes a different approach. The tool loads change log data as Parquet files into a Mirror Database which Fabric then converts to Delta tables (called CData Sync). This extra layer might seem like unnecessary complexity, but that intermediate state creates opportunities that direct-write patterns don’t easily support.
When Staged Paths Add Value
The Mirror Database pattern provides change log visibility before final Delta conversion. For regulated industries, this intermediate proof point helps demonstrate data lineage. When data quality issues surface, you can examine Parquet files to determine whether problems originated at the source or during conversion.
This architecture also supports complex transformations that benefit from file-based processing. For example, a healthcare organization can use this pattern to apply de-identification rules by intercepting Parquet files, applying their logic, and then allowing modified files to flow into Delta tables. The staged architecture can make this insertion point natural rather than forced.
For multi-destination scenarios, extracting once and routing from Parquet files reduces API calls and source system load. You’re not paying for multiple extractions or managing multiple rate limits.
When Direct Paths Accelerate Value
Direct-write tools minimize latency between source changes and queryable data. After initial seeds are complete, subsequent syncs refresh Delta tables within minutes. This matters for operational dashboards where decision velocity depends on data freshness.
Operational simplicity compounds over time. Fewer components mean fewer things to monitor, maintain, and understand. Each additional architectural layer requires documentation, training, and specialized knowledge. Direct-write patterns let teams focus on analytics rather than pipeline orchestration.
Some direct-write tools also understand data model dependencies. Fivetran provides an entity relationship diagram (ERD) and automatically includes related tables, for example when you select Orders, it brings along order_line, fulfillment, fulfillment_line, and other tables. This relationship intelligence functions as built-in risk mitigation. You can’t accidentally create an incomplete data model because the tool encodes domain expertise that would otherwise require institutional knowledge.
The Economics of Choice
At the time of this writing, CData Sync charges a fixed-price per connector, and you’ll need at least 2 connectors for source and target. This includes unlimited data volume movement between connectors. This offers budget predictability for organizations with stable workloads. CData list price may appear high depending on your needs but alternatively you can contact CData for a quote matching your specific needs.
Fivetran uses Monthly Active Rows pricing that scales with data activity. For many mid-sized implementations, this comes in below fixed-price alternatives. Variable pricing aligns costs with value; if data volumes are low, you pay less. If business grows, costs scale proportionally.
Beyond licensing, consider operational overhead. Staged architectures require monitoring multiple transformation points. Direct-write patterns consolidate complexity into a single tool. The skill requirements differ too: staged architectures benefit from teams comfortable with file-based processing, while direct-write patterns need less specialized knowledge.
Choosing Your Path
- Start with transformation complexity. Do your business rules require intermediate processing? Staged architectures provide natural insertion points. Direct transformations work within Delta tables.
- Consider audit requirements. Need to prove data change sequences? Mirror databases with change log visibility provide evidence that direct-write patterns don’t naturally create.
- Evaluate time-to-query priorities. Near-real-time visibility driving business decisions? Minimize processing stages.
- Assess team expertise. Engineers comfortable with multi-stage pipelines? Staged architectures become viable. Otherwise, factor in learning curves and implementation risk.
- Think broadly about integration strategy. Solving for one system or building patterns for dozens of sources? Fixed per-connector pricing becomes expensive at scale. Variable pricing might scale more efficiently but requires accurate volume projections.
Organizations often use both approaches. Direct-write for operational dashboards where latency matters, staged architectures for compliance-sensitive data requiring audit visibility.
Architectural Maturity Means Choice
The Microsoft Fabric ecosystem has matured beyond “best practice” into “best fit.” Direct-write and staged-path patterns each offer distinct advantages depending on context.
Before committing to an integration approach, map requirements against both patterns. Consider full costs including operational overhead and team expertise. Test assumptions with trials rather than theoretical analysis.
The right path to Delta tables aligns with how your organization operates, not what works for someone else’s context.
