Enriched event data in your warehouse in real time

The problem with current data warehouse pipelines

Most analytics data reaches your warehouse through a chain of lossy, delayed steps: client-side tags capture a subset of events, a CDP or ETL tool batches them, and hours later a partial picture arrives in your tables.

Pipeline configuration

The same pipeline that delivers events to GA4 and Meta also streams them to your warehouse. Configure what fields to include, how to transform them, and which events to send.

# bigquery-all-events.yaml
name: bigquery_all_events
integration: google-bigquery
trigger:
  event: "*"  # All event types

parameters:
  project_id: "your-gcp-project"
  dataset: "datafly_events"
  table: "events"

global:
  anonymous_id:
    source: anonymous_id
    mode: direct
  user_id:
    source: user_id
    mode: direct
  session_id:
    source: context.session.id
    mode: direct

events:
  "*":
    mappings:
      event_name:
        source: event
        mode: direct
      event_type:
        source: type
        mode: direct
      event_timestamp:
        source: timestamp
        mode: direct
      page_url:
        source: context.page.url
        mode: direct
      page_title:
        source: context.page.title
        mode: direct
      referrer:
        source: context.page.referrer
        mode: direct
      properties:
        source: properties
        mode: direct
      geo_country:
        source: context.geo.country
        mode: direct
      geo_city:
        source: context.geo.city
        mode: direct
      device_type:
        source: context.device.type
        mode: direct
      browser:
        source: context.device.browser
        mode: direct

What teams build with warehouse data

Frequently asked questions

Which data warehouses are supported?

BigQuery, Snowflake, Redshift, Databricks, S3, Google Cloud Storage, and Azure Blob Storage are first-class destinations with native streaming support. Any warehouse or object store reachable over HTTPS can be added via custom webhook delivery if a native connector does not yet exist.

How does delivery latency compare to batch ETL?

Events typically arrive in the warehouse within 1-10 seconds depending on destination. BigQuery streaming inserts are sub-second. Snowflake via Snowpipe Streaming is 1-2 seconds. Compared to nightly batch ETL or hourly Fivetran syncs, downstream models, dashboards, and ML pipelines run on data that is genuinely current rather than 3-24 hours stale.

How is the data structured in the warehouse?

Events land in a normalised events table with structured columns (event_id, user_id, timestamp, event_name, properties JSON, context JSON, consent flags, classification metadata). The schema is consistent across all sources (web, mobile, server) so downstream queries, dbt models, and ML feature pipelines do not need source-specific logic.

Are PII transformations applied before warehouse delivery?

Yes. The Org Data Layer's PII rules apply uniformly across every destination, including warehouses. Email and phone are SHA-256 hashed by default, IP addresses are masked, payment cards / DOB / addresses are stripped. You can override per-destination if a specific warehouse needs the raw values for compliance reasons (rare), but the secure-by-default posture means warehouse data is procurement-friendly without per-team configuration.

Does Datafly Signal replace dbt or my data modelling layer?

No, it complements them. Datafly Signal handles the collect → enrich → deliver layer, with events landing in a clean, well-typed events table. Your dbt project then transforms that table into your business models, marts, and metrics. Most teams point dbt staging models directly at the Datafly-delivered events table without intermediate ELT.