Data Mapping — Source resolution

Source resolution answers the question "which source column(s) become this field on the lens?" — before transforms run, before null handling.

from: undefined (identity)

Omit from and the source column name is the same as the output field name:

fields: {
  // Reads source column 'email' → lens field 'email'
  email: { type: 'text' },
}

Identity is the default. Most fields on most lenses use it.

from: string (rename)

Point at a different source column:

fields: {
  // Reads source column 'name_en' → lens field 'productName'
  productName: { type: 'text', from: 'name_en' },
}

The source table still contains name_en; the index-level view is productName. Callers of beam.products.query(...) get row.productName, never row.name_en.

from: string[] (merge)

Combine multiple source columns into one output field. Requires merge.

merge: 'concat'

Joins non-null parts with a separator. Nulls are dropped; an empty result is the empty string.

fields: {
  fullName: {
    type:      'text',
    from:      ['first_name', 'last_name'],
    merge:     'concat',
    separator: ' ',
  },
}

// source row: { first_name: 'Priya',  last_name: 'Shah',  ... }  → 'Priya Shah'
// source row: { first_name: 'Marco',  last_name: null,    ... }  → 'Marco'
// source row: { first_name: null,     last_name: null,    ... }  → ''

separator is required when merge: 'concat'. Common values:

merge: 'coalesce'

Returns the first non-null part. If every part is null, the result is null (then nullAs kicks in if set).

fields: {
  displayName: {
    type:  'text',
    from:  ['nickname', 'preferred_name', 'first_name'],
    merge: 'coalesce',
  },
}

// source row: { nickname: 'Pri',   preferred_name: 'Priya', first_name: 'Priyanka' }  → 'Pri'
// source row: { nickname: null,    preferred_name: 'Priya', first_name: 'Priyanka' }  → 'Priya'
// source row: { nickname: null,    preferred_name: null,    first_name: 'Priyanka' }  → 'Priyanka'
// source row: { nickname: null,    preferred_name: null,    first_name: null }        → null

No separator needed — coalesce returns one value as-is.

Transform runs on the resolved value

Crucial detail: when you use from: string[] + merge + transform, the transform runs on the merged string, not each part individually.

fullNameUpper: {
  type:      'text',
  from:      ['first_name', 'last_name'],
  merge:     'concat',
  separator: ' ',
  transform: { type: 'uppercase' },      // runs on 'Priya Shah' → 'PRIYA SHAH'
}

Not on 'Priya' and 'Shah' separately. If you need per-part transformation (e.g. lowercase each part before merging), declare them as separate fields first:

// Alternative — transform per part
firstLower: { type: 'text', from: 'first_name', transform: { type: 'lowercase' } },
lastLower:  { type: 'text', from: 'last_name',  transform: { type: 'lowercase' } },
// Now firstLower + lastLower are lowercase before any downstream usage

What happens when from doesn't exist

Validation does NOT check this at config time. Your lens will push successfully with from: 'typo_column' on every field. At ingest time:

• The bridge fetch doesn't raise — it just doesn't return that column.
• Mapping resolves to undefined.
• undefinedAs kicks in (if set).
• If undefinedAs isn't set, nullAs doesn't either (undefined ≠ null).
• The output row has undefined for that field, which downstream code reads as missing.

Catch this in testing. The CLI's semilayer generate emits TypeScript types from your lens config — if you run codegen against a freshly pushed lens and query a field whose from is wrong, you get typed undefineds at runtime, but no compile-time signal. The right fix is always to reconcile the mapping against the actual source schema before shipping.

When from is an array of 1

from: ['first_name']      // config error — must be ≥ 2

Arrays of length 1 are a validation error. Use from: 'first_name' (string) instead. If you genuinely need the single-source-with-merge shape (e.g. you want a placeholder for adding more sources later), add the second column now and tolerate one of them always being null.

Ordering matters

Source column order in from: [] determines:

• concat: the order of the output string's parts.
• coalesce: the priority — the first non-null wins.

Changing the order is semantically meaningful — ['first_name', 'last_name'] produces 'Priya Shah'; ['last_name', 'first_name'] produces 'Shah Priya'. Keep them aligned with the rendering intent.

Next: Transforms — what you can do to the resolved value.