Flatten Transform

The flatten transform ≥ 3.1 maps array-valued fields to a set of individual data objects, one per array entry. This transform generates a new data stream in which each data object consists of an extracted array value as well as all the original fields of the corresponding input data object.

Note: The flatten transform only applies to array-typed data fields. If your data objects instead contain nested sub-objects with fields known at design time, you may wish to use a fold or project transform instead.

Transform Parameters

Property Type Description
fields Field[ ] Required. An array of one or more data fields containing arrays to flatten. If multiple fields are specified, their array values should have a parallel structure, ideally with the same length. If the lengths of parallel arrays do not match, the longest array will be used with null values added for missing entries.
index String The output field name for the zero-based index of the array values. If unspecified, an index field is not added. ≥ 5.8
as String[ ] The output field names for extracted array values. If unspecified, the field name of the corresponding array field is used.

Usage

Single-Field Flattening

This example flattens the array-valued field named foo. Note that all fields except foo are repeated in every output datum.

{"type": "flatten", "fields": ["foo"]}

Input data:

[
  {"name": "alpha", "data": 123, "foo": [1, 2]},
  {"name": "beta",  "data": 456, "foo": [3, 4, 5]}
]

Result:

[
  {"name": "alpha", "data": 123, "foo": 1},
  {"name": "alpha", "data": 123, "foo": 2},
  {"name": "beta",  "data": 456, "foo": 3},
  {"name": "beta",  "data": 456, "foo": 4},
  {"name": "beta",  "data": 456, "foo": 5}
]

Adding an Index Field

{"type": "flatten", "fields": ["foo"], "index": "idx"}

This example adds an field containing the array index that each item originated from.

[
  {"name": "alpha", "data": 123, "foo": [1, 2]},
  {"name": "beta",  "data": 456, "foo": [3, 4, 5]}
]

Result:

[
  {"name": "alpha", "data": 123, "foo": 1, "idx": 0},
  {"name": "alpha", "data": 123, "foo": 2, "idx": 1},
  {"name": "beta",  "data": 456, "foo": 3, "idx": 0},
  {"name": "beta",  "data": 456, "foo": 4, "idx": 1},
  {"name": "beta",  "data": 456, "foo": 5, "idx": 2}
]

Multi-Field Flattening

{"type": "flatten", "fields": ["foo", "bar"]}

This example simultaneously flattens the array-valued fields foo and bar. Given the input data

[
  {"key": "alpha", "foo": [1, 2],    "bar": ["A", "B"]},
  {"key": "beta",  "foo": [3, 4, 5], "bar": ["C", "D"]}
]

this example produces the output:

[
  {"key": "alpha", "foo": 1, "bar": "A"},
  {"key": "alpha", "foo": 2, "bar": "B"},
  {"key": "beta",  "foo": 3, "bar": "C"},
  {"key": "beta",  "foo": 4, "bar": "D"},
  {"key": "beta",  "foo": 5, "bar": null}
]