Overview
IndicativeIO is a layer of Indicative's ETL stack that allows you to define transformation rules to be able to turn customer data into events and properties, known as a Data Model. This Data Model is defined by writing a YAML file, whose properties are documented below. This is used for batch load integrations, like BigQuery or S3. If your integration is batch load, you can request access to GitLab to modify your YAML file by contacting your Customer Success Manager. This is available for Enterprise customers.
PipelineObject
PipelineObject is the top level of defining a data model. It contains the following fields.
Field Name |
|
Required? |
|
Field Type |
|
Description |
name |
|
Yes |
|
string |
|
The name of the integration |
model |
|
Yes |
|
|
Defines common fields for the integration, like timestamp and userID. |
|
events |
|
Yes |
|
list<EventObject> |
|
A list of events to output |
properties |
|
No |
|
list<PropertyObject> |
|
A list of properties to attach to output events |
globalPredicates |
|
No |
|
list<PredicateObject> |
|
A list of conditions that may be references in any other predicate expression |
globalTransformations |
|
No |
|
list<TransformationObject> |
|
A list of event or property value transformations to apply across the board |
ModelObject
ModelObject holds globally applied information about the data model. This includes how to derive event time, event unique key, and whether or not we use aliasing. There are two types of model objects:
Common Fields
Field Name |
|
Required? |
|
Field Type |
|
Description |
type |
|
Yes |
|
string |
|
This MUST be either "standard" or "aliasing". Refer to StandardModelObject and AliasingModelObject for other required fields. |
time |
|
Yes |
|
string|TimeModel |
|
Describes information about how to derive the timestamp for events. If a string is passed in, it references the field name to derive the timestamp from. |
Type: StandardModelObject
StandardModelObject is used when defining data models that do not require aliasing. It requires a single field, userID.
Field Name |
|
Required? |
|
Field Type |
|
Description |
userID |
|
Yes |
|
string |
|
The field name from which to derive the uniqueKey. |
Type: AliasingModelObject
StandardModelObject is used when defining data models that do require aliasing. It requires two fields, anonID and userID.
Field Name |
|
Required? |
|
Field Type |
|
Description |
anonID |
|
Yes |
|
string |
|
The field name from which to derive the anonymous ID. |
userID |
|
Yes |
|
string |
|
The field name from which to derive the user ID. |
TimeObject
TimeObject is used to define the global timestamp for generated events. If a string is passed where a TimeObject is expected, that value is used as "field".
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
Yes |
|
string |
|
The field name from which to derive the timestamp. |
format |
|
No |
|
string |
|
The format of the timestamp in "field". Useful for timestamps with uncommon formatting. Defaults to ISO 8601 format. Example: "YYYY-MM-dd hh:mm:ss" |
zone |
|
No |
|
string |
|
The timezone of the timestamp in "field". Default to "UTC". Example: "America/New_York" |
with |
|
No |
|
string |
|
Used to justify the time component of the timestamp in "field" to either the start or end of the day. The only two valid values are "TimeAtStartOfDay" and "TimeAtEndOfDay". |
plus |
|
No |
|
list<TimeArithmeticObject> |
|
A list of arithmetic operations to apply to the timestamp derived from "field". For example, allows you to add (or subtract) an hour from the derived timestamp. |
TimeArithmeticObject
TimeArithmeticObject is referenced by TimeObject to add or subtract constant time periods from the derived timestamp. It required two arguments, "amount" and "unit"
Field Name |
|
Required? |
|
Field Type |
|
Description |
amount |
|
Yes |
|
number |
|
The number of units to adjust the derived timestamp by. Can be positive or negative. Positive values add to the timestamp and negative values subtract from the timestamp. |
unit |
|
Yes |
|
string |
|
The time unit to add or subtract from the timestamp. Must be one of the following:
|
EventObject
EventObject is used to define which events are output by this data model and what conditions need to be true to output each specific event. It also includes utilities to be able to transform the event names. Optionally, you can set a TimeObject on an event-by-event basis if you'd like fine-tuned control to have different timestamps for different events.
Field Name |
|
Required? |
|
Field Type |
|
Description |
name |
|
Yes |
|
string |
|
The name of the event to output |
time |
|
No |
|
string|TimeObject |
|
Describes information about how to derive the timestamp for this specific event. If a string is passed in, it references the field name to derive the timestamp from. |
predicates |
|
No |
|
list<PredicateObject> |
|
A list of predicates that must evaluate to true for this event to be emitted |
transformations |
|
No |
|
list<TransformationObject> |
|
A list of event name transformations to apply. When used as part of an EventObject, the only one that really makes sense to use is "Interpolate" |
groups |
|
No |
|
list<string> |
|
A list of groups to tag this event as. Groups are used to tie properties to specific events, but not others. These values must correspond to values in the "eventGroups" field of a PropertyObject. |
PredicateObject
PredicateObjects are expressions that evaluate to either true or false. They're used when we want to do something (i.e. create an event) under specific circumstances, not all circumstances. There are many different types of PredicateObjects, listed below:
Type |
|
Aliases |
|
Description |
|
eq, equals, equal, =, ==, ==== |
|
Used to test whether a field is equal to a supplied value |
|
|
neq, not_equals, notequals, ne, not equals, !=, <> |
|
Used to test whether a field is not equal to a supplied value |
|
|
contains |
|
Used to test whether a field contains a supplied value |
|
|
not_contains, notcontains, not contains |
|
Used to test whether a field does not contain a supplied value |
|
|
defined, def, exists |
|
Used to test whether a field is defined |
|
|
not_defined, notdefined, not defined, undef, undefined, not_exists, notexists, not exists |
|
Used to test whether a field is not defined |
|
|
and, &, && |
|
Used to evaluate the logical "and" operation of two other predicates. |
|
|
or, |, || |
|
Used to evaluate the logical "or" operation of two other predicates. |
|
|
not, !, ~ |
|
Used to evaluate the negation of another supplied predicate |
|
|
any |
|
Used to determine whether any of the supplied predicates evaluate to true |
|
|
global, custom |
|
Used to reference a predicate defined in under PipelineObject.globalPredicates |
Common Fields
Field Name |
|
Required? |
|
Field Type |
|
Description |
type |
|
Yes |
|
string |
|
This MUST be one of the above options. Refer to the specific options below for other required fields |
name |
|
No |
|
string |
|
The name of the predicate. You must set a name when defining as a member of globalPredicates. When using a GlobalPredicate, this must correspond to refName |
Type: EqualsPredicate
Used to test whether a field is equal to a supplied value
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
Yes |
|
string |
|
The field to evaluate in the predicate |
value |
|
Yes |
|
string |
|
The value to which the field is compared |
Example
events: |
Type: NotEqualsPredicate
Used to test whether a field is not equal to a supplied value
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
Yes |
|
string |
|
The field to evaluate in the predicate |
value |
|
Yes |
|
string |
|
The value to which the field is compared |
Example
events: |
Type: ContainsPredicate
Used to test whether a field contains a supplied value
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
Yes |
|
string |
|
The field to evaluate in the predicate |
value |
|
Yes |
|
string |
|
The value to which the field is compared |
Example
events: |
Type: NotContainsPredicate
Used to test whether a field does not contain a supplied value
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
Yes |
|
string |
|
The field to evaluate in the predicate |
value |
|
Yes |
|
string |
|
The value to which the field is compared |
Example
events: |
Type: DefinedPredicate
Used to test whether a field is defined
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
Yes |
|
string |
|
The field to check whether it's defined |
Example
events: |
Type: NotDefinedPredicate
Used to test whether a field is not defined
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
Yes |
|
string |
|
The field to check whether it's not defined |
Example
events: |
Type: AndPredicate
Used to evaluate the logical "and" operation of two other predicates.
Field Name |
|
Required? |
|
Field Type |
|
Description |
left |
|
Yes |
|
|
The first predicate to evaluate |
|
right |
|
Yes |
|
|
The second predicate to evaluate |
Example
events: |
Type: OrPredicate
Used to evaluate the logical "or" operation of two other predicates.
Field Name |
|
Required? |
|
Field Type |
|
Description |
left |
|
Yes |
|
|
The first predicate to evaluate |
|
right |
|
Yes |
|
|
The second predicate to evaluate |
Example
events: |
Type: NotPredicate
Used to evaluate the negation of another supplied predicate
Field Name |
|
Required? |
|
Field Type |
|
Description |
predicate |
|
Yes |
|
|
The predicate to negate |
Example
events: |
Type: AnyPredicate
Used to determine whether any of the supplied predicates evaluate to true
Field Name |
|
Required? |
|
Field Type |
|
Description |
predicates |
|
Yes |
|
list<PredicateObject> |
|
The list of predicates to evaluate |
Example
events: |
Type: GlobalPredicate
Used to reference a predicate defined in under PipelineObject.globalPredicates
Field Name |
|
Required? |
|
Field Type |
|
Description |
refName |
|
Yes |
|
string |
|
The name of the global predicate to evaluate. This must correspond to the name field of a predicate |
Example
globalPredicates: |
PropertyObject
PropertyObjects are used to define properties to be tagged on events. Properties can be used tied to specific events (and not others) by using the "eventGroups" field. There are several different types of properties, listed below. Note that there are two overarching classifications of properties: those that output a single value and those that output multiple values. An example of a multi-value property is UserAgentProperty, which outputs many separate pieces of information derived from a single user agent value.
Type |
|
Single vs Multi? |
|
Aliases |
|
Description |
|
Single |
|
default, Special: the absence of a type |
|
Uses the value of a field in the input as the property value, with no transformation |
|
|
Single |
|
defined |
|
Uses "true" or "false" as the property value, depending on whether the input field is defined. |
|
|
Single |
|
datetime, date_time |
|
Parses the value of a field as a date-time, and outputs it in an easier to read format, like "YYYY-MM-dd hh:mm:ss" |
|
|
Single |
|
json |
|
Parses the value of a field as a JSON object, and extracts the supplied path from that object. |
|
|
Multi |
|
alljson, all_json |
|
Parses the value of a field as a JSON object, and extracts all possible keys from that object as properties |
|
|
Multi |
|
useragent, user_agent, user agent |
|
Parses the value of a field as a user agent, and outputs information extracted from that user agent. See below for specifics. |
|
|
Single |
|
random |
|
Uses a random string as the property value |
Common Fields
Field Name |
|
Required? |
|
Field Type |
|
Description |
type |
|
Yes, with caveats |
|
string |
|
Type is not required when using a DefaultProperty. If you'd like to use another type of property, you must specify type. If populated, this MUST be one of the above options. Refer to the specific options below for other required fields |
name |
|
Yes |
|
string |
|
The name of the property in Indicative. As a shorthand, you can define a property but using only a string, which is interpreted as the value for name. In that case, type is assumed to be DefaultProperty. |
eventGroups |
|
No |
|
list<string> |
|
A list of groups to tag this property as. Groups are used to tie properties to specific events, but not others. These values must correspond to values in the "groups" field of an EventObject. |
Type: DefaultProperty
The default property to use. Uses the value of a field in the input as the property value. In the YAML, you can use a shorthand by just specifying a string, which creates a DefaultProperty based on that field name.
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
No, with caveats |
|
string |
|
The field to derive the property value from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from. |
Example 1
properties: |
Example 2
properties: |
Example 3
properties: type: default |
Example 4
properties: field: email |
Type: DefinedProperty
DefinedProperty returns either "true" or "false" depending on whether there exists data in the specified field.
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
No, with caveats |
|
string |
|
The field to check whether there exists a value. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from. |
Example
properties: field: app_user_token |
Type: DateTimeProperty
DateTimeProperty parses the value of a field as a date-time, and outputs it in an easier to read format, like "YYYY-MM-dd hh:mm:ss".
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
No, with caveats |
|
string |
|
The field to derive the datetime from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from. |
format |
|
Yes |
|
string |
|
The format of the datetime to output. |
zone |
|
No |
|
string |
|
The timezone to use when outputting the datetime. |
Example
properties: field: created format: YYYY-MM-dd zone: UTC |
Type: JSONProperty
JSONProperty parses the value of the supplied field as a JSON object, and extracts the supplied path from that object.
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
No, with caveats |
|
string |
|
The field to derive the JSON object from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from. |
jsonPath |
|
Yes |
|
string |
|
The path in the JSON from which to extract the value. For example, "user.address.city". |
Example
properties: field: parameters jsonPath: email |
Type: AllJSONProperty
AllJSONProperty parses the value of the supplied field as a JSON object, and extracts all of keys as properties, recursively. The name is prepended to the output property values. For example, if "name" is "parameters", then the output properties will start with "parameters.", like "parameters.platform" or "parameters.email".
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
No, with caveats |
|
string |
|
The field to derive the JSON object from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from. |
exclude |
|
No |
|
list<string> |
|
A list of JSON key paths to exclude from being output. For example, "email". |
Example
properties: field: parameters exclude: - address.city |
Type: UserAgentProperty
UserAgentProperty parses the supplied field value as a user agent, and outputs properties derived from that user agent. The name is prepended to the output property values. For example, if "name" is "ua", then the output properties will start with "ua.", like "ua.browseNamer" or "ua.osName". Seven properties are currently output from this property type
- User Agent (the full user agent supplied)
- Browser Name (e.g. Chrome 64)
- Browser Type (e.g. Chrome, Firefox, Safari)
- Browser Full Name (e.g. Chrome 64.0.3282.140)
- OS Name (e.g. Windows 10, Mac OS X)
- OS Type (e.g. Windows, Mac OS X)
- Device Type (e.g. Computer, Mobile, Tablet)
Field Name |
|
Required? |
|
Field Type |
|
Description |
field |
|
No, with caveats |
|
string |
|
The field to derive the user agent from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from. |
exclude |
|
No |
|
list<string> |
|
A list of JSON key paths to exclude from being output. For example, "email". |
Example
properties: field: header_user_agent |
Type: RandomProperty
RandomProperty is used to add a property with a random value. There are no parameters.
Example
properties: |
TransformationObject
TransformationObjects are used as a way to transform the values of event names and property values.
Type |
|
Aliases |
|
Description |
LowerCaseTransform |
|
lowercase, lower_case, lower |
|
Transforms the value to all lower case characters. |
UpperCaseTransform |
|
uppercase, upper_case, upper |
|
Transforms the value to all upper case characters. |
PercentageTransform |
|
percentage, percent |
|
If the value is a number, treat it as a ratio of 1 and formats it as a percentage, rounded. For example, .85 becomes 85% and .857 becomes 86% |
MapReplaceTransform |
|
mapreplace, replace |
|
Replaces the value with another specified value. Useful when you'd like to rename specific values. Use * to replace all values. |
|
alljson, all_json |
|
Parses the value of a field as a JSON object, and extracts all possible keys from that object as properties |
|
|
useragent, user_agent, user agent |
|
Parses the value of a field as a user agent, and outputs information extracted from that user agent. See below for specifics. |
|
|
random |
|
Uses a random string as the property value |
Common Fields
Field Name |
|
Required? |
|
Field Type |
|
Description |
type |
|
Yes, with caveats |
|
string |
|
Type is not required when using a DefaultProperty. If you'd like to use another type of property, you must specify type. If populated, this MUST be one of the above options. Refer to the specific options below for other required fields |
name |
|
Yes |
|
string |
|
The name of the property in Indicative. As a shorthand, you can define a property but using only a string, which is interpreted as the value for name. In that case, type is assumed to be DefaultProperty. |
eventGroups |
|
No |
|
list<string> |
|
A list of groups to tag this property as. Groups are used to tie properties to specific events, but not others. These values must correspond to values in the "groups" field of an EventObject. |