Data Warehouse Integrations - IndicativeIO Data Model – Indicative

Overview

IndicativeIO is a layer of Indicative's ETL stack that allows you to define transformation rules to be able to turn customer data into events and properties, known as a Data Model. This Data Model is defined by writing a YAML file, whose properties are documented below. This is used for batch load integrations, like BigQuery or S3. If your integration is batch load, you can request access to GitLab to modify your YAML file by contacting your Customer Success Manager. This is available for Enterprise customers.

PipelineObject

PipelineObject is the top level of defining a data model. It contains the following fields.

Field Name	Required?	Field Type	Description
name	Yes	string	The name of the integration
model	Yes	ModelObject	Defines common fields for the integration, like timestamp and userID.
events	Yes	list<EventObject>	A list of events to output
properties	No	list<PropertyObject>	A list of properties to attach to output events
globalPredicates	No	list<PredicateObject>	A list of conditions that may be references in any other predicate expression
globalTransformations	No	list<TransformationObject>	A list of event or property value transformations to apply across the board

ModelObject

ModelObject holds globally applied information about the data model. This includes how to derive event time, event unique key, and whether or not we use aliasing. There are two types of model objects:

StandardModelObject
AliasingModelObject

Common Fields

Field Name	Required?	Field Type	Description
type	Yes	string	This MUST be either "standard" or "aliasing". Refer to StandardModelObject and AliasingModelObject for other required fields.
time	Yes	string\|TimeModel	Describes information about how to derive the timestamp for events. If a string is passed in, it references the field name to derive the timestamp from.

Type: StandardModelObject

StandardModelObject is used when defining data models that do not require aliasing. It requires a single field, userID.

Field Name		Required?		Field Type		Description
userID		Yes		string		The field name from which to derive the uniqueKey.

Type: AliasingModelObject

StandardModelObject is used when defining data models that do require aliasing. It requires two fields, anonID and userID.

Field Name	Required?	Field Type	Description
anonID	Yes	string	The field name from which to derive the anonymous ID.
userID	Yes	string	The field name from which to derive the user ID.

TimeObject

TimeObject is used to define the global timestamp for generated events. If a string is passed where a TimeObject is expected, that value is used as "field".

Field Name	Required?	Field Type	Description
field	Yes	string	The field name from which to derive the timestamp.
format	No	string	The format of the timestamp in "field". Useful for timestamps with uncommon formatting. Defaults to ISO 8601 format. Example: "YYYY-MM-dd hh:mm:ss"
zone	No	string	The timezone of the timestamp in "field". Default to "UTC". Example: "America/New_York"
with	No	string	Used to justify the time component of the timestamp in "field" to either the start or end of the day. The only two valid values are "TimeAtStartOfDay" and "TimeAtEndOfDay".
plus	No	list<TimeArithmeticObject>	A list of arithmetic operations to apply to the timestamp derived from "field". For example, allows you to add (or subtract) an hour from the derived timestamp.

TimeArithmeticObject

TimeArithmeticObject is referenced by TimeObject to add or subtract constant time periods from the derived timestamp. It required two arguments, "amount" and "unit"

Field Name

Required?

Field Type

Description

amount

Yes

number

The number of units to adjust the derived timestamp by. Can be positive or negative. Positive values add to the timestamp and negative values subtract from the timestamp.

unit

Yes

string

The time unit to add or subtract from the timestamp. Must be one of the following:

Eras
Centuries
Weekyears
Years
Months
Weeks
Days
Halfdays
Hours
Minutes
Seconds
Millis

EventObject

EventObject is used to define which events are output by this data model and what conditions need to be true to output each specific event. It also includes utilities to be able to transform the event names. Optionally, you can set a TimeObject on an event-by-event basis if you'd like fine-tuned control to have different timestamps for different events.

Field Name	Required?	Field Type	Description
name	Yes	string	The name of the event to output
time	No	string\|TimeObject	Describes information about how to derive the timestamp for this specific event. If a string is passed in, it references the field name to derive the timestamp from.
predicates	No	list<PredicateObject>	A list of predicates that must evaluate to true for this event to be emitted
transformations	No	list<TransformationObject>	A list of event name transformations to apply. When used as part of an EventObject, the only one that really makes sense to use is "Interpolate"
groups	No	list<string>	A list of groups to tag this event as. Groups are used to tie properties to specific events, but not others. These values must correspond to values in the "eventGroups" field of a PropertyObject.

PredicateObject

PredicateObjects are expressions that evaluate to either true or false. They're used when we want to do something (i.e. create an event) under specific circumstances, not all circumstances. There are many different types of PredicateObjects, listed below:

Type	Aliases	Description
EqualsPredicate	eq, equals, equal, =, ==, ====	Used to test whether a field is equal to a supplied value
NotEqualsPredicate	neq, not_equals, notequals, ne, not equals, !=, <>	Used to test whether a field is not equal to a supplied value
ContainsPredicate	contains	Used to test whether a field contains a supplied value
NotContainsPredicate	not_contains, notcontains, not contains	Used to test whether a field does not contain a supplied value
DefinedPredicate	defined, def, exists	Used to test whether a field is defined
NotDefinedPredicate	not_defined, notdefined, not defined, undef, undefined, not_exists, notexists, not exists	Used to test whether a field is not defined
AndPredicate	and, &, &&	Used to evaluate the logical "and" operation of two other predicates.
OrPredicate	or, \|, \|\|	Used to evaluate the logical "or" operation of two other predicates.
NotPredicate	not, !, ~	Used to evaluate the negation of another supplied predicate
AnyPredicate	any	Used to determine whether any of the supplied predicates evaluate to true
GlobalPredicate	global, custom	Used to reference a predicate defined in under PipelineObject.globalPredicates

Common Fields

Field Name	Required?	Field Type	Description
type	Yes	string	This MUST be one of the above options. Refer to the specific options below for other required fields
name	No	string	The name of the predicate. You must set a name when defining as a member of globalPredicates. When using a GlobalPredicate, this must correspond to refName

Type: EqualsPredicate

Used to test whether a field is equal to a supplied value

Field Name	Required?	Field Type	Description
field	Yes	string	The field to evaluate in the predicate
value	Yes	string	The value to which the field is compared

Example

events:
- name: registration
   predicates:
     - type: eq
       field: page
       value: signup

Type: NotEqualsPredicate

Used to test whether a field is not equal to a supplied value

Field Name	Required?	Field Type	Description
field	Yes	string	The field to evaluate in the predicate
value	Yes	string	The value to which the field is compared

Example

events:
- name: registration
   predicates:
     - type: neq
       field: page
       value: signup

Type: ContainsPredicate

Used to test whether a field contains a supplied value

Field Name	Required?	Field Type	Description
field	Yes	string	The field to evaluate in the predicate
value	Yes	string	The value to which the field is compared

Example

events:
- name: registration
   predicates:
     - type: contains
       field: page
       value: register

Type: NotContainsPredicate

Used to test whether a field does not contain a supplied value

Field Name	Required?	Field Type	Description
field	Yes	string	The field to evaluate in the predicate
value	Yes	string	The value to which the field is compared

Example

events:
- name: non-homepage view
   predicates:
     - type: not contains
       field: page name
       value: homepage

Type: DefinedPredicate

Used to test whether a field is defined

Field Name		Required?		Field Type		Description
field		Yes		string		The field to check whether it's defined

Example

events:
- name: page view
   predicates:
     - type: defined
       field: page_url

Type: NotDefinedPredicate

Used to test whether a field is not defined

Field Name		Required?		Field Type		Description
field		Yes		string		The field to check whether it's not defined

Example

events:
- name: non-page view
   predicates:
     - type: not defined
       field: page_url

Type: AndPredicate

Used to evaluate the logical "and" operation of two other predicates.

Field Name	Required?	Field Type	Description
left	Yes	PredicateObject	The first predicate to evaluate
right	Yes	PredicateObject	The second predicate to evaluate

Example

events:
- name: registration
   predicates:
     - type: and
       left:
         type: eq
         field: page
         value: register
       right:
         type: eq
         field: platform
         value: web

Type: OrPredicate

Used to evaluate the logical "or" operation of two other predicates.

Field Name	Required?	Field Type	Description
left	Yes	PredicateObject	The first predicate to evaluate
right	Yes	PredicateObject	The second predicate to evaluate

Example

events:
- name: registration
   predicates:
     - type: or
       left:
         type: eq
         field: page
         value: register
       right:
         type: eq
         field: page
         value: signup

Type: NotPredicate

Used to evaluate the negation of another supplied predicate

Field Name		Required?		Field Type		Description
predicate		Yes		PredicateObject		The predicate to negate

Example

events:
- name: non-page view
   predicates:
     - type: not
       predicate:
         type: defined
         field: page_url

Type: AnyPredicate

Used to determine whether any of the supplied predicates evaluate to true

Field Name		Required?		Field Type		Description
predicates		Yes		list<PredicateObject>		The list of predicates to evaluate

Example

events:
- name: registration
   predicates:
     - type: any
       predicates:
         - type: eq
           field: page
           value: register
         - type: eq
           field: page
           value: signup

Type: GlobalPredicate

Used to reference a predicate defined in under PipelineObject.globalPredicates

Field Name		Required?		Field Type		Description
refName		Yes		string		The name of the global predicate to evaluate. This must correspond to the name field of a predicate

Example

globalPredicates:
- name: register_page
   type: eq
   field: page
   value: register
events:
- name: registration
   predicates:
     - type: global
       refName: register_page

PropertyObject

PropertyObjects are used to define properties to be tagged on events. Properties can be used tied to specific events (and not others) by using the "eventGroups" field. There are several different types of properties, listed below. Note that there are two overarching classifications of properties: those that output a single value and those that output multiple values. An example of a multi-value property is UserAgentProperty, which outputs many separate pieces of information derived from a single user agent value.

Type	Single vs Multi?	Aliases	Description
DefaultProperty	Single	default, Special: the absence of a type	Uses the value of a field in the input as the property value, with no transformation
DefinedProperty	Single	defined	Uses "true" or "false" as the property value, depending on whether the input field is defined.
DateTimeProperty	Single	datetime, date_time	Parses the value of a field as a date-time, and outputs it in an easier to read format, like "YYYY-MM-dd hh:mm:ss"
JSONProperty	Single	json	Parses the value of a field as a JSON object, and extracts the supplied path from that object.
AllJSONProperty	Multi	alljson, all_json	Parses the value of a field as a JSON object, and extracts all possible keys from that object as properties
UserAgentProperty	Multi	useragent, user_agent, user agent	Parses the value of a field as a user agent, and outputs information extracted from that user agent. See below for specifics.
RandomProperty	Single	random	Uses a random string as the property value

Common Fields

Field Name	Required?	Field Type	Description
type	Yes, with caveats	string	Type is not required when using a DefaultProperty. If you'd like to use another type of property, you must specify type. If populated, this MUST be one of the above options. Refer to the specific options below for other required fields
name	Yes	string	The name of the property in Indicative. As a shorthand, you can define a property but using only a string, which is interpreted as the value for name. In that case, type is assumed to be DefaultProperty.
eventGroups	No	list<string>	A list of groups to tag this property as. Groups are used to tie properties to specific events, but not others. These values must correspond to values in the "groups" field of an EventObject.

Type: DefaultProperty

The default property to use. Uses the value of a field in the input as the property value. In the YAML, you can use a shorthand by just specifying a string, which creates a DefaultProperty based on that field name.

Field Name		Required?		Field Type		Description
field		No, with caveats		string		The field to derive the property value from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.

Example 1

properties:
- email

Example 2

properties:
- name: email

Example 3

properties:
- name: email

type: default

Example 4

properties:
- name: user_email

field: email

Type: DefinedProperty

DefinedProperty returns either "true" or "false" depending on whether there exists data in the specified field.

Field Name		Required?		Field Type		Description
field		No, with caveats		string		The field to check whether there exists a value. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.

Example

properties:
- name: is_app_user

field: app_user_token
type: defined

Type: DateTimeProperty

DateTimeProperty parses the value of a field as a date-time, and outputs it in an easier to read format, like "YYYY-MM-dd hh:mm:ss".

Field Name	Required?	Field Type	Description
field	No, with caveats	string	The field to derive the datetime from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.
format	Yes	string	The format of the datetime to output.
zone	No	string	The timezone to use when outputting the datetime.

Example

properties:
- name: signup_date

field: created
type: datetime

format: YYYY-MM-dd

zone: UTC

Type: JSONProperty

JSONProperty parses the value of the supplied field as a JSON object, and extracts the supplied path from that object.

Field Name	Required?	Field Type	Description
field	No, with caveats	string	The field to derive the JSON object from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.
jsonPath	Yes	string	The path in the JSON from which to extract the value. For example, "user.address.city".

Example

properties:
- name: email

field: parameters
type: json

jsonPath: email

Type: AllJSONProperty

AllJSONProperty parses the value of the supplied field as a JSON object, and extracts all of keys as properties, recursively. The name is prepended to the output property values. For example, if "name" is "parameters", then the output properties will start with "parameters.", like "parameters.platform" or "parameters.email".

Field Name	Required?	Field Type	Description
field	No, with caveats	string	The field to derive the JSON object from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.
exclude	No	list<string>	A list of JSON key paths to exclude from being output. For example, "email".

Example

properties:
- name: parameters

field: parameters
type: all_json

exclude:

- email

- address.city

Type: UserAgentProperty

UserAgentProperty parses the supplied field value as a user agent, and outputs properties derived from that user agent. The name is prepended to the output property values. For example, if "name" is "ua", then the output properties will start with "ua.", like "ua.browseNamer" or "ua.osName". Seven properties are currently output from this property type

User Agent (the full user agent supplied)
Browser Name (e.g. Chrome 64)
Browser Type (e.g. Chrome, Firefox, Safari)
Browser Full Name (e.g. Chrome 64.0.3282.140)
OS Name (e.g. Windows 10, Mac OS X)
OS Type (e.g. Windows, Mac OS X)
Device Type (e.g. Computer, Mobile, Tablet)

Field Name	Required?	Field Type	Description
field	No, with caveats	string	The field to derive the user agent from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.
exclude	No	list<string>	A list of JSON key paths to exclude from being output. For example, "email".

Example

properties:
- name: ua

field: header_user_agent
type: user_agent

Type: RandomProperty

RandomProperty is used to add a property with a random value. There are no parameters.

Example

properties:
- name: rand
type: random

TransformationObject

TransformationObjects are used as a way to transform the values of event names and property values.

Type	Aliases	Description
LowerCaseTransform	lowercase, lower_case, lower	Transforms the value to all lower case characters.
UpperCaseTransform	uppercase, upper_case, upper	Transforms the value to all upper case characters.
PercentageTransform	percentage, percent	If the value is a number, treat it as a ratio of 1 and formats it as a percentage, rounded. For example, .85 becomes 85% and .857 becomes 86%
MapReplaceTransform	mapreplace, replace	Replaces the value with another specified value. Useful when you'd like to rename specific values. Use * to replace all values.
AllJSONProperty	alljson, all_json	Parses the value of a field as a JSON object, and extracts all possible keys from that object as properties
UserAgentProperty	useragent, user_agent, user agent	Parses the value of a field as a user agent, and outputs information extracted from that user agent. See below for specifics.
RandomProperty	random	Uses a random string as the property value

Common Fields

Field Name	Required?	Field Type	Description
type	Yes, with caveats	string	Type is not required when using a DefaultProperty. If you'd like to use another type of property, you must specify type. If populated, this MUST be one of the above options. Refer to the specific options below for other required fields
name	Yes	string	The name of the property in Indicative. As a shorthand, you can define a property but using only a string, which is interpreted as the value for name. In that case, type is assumed to be DefaultProperty.
eventGroups	No	list<string>	A list of groups to tag this property as. Groups are used to tie properties to specific events, but not others. These values must correspond to values in the "groups" field of an EventObject.