Data Warehouse Integrations - IndicativeIO Data Model

Overview

IndicativeIO is a layer of Indicative's ETL stack that allows you to define transformation rules to be able to turn customer data into events and properties, known as a Data Model. This Data Model is defined by writing a YAML file, whose properties are documented below.  This is used for batch load integrations, like BigQuery or S3.  If your integration is batch load, you can request access to GitLab to modify your YAML file by contacting your Customer Success Manager.  This is available for Enterprise customers.

PipelineObject

PipelineObject is the top level of defining a data model. It contains the following fields.

Field Name

 

Required?

 

Field Type

 

Description

name

 

Yes

 

string

 

The name of the integration

model

 

Yes

 

ModelObject

 

Defines common fields for the integration, like timestamp and userID.

events

 

Yes

 

list<EventObject>

 

A list of events to output

properties

 

No

 

list<PropertyObject>

 

A list of properties to attach to output events

globalPredicates

 

No

 

list<PredicateObject>

 

A list of conditions that may be references in any other predicate expression

globalTransformations 

 

No

 

list<TransformationObject>

 

A list of event or property value transformations to apply across the board

ModelObject

ModelObject holds globally applied information about the data model. This includes how to derive event time, event unique key, and whether or not we use aliasing. There are two types of model objects:

Common Fields

Field Name

 

Required?

 

Field Type

 

Description

type

 

Yes

 

string

 

This MUST be either "standard" or "aliasing". Refer to StandardModelObject and AliasingModelObject for other required fields.

time

 

Yes

 

string|TimeModel

 

Describes information about how to derive the timestamp for events. If a string is passed in, it references the field name to derive the timestamp from.

 

Type: StandardModelObject

StandardModelObject is used when defining data models that do not require aliasing. It requires a single field, userID.

Field Name

 

Required?

 

Field Type

 

Description

userID

 

Yes

 

string

 

The field name from which to derive the uniqueKey.

 

Type: AliasingModelObject

StandardModelObject is used when defining data models that do require aliasing. It requires two fields, anonID and userID.

Field Name

 

Required?

 

Field Type

 

Description

anonID

 

Yes

 

string

 

The field name from which to derive the anonymous ID.

userID

 

Yes

 

string

 

The field name from which to derive the user ID.

TimeObject

TimeObject is used to define the global timestamp for generated events. If a string is passed where a TimeObject is expected, that value is used as "field".

Field Name

 

Required?

 

Field Type

 

Description

field

 

Yes

 

string

 

The field name from which to derive the timestamp.

format

 

No

 

string

 

The format of the timestamp in "field". Useful for timestamps with uncommon formatting. Defaults to ISO 8601 format. Example: "YYYY-MM-dd hh:mm:ss"

zone

 

No

 

string

 

The timezone of the timestamp in "field". Default to "UTC". Example: "America/New_York"

with

 

No

 

string

 

Used to justify the time component of the timestamp in "field" to either the start or end of the day. The only two valid values are "TimeAtStartOfDay" and "TimeAtEndOfDay".

plus

 

No

 

list<TimeArithmeticObject>

 

A list of arithmetic operations to apply to the timestamp derived from "field". For example, allows you to add (or subtract) an hour from the derived timestamp.

TimeArithmeticObject

TimeArithmeticObject is referenced by TimeObject to add or subtract constant time periods from the derived timestamp. It required two arguments, "amount" and "unit"

Field Name

 

Required?

 

Field Type

 

Description

amount

 

Yes

 

number

 

The number of units to adjust the derived timestamp by. Can be positive or negative. Positive values add to the timestamp and negative values subtract from the timestamp.

unit

 

Yes

 

string

 

The time unit to add or subtract from the timestamp. Must be one of the following:

  • Eras
  • Centuries
  • Weekyears
  • Years
  • Months
  • Weeks
  • Days
  • Halfdays
  • Hours
  • Minutes
  • Seconds
  • Millis

EventObject

EventObject is used to define which events are output by this data model and what conditions need to be true to output each specific event. It also includes utilities to be able to transform the event names. Optionally, you can set a TimeObject on an event-by-event basis if you'd like fine-tuned control to have different timestamps for different events.

Field Name

 

Required?

 

Field Type

 

Description

name

 

Yes

 

string

 

The name of the event to output

time

 

No

 

string|TimeObject

 

Describes information about how to derive the timestamp for this specific event. If a string is passed in, it references the field name to derive the timestamp from.

predicates

 

No

 

list<PredicateObject>

 

A list of predicates that must evaluate to true for this event to be emitted

transformations

 

No

 

list<TransformationObject>

 

A list of event name transformations to apply. When used as part of an EventObject, the only one that really makes sense to use is "Interpolate"

groups

 

No

 

list<string>

 

A list of groups to tag this event as. Groups are used to tie properties to specific events, but not others. These values must correspond to values in the "eventGroups" field of a PropertyObject.

PredicateObject

PredicateObjects are expressions that evaluate to either true or false. They're used when we want to do something (i.e. create an event) under specific circumstances, not all circumstances. There are many different types of PredicateObjects, listed below:

 

Type

 

Aliases

 

Description

EqualsPredicate

 

eq, equals, equal, =, ==, ====

 

Used to test whether a field is equal to a supplied value

NotEqualsPredicate

 

neq, not_equals, notequals, ne, not equals, !=, <>

 

Used to test whether a field is not equal to a supplied value

ContainsPredicate

 

contains

 

Used to test whether a field contains a supplied value

NotContainsPredicate

 

not_contains, notcontains, not contains

 

Used to test whether a field does not contain a supplied value

DefinedPredicate

 

defined, def, exists

 

Used to test whether a field is defined

NotDefinedPredicate

 

not_defined, notdefined, not defined, undef, undefined, not_exists, notexists, not exists

 

Used to test whether a field is not defined

AndPredicate

 

and, &, &&

 

Used to evaluate the logical "and" operation of two other predicates.

OrPredicate

 

or, |, ||

 

Used to evaluate the logical "or" operation of two other predicates.

NotPredicate

 

not, !, ~

 

Used to evaluate the negation of another supplied predicate

AnyPredicate

 

any

 

Used to determine whether any of the supplied predicates evaluate to true

GlobalPredicate

 

global, custom

 

Used to reference a predicate defined in under PipelineObject.globalPredicates

 

Common Fields

Field Name

 

Required?

 

Field Type

 

Description

type

 

Yes

 

string

 

This MUST be one of the above options. Refer to the specific options below for other required fields

name

 

No

 

string

 

The name of the predicate. You must set a name when defining as a member of globalPredicates. When using a GlobalPredicate, this must correspond to refName

 

Type: EqualsPredicate

Used to test whether a field is equal to a supplied value 

Field Name

 

Required?

 

Field Type

 

Description

field

 

Yes

 

string

 

The field to evaluate in the predicate

value

 

Yes

 

string

 

The value to which the field is compared

Example

events:
 - name: registration
   predicates:
     - type: eq
       field: page
       value: signup

 

Type: NotEqualsPredicate

Used to test whether a field is not equal to a supplied value 

Field Name

 

Required?

 

Field Type

 

Description

field

 

Yes

 

string

 

The field to evaluate in the predicate

value

 

Yes

 

string

 

The value to which the field is compared

Example

events:
 - name: registration
   predicates:
     - type: eq
       field: page
       value: signup

Type: ContainsPredicate

Used to test whether a field contains a supplied value

Field Name

 

Required?

 

Field Type

 

Description

field

 

Yes

 

string

 

The field to evaluate in the predicate

value

 

Yes

 

string

 

The value to which the field is compared

Example

events:
 - name: registration
   predicates:
     - type: contains
       field: page
       value: register

 

Type: NotContainsPredicate

Used to test whether a field does not contain a supplied value

Field Name

 

Required?

 

Field Type

 

Description

field

 

Yes

 

string

 

The field to evaluate in the predicate

value

 

Yes

 

string

 

The value to which the field is compared

Example

events:
 - name: non-homepage view
   predicates:
     - type: not contains
       field: page name
       value: homepage

 

Type: DefinedPredicate

Used to test whether a field is defined

Field Name

 

Required?

 

Field Type

 

Description

field

 

Yes

 

string

 

The field to check whether it's defined

Example

events:
 - name: page view
   predicates:
     - type: defined
       field: page_url

 

Type: NotDefinedPredicate

Used to test whether a field is not defined

Field Name

 

Required?

 

Field Type

 

Description

field

 

Yes

 

string

 

The field to check whether it's not defined

Example

events:
 - name: non-page view
   predicates:
     - type: not defined
       field: page_url

 

Type: AndPredicate

Used to evaluate the logical "and" operation of two other predicates.

Field Name

 

Required?

 

Field Type

 

Description

left

 

Yes

 

PredicateObject

 

The first predicate to evaluate

right

 

Yes

 

PredicateObject

 

The second predicate to evaluate

Example

events:
 - name: registration
   predicates:
     - type: and
       left:
         type: eq
         field: page
         value: register
       right:
         type: eq
         field: platform
         value: web

 

Type: OrPredicate

Used to evaluate the logical "or" operation of two other predicates.

Field Name

 

Required?

 

Field Type

 

Description

left

 

Yes

 

PredicateObject

 

The first predicate to evaluate

right

 

Yes

 

PredicateObject

 

The second predicate to evaluate

Example

events:
 - name: registration
   predicates:
     - type: or
       left:
         type: eq
         field: page
         value: register
       right:
         type: eq
         field: page
         value: signup

 

Type: NotPredicate

Used to evaluate the negation of another supplied predicate

Field Name

 

Required?

 

Field Type

 

Description

predicate

 

Yes

 

PredicateObject

 

The predicate to negate

Example

events:
 - name: non-page view
   predicates:
     - type: not
       predicate:
         type: defined
         field: page_url

 

Type: AnyPredicate

Used to determine whether any of the supplied predicates evaluate to true

Field Name

 

Required?

 

Field Type

 

Description

predicates

 

Yes

 

list<PredicateObject>

 

The list of predicates to evaluate

Example

events:
 - name: registration
   predicates:
     - type: any
       predicates:
         - type: eq
           field: page
           value: register
         - type: eq
           field: page
           value: signup

 

Type: GlobalPredicate

Used to reference a predicate defined in under PipelineObject.globalPredicates

Field Name

 

Required?

 

Field Type

 

Description

refName

 

Yes

 

string

 

The name of the global predicate to evaluate. This must correspond to the name field of a predicate

Example

globalPredicates:
 - name: register_page
   type: eq
   field: page
   value: register
events:
 - name: registration
   predicates:
     - type: global
       refName: register_page

PropertyObject

PropertyObjects are used to define properties to be tagged on events. Properties can be used tied to specific events (and not others) by using the "eventGroups" field. There are several different types of properties, listed below. Note that there are two overarching classifications of properties: those that output a single value and those that output multiple values. An example of a multi-value property is UserAgentProperty, which outputs many separate pieces of information derived from a single user agent value.

Type

 

Single vs Multi?

 

Aliases

 

Description

DefaultProperty

 

Single

 

default, Special: the absence of a type

 

Uses the value of a field in the input as the property value, with no transformation

DefinedProperty

 

Single

 

defined

 

Uses "true" or "false" as the property value, depending on whether the input field is defined.

DateTimeProperty

 

Single

 

datetime, date_time

 

Parses the value of a field as a date-time, and outputs it in an easier to read format, like "YYYY-MM-dd hh:mm:ss"

JSONProperty

 

Single

 

json

 

Parses the value of a field as a JSON object, and extracts the supplied path from that object.

AllJSONProperty

 

Multi

 

alljson, all_json

 

Parses the value of a field as a JSON object, and extracts all possible keys from that object as properties

UserAgentProperty

 

Multi

 

useragent, user_agent, user agent

 

Parses the value of a field as a user agent, and outputs information extracted from that user agent. See below for specifics.

RandomProperty

 

Single

 

random

 

Uses a random string as the property value

Common Fields

Field Name

 

Required?

 

Field Type

 

Description

type

 

Yes, with caveats

 

string

 

Type is not required when using a DefaultProperty. If you'd like to use another type of property, you must specify type. If populated, this MUST be one of the above options. Refer to the specific options below for other required fields

name

 

Yes

 

string

 

The name of the property in Indicative. As a shorthand, you can define a property but using only a string, which is interpreted as the value for name. In that case, type is assumed to be DefaultProperty.

eventGroups

 

No

 

list<string>

 

A list of groups to tag this property as. Groups are used to tie properties to specific events, but not others. These values must correspond to values in the "groups" field of an EventObject.

 

Type: DefaultProperty

The default property to use. Uses the value of a field in the input as the property value. In the YAML, you can use a shorthand by just specifying a string, which creates a DefaultProperty based on that field name.

Field Name

 

Required?

 

Field Type

 

Description

field

 

No, with caveats

 

string

 

The field to derive the property value from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.

Example 1

properties:
 - email

Example 2

properties:
 - name: email

Example 3

properties:
 - name: email

   type: default

Example 4

properties:
 - name: user_email

   field: email

 

Type: DefinedProperty

DefinedProperty returns either "true" or "false" depending on whether there exists data in the specified field.

Field Name

 

Required?

 

Field Type

 

Description

field

 

No, with caveats

 

string

 

The field to check whether there exists a value. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.

Example

properties:
 - name: is_app_user

   field: app_user_token
   type: defined

 

Type: DateTimeProperty

DateTimeProperty parses the value of a field as a date-time, and outputs it in an easier to read format, like "YYYY-MM-dd hh:mm:ss".

Field Name

 

Required?

 

Field Type

 

Description

field

 

No, with caveats

 

string

 

The field to derive the datetime from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.

format

 

Yes

 

string

 

The format of the datetime to output.

zone

 

No

 

string

 

The timezone to use when outputting the datetime.

Example

properties:
 - name: signup_date

   field: created
   type: datetime

   format: YYYY-MM-dd

   zone: UTC

 

Type: JSONProperty

JSONProperty parses the value of the supplied field as a JSON object, and extracts the supplied path from that object.

Field Name

 

Required?

 

Field Type

 

Description

field

 

No, with caveats

 

string

 

The field to derive the JSON object from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.

jsonPath

 

Yes

 

string

 

The path in the JSON from which to extract the value. For example, "user.address.city".

Example

properties:
 - name: email

   field: parameters
   type: json

   jsonPath: email

 

Type: AllJSONProperty

AllJSONProperty parses the value of the supplied field as a JSON object, and extracts all of keys as properties, recursively. The name is prepended to the output property values. For example, if "name" is "parameters", then the output properties will start with "parameters.", like "parameters.platform" or "parameters.email".

Field Name

 

Required?

 

Field Type

 

Description

field

 

No, with caveats

 

string

 

The field to derive the JSON object from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.

exclude

 

No

 

list<string>

 

A list of JSON key paths to exclude from being output. For example, "email".

Example

properties:
 - name: parameters

   field: parameters
   type: all_json

   exclude:

     - email

     - address.city

 

Type: UserAgentProperty

UserAgentProperty parses the supplied field value as a user agent, and outputs properties derived from that user agent. The name is prepended to the output property values. For example, if "name" is "ua", then the output properties will start with "ua.", like "ua.browseNamer" or "ua.osName". Seven properties are currently output from this property type

  • User Agent (the full user agent supplied)
  • Browser Name (e.g. Chrome 64)
  • Browser Type (e.g. Chrome, Firefox, Safari)
  • Browser Full Name (e.g. Chrome 64.0.3282.140)
  • OS Name (e.g. Windows 10, Mac OS X)
  • OS Type (e.g. Windows, Mac OS X)
  • Device Type (e.g. Computer, Mobile, Tablet)

Field Name

 

Required?

 

Field Type

 

Description

field

 

No, with caveats

 

string

 

The field to derive the user agent from. If no value is specified here, it is assumed that the value of name is the name of the field to derive the property value from.

exclude

 

No

 

list<string>

 

A list of JSON key paths to exclude from being output. For example, "email".

Example

properties:
 - name: ua

   field: header_user_agent
   type: user_agent

Type: RandomProperty

RandomProperty is used to add a property with a random value. There are no parameters.

Example

properties:
 - name: rand
   type: random

 

TransformationObject

TransformationObjects are used as a way to transform the values of event names and property values. 

Type

 

Aliases

 

Description

LowerCaseTransform

 

lowercase, lower_case, lower

 

Transforms the value to all lower case characters.

UpperCaseTransform

 

uppercase, upper_case, upper

 

Transforms the value to all upper case characters.

PercentageTransform

 

percentage, percent

 

If the value is a number, treat it as a ratio of 1 and formats it as a percentage, rounded. For example, .85 becomes 85% and .857 becomes 86%

MapReplaceTransform

 

mapreplace, replace

 

Replaces the value with another specified value. Useful when you'd like to rename specific values. Use * to replace all values.

AllJSONProperty

 

alljson, all_json

 

Parses the value of a field as a JSON object, and extracts all possible keys from that object as properties

UserAgentProperty

 

useragent, user_agent, user agent

 

Parses the value of a field as a user agent, and outputs information extracted from that user agent. See below for specifics.

RandomProperty

 

random

 

Uses a random string as the property value

Common Fields

Field Name

 

Required?

 

Field Type

 

Description

type

 

Yes, with caveats

 

string

 

Type is not required when using a DefaultProperty. If you'd like to use another type of property, you must specify type. If populated, this MUST be one of the above options. Refer to the specific options below for other required fields

name

 

Yes

 

string

 

The name of the property in Indicative. As a shorthand, you can define a property but using only a string, which is interpreted as the value for name. In that case, type is assumed to be DefaultProperty.

eventGroups

 

No

 

list<string>

 

A list of groups to tag this property as. Groups are used to tie properties to specific events, but not others. These values must correspond to values in the "groups" field of an EventObject.

 

0 users found this helpful