Job configuration
A job contains a number of configuration options:
id - the unique id for the job, automatically generated
customer - an optional field to group multiple jobs
name - mandatory a job name
notifications - an option to send notifyications when particular events occur, e.g. suspended, completed
itemsToProcess - how many items that will be processed by the job, used in status reporting
runners - the runners that will be used in the job
extract - runners that do the extracting of information from the source system
load - runners that either transform the metadata of the task or update or upload the item to the target system
schedule - an option for initiating runs at a specified time, including recurring initiation
timeout - allows a run to wait for an update longer than the default five minutes
tags - an array of descriptive tags, yet to be formalised
Example job configuration
{
"id": "bXnaYpDhMJ3-NLEDH_hZo"
"name": "File copy",
"runners": {
"extract": [
{
"type": "FileSystemExtractor",
"config": {
"extractRoot": "C:\\code\\test\\fs-extract"
}
}
],
"load": [
{
"type": "FileSystemLoader",
"config": {
"loadRoot": "C:\\code\\test\\fs-load"
}
}
]
},
}Runners
Runners are divided into extract and load sections. As there is no functional difference between these arrays the convention is that that runners focused on extracting data from source systems are added to the extract array and other runners are added to the load array.
More information on runner configuration is available here.
Notifications
A job can have multiple notifications. A notification consists of the following:
type - how the notification will be delivered, e.g. SMTP, Slack channel
events - the events that will fire a notification, e.g. suspended, completed
config - information needed to deliver the notification based on the type
Example notification:
"notifications": [
{
"type": "slack",
"config": {
"url": "https://hooks.slack.com/services/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"
},
"events": ["runSuspended"]
}
],Check the notifications page for information on the currently supported events and notification types.
Schedule
The schedule attribute allows defining a run scheduled in the future. The example below will create a run at 4am every Monday morning.
"schedule":{"enabled": true,"cron":"0 4 * * 1"}