This page details a number of common configuration options that are available within many or all runners.
Ignoring extract items
All extractors have the filters configuration option to control which extract items are ignored or included. Filters tell the extractor which items should not be raised as a task and use a syntax based on the URL search paramters. The paramter name is the attribute in the task message containing the value used in the filter and the paramter value holds the filter arguments.
Filter arguments
Filter arguments are made up of up to three components:
case sensitive - whether the test is case sensitive or not, defaults to true
comparison operator - what test is used, defaults to equal. Support operators are:
in - includes
ni - not includes
eq - equals
ne - not equals
gt - greater than
ge - greater than or equal to
lt - less than
le - less than or equal to
sw - starts with
!sw - not starts with
ew - ends with
!ew - not ends with
re - regular expresion - not implemented yet
test value - mandatory, the value to be tested against
Filter joins
By default multiple filter parameters are evaluated as or combinations. To evaluate as an andcombination use &join=and& to join the filters.
Examples
extractItem.name=in|.txt
Exclude all items that include .txt in the name field
extractItem.name=false|!ew|.pdf
Exclude items where the name field does not end with .pdf, case insensitive
extractItem.name=ew|.txt&join=and&extractItem.path=in|DRAFTS
Exclude all items that end with .txt in the name field and have DRAFTS in the path
extractItem.name=ew|[".txt",".pdf"]
Exclude all items where the name ends in .txt or .pdf
extractItem.stats.size=lt|100
Exclude all files with a size less than 100 bytes
Error handling and retry
By default, when a runner hits an error it records the error message and stops processing the task, setting it's status to suspended. You can then manually restart the task via the client or API.
By adding a retry policy the runner can automatically retry the task. Retries can be configured to control the number of retries, the delay between the retries and what error messages would not be retried. This configuration option is available for all runners.
This simplest retry policy is to set a number of retry attempts. There will be a five second delay between each retry by default. Example of this with two retry attempts:
{
"type": "PSTLoader",
"config": {...},
"retry": {
"limit": 2
}
}You can also specify a different amount of delay, as well as the ability to increase the delay between each retry. The delay is specified in milliseconds and the backoff option multiplies the delay by the number of retries.
{
"type": "PSTLoader",
"config": {...},
"retry": {
"limit": 2,
"delay": 10000,
"pattern": "backoff"
}
}In addition to changing the number of retries and the delay between retries you can also specify error messages that will not be retried and can either be suspended or quarantined. A task with an error message matching a quarantined message is stopped and a quarantine lookup is created. An item with a quarantine lookup will not be processed again until the quarantine is removed by deleting the lookup. Suspended errors can be retried by restarting the task via the client or API.
{
"type": "PSTLoader",
"config": {...},
"retry": {
"limit": 2,
"delay": 10000,
"pattern": "backoff",
"suspendMessages": [
"System is unavailable"
],
"quarantineMessages": [
"Item already exists"
]
}
}The quarantine functionality allows you to stop a particular item in the migration from being processed. This would be used when there's a problem that needs to be resolved before another attempt is made to process the item.
Quarantining a task creates a lookup using the extractId created by the extract runner and the quarantine name specified in the extractor configuration. Once quarantined, the item will not be extracted again until the quarantine lookup is deleted.
Items can be quarantined via the client, via an error retry configuration or via the API. To quarantine an item in the client find the task that has the issue and click the Quarantine button.
Example extract configuration is:
{
"type": "PSTLoader",
"config": {
"quarantinedLookupName": "HRMigrationQuarantined",
}
}To quarantine an item based on error message use the retry object in the runner config:
{
"type": "OTCSLoader",
"config": {...},
"retry": {
"limit": 2,
"quarantineMessages": [
"Item already exists"
]
}
}Batches
All extractors provide the option to split a large job into multiple runs. To set up batches specify the batch size and whether the next run is automatically queued:
sizemandatory number - how many tasks generated in the batchqueueNextRunboolean - whether the next run is automatically generated. Defaults to false
Example config is:
{
"type": "PSTLoader",
"config": {
"hashName": "HRMigration",
"batchConfig": {
"size": 5000,
"queueNextRun": true
},
}
}Database batch skipping
When using batch config the runner will automatically skip records that were processed in previous batches. For database extractors this may not be required as the tracking of what has been extracted can be handled by the database. In this scenario set the disableSkip config option to true and batch skipping will be disabled.
Repeating runs
Most extract runners provide the ability to repeat the extract and ignore unchanged data. To allow this to happen the configuration section requires a hashName. The hashName will be used to store a value as a lookup that is used to compare the data on later runs to see if it has changed.
The lookup is made up of:
name - the hashName supplied in the runner config
key - a unique identifier to the piece of data
value - the value used to compare if the item has changed between runs
Each runner will generate the value used for the comparison based on the data structures of the extract system. For example the FileSystemExtractor generates uses these values:
name - the hashName supplied in the runner config
key - a hash of the path to the file or directory
value - a hash of an object with at least the following information:
size - the file size in bytes
mtimeMs - the time the file was last modified
birthtimeMs - the time the file was created
To support the ability to ignore unchanged items in multiple runs add the following to the runner config:
{
"type": "PSTLoader",
"config": {
...
"hashName": "HRMigration",
...
}
}This would create an extract hash object in the message like:
{
"message": {
...
"extractHash": {
"name": "HRMigration",
"key": "ef15711093fce44857f960c27f239414ee7beaaa",
"value": {
"source": "C:\\code\\test\\fs-extract\\built-file3.txt",
"hash": "78e983c58916366b6534c63b96d28cfccf00aaf9",
},
},
...
}As these values are lookups they can be reset by using the Lookups section in the client. Find the lookups with the name specified and delete these lookups to reset the check.
Disable extract hash checking
By default, an extractor checks each item being extracted to see if it has already been extracted. It does this by checking to see if there is a lookup with the extractHash name and the extractId of the item being extracted. If there is a lookup for each loader in the run, no task is raised for the item.
In some jobs a transformer may raise multiple tasks per extractId. In this scenario the extractor is unable to determine if all the tasks were raised in a previous task. The isProcessed check by the extractor should be cancelled and the transformer should determine if all tasks have been raised. To do this set checkProcessed to false in the extractor config object
{
"type": "SQLLiteExtractor",
"config": {
"hashName": "HRMigration",
"batchConfig": {
"checkProcessed": false,
"size": 50000,
"queueNextRun": true
},
}
}Text replacement
Strings can be processed by the parseTemplateString function from the task-runner base class. This function will find all occurences of ${.*} in the supplied text string.
For each occurrence it will assume the value is in the message attribute of the task and will split the value by periods. It will then iterate through the segments to check if that attribute exists in the message object.
For example the query UPDATE status SET updated = '${MockLoaderCompleted}' WHERE id = '${item.id}' will look for this.task.message.MockLoaderCompleted and this.task.message.item.id and replace these with the values of the attributes.
Queues
By default all tasks are sent to a message queue called Tasks and all instances create a Tasks consumer. In some situations you may want a runner to use an alternative queue or configure an instance to consume queues other than Tasks.
An example is a queue for tasks processed after working hours. In this case you create a queue, say AfterHours. Consumers of this queue can then be configured to only process tasks after working hours. To send tasks for a particular runner to the other queue, update the queue attribute of the runner
"runners":{
...
"load": [
{
"type": "OTCSLoader",
"config": {
...
},
"queue": "AfterHours"
}
]
}Then using the client or API create a consumer with the operating hours defined. In the example below tasks using the AterHours queue will only run after 6pm and before 8am Monday to Friday.
[
...,
{
"name": "AfterHours",
"configCheck": 60000,
"isConsumer": true,
"limit": 1000,
"operatingHours": [
{
"running": false,
"prefetch": 4,
"cron": "0 8 * * 1-5"
},
{
"running": true,
"prefetch": 4,
"cron": "0 18 * * 1-5"
}
],
"prefetch": 4,
"reconnect": {
"attempts": 0,
"enabled": false,
"maxAttempts": 5,
"wait": 1000,
"backoff": 2
},
"type": "BetterQueue"
}
]