Installing TaskManager for background tasking

By Release 19, a lot of processing has been put to the background and a task manager was created in order to manage and execute these tasks in the background.

How does it work?

Tasks are generated by the RDA Registry Software and the task parameters are provided in the dbs_registry.tasks table. Every so often, the ARDC TaskManager will hit the database looking for any tasks under the PENDING status and hit a URL to execute the task. This is repeated until there are no tasks under the PENDING state.

Installation

CODE

cd /opt
git clone https://github.com/au-research/ANDS-TaskManager.git ands-taskmanager
cd ands-taskmanager
mkdir log

Configuration

CODE

polling_frequency = 5 #how often does the task manager hit the database
max_thread_count = 5 #how many concurrent threads are run
max_up_seconds_per_task = 7200 #when a task run past this number of seconds, consider it failed
run_dir = '/opt/ands-taskmanager/' #the directory of the task manager
admin_email_addr = "" #for reporting purposes
response_url='https//localhost/api/task/exe/' #the exe to execute a task eg. http://localhost/api/task/exe/:taskid
maintenance_request_url = 'http://localhost/api/task/run/' #maintenance task run when there's no task required
data_store_path = run_dir + 'result_contents'
log_dir= run_dir + 'log'
log_level = "INFO"
db_host='localhost'
db_user='webuser'
db_passwd=''
db='dbs_registry'
tasks_table='tasks'

Install as a service

It is recommended to install the ANDS-TaskManager as a service for stability purposes as well and improve the up time of the Task Manager

CODE

cp ands-taskprocessor /etc/init.d/ands-taskmanager
chmod 755 /etc/init.d/ands-taskmanager
chkconfig --add ands-taskmanager
chkconfig ands-taskmanager on
service ands-taskmanager start