task_processing: Generic task processing

Interfaces and shared infrastructure for generic task processing @ Yelp

Generated Docs

task_processing

task_processing package

Subpackages
task_processing.interfaces package
Submodules
task_processing.interfaces.event module
class task_processing.interfaces.event.Event

Bases: pyrsistent._precord.PRecord

static validate_task_event(r)
task_processing.interfaces.event.control_event(**kwargs)
task_processing.interfaces.event.json_deserializer(dct)
task_processing.interfaces.event.json_serializer(o)
task_processing.interfaces.event.task_event(**kwargs)
task_processing.interfaces.runner module
class task_processing.interfaces.runner.Runner

Bases: object

TASK_CONFIG_INTERFACE

alias of task_processing.interfaces.task_executor.DefaultTaskConfigInterface

kill(task_id)
run(task_config)
task_processing.interfaces.task_executor module
class task_processing.interfaces.task_executor.DefaultTaskConfigInterface

Bases: pyrsistent._precord.PRecord

class task_processing.interfaces.task_executor.TaskExecutor

Bases: object

The core interface for Task Processing This is the class you want to implement to add a new TaskExecutor

TASK_CONFIG_INTERFACE

The interface, specified as a PRecord of objects that you will be passing as task_configs to run

alias of DefaultTaskConfigInterface

get_event_queue()

Get queue of events

Returns:TBD
kill(task_id)

Kill the specified task

Parameters:task_id (str) – The task that you want to kill
run(task_config)

Run the supplied task

Parameters:task_config – An object satistfying the TASK_CONFIG_INTERFACE

The executor should start running the provided task and return the task id.

Returns str task_id:
 Callers get the id of the task that was run

to check status or kill it later

stop()

Stop the executor stack

Module contents
task_processing.plugins package
Subpackages
task_processing.plugins.mesos package
Submodules
task_processing.plugins.mesos.execution_framework module
task_processing.plugins.mesos.mesos_executor module
task_processing.plugins.mesos.translator module
Module contents
Module contents
task_processing.runners package
Submodules
task_processing.runners.async module
class task_processing.runners.async.Async(executor, callbacks=None)

Bases: task_processing.interfaces.runner.Runner

callback_loop()
kill(task_id)
run(task_config)
stop()
exception task_processing.runners.async.AsyncError

Bases: Exception

class task_processing.runners.async.EventHandler(predicate, cb)

Bases: tuple

cb

Alias for field number 1

predicate

Alias for field number 0

task_processing.runners.promise module
class task_processing.runners.promise.Promise(executor, futures_executor)

Bases: task_processing.interfaces.runner.Runner

kill(task_id)
run(task_config)

Schedules execution of the supplied task

Parameters:task_config – An object satisfying the TASK_CONFIG_INTERFACE
Returns:A Future object representing the execution of the task.
stop()
task_processing.runners.subscription module
class task_processing.runners.subscription.Subscription(executor, queue)

Bases: task_processing.interfaces.runner.Runner

event_producer()
kill(task_id)
run(task_config)
stop()
task_processing.runners.sync module
class task_processing.runners.sync.Sync(executor)

Bases: task_processing.interfaces.runner.Runner

kill(task_id)
run(task_config)
stop()
Module contents
Module contents

Indices and tables