# distributed-task

A python library that provides a system for managing distributed tasks. Designed to allow for parallelization and task persistence - ensures that any tasks that are forcibly interrupted will eventually run to completion when another instance of the program is running.

## Quick-start

`examples/quick_start.py`

```python3
from distributed_task import ShelfDistributedTaskManager
manager = ShelfDistributedTaskManager("distributed-task-manager-store")

@manager.distributed_task("my_task:1")
def my_task(should_raise):
  if should_raise:
    raise Exception()
  return "output"

# Call 'my_task' as if it is a normal function:
my_task(False) # Returns "output"
my_task(True) # Raises 'TaskException' (the underlying exception is handled by the worker thread)

# Or use the underlying 'start' and 'await' methods for finer control and parallelism:
task_run_id_1 = my_task.start(False)
task_run_id_2 = my_task.start(True)
my_task.await(task_run_id_1) # Returns "output"
my_task.await(task_run_id_2) # Raises 'TaskException'
```

## Discussion

Consider the following complete program:

`examples/do_slow_things_1.py`

```python3
import sys
import time

def slow_thing(i):
  time.sleep(2)
  return "thing-%i" % i

def slow_things(n, output_file_path):
  things = [slow_thing(i) for i in range(n)]
  with open(output_file_path, "w") as file:
    for thing in things:
      print(thing, file=file)

if __name__ == "__main__":
  slow_things(3, sys.argv[1])
```

You can run the program and see the output as follows:

```bash
rm -rf run && mkdir run
python3 examples/do_slow_things_1.py run/slow_things_a.txt
cat run/slow_things_a.txt
# thing-0
# thing-1
# thing-2
```

There are a couple of issues here:
* The calls to `slow_thing` can be parallelized
* If we get interrupted in the midst of a call to `slow_thing`, the work is abandoned, and our output file never gets generated

The parallelization issue is easily solved with threads or processes, but what about the task persistence issue? You can see the problem if you run the following commands:

```bash
rm -rf run && mkdir run
python3 examples/do_slow_things_1.py run/slow_things_b.txt &
do_slow_things_pid=$!
sleep 1 # To make sure we are in the midst of our 'slow_things' function call
kill $do_slow_things_pid
cat run/slow_things_b.txt
# cat: run/slow_things_b.txt: No such file or directory
```

As expected, the output file `slow_things_b.txt` will never get created. The desired behavior is that next time the program runs (regardless of whetever input it receives that next time), it will realize it never finished the older task, and it will finish that task before continuing.

Note: the issue of task persistence makes a lot more sense in the context of a long-running service, rather than a single-use program, but it's presented this way so as to be easy to experiment with.

### Persistence

Let's make the following changes to our program to introduce a `ShelfDistributedTaskManager`. Note that we require a local path in which we can persist our data. The manager locks access to that data, so there is no conflict with other processes using the same path. The `manager.distributed_task` decorator also requires a unique task type ID as input. We do this instead of relying on just the funciton name in case the function implementation changes - a task submitted with an expectation of one implementation should not be handled by a different implementation (if you change a function implementation, you should also change its type ID).

`examples/do_slow_things_2.py`

```python3
import sys
import time

from distributed_task import ShelfDistributedTaskManager
manager = ShelfDistributedTaskManager("distributed-task-manager-store")

@manager.distributed_task("slow_thing:1")
def slow_thing(i):
  time.sleep(2)
  return "thing-%i" % i

@manager.distributed_task("slow_things:1")
def slow_things(n, output_file_path):
  things = [slow_thing(i) for i in range(n)]
  with open(output_file_path, "w") as file:
    for thing in things:
      print(thing, file=file)

if __name__ == "__main__":
  slow_things(3, sys.argv[1])
```

Now let's repeat the same test from earlier with our new program, but after terminating it, start another instance (with different parameters). This example shows that the persistence issue has been addressed.

```bash
rm -rf run && mkdir run
python3 examples/do_slow_things_2.py run/slow_things_c.txt &
do_slow_things_pid=$!
sleep 5 # To make sure we are in the midst of our 'slow_things' function call
kill $do_slow_things_pid
cat run/slow_things_c.txt
# cat: run/slow_things_c.txt: No such file or directory
sleep 16 # To make sure the task is no longer in progress (default timeout is 15s)
time python3 examples/do_slow_things_2.py run/slow_things_d.txt
cat run/slow_things_c.txt
# thing-0
# thing-1
# thing-2
cat run/slow_things_d.txt
# thing-0
# thing-1
# thing-2
```

### Sub-tasks

The preceding example includes sub-tasks, which makes the problem more nuanced. During our initial `sleep 5` statement, two `slow_thing` tasks completed before we terminated the program. When we run the program again and it starts catching up on the old tasks, it should recognize that the incomplete `slow_things` task has already completed two `slow_thing` sub-tasks. To do this, we keep track of the parent task run ID any time a new task is started. Then when we try to start a sub-task, we check to see if a sub-task with the same input has already been started in the context of that parent. __This strategy makes the assumption that all sub-tasks are deterministic__ – if more than one sub-task of the same type is started with the same input, successive calls will latch on to the existing task without re-evaluating the task code.

#### Threads

A task is a sub-task if it is started by the worker of an existing task - basically if any code with your `manager.distributed_task` decorated function calls another similarly-decorated function. Note however that if you create separate child _threads_ within the context of a distributed task worker, then there is no way to determine the parent context. Your code will still work, but the pseudo "sub-tasks" will be re-evaluated if the parent gets interrupted. The following section describes a better method for parallelizing your tasks using the `manager.distributed_task` decorator. 

### Parallelization

When you decorate a task function with the `manager.distributed_task` decorator, it actually wraps the enclosed function up in a `DistributedTask` object and returns that object. This object is still callable, so you can write your code as if you're interracting with the underlying function, but what is actually happening is something like the following:

* _Worker(s) start polling for tasks when the program starts..._
* Decorated function is called with input
* New task of the type specified in the decorator is submitted to manager with the given input
* Start polling for the result of that task...
* _A worker checks the task out_
* _The worker runs the task_
* _The worker writes the result back to the manager_
* The poll finally returns the result (or raises an exception)

Note that any exceptions will lose their underlying type and get thrown as a `TaskException` - the original exception and relevant stack trace can be found with the worker.

The underlying `DistributedTask` object provides the methods `start` and `await` for more fine-grained control over distribtued tasks. You can use these functions for parallelization. The following example creates multiple worker threads instead of just one. Note that this example uses the class `SingleProcessDistributedTaskManager` as its implementation of `AbstractDistributedTaskManager` (this implementation does not persist tasks).

`examples/parallel.py`

```python3
import time

from distributed_task import SingleProcessDistributedTaskManager
manager = SingleProcessDistributedTaskManager()

@manager.distributed_task("slow_thing:2", n_worker_threads=5)
def slow_thing():
  time.sleep(2)

if __name__ == "__main__":
  task_run_ids = [slow_thing.start() for i in range(5)]
  for task_run_id in task_run_ids:
    slow_thing.await(task_run_id)
```

### Controlling workers

You may want to change how your workers run. By default, they run in threads waiting on events (there is no busy-waiting). To control the workers yourself, set the `n_worker_threads` parameter of the `manager.distributed_task` decorator to `0`, and call the zero-argument `worker` method on the `DistributedTask` object returned by the decorator. 

```python3
from distributed_task import SingleProcessDistributedTaskManager
manager = SingleProcessDistributedTaskManager()

@manager.distributed_task("my_task:1", n_worker_threads=0)
def my_task():
  pass

if __name__ == "__main__":
  # Run forever waiting for tasks that will never come...
  my_task.worker()
```

## Test

```
python3 -m unittest discover -b -v tst
```

## Build

```
pip3 uninstall distributed-task -y
python3 setup.py clean --all && python3 setup.py build && python3 setup.py install --force
rm -rf build dist distributed_task.egg-info

# As a one-liner:
pip3 uninstall distributed-task -y && python3 setup.py clean --all && python3 setup.py build && python3 setup.py install --force && rm -rf build dist distributed_task.egg-info
```

## TODO

* Error reporting - perhaps allow a custom `on_error` handler that workers call upon finishing a task, or the manager monitor calls upon discovering a task that has timed out

* Limit the number of tasks that can be submitted to keep our data file manageable

* Consider adding optional encryption so the input / output is not stored in plain text (at the worker level)

* Add a cloud-based manager strategy implementation like SWF
