Execution Model

Job Hierarchies

In GoCollaborate, Task serves as the minimum unit of the transaction to process, hence all operations will converge to a map of Tasks by the end of each Stage. However, the result set from an accomplished Task could be passed on to the next Stage as specified in Job. This enables both reading and writing data files to persistent memory for successive execution.

Job

A Job is a set of tasks to be processed when its handler is called

Stage

A Stage is a sub-set of Job where the tasks therein have no concurrent dependencies on one another.

Task

A Task is the minimal unit of computation on the given dataset.

TaskType

TaskType specifies the expected timeout type of each task to be executed, currently, ROUTINE and PERMANENT type are not supported.

TaskPriority

TaskPriority specifies the priority of each task, workers in the worker pool will pull tasks from Master based on this order.

Function

A Function instance is a First-Class Function defined for task processing. It looks something like this:

func(source *task.Collection,
        result *task.Collection,
        context *task.TaskContext) bool {
        // do whatever
}

Executors

The Executor is the base struct of task operation handlers such as Mapper, Reducer or Combiner, etc.

Mapper

A Mapper should always implement the Mapper Interface, where it maps the original task set to an incremental map of tasks.

Reducer

A Reducer should always implement the Reducer Interface, where it maps the original task set to a decremental map of tasks.

Shuffle

In most of the Java implementations of MapReduce algorithm, there is usually a separate Shuffle phase which is responsible for re-arranging the order of mapped dataset. In Golang, however, this is not mandatory. The range literal in Golang will automatically shuffle the existing map during runtime execution (reference).

Mechanisms

Non-Blocking Queue

The execution of tasks in GoCollaborate is non-blocking, meaning that you can still write an HTTP response while your previously-triggered tasks are still under processing. This will reduce the performance issue caused by a batch-processing task unit.

Project Structure Auto-mapping

To better organize your API exposure, the GoCollaborate framework automatically maps the routes to your project structure.

For example, the handlers defined in /core directory are mapped to the route /core/handler_name, by which you can focus on writing the handler of your task logic instead of considering unnecessary exposure structure.