We created an algorithm that internally creates heatmaps based on the data. The algorithm refines the areas into smaller areas. By comparing these areas, it decides if the refinement makes sense and leads to a morge complete dataset, or if the refinement can be rejected.
The result is a list of usable heatmaps with data, varying in size.
Because the algorithm must handle huge amounts of data, and the process may take very long, we cannot execute it in a single process. There would simply not be enough memory, and it would not be fault tolerant.
Hence the process creates many smaller tasks, that can be executed standalone and asynchronously. Each subtask can create it's own subtask. This makes the data aggregation fault tolerant, and restartable.
To control this process, we used de 'workflow' and 'state machine' design patterns. These check the state of the algorithm, and decide what transitions the system can go through.