Geo algorithm

We developed a geo algorithm to solve a problem with the following properties:

• We can obtain certain data based on geo coordinates.
• Before the data request, and afterwards, the completeness of the data set is unknown.
• What is known, is that if the geographical area for which the data was requested is smaller, the data is more accurate
• There are limitations on how many requests we can do
  
• The resulting data set may be very large

We created an algorithm that internally creates heatmaps based on the data. The algorithm refines the areas into smaller areas. By comparing these areas, it decides if the refinement makes sense and leads to a morge complete dataset, or if the refinement can be rejected.

The result is a list of usable heatmaps with data, varying in size.

Because the algorithm must handle huge amounts of data, and the process may take very long, we cannot execute it in a single process. There would simply not be enough memory, and it would not be fault tolerant.

Hence the process creates many smaller tasks, that can be executed standalone and asynchronously. Each subtask can create it's own subtask. This makes the data aggregation fault tolerant, and restartable.

To control this process, we used de 'workflow' and 'state machine' design patterns. These check the state of the algorithm, and decide what transitions the system can go through.