Basic usage

Let’s suppose that you have developed a number of simple Python scripts that do some useful data processing. For example, the first script requests satellite metadata from provider by certain parameters. Then other scripts load data, calibrate it, produce RGB image and download it to the site storage. Each of these scripts is developed separately and can be easily debugged. Intermediate files that used by several scripts could be formed during the processing. Everything seems to be working fine manually but the production needs to be automated. In such cases, alt_processing comes to the rescue.

Two tasks are configured:

• Periodic Task includes a script, that discovers new data and generates the Event for each new satellite scene with its parameters.
• Event Task includes a chain of scripts that process new scene: load, calibrate, form RGB image, and deliver results.

What is important is that only slight modification of existing scripts is needed. Everything is working the same way it would work during a manual process.

For each Event the Launcher automatically creates Processing as chain of Scripts, makes temporary working directories for the whole Processing and each Script, run Scripts in sequence according to available resources, rerun Script in the case of a soft error, delete temporary directories after execution to keep required amount of free space. User web interface shows the status of Processings, gathers statistic of success and failures, allows to manually rerun certain Processing and Scripts, pauses Tasks and much more. The same functions could be done via API and even from running Script that gives utter control over processing features.

System starts working but as always something goes wrong sometimes. For example one of the Script fails from time to time and log file does not help much. In this case before temporary directory is not deleted there is an opportunity to run and debug failed Script in the terminal. This speeds up development and debug process simultaneous with System production exploitation.