Page 1 of 1

Batch Processing Sentinel-2 in l2gen

Posted: Wed Dec 05, 2018 12:57 am America/New_York
by dogun
Wondering if there's a way to batch process multiple Sentinel-2 scenes of different areas in l2gen? Also, is it possible to make l2gen with Sentinel-2 scenes run faster (multiprocessor?).

Thanks!

Batch Processing Sentinel-2 in l2gen

Posted: Wed Dec 05, 2018 7:31 am America/New_York
by gnwiii
If you have many files to process and have a suitable multi-processor system you can run multiple jobs.   GNU parallel is designed for just this sort of task:

"GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel."

Note that l2gen processing is quite I/O intensive and many modern muti-core CPU's can saturate disk IO.  You may find it helpful to write output to a separate filesystem.
A common strategy to determine how many jobs can be run in parallel is to start with 2 simultaneous, then 4, 8, ... until thruput falls off.