And again today:
modis_GEO.py --parfile=./MOD00.P2020042.1325.ANGOLA_FULL.L1A_LAC.atteph.fixed MOD00.P2020042.1325.ANGOLA_FULL.L1A_LAC -o MOD00.P2020042.1325.ANGOLA_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
STATUS: $ocssw_return_2 = ./MOD00.P2020042.1325.ANGOLA_FULL.L1A_LAC.atteph.fixed
{'geogen': {}}
Missing attitude files!
Missing ephemeris files!
$ocssw_command_2 = modis_GEO.py --parfile=./MOD00.P2020043.0815.WGOM_FULL.L1A_LAC.atteph.fixed MOD00.P2020043.0815.WGOM_FULL.L1A_LAC -o MOD00.P2020043.0815.WGOM_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
STATUS: $ocssw_return_2 = ./MOD00.P2020043.0815.WGOM_FULL.L1A_LAC.atteph.fixed
{'geogen': {}}
Missing attitude files!
Missing ephemeris files!
Ancillary file delay
-
oo_processing
- Posts: 338
- Joined: Wed Apr 06, 2005 12:11 pm America/New_York
- Has thanked: 10 times
- Been thanked: 3 times
-
OB ODPS - jgwilding
- Subject Matter Expert
- Posts: 139
- Joined: Fri Feb 19, 2021 1:09 pm America/New_York
- Been thanked: 1 time
Ancillary file delay
For these granules, it appears that these would be the ATT/EPH files:
PM1ATTNR.P2020042.1200.003
PM1ATTNR.P2020043.0800.003
PM1EPHND.P2020042.1200.003
These appear to be available via https-getfile. Are you still seeing the problem now?
john
PM1ATTNR.P2020042.1200.003
PM1ATTNR.P2020043.0800.003
PM1EPHND.P2020042.1200.003
These appear to be available via https-getfile. Are you still seeing the problem now?
john
-
oo_processing
- Posts: 338
- Joined: Wed Apr 06, 2005 12:11 pm America/New_York
- Has thanked: 10 times
- Been thanked: 3 times
Ancillary file delay
No, the problem clears up eventually.
I suspect that the subscriptions are filled prior to the anc data being available, or in the database that the scripts query.
But it is an ongoing problem.
Brock
I suspect that the subscriptions are filled prior to the anc data being available, or in the database that the scripts query.
But it is an ongoing problem.
Brock
-
OB.DAACx - SeanBailey
- Posts: 1519
- Joined: Wed Sep 18, 2019 6:15 pm America/New_York
- Been thanked: 9 times
Ancillary file delay
Brock,
Your suspicion is pretty spot on. It's a long-ish story, the details of which would bore all but those most keen on data management intricacies, but the short version is this. The back-end of the subscription service has been modified. Previously, L0 and L1 subscriptions were not staged until the L2 file had been processed. This was a silly thing. So, now the files are staged sooner.
The consequence is that we may not have received the necessary attitude and ephemeris files before you see the subscription data are available. If you try to process those L0 data before the att/eph are available, you will get an error. The att/eph are usually close behind the L0, so if you do get an error, retry the failed process after a pause - 30 to 60 minutes should suffice.
Sean
Your suspicion is pretty spot on. It's a long-ish story, the details of which would bore all but those most keen on data management intricacies, but the short version is this. The back-end of the subscription service has been modified. Previously, L0 and L1 subscriptions were not staged until the L2 file had been processed. This was a silly thing. So, now the files are staged sooner.
The consequence is that we may not have received the necessary attitude and ephemeris files before you see the subscription data are available. If you try to process those L0 data before the att/eph are available, you will get an error. The att/eph are usually close behind the L0, so if you do get an error, retry the failed process after a pause - 30 to 60 minutes should suffice.
Sean
-
oo_processing
- Posts: 338
- Joined: Wed Apr 06, 2005 12:11 pm America/New_York
- Has thanked: 10 times
- Been thanked: 3 times
Ancillary file delay
Sean,
This becomes very tedious and fast. I have scripts that run every 1/2 hour and if subscription file is stages, I grab it.
And I restart failed jobs every hour, so some go through several failures. And as a result, it appears that the .usr_cookies file is often corrupted
When the subscriptions were set up, there were check boxes asking to wait for anc data to become available.
Can those be toggled now, and it they are, will my babysitting job I now have, end? :eek:
Just curious.
Cheers,
Brock
This becomes very tedious and fast. I have scripts that run every 1/2 hour and if subscription file is stages, I grab it.
And I restart failed jobs every hour, so some go through several failures. And as a result, it appears that the .usr_cookies file is often corrupted
When the subscriptions were set up, there were check boxes asking to wait for anc data to become available.
Can those be toggled now, and it they are, will my babysitting job I now have, end? :eek:
Just curious.
Cheers,
Brock
-
OB.DAACx - SeanBailey
- Posts: 1519
- Joined: Wed Sep 18, 2019 6:15 pm America/New_York
- Been thanked: 9 times
Ancillary file delay
Brock,
The cookie is really only needed for a session. You could delete it once the session is done and recreate it for the next. That'll get you past the potential corruption problem.
There is no ancillary data for L0 files, so there is no way to wait for something that doesn't exist. If your subscriptions were for L1 data, then the att/eph would most definitely be availble, as we don't catalog the L1 files until we can generate the geolocation file.
As for the "wait for anc", never was such a thing. There was (and still is) a wait for refined, but that only applies to L2 subscriptions. There was also an "include ancillary files", but that option was eliminated when getanc.py was created.
Sean
The cookie is really only needed for a session. You could delete it once the session is done and recreate it for the next. That'll get you past the potential corruption problem.
There is no ancillary data for L0 files, so there is no way to wait for something that doesn't exist. If your subscriptions were for L1 data, then the att/eph would most definitely be availble, as we don't catalog the L1 files until we can generate the geolocation file.
As for the "wait for anc", never was such a thing. There was (and still is) a wait for refined, but that only applies to L2 subscriptions. There was also an "include ancillary files", but that option was eliminated when getanc.py was created.
Sean