So,
I've run into an issue where some of the files requested come back as bzip files:
[bmurch@optics0 014]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --interface 2607:fe50:0:6330::117 --retry 5 --retry-delay 2 --max-time 0 --output ~/N202001100_MET_NCEPR2_6h.hdf https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/N202001100_MET_NCEPR2_6h.hdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 829k 100 829k 0 0 654k 0 0:00:01 0:00:01 --:--:-- 654k
[bmurch@optics0 014]$ ll ~/N202001100_MET_NCEPR2_6h.hdf
-rw-rw-r-- 1 bmurch usfuser 849466 Jan 18 00:27 /home/b/bmurch/N202001100_MET_NCEPR2_6h.hdf
[bmurch@optics0 014]$ file /home/b/bmurch/N202001100_MET_NCEPR2_6h.hdf
/home/b/bmurch/N202001100_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
Not all by any means, but enough to kiil off my processing. eg:
[bmurch@optics0 010]$ file * | grep bzip
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/010/N202001000_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/010/N202001006_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/010/N202001012_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/010/N202001018_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
[bmurch@optics0 013]$ file * | grep bzip
N202001300_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
N202001306_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
N202001312_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
N202001318_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
[bmurch@optics0 013]$ cd ../014
[bmurch@optics0 014]$ file * | grep bzip
N202001400_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
N202001406_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
N202001412_MET_NCEPR2_6h.hdf: bzip2 compressed data, block size = 100k
[bmurch@optics0 014]$ cd ../015/
[bmurch@optics0 015]$ file * | grep bzip
N2020015_SST_OIV2AV_24h.nc: bzip2 compressed data, block size = 100k
[bmurch@optics0 015]$ cd ../016/
[bmurch@optics0 016]$ file * | grep bzip
N2020016_SST_OIV2AV_24h.nc: bzip2 compressed data, block size = 100k
[bmurch@optics0 016]$ cd ../017/
[bmurch@optics0 017]$ file * | grep bzip
Some anc files d/l as bzip file
-
- Posts: 338
- Joined: Wed Apr 06, 2005 12:11 pm America/New_York
- Has thanked: 10 times
- Been thanked: 3 times
-
- Posts: 1519
- Joined: Wed Sep 18, 2019 6:15 pm America/New_York
- Been thanked: 9 times
Some anc files d/l as bzip file
The getacn.py script retruns the uncompressed filename. When the script downloads the file, it checks to see if the file is compressed and if so uncompresses it.
Since the download portion is currently not working, when downloading you will need to check the content-disposition header (or use the
Sean
Since the download portion is currently not working, when downloading you will need to check the content-disposition header (or use the
file
command) to see if the file returned is compressed.Sean