Issues Integrating NASA Earthdata APIs for Real-Time Geospatial Data on My Website

[joebob9890]

Hello everyone,

I’m working on integrating NASA’s Earthdata APIs into my website to display real-time geospatial data (e.g., MODIS imagery, land surface temperature, etc.). The goal is to create an interactive map where users can view specific datasets based on their region of interest.

Here’s what I’ve done so far:

I registered my app and obtained an API key.
I’m using Leaflet.js for the map interface.
I set up the API calls using JavaScript to fetch data (e.g., GeoJSON for map layers).
However, I’ve encountered the following issues:

API Rate Limiting: I sometimes receive 429 Too Many Requests errors when loading multiple layers simultaneously. Is there a recommended way to handle this efficiently, like batching requests?
Data Projection Issues: The GeoJSON data fetched doesn’t align correctly with my Leaflet map. My map uses Web Mercator (EPSG:3857). Is there a way to reproject the data dynamically to ensure proper alignment?
Slow Load Times: Large datasets take a significant amount of time to load, which affects user experience. Are there best practices for caching or reducing data size without losing critical details?
I’d appreciate any guidance, examples, or links to relevant documentation to resolve these issues.

Thank you in advance!

[LP DAAC - dgolon]

Hi @joebob9890 We need a little more detail. Such as: Are there any specific APIs you are trying to use? How many requests are you making per hour and per day? This could be the cause of the rate limits you are seeing if you are requesting above the rate limit. How often are you wanting to generate new maps, are these generated on-demand or statically updated? How many datasets are you trying to incorporate? What time frames are you looking for (i.e. generating across the whole archive or within the last day, week, year)? What is the area of interest (i.e. creating global maps or maps over a specific user defined AOI, or over pre-defined areas such as countries or counties)? Are there specific MODIS datasets you are using, based on your mention of LST I am assuming the MOD/MYD11s and/or MOD/MYD21s any others? Also noting, we strongly recommend you start looking into VIIRS as well as MODIS will be sunsetting in the next couple of years. All of that will factor into the speed issue you are seeing.

Once we have answers to those questions, I'll talk to a few of our developers to see if they have any advice. Please note a lot of our team, and teams at other DAACs, are at AGU this week, so responses will be delayed. We'll also leave this open to the community, and any other DAACS that work with the APIs, in case they have any advice in the meantime.

[joebob9890]

joebob9890 wrote:
> Hello everyone,
>
> I’m working on integrating NASA’s Earthdata APIs into my website to display
> real-time geospatial data (e.g., MODIS imagery, land surface temperature,
> etc.). The goal is to create an interactive map where users can view
> specific datasets based on their region of interest.
>
> Here’s what I’ve done so far:
>
> I registered my app and obtained an API key.
> I’m using Leaflet.js for the map interface.
> I set up the API calls using JavaScript to fetch data (e.g., GeoJSON for
> map layers).
> However, I’ve encountered the following issues:
>
> API Rate Limiting: I sometimes receive 429 Too Many Requests errors when
> loading multiple layers simultaneously. Is there a recommended way to
> handle this efficiently, like batching requests?
> Data Projection Issues: The GeoJSON data fetched doesn’t align correctly
> with my Leaflet map. My map uses Web Mercator (EPSG:3857). Is there a way
> to reproject the data dynamically to ensure proper alignment?
> Slow Load Times: Large datasets take a significant amount of time to load,
> which affects user experience. Are there best practices for caching or
> reducing data size without losing critical details?
> I’d appreciate any guidance, examples, or links to relevant documentation
> to resolve these issues.
>
> Thank you in advance!
Thank you for the detailed response! I really appreciate the guidance and understand that many of the team members are currently busy with AGU. Let me provide more details to address your questions:

Specific APIs: I’m primarily working with the MODIS APIs for MOD/MYD11 (Land Surface Temperature) and MOD/MYD21 (Surface Reflectance). Based on your recommendation, I will explore VIIRS data for future integration.

Rate Limits: On average, I’m making around 300-400 requests per hour and approximately 2,000-3,000 requests per day. These requests vary depending on user interaction, as maps are generated on-demand based on specific user-selected Areas of Interest (AOIs).

Map Generation Frequency: The maps are dynamically generated. Users can filter datasets by date range (e.g., last 7 days, last 30 days) and visualize specific parameters in real-time.

Datasets and Timeframes: Alongside MODIS LST datasets, I plan to incorporate vegetation indices and NDVI products. The focus is on recent data (typically the past 30 days) for real-time analysis.

Area of Interest: The AOIs are user-defined regions. However, I also support predefined AOIs, such as country borders or custom polygons uploaded by users.

Given these specifics, I aim to optimize data retrieval and display processes to ensure a seamless user experience. If it would help, I’d be happy to share the URL of my website so you can get a better idea of the interface and how I’ve integrated the APIs so far. Perhaps seeing it in action might make it easier to identify potential improvements or best practices. Thanks again for your support! Looking forward to any suggestions you or the community might have.

[ijallison]

Hello, can you provide some example queries including what parameters you are using?

It sounds like you are using the CMR.

CMR rate limits kick in when a large number of queries with particular, targeted parameter patterns hit CMR. We've identified these query signatures as problematic when the total number of these queries exceeds some limit. We do not throttle based on a particular user per se; it is based on overall volume of those queries from all users.

That is, your search rate might be fine on its own (I will need to know what your searches look like to determine this), but if there are many searches with similar signatures hitting CMR at the same time, you will start to see 429s. This is to protect CMR stability and user experience.

Something that could help us out is if you include the Client-Id header to your searches - this will allow us to easily drill down on your searches in our logs.

Regarding slow load times - again, I will need to see your queries before knowing what guidance I could give you. Generally, if your running searches on large datasets, you may see some latency.

[jteague1]

Hi, in order to help troubleshoot your 429 errors, can you provide the following:
- You should be getting additional descriptive error text, along with the 429 status; can you check and confirm this? Ideally, this error text would provide additional context for the error
- Can you post an example of your CMR search query that is generating the 429? Note, if you are using scrolling to iterate over the CMR result set this could be a unique source of the 429.
John Teague
CMR Operations