LAADS Web Tutorials

Mechanisms to search, access, and acquire publicly available data products from LAADS Web

The following tutorial describes the process-flows involved in searching for, accessing, and acquiring MODIS, VIIRS, and other publicly available data products from the LAADS Web. It describes how users may accomplish these tasks through three different processes (click to expand sections):

Search for Data

Designed for new users, and those who wish to search and order particular products in specific geographic space-time windows, this interface provides a five-step process. Before we describe these steps, you need to fulfill a couple of requirements:

  • Complete user registration with NASA’s Earthdata User Registration System (URS): Select and enter a username and password.
  • Following user registration, you need to enable the LAADS Web application in your URS profile: Go to the Earthdata URS Profile page; under the “Applications” tab, click “Authorized Apps” and add “LAADS Web” to the list of enabled applications.

After you complete these two steps, log in to LAADS Web by selecting “Earthdata Login” under “Profile” in the top-right, and provide your login credentials.

The Search and Order interface is defined by five logically sequenced, numbered steps, from left to right on the top: 1 PRODUCTS, 2 TIME, 3 LOCATION, 4 FILES, and 5 REVIEW & ORDER. The following paragraphs briefly explain each of these steps, which collectively describe the end-to-end process flow to search, select, and place an order.

1 PRODUCTS: All public LAADS products are available for selection via two drop-down menus: “All Sensors” and “All Searchable Collections.” The first is a selection of all the source sensors or instruments whose data products are available. The second is the list of all the available data collections. In LAADS’ parlance, a collection equates to both a data processing version number and a unique set of data from a particular production cycle. Selecting a particular sensor would automatically reveal only the relevant collection(s) associated with that sensor-derived products. Based on your selection of sensor and collection, a table is presented with the short- and long-names of products that satisfy your chosen sensor-collection criteria. Each product has an info glyph or icon to its right, which is a link to detailed information for that product.

2 TIME: You can either define (via drop-down selections) a single date or a date range for your temporal search window. Under the +Advanced link, users may also define single date(s) and time(s) or date ranges without the time component.

3 LOCATION: You can define your geographic search window by a number of means:

  • Select or manually define your horizontal and vertical tile addresses (e.g., H08 V05)
  • Draw your own custom-defined polygon
  • Manually enter the latitude/longitude coordinates defining your search-space
  • Select by country
  • Select by a variety of validation sites

4 FILES: Assuming that you have entered or selected all the requisite inputs for the first three steps, clicking FILES will initiate the search, and return data granules that meet the selected criteria. The search results are returned in a table that include the following columns:

Filename | Info| Product (collection) | Image | Date/Time | Download

  • Filename: The filename with short-name, acquisition year and day, time, tile address, collection number, production year, date, and time, and format suffix

(For details, check: https://tinyurl.com/y8cfr6h8)

  • Info: The Info button provides a number of file-level details for each granule including File name, File size, Checksum, etc. A checksum is a specific numeric value that is derived from a digital data object (such as a data file), which helps detect errors that may have occurred during data transmission or storage. Users can run a checksum on their acquired files to match with the published (checksum) values to verify the data integrity.
  • Product (collection): Product shortname and collection number in parenthesis.
  • Image: Browse image for the particular dataset, if applicable. Not all products have supported browse images.
  • Date/Time: The defined date and time
  • Download: Direct download link that also indicates the file size in megabytes (MB)

You may select one, more, or all files, and click REVIEW & ORDER.

5 REVIEW & ORDER: This page offers users a number of options that may vary with each product:

  1. Apply post-processing: Post-processing may include many data transformation options such as parameter subsetting to deliver only those Science Data Set (SDS) variables that you require, spatial subsetting to reduce the geographic footprint, format conversion, resampling, etc. Offered post-processing options vary with each product type.
  2. Select delivery method: You can select a delivery method, the default being “Pull” and hit the “Submit Order” button. Once your order is processed and ready, you are notified by e-mail with details on how to download your ordered data products.
  3. Manage your orders: You can access your User Profile (the last link called “Past Orders” on the left-side in the search and order interface) to check the status of your order, resubmit a previously submitted order, kill an order, etc.

Users should keep in mind that if their search criteria identifies more than 2000 files, then no results are returned; instead you are asked to revise your search criteria.

The LAADS Online archive refers to cached data on spinning disk that offers all public product collections that are readily available to download without a registered user having to conduct a search. We offer two pathways to navigate through this online archive to find and acquire the requisite data for your requirements:

Science domain-driven navigation

View Science domain-driven navigation

This newly implemented capability is designed to streamline users to intuitively follow the science domain attributes to target their datasets of interest. This (and the /allData tree) method is appropriate for seasoned users who know what datasets they require for their research or applications, and can afford to forego any post-processing transformations. It also is beneficial for those who depend on a constantly acquired time-series of products for regularly computed application requirements.

This capability allows users to naturally follow the taxonomic structure by selecting their discipline (e.g., atmosphere, land, Level-1, etc.), product suite (e.g., Aerosol, Land Surface Reflectance, etc.), particular products, their year, and day-of-year, etc. For instance, the following pathway provides a science domain-driven access to the SNPP VIIRS Deep Blue Aerosol L2 products from January 4th 2015:

http://ladsweb.modaps.eosdis.nasa.gov/archive/Science Domain/Atmosphere/Aerosol/VIIRS SNPP C1.0 - Deep Blue Aerosol L2 6-Min Swath 6 km/2015/004/

Once users are on this page that lists all datasets for that particular day-of -year, they have the option to select and download one or more datafiles. They can also avail of the multi-file download capability that allows them to click individual table rows (or select multiple rows by holding down the Shift key) to select files, and later click "Download Selected" to confirm their multi-file download.

/allData tree-based navigation

View /allData tree-based navigation

Registered users can navigate through the LAADS /allData tree, and directly download their preferred data products through this online archive. Prior to being able to do that, they need to know a couple of things. They include the product’s collection number and its short-name. In this context, it is important to clarify the existence of another attribute called “Archive Set,” a numeric identifier that primarily relates to archive storage. Users strictly shouldn’t have to know or keep track of archive set numbers, but currently need to understand how they map to collection numbers. The table below presents the LAADS DAAC’s public product collection numbers and their mapping to archive set numbers. Product short-names are used within the archive directory structure, but they are not very descriptive and sometimes difficult to remember, so many users use the LAADS product descriptions to find each product’s short-name.

Collection Archive Set number Mission | Instrument
MODIS Collection 6 6 Terra MODIS, Aqua MODIS
MODIS Collection 5.5 55 Terra MODIS
MODIS Collection 6.1 61 Terra MODIS, Aqua MODIS
VIIRS Collection 1 (Land) 5000 SNPP VIIRS
VIIRS Collection 1 (L1, Atmos.) 5110 SNPP VIIRS
VIIRS Collection 1.1 (Atmos.) 5111 SNPP VIIRS
VIIRS Collection 2 5200 JPSS-1 VIIRS
LTDR Collection 4 464 NOAA AVHRR
LTDR Collection 5 465 NOAA AVHRR
NACP Collection 4 404 Terra MODIS
SLSTR & OLCI Collection 1 450 Copernicus Sentinel SLSTR & OLCI
MERIS Collection 1 490 Envisat MERIS

Assuming users know the collection number and short-name of their desired products, it is fairly straightforward to navigate through the online archive; For instance, the following URL provides the location that lists all 240 6-minute datasets for the SNPP VIIRS Collection 1 (Archive Set 5110) Day-Night Band Geolocation products (VNP03DNB) for 2015, Day-Of-Year 004:

http://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5110/VNP03DNB/2015/004/

When you choose to download a certain dataset, it gives you the option to either open the file (by selecting Open with) with a particular image-handling software package, or save it to your local system’s default Download directory. A couple of simple, open-source packages to visualize your downloaded products include Panoply and HDFView; You need to know the temporal frequency (e.g., 6-minute, Daily, 8-day, Monthly, etc.) of your preferred products to navigate to the correct location to find the listed data granules.

Using scripts provide an efficient means to acquire multiple data products from the same collection/product, data-day, mixed collections/products, assorted data-days, etc. Common methods include wget and curl. Wget (for WWW get) is an open-source program from the GNU Project that is designed to help download content from web servers. Curl (for Client URL) is another open-source software that is used in command-line and scripts to transfer data using different network protocols. Consult LAADS Data Download Scripts for help, code samples, and additional information regarding creating and using download scripts.

According to W3C, a Web service is a software system designed to support interoperable machine‐to-machine interaction over a network. Web services use Extensible Markup Language (XML) syntax to construct their messages. The advantages of an XML-based Web service architecture include the following:

  • platform independence
  • physical location independence (i.e., where the message is being sent from)
  • application language independence
  • client's knowledge independence (i.e., what kind of processor the server uses)

Web services use Hypertext Transfer Protocol (HTTP) as the underlying communication protocol that facilitates how applications send XML messages to Web services. Representational State Transfer (REST) and Simple Object Access Protocol (SOAP) comprise messaging protocols to exchange structured information to implement web services within a network.

The LAADS DAAC currently offers a handful of Web services that more experienced users can use to programmatically access and retrieve their desired data products. They include a couple that qualify as open standards (OGC WCS and OPeNDAP) and an In-house API called LAADS Web Services. An API is an acronym for Application Programming Interface, which is a software interface that allows two applications to interact with each other without any user intervention. They facilitate communication between one set of products or services with another set while remaining agnostic regarding implementation details.

What you should know before you use any of the following three Web services

  1. A critical detail that users should keep in mind while using these Web services relates to the Collection vs. Archive set distinction that is described above in the LAADS Online archive tutorial. To recap, the Collection number reflects the version ID of the current production cycle for a certain product while the Archive set number points to an archive storage detail. The mapping between these two attributes is what the above table provides. When you develop your API for all three Web services, the URL require a collection=value parameter for some cases — you need to ensure that you always set the value to “ArchiveSet.”
  2. You should use a client that understands both how to make a request to the service and how to process the results of that service.
    • All services provide documentation on how to write a client, while some services already have clients that support making a request to the service and processing its results.
    • All services respond to HTTP requests from web browsers.
  3. Each of the following Web services have unique capabilities, and users should consider when and why they may want to use each of them. Each individual section includes some general pointers to aid users in their decision as they consider its use.

Open Geospatial Consortium (OGC) – Web Coverage Service (WCS)

OGC’s WCS standard enables Web-mediated retrieval of coverages. A coverage is a digital representation of a spatio-temporal object that usually comprises the content of a geographic information system (GIS) process or service. The MODAPS WCS API follows the OGC WCS API, although our implementation (at version 1.0) does not conform to the latest version of the standard, which is now at version 2.1. Users would require an XML parser to process the output of the service. Some GIS tools (for instance ESRI tools and those based on GDAL library) can understand the protocol. Similar to the benefits from other Web services, this service provides the ability to directly stream filtered, subsetted, reformatted and regridded data that conform to user specifications.

MODAPS WCS API that provides access to MODIS Level-1 and Atmosphere data products

OGC’s WCS Standard

OpenAPI for Coverages — GitHub

Open-source Project for a Network Data Access Protocol (OPeNDAP)

OPeNDAP, an acronym for “Open-source Project for a Network Data Access Protocol” is a data transport architecture and protocol used in the earth science modelling, research, and applications communities. OPeNDAP was designed and implemented to provide researchers universal access to science data products using the same client/server model provided by the World-Wide Web and the Internet. Clients can submit specifically tailored requests to web servers supporting web pages that handle and manage the science data products. Client-specified requests for datasets conforming to a certain format, space-time, or variable-subset are directly delivered to the requesting client.

OPeNDAP provides a data server architecture that provides access to data stored on remote servers that end-users can access and use within their own client software. See a list of all the available OPeNDAP client software and their libraries. OPeNDAP data requests are submitted via URLs; the URL’s syntax contains all the elements, i.e., product name, science dataset or variable name, spatial and temporal details for the requesting action, etc. A major advantage bestowed by OPeNDAP is the ability to perform data filtering, subsetting, and data format transformation operations directly from the client. OPeNDAP is limited in its ability to conduct searches, and cannot search by area of interest.

All the publicly available LAADS DAAC product collections are available via OPeNDAP

For additional information about OPeNDAP, check the following sources:

OPeNDAP User Guide from the OPeNDAP organization

NASA Earthdata OPeNDAP User Guide

How to use OPeNDAP to access data – video tutorial from NASA JPL PO.DAAC

OPeNDAP URL syntax examples that demonstrate how to download particular LAADS datasets:

  1. Download Aqua-MODIS MYD04_3K, Collection-6.1 product from year 2015, day 130, time 0800 in netCDF
http://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/61/MYD04_3K/2015/130/MYD04_3K.A2015130.0800.061.2018049210111.hdf.nc
  1. Download SNPP-VIIRS VNP02IMG, Collection-1 product from year 2015, day 40, time 0512 in ascii
http://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/5110/VNP02IMG/2015/040/VNP02IMG.A2015040.0512.001.2017259174514.nc.ascii
  1. Download Terra-MODIS MOD03, Collection-6 product from year 2013, day 56, time 0000 in netCDF:
http://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD03/2013/056/MOD03.A2013056.0000.006.2013056075649.hdf.nc
  1. Download Terra-MODIS MOD03, Collection-6 product from year 2013, day 56, time 0000 in ASCII format:
http://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD03/2013/056/MOD03.A2013056.0000.006.2013056075649.hdf.ascii
  1. Download Terra-MODIS MOD03, Collection-6 product from year 2013, day 56, time 0000 in netCDF but only include the Scan_Offset variable in a range from 0 to 4059 with a stepping of 1 in the first dimension, and a range of 0 to 2707 with a stepping of 1 in the second dimension.
http://ladsweb.modaps.eosdis.nasa.gov/opendap/allData/6/MOD03/2013/056/MOD03.A2013056.0000.006.2013056075649.hdf.nc?Scan_Offset%5b0:1:4059%5d%5b0:1:2707

In-house API – LAADS Web Service (LWS) Classic

LWS Classic is an in-house-developed, custom SOAP- and REST-based API that allows users to search, order, and download MODIS and VIIRS Level-1 and higher-level atmosphere data products through a programmable interface. It requires an XML parser to process the output of the service. It provides extensive search capabilities, and the standard set of data transformation functions. It does, though, require extra steps to order and poll for completion, or directly download subsetted, reformatted, and regridded data. Consult the following links for additional information:

LWS Classic Quick Start Guide

LWS Classic API