How to Access LAADS Data
Mechanisms to search, access, and acquire publicly available data products from LAADS Web
Before we begin:
- Make sure you have an Earthdata login before you start using the LAADS search and order interface.
- Log in to LAADS Web by selecting “Earthdata Login” under “Profile” in the top-right and provide your login credentials.
Now that you are logged in, open the LAADS Search and Order interface:
1 Select the data product(s) you need under Products.
All public LAADS products are available for selection via two drop-down menus: “All Sensors” and “All Searchable Collections.”
The first is a selection of all the source sensors or instruments whose data products are available. The second is the list of all the available data collections. In LAADS’ parlance, a collection equates to both a data processing version number and a unique set of data from a particular production cycle.
Selecting a particular sensor automatically reveals only the relevant collection(s) associated with that sensor. Based on your selection of sensor and collection, a table is presented with the short- and long-names of products that satisfy your chosen sensor-collection criteria. Each product has an info glyph or icon to its right, which is a link to detailed information for that product.
2 Select the temporal resolution.
You can either define (via drop-down selections) a single date or a date range for your temporal search window. Under the +Advanced link, users may also define single date(s) and time(s) or date ranges without the time component.
3 Select the data location.
You can define your geographic search window in several ways:
- Select or manually define your horizontal and vertical tile addresses (e.g., H08 V05)
- Draw your own custom-defined polygon
- Manually enter the latitude/longitude coordinates defining your search-space
- Select by country
- Select by a variety of validation sites
4 Generate the product files.
FILES will initiate the search and return data granules that meet the selected criteria. The search results are returned in a table that include the following columns:
Filename | Info| Product (collection) | Image | Date/Time | Download
- Filename: The filename with short-name, acquisition year and day, time, tile address, collection number, production year, date, and time, and format suffix
(For details, check: https://tinyurl.com/y8cfr6h8 )
- Info: The Info button provides several file-level details for each granule including File name, File size, Checksum, etc. A checksum is a specific numeric value that is derived from a digital data object (such as a data file), which helps detect errors that may have occurred during data transmission or storage. Users can run a checksum on their acquired files to match with the published (checksum) values to verify the data integrity.
- Product (collection): Product short-name and collection number in parenthesis.
- Image: Browse image for the dataset, if applicable. Not all products have supported browse images.
- Date/Time: The defined date and time.
- Download: Direct download link that also indicates the file size in megabytes (MB).
You may select one, more, or all files, and click REVIEW & ORDER.
5 Review and order the selected data products.
This page offers users several options that may vary with each product:
- Apply post-processing: post-processing may include many data transformation options such as parameter subsetting to deliver only those Science Data Set (SDS) variables that you require, spatial subsetting to reduce the geographic footprint, format conversion, resampling, etc. Offered post-processing options vary with each product type.
- Select delivery method: You can select a delivery method, the default being “Pull” and hit the “Submit Order” button. Once your order is processed and ready, you are notified by e-mail with details on how to download your ordered data products.
- Manage your orders: You can access your User Profile (the last link called “Past Orders” on the left-side in the search and order interface) to check the status of your order, resubmit a previously submitted order, kill an order, etc.
*Keep in mind that if your search criteria identifies more than 2000 files, no results are returned; instead you are asked to revise your search criteria.
The LAADS Online archive refers to cached data on spinning disk that offers all public product collections that are readily available to download without a registered user having to conduct a search. We offer two pathways to navigate through this online archive to find and acquire the requisite data for your requirements:
Science domain-driven navigation
This newly implemented capability is designed to streamline users to intuitively follow the science domain attributes to target their datasets of interest. This (and the /allData tree) method is appropriate for seasoned users who know what datasets they require for their research or applications, and can afford to forego any post-processing transformations. It also is beneficial for those who depend on a constantly acquired time-series of products for regularly computed application requirements.
This capability allows users to naturally follow the taxonomic structure by selecting their discipline (e.g., atmosphere, land, Level-1, etc.), product suite (e.g., Aerosol, Land Surface Reflectance, etc.), particular products, their year, and day-of-year, etc. For instance, the following pathway provides a science domain-driven access to the SNPP VIIRS Deep Blue Aerosol L2 products from January 4th 2015:
http://ladsweb.modaps.eosdis.nasa.gov/archive/Science Domain/Atmosphere/Aerosol/VIIRS SNPP C1.0 - Deep Blue Aerosol L2 6-Min Swath 6 km/2015/004/
Once users are on this page that lists all datasets for that particular day-of -year, they have the option to select and download one or more datafiles. They can also avail of the multi-file download capability that allows them to click individual table rows (or select multiple rows by holding down the Shift key) to select files, and later click "Download Selected" to confirm their multi-file download.
/allData tree-based navigation
Registered users can navigate through the LAADS /allData tree, and directly download their preferred data products through this online archive. Prior to being able to do that, they need to know a couple of things. They include the product’s collection number and its short-name. In this context, it is important to clarify the existence of another attribute called “Archive Set,” a numeric identifier that primarily relates to archive storage. Users strictly shouldn’t have to know or keep track of archive set numbers, but currently need to understand how they map to collection numbers. The table below presents the LAADS DAAC’s public product collection numbers and their mapping to archive set numbers. Product short-names are used within the archive directory structure, but they are not very descriptive and sometimes difficult to remember, so many users use the LAADS product descriptions to find each product’s short-name.
|Collection||Archive Set number||Mission | Instrument|
|MODIS Collection 6||6||Terra MODIS, Aqua MODIS|
|MODIS Collection 5.5||55||Terra MODIS|
|MODIS Collection 6.1||61||Terra MODIS, Aqua MODIS|
|VIIRS Collection 1 (Land)||5000||SNPP VIIRS|
|VIIRS Collection 1 (L1, Atmos.)||5110||SNPP VIIRS|
|VIIRS Collection 1.1 (Atmos.)||5111||SNPP VIIRS|
|VIIRS Collection 2||5200||JPSS-1 VIIRS|
|LTDR Collection 4||464||NOAA AVHRR|
|LTDR Collection 5||465||NOAA AVHRR|
|NACP Collection 4||404||Terra MODIS|
|SLSTR & OLCI Collection 1||450||Copernicus Sentinel SLSTR & OLCI|
|MERIS Collection 1||490||Envisat MERIS|
Assuming users know the collection number and short-name of their desired products, it is fairly straightforward to navigate through the online archive; For instance, the following URL provides the location that lists all 240 6-minute datasets for the SNPP VIIRS Collection 1 (Archive Set 5110) Day-Night Band Geolocation products (VNP03DNB) for 2015, Day-Of-Year 004:
When you choose to download a certain dataset, it gives you the option to either open the file (by selecting Open with) with a particular image-handling software package, or save it to your local system’s default Download directory. A couple of simple, open-source packages to visualize your downloaded products include Panoply and HDFView; You need to know the temporal frequency (e.g., 6-minute, Daily, 8-day, Monthly, etc.) of your preferred products to navigate to the correct location to find the listed data granules.
Using scripts provide an efficient means to acquire multiple data products from the same collection/product, data-day, mixed collections/products, assorted data-days, etc. Common methods include wget and curl. Wget (for WWW get) is an open-source program from the GNU Project that is designed to help download content from web servers. Curl (for Client URL) is another open-source software that is used in command-line and scripts to transfer data using different network protocols. Consult LAADS Data Download Scripts for help, code samples, and additional information regarding creating and using download scripts.
According to W3C, a Web service is a software system designed to support interoperable machine‐to-machine interaction over a network. Web services use Extensible Markup Language (XML) syntax to construct their messages. The advantages of an XML-based Web service architecture include the following:
- platform independence
- physical location independence (i.e., where the message is being sent from)
- application language independence
- client's knowledge independence (i.e., what kind of processor the server uses)
Web services use Hypertext Transfer Protocol (HTTP) as the underlying communication protocol that facilitates how applications send XML messages to Web services. Representational State Transfer (REST) and Simple Object Access Protocol (SOAP) comprise messaging protocols to exchange structured information to implement web services within a network.
The LAADS DAAC currently offers a handful of Web services that more experienced users can use to programmatically access and retrieve their desired data products. They include a couple that qualify as open standards (OGC WCS and OPeNDAP) and an In-house API called LAADS Web Services. An API is an acronym for Application Programming Interface, which is a software interface that allows two applications to interact with each other without any user intervention. They facilitate communication between one set of products or services with another set while remaining agnostic regarding implementation details.
What you should know before you use any of the following three Web services
- A critical detail that users should keep in mind while using these Web services relates to the Collection vs. Archive set distinction that is described above in the LAADS Online archive tutorial. To recap, the Collection number reflects the version ID of the current production cycle for a certain product while the Archive set number points to an archive storage detail. The mapping between these two attributes is what the above table provides. When you develop your API for all three Web services, the URL require a
collection=valueparameter for some cases — you need to ensure that you always set the value to “ArchiveSet.”
- You should use a client that understands both how to make a request to the service and how to process the results of that service.
- All services provide documentation on how to write a client, while some services already have clients that support making a request to the service and processing its results.
- All services respond to HTTP requests from web browsers.
- Each of the following Web services have unique capabilities, and users should consider when and why they may want to use each of them. Each individual section includes some general pointers to aid users in their decision as they consider its use.
OGC’s WCS standard enables Web-mediated retrieval of coverages. A coverage is a digital representation of a spatio-temporal object that usually comprises the content of a geographic information system (GIS) process or service. The MODAPS WCS API follows the OGC WCS API, although our implementation (at version 1.0) does not conform to the latest version of the standard, which is now at version 2.1. Users would require an XML parser to process the output of the service. Some GIS tools (for instance ESRI tools and those based on GDAL library) can understand the protocol. Similar to the benefits from other Web services, this service provides the ability to directly stream filtered, subsetted, reformatted and regridded data that conform to user specifications.
Open-source Project for a Network Data Access Protocol (OPeNDAP)
OPeNDAP, an acronym for “Open-source Project for a Network Data Access Protocol” is a data transport architecture and protocol used in the earth science modelling, research, and applications communities. OPeNDAP was designed and implemented to provide researchers universal access to science data products using the same client/server model provided by the World-Wide Web and the Internet. Clients can submit specifically tailored requests to web servers supporting web pages that handle and manage the science data products. Client-specified requests for datasets conforming to a certain format, space-time, or variable-subset are directly delivered to the requesting client.
OPeNDAP provides a data server architecture that provides access to data stored on remote servers that end-users can access and use within their own client software. See a list of all the available OPeNDAP client software and their libraries. OPeNDAP data requests are submitted via URLs; the URL’s syntax contains all the elements, i.e., product name, science dataset or variable name, spatial and temporal details for the requesting action, etc. A major advantage bestowed by OPeNDAP is the ability to perform data filtering, subsetting, and data format transformation operations directly from the client. OPeNDAP is limited in its ability to conduct searches, and cannot search by area of interest.
For additional information about OPeNDAP, check the following sources:
How to use OPeNDAP to access data – video tutorial from NASA JPL PO.DAAC
OPeNDAP URL syntax examples that demonstrate how to download particular LAADS datasets:
- Download Aqua-MODIS MYD04_3K, Collection-6.1 product from year 2015, day 130, time 0800 in netCDF
- Download SNPP-VIIRS VNP02IMG, Collection-1 product from year 2015, day 40, time 0512 in ascii
- Download Terra-MODIS MOD03, Collection-6 product from year 2013, day 56, time 0000 in netCDF:
- Download Terra-MODIS MOD03, Collection-6 product from year 2013, day 56, time 0000 in ASCII format:
- Download Terra-MODIS MOD03, Collection-6 product from year 2013, day 56, time 0000 in netCDF but only include the Scan_Offset variable in a range from 0 to 4059 with a stepping of 1 in the first dimension, and a range of 0 to 2707 with a stepping of 1 in the second dimension.
In-house API – LAADS Web Service (LWS) Classic
LWS Classic is an in-house-developed, custom SOAP- and REST-based API that allows users to search, order, and download MODIS and VIIRS Level-1 and higher-level atmosphere data products through a programmable interface. It requires an XML parser to process the output of the service. It provides extensive search capabilities, and the standard set of data transformation functions. It does, though, require extra steps to order and poll for completion, or directly download subsetted, reformatted, and regridded data. Consult the following links for additional information: