How to Access LAADS Data

Before we begin:

• Make sure you have an Earthdata login before you start using the LAADS search and order interface.
• Log in to LAADS Web by selecting "Earthdata Login" under "Profile" in the top-right and provide your login credentials.

Now that you are logged in, open the LAADS Search and Order interface:

Search for Data

1 Select the data product(s) you need under Products.

All public LAADS products are available for selection via two drop-down menus: "All Sensors" and "All Searchable Collections."

The first is a selection of all the source sensors or instruments whose data products are available. The second is the list of all the available data collections. In LAADS' parlance, a collection equates to both a data processing version number and a unique set of data from a particular production cycle.

Selecting a particular sensor automatically reveals only the relevant collection(s) associated with that sensor. Based on your selection of sensor and collection, a table is presented with the short- and long-names of products that satisfy your chosen sensor-collection criteria. Each product has an info glyph or icon to its right, which is a link to detailed information for that product.

2 Select the temporal resolution.

You can either define (via drop-down selections) a single date or a date range for your temporal search window. Under the +Advanced link, users may also define single date(s) and time(s) or date ranges without the time component.

3 Select the data location.

You can define your geographic search window in several ways:

• Select or manually define your horizontal and vertical tile addresses (e.g., H08 V05)
• Draw your own custom-defined polygon
• Manually enter the latitude/longitude coordinates defining your search-space
• Select by country
• Select by a variety of validation sites

4 Generate the product files.

FILES will initiate the search and return data granules that meet the selected criteria. The search results are returned in a table that include the following columns:

Filename | Info| Product (collection) | Image | Date/Time | Download

• Filename: The filename with short-name, acquisition year and day, time, tile address, collection number, production year, date, and time, and format suffix
  
  (For details, check: https://tinyurl.com/y8cfr6h8 )
• Info: The Info button provides several file-level details for each granule including File name, File size, Checksum, etc. A checksum is a specific numeric value that is derived from a digital data object (such as a data file), which helps detect errors that may have occurred during data transmission or storage. Users can run a checksum on their acquired files to match with the published (checksum) values to verify the data integrity.
• Product (collection): Product short-name and collection number in parenthesis.
• Image: Browse image for the dataset, if applicable. Not all products have supported browse images.
• Date/Time: The defined date and time.
• Download: Direct download link that also indicates the file size in megabytes (MB).

You may select one, more, or all files, and click REVIEW & ORDER.

5 Review and order the selected data products.

This page offers users several options that may vary with each product:

1. Apply post-processing: post-processing may include many data transformation options such as parameter subsetting to deliver only those Science Data Set (SDS) variables that you require, spatial subsetting to reduce the geographic footprint, format conversion, resampling, etc. Offered post-processing options vary with each product type. Some post-processing examples are below.

   • Sds

     select only the data sets that you want from a file. Most LAADS products contain many data sets (some contain thousands of data sets). If you will not need most of them, choose only the ones you will use. This can reduce the size and number of files you will need to download.

   • Geo

     select only the data from the geographic region you specify. Even if you searched for data from only a specific region in step (3), the files can contain more data, with only some of it actually intersecting the region you specified. By selecting this option, we will slice out all data that are not in your specified region. This can further reduce the size of files you will need to download, and will also reduce the amount of data you need to process.

   • Mosaic

     this option can combine the data from multiple files into one file, with all the data correctly geolocated. Some caveats:

     • all combined files will be from the same date (for lists of files from multiple dates, there will be a result file for each separate date)
     • data sets remain separate -- our tool does not combine data from different data sets together
     • where files overlap geographically, we will favor pixels based on whether the pixel contains a measurement value or not, and then favor the pixel nearest to the satellite's nadir position

   • Reformat

     change the format of the file. Currently, we offer the GeoTIFF file format, commonly used in Geographic Information System (GIS) tools.

   • Reproject

     change the Map Projection or Sample Size. Many maps and data sets from other instruments do not use the Geographic, or Cartesian, coordinate system. When overlaying data sets, they will line up better if they use the same coordinate system. The Reproject option allows one to specify parameters that govern the reproject operation. Refer to the U.S.Geological Survey Professional Paper 1395, "Map Projections -- A Working Manual", U.S. Government Printing Office, Washington: (C)1987, for detailed information on map projections.

   • Control

     Include process log files with the output. Most users will not find this option particularly useful.

2. Select delivery method: You can select a delivery method, the default being "Pull" and hit the "Submit Order" button. Once your order is processed and ready, you are notified by e-mail with details on how to download your ordered data products.
3. Manage your orders: You can access your User Profile (the last link called "Past Orders" on the left-side in the search and order interface) to check the status of your order, resubmit a previously submitted order, kill an order, etc.

*Keep in mind that if your search criteria identifies more than 2000 files, no results are returned; instead you are asked to revise your search criteria.

The LAADS Online archive refers to cached data on spinning disk that offers all public product collections that are readily available to download without a registered user having to conduct a search. We offer two pathways to navigate through this online archive to find and acquire the requisite data for your requirements:

• Science domain-driven navigation
• /allData tree-based navigation

Science domain-driven navigation

View Science domain-driven navigation

This newly implemented capability is designed to streamline users to intuitively follow the science domain attributes to target their datasets of interest. This (and the /allData tree) method is appropriate for seasoned users who know what datasets they require for their research or applications, and can afford to forego any post-processing transformations. It also is beneficial for those who depend on a constantly acquired time-series of products for regularly computed application requirements.

This capability allows users to naturally follow the taxonomic structure by selecting their discipline (e.g., atmosphere, land, Level-1, etc.), product suite (e.g., Aerosol, Land Surface Reflectance, etc.), particular products, their year, and day-of-year, etc. For instance, the following pathway provides a science domain-driven access to the SNPP VIIRS Deep Blue Aerosol L2 products from January 4^th 2015:

/archive/Science Domain/Atmosphere/Aerosol/VIIRS SNPP C1.0 - Deep Blue Aerosol L2 6-Min Swath 6 km/2015/004/

Once users are on this page that lists all datasets for that particular day-of -year, they have the option to select and download one or more datafiles. They can also avail of the multi-file download capability that allows them to click individual table rows (or select multiple rows by holding down the Shift key) to select files, and later click "Download Selected" to confirm their multi-file download.

/allData tree-based navigation

View /allData tree-based navigation

Registered users can navigate through the LAADS /allData tree, and directly download their preferred data products through this online archive. Prior to being able to do that, they need to know a couple of things. They include the product’s collection number and its short-name. In this context, it is important to clarify the existence of another attribute called “Archive Set,” a numeric identifier that primarily relates to archive storage. Users strictly shouldn’t have to know or keep track of archive set numbers, but currently need to understand how they map to collection numbers. The table below presents the LAADS DAAC’s public product collection numbers and their mapping to archive set numbers. Product short-names are used within the archive directory structure, but they are not very descriptive and sometimes difficult to remember, so many users use the LAADS product descriptions to find each product’s short-name.

Assuming users know the collection number and short-name of their desired products, it is fairly straightforward to navigate through the online archive; For instance, the following URL provides the location that lists all 240 6-minute datasets for the SNPP VIIRS Collection 1 (Archive Set 5110) Day-Night Band Geolocation products (VNP03DNB) for 2015, Day-Of-Year 004:

/archive/allData/5110/VNP03DNB/2015/004/

When you choose to download a certain dataset, it gives you the option to either open the file (by selecting Open with) with a particular image-handling software package, or save it to your local system’s default Download directory. A couple of simple, open-source packages to visualize your downloaded products include Panoply and HDFView; You need to know the temporal frequency (e.g., 6-minute, Daily, 8-day, Monthly, etc.) of your preferred products to navigate to the correct location to find the listed data granules.

Using scripts provide an efficient means to acquire multiple data products from the same collection/product, data-day, mixed collections/products, assorted data-days, etc. Common methods include wget and curl. Wget (for WWW get) is an open-source program from the GNU Project that is designed to help download content from web servers. Curl (for Client URL) is another open-source software that is used in command-line and scripts to transfer data using different network protocols. Consult LAADS Data Download Scripts for help, code samples, and additional information regarding creating and using download scripts.

ESDIS, our parent organization, requires that we track who downloads files. In addition, MODIS and VIIRS Science teams require that we protect some data from download by the general public but permit it for authorized users.

To meet these requirements, ESDIS has implemented Earthdata Profile (URS), a profile manager that helps us keep a record of anyone using our services. In addition to URS, LAADS manages authorizations that restrict access to certain resources by user and by type of access.

In order to access restricted resources (e.g. SENTINEL-3) a user would need to first get authorization from the owner of that resource and log in. Users that are currently logged into LAADS should not need to log out and log back in in order to see the changes.

Deprecation of FTP

File Transfer Protocol (FTP) has been around for many generations and has been used successfully for transfering files via FTP clients and numerous scripting languages. Unfortunately, FTP is also considered a security risk by many cybersecurity experts.

NASA’s Goddard Space Flight Center will be blocking all requests to public facing FTP servers—including LAADS DAAC and LANCE NRT—as of 20 April 2018.

Downloading via HTTP

Hypertext Transfer Protocol (HTTP) is the protocol that drives most web site internet traffic today. A variant of the protocol, called “HTTPS”, “S” for “Secure”, has been chosen to replace FTP.

HTTPS encrypts all transactions between client and server. This makes it extremely difficult for third parties to intercept what is being transferred.

All HTTPS downloads will require either an active login session or an token (i.e. token) passed in via an Authorization HTTP header. LAADS DAAC currently requires HTTPS downloading of all data.

Earthdata Profile

Each user must create an Earthdata URS profile in order to download files. Each URS profile is tracked in LAADS by email address, not URS usernames.

Authorization

After creating an Earthdata profile, you may request authorization to access a resource from the owner of that resource. The following rubrick will guide you on the process for each type of resource:

Tokens

Users wishing to download via a browser simply need to log in and click links within the LAADS Archive. Scripted downloads will need to use LAADS tokens in order to be properly authorized.

LAADS tokens are alphanumeric string values that identify who you are. Tokens get passed in the Authorization header of each HTTP GET request. See code samples below.

Requesting a Token

Any user that does not already have a token for LAADS DAAC can perform the following steps:

1. login by going to Profile -> Earthdata Login
2. select Profile -> Generate Token from the top menu
3. copy the token from this window and store it somewhere safe and secure
4. if you lose your token, repeat the steps above to generate a replacement

Retrieving a Token

LAADS does not store previously generated tokens. In the event of a lost token, please generate a new one. Users can generate as many tokens as needed.

How to use a Token

This example uses the curl command to make a request for the file on our web server at the URL https://ladsweb.modaps.eosdis.nasa.gov/PATH_TO_MY_FILE

curl -v -H 'Authorization: Bearer MY_TOKEN' '-C - https://ladsweb.modaps.eosdis.nasa.gov/PATH_TO_MY_FILE' > result

The example above passes your token via the Authorization HTTP header while utilizing the Bearer schema. When finished, the resulting download will be written to a file called “result” in whatever directory (folder) you run the command from.

The token located in the header is how LAADS identifies users. If the token is invalid, missing, or an -h is used instead of -H, the curl command will not work. The -v parameter tells curl to be verbose so extra information about the request and response will be printed out. If this extra information is not needed the -v parameter can be left off.

Curl is available for all current operating systems, including Linux, MacOS, and Microsoft Windows.

Note

1. all characters in the command are important, including dashes, colons, and quotation marks
2. copy your token in place of the string MY_TOKEN
3. copy the path to the file you need in place of the string PATH_TO_MY_FILE
4. Most LAADS DAAC file paths look like
   
   archive/allData/ARCHIVE SET/PRODUCT/YEAR/DAY_OF_YEAR/FILENAME
   
   An example of an existing file is below.
   
   https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD02QKM/2007/018/MOD02QKM.A2007018.0105.006.2014227230926.hdf
   
   This path should return a MODIS Terra quarter kilometer (250 m) top of atmosphere reflectance product for year 2007, day-of-year 018 (i.e. January 18), from collection 6.

According to W3C, a Web service is a software system designed to support interoperable machine‐to-machine interaction over a network. Web services use Extensible Markup Language (XML) syntax to construct their messages. The advantages of an XML-based Web service architecture include the following:

• platform independence
• physical location independence (i.e., where the message is being sent from)
• application language independence
• client's knowledge independence (i.e., what kind of processor the server uses)

Web services use Hypertext Transfer Protocol (HTTP) as the underlying communication protocol that facilitates how applications send XML messages to Web services. Representational State Transfer (REST) and Simple Object Access Protocol (SOAP) comprise messaging protocols to exchange structured information to implement web services within a network.

The LAADS DAAC currently offers a handful of Web services that more experienced users can use to programmatically access and retrieve their desired data products. They include a few that qualify as open standards (OGC, WCS, and OPeNDAP) and an In-house API called LAADS Web Services. An API is an acronym for Application Programming Interface, which is a software interface that allows two applications to interact with each other without any user intervention. They facilitate communication between one set of products or services with another set while remaining agnostic regarding implementation details.

What you should know before you use any of the following three web services:

1. A critical detail that users should keep in mind while using these Web services relates to the Collection vs. Archive set distinction that is described above in the LAADS Online archive tutorial. To recap, the Collection number reflects the version ID of the current production cycle for a certain product while the Archive set number points to an archive storage detail. The mapping between these two attributes is what the above table provides. When you develop your API for all three Web services, the URL require a collection=value parameter for some cases — you need to ensure that you always set the value to “ArchiveSet.”
2. You should use a client that understands both how to make a request to the service and how to process the results of that service.
   • All services provide documentation on how to write a client, while some services already have clients that support making a request to the service and processing its results.
   • All services respond to HTTP requests from web browsers.
3. Each of the following Web services have unique capabilities, and users should consider when and why they may want to use each of them. Each individual section includes some general pointers to aid users in their decision as they consider its use.

Open Geospatial Consortium (OGC) – Web Coverage Service (WCS)

OGC’s WCS standard enables Web-mediated retrieval of coverages. A coverage is a digital representation of a spatio-temporal object that usually comprises the content of a geographic information system (GIS) process or service. The MODAPS WCS API follows the OGC WCS API, although our implementation (at version 1.0) does not conform to the latest version of the standard, which is now at version 2.1. Users would require an XML parser to process the output of the service. Some GIS tools (for instance ESRI tools and those based on GDAL library) can understand the protocol. Similar to the benefits from other Web services, this service provides the ability to directly stream filtered, subsetted, reformatted and regridded data that conform to user specifications.

MODAPS WCS API that provides access to MODIS Level-1 and Atmosphere data products

OGC's WCS Standard

OpenAPI for Coverages — GitHub

Open-source Project for a Network Data Access Protocol (OPeNDAP)

OPeNDAP, an acronym for “Open-source Project for a Network Data Access Protocol” is a data transport architecture and protocol used in the earth science modelling, research, and applications communities. OPeNDAP was designed and implemented to provide researchers universal access to science data products using the same client/server model provided by the World-Wide Web and the Internet. Clients can submit specifically tailored requests to web servers supporting web pages that handle and manage the science data products. Client-specified requests for datasets conforming to a certain format, space-time, or variable-subset are directly delivered to the requesting client.

OPeNDAP provides a data server architecture that provides access to data stored on remote servers that end-users can access and use within their own client software. See a list of all the available OPeNDAP client software and their libraries. OPeNDAP data requests are submitted via URLs; the URL's syntax contains all the elements, i.e., product name, science dataset or variable name, spatial and temporal details for the requesting action, etc. A major advantage bestowed by OPeNDAP is the ability to perform data filtering, subsetting, and data format transformation operations directly from the client. OPeNDAP is limited in its ability to conduct searches, and cannot search by area of interest.

All the publicly available LAADS DAAC product collections are available via OPeNDAP

For additional information about OPeNDAP, check the following sources:

OPeNDAP User Guide from the OPeNDAP organization

NASA Earthdata OPeNDAP User Guide

How to use OPeNDAP to access data – video tutorial from NASA JPL PO.DAAC

OPeNDAP URL syntax examples that demonstrate how to download particular LAADS datasets:

1. Download Aqua-MODIS MYD04_3K, Collection-6.1 product from year 2015, day 130, time 0800 in netCDF

/opendap/RemoteResources/laads/allData/61/MYD04_3K/2015/130/MYD04_3K.A2015130.0800.061.2018049210111.hdf.nc

2. Download SNPP-VIIRS VNP02IMG, Collection-1 product from year 2015, day 40, time 0512 in ascii

/opendap/RemoteResources/laads/allData/5110/VNP02IMG/2015/040/VNP02IMG.A2015040.0512.001.2017259174514.nc.ascii

3. Download Terra-MODIS MOD03, Collection-6 product from year 2013, day 56, time 0000 in netCDF:

/opendap/RemoteResources/laads/allData/6/MOD03/2013/056/MOD03.A2013056.0000.006.2013056075649.hdf.nc

4. Download Terra-MODIS MOD03, Collection-6 product from year 2013, day 56, time 0000 in ASCII format:

/opendap/RemoteResources/laads/allData/6/MOD03/2013/056/MOD03.A2013056.0000.006.2013056075649.hdf.ascii

5. Download Terra-MODIS MOD03, Collection-6 product from year 2013, day 56, time 0000 in netCDF but only include the Scan_Offset variable in a range from 0 to 4059 with a stepping of 1 in the first dimension, and a range of 0 to 2707 with a stepping of 1 in the second dimension.

/opendap/RemoteResources/laads/allData/6/MOD03/2013/056/MOD03.A2013056.0000.006.2013056075649.hdf.nc?Scan_Offset%5b0:1:4059%5d%5b0:1:2707

In-house API – LAADS Web Service (LWS) Classic

LWS Classic is an in-house-developed, custom SOAP- and REST-based API that allows users to search, order, and download MODIS and VIIRS Level-1 and higher-level atmosphere data products through a programmable interface. It requires an XML parser to process the output of the service. It provides extensive search capabilities, and the standard set of data transformation functions. It does, though, require extra steps to order and poll for completion, or directly download subsetted, reformatted, and regridded data. Consult the following links for additional information:

LWS Classic Quick Start Guide

LWS Classic API