LAADS DAAC in the Cloud
Introduction
As part of NASA’s open-science policy and related goals, the LAADS DAAC is migrating its Level-1 and Atmosphere product collections to the Earthdata Cloud. All these data collections and services remain freely available to the global user community.
This page, LAADS DAAC in the Cloud, is intended to help users discover and access all LAADS DAAC data collections in the Earthdata Cloud. It addresses the following four topics:
- What are the benefits of working with Earth science data in the Cloud?
- What is the migration status of the LAADS DAAC’s products in the Cloud?
- How would a user access and download any desired data product from the Cloud?
- What are the answers to frequently asked questions?
As the Cloud evolves with the addition of new datasets, services, and related functionalities, this page will be updated with new information.
Benefits
- Easy access to data: Data users will be able to access data directly in the cloud, making the need to download volumes of data unnecessary. However, users will still be able to download data if they choose.
- Rapid deployment: Users can bring their algorithms and processing software to the cloud and work directly with the data in the cloud, simplifying procurement and hardware support while expediting science discovery.
- Scalability: The size and use of the archive can expand easily and rapidly as needed.
- Flexibility: Mission needs can dictate options for selecting operating systems, programming languages, databases, and other criteria to enable the best use of mission data.
- Reduced redundancy: The use of a common infrastructure with cloud native services will reduce redundant tools and services, enable sharing, and enforce the use of community standards as well as uniform policies and processes.
- Cost effectiveness: NASA pays only for the storage and services used. Along with scalability benefits, this allows the amount of storage or services to be continually adjusted to ensure that data and services are effectively provided at the lowest possible cost to NASA. Under NASA’s full and open data policy, all NASA data will continue to be free to access and download. However, users who wish to store data in their own Amazon Web Services (AWS) cloud instance or cloud storage, are responsible for covering those cost. For more information on this topic, see “Understanding and Managing Costs in the AWS Cloud.”
Earthdata Cloud also benefits the scientific community that uses NASA Earth science data for research. By making NASA data, algorithmic code, and metadata available in the cloud, the scientific processes of NASA researchers will become more transparent and their results more reproducible, which in turn lends clarity and validity to the scientific process. In addition, the use of standardized software and code makes it easier for new users to learn to interact with the data and become more involved in the scientific process.
Migration Status
The LAADS DAAC is currently populating its Phase-1 datasets in the Earthdata Cloud. These datasets will be available in late 2022.
The following diagram and tables depict the proposed timelines for migrating data products, beginning with a historical selection that includes a set of the most frequently used products. The subsequent migration includes in-production datasets grouped by their popularity. Even as the LAADS DAAC Cloud grows and becomes operational, users may continue to search, access, and download all data products via the on-premises channel as well.
LAADS DAAC Phased Dataset Migration (Historical Data)
|
LAADS DAAC Phased Dataset Migration (Historical Data) |
|||
|
Total |
4,534 |
||
|
Phase |
Collection |
Year |
Total Volume (TB) |
|
1.1 |
MODIS C61 L1, Geolocation and Cloud Mask (Cohort-2) |
2022 |
1,260 |
|
1.2 |
LAADS DAAC product “Top 50 + 25” List |
2022 |
355 |
|
2.0 |
MODIS C61 L2 and L3 Atmosphere Products |
2023 |
35 |
|
3.1 |
MODIS C61 L2 Land Surface Reflectance Product |
2024 |
706 |
|
3.2 |
VIIRS C2 SNPP and C2.1 NOAA20/J1 L1 Products |
2024 |
456 |
|
4.1 |
SNPP/J1 VIIRS C1/C1.1/C2 Atmosphere Products |
2025 |
572 |
|
4.2 |
C6 Long Term Data Records from NOAA POES and ESA MetOps |
2025 |
17 |
|
4.3 |
MERIS, Sentinel 3A, and Sentinel 3B L1 Products |
2025 |
1,133 |
|
4.4 |
Airborne Data |
2025 |
0.37 |
LAADS DAAC Phase 1.1 (Cohort-2) Datasets
|
LAADS DAAC Phase 1.1 (Cohort-2) Datasets |
||||
|
Shortname |
Platform |
Instrument |
Description |
Availability |
|
MOD021KM |
Terra |
MODIS |
Level 1B Calibrated Radiances - 1km |
December 2022 |
|
MYD021KM |
Aqua |
MODIS |
Level 1B Calibrated Radiances - 1km |
December 2022 |
|
MOD02HKM |
Terra |
MODIS |
Level 1B Calibrated Radiances - 500m |
December 2022 |
|
MYD02HKM |
Aqua |
MODIS |
Level 1B Calibrated Radiances - 500m |
December 2022 |
|
MOD02QKM |
Terra |
MODIS |
Level 1B Calibrated Radiances - 250m |
December 2022 |
|
MYD02QKM |
Aqua |
MODIS |
Level 1B Calibrated Radiances - 250m |
December 2022 |
|
MOD03 |
Terra |
MODIS |
Geolocation - 1km |
December 2022 |
|
MYD03 |
Aqua |
MODIS |
Geolocation - 1km |
December 2022 |
|
MOD35_L2 |
Terra |
MODIS |
Cloud Mask and Spectral Test Results 5-Min L2 Swath 250 and 1km |
December 2022 |
LAADS DAAC Phase 1.2 Top 50+25 Datasets
|
LAADS DAAC Phase 1.2 Top 50 + 25 Datasets |
||||
|
Shortname |
Platform |
Instrument |
Description |
Availability |
|
MOD04_3K |
Terra |
MODIS |
Aerosol 5-Min L2 Swath 3km |
December 2022 |
|
MYD04_03K |
Aqua |
MODIS |
Aerosol 5-Min L2 Swath 3km |
December 2022 |
|
MOD06_L2 |
Terra |
MODIS |
Cloud Properties |
December 2022 |
|
MYD06_L2 |
Aqua |
MODIS |
Cloud Properties |
December 2022 |
|
MOD07_L2 |
Terra |
MODIS |
Temperature and Water Vapor Profiles 5-Min l2 Swath 5km |
December 2022 |
|
MYD07_L2 |
Aqua |
MODIS |
Temperature and Water Vapor Profiles 5-Min L2 Swath 5km |
December 2022 |
|
VNP46A1 |
Suomi-NPP |
VIIRS |
VIIRS/NPP Daily Gridded Day Night Band 500m Linear Lat. Long. Grid Night |
December 2022 |
|
VNP46A2 |
Suomi-NPP |
VIIRS |
VIIRS/NPP Gap-Filled Lunar BRDF-Adjusted Nighttime Lights Daily L3 Global 500m Linear Lat. Long. Grid |
December 2022 |
Frequently Asked Questions
|
What are the advantages of using data from the Earthdata Cloud? 1) Improved performance: Quick access and subsetting of a large volume of data compared to data and services on-premises. 2) Multiple data access methods: In addition to existing LAADS DAAC data access methods, a new way to access data, S3 access, is available to users in the AWS us-west-2 region. 3) Multidisciplinary data access: Find multidisciplinary NASA data from multiple DAACs in one place, the Earthdata Cloud. 4) Analysis next to data: Users in the AWS us-west-2 region can import their algorithms and processing software and work directly with NASA data in the cloud, expediting science discovery. Note that users are responsible for procuring their own cloud computing resources. 5) Expanded services: Expanded services, such as CMR-STAC, Harmony, and Zarr creation will be available to users in the future.* |
|
What is S3 access? Amazon's Simple Storage Service (S3) is an object storage service that offers scalability, data availability, security, and performance. S3 access is a way of accessing AWS cloud data from S3 storage using cloud native tools. S3 access (also referred to as "direct S3 access") is only available to users in the same cloud region as LAADS DAAC data (AWS us-west-2). Access to LAADS DAAC cloud data on S3 is described in the How-To section. |
|
I am a user with a cloud account, how can I find out which LAADS DAAC datasets are available for direct S3 access in the Earthdata Cloud? The LAADS DAAC Migration Status Section describes the LAADS DAAC phased dataset projected release dates. You may also explore LAADS DAAC data through the Earthdata Search portal by filtering. To filter for LAADS DAAC Data in the Cloud, in the left panel under Organizations, select Level-1 and Atmosphere Archive & Distribution System (LAADS) and under Features, select Available from AWS Cloud. Step-by-step instructions for obtaining the S3 bucket URL for a LAADS DAAC collection from Earthdata Search are found in the How To section. |
|
I am a user with a cloud account, how can I access LAADS DAAC data in the Earthdata Cloud? LAADS data stored in the Earthdata Cloud can be accessed via S3 buckets. Users in the AWS us-west-2 region may use "S3 buckets" (bins of files, called objects) to access Earthdata Cloud data from S3 storage using cloud native tools. S3 access allows users to use Python and other language libraries to explicitly access files (or portions of files) in the LAADS DAAC cloud archive, list the contents of the bucket, and also interrogate and interact with cloud-native formats. You will need a URL path to the specific LAADS DAAC collection you would like to access using S3. The structure for an S3 URL looks like: s3://prod-lads/MOD021KM/MOD021KM.A2000055.0000.061.2017171194852.hdf An example of an S3 URL for a LAADS DAAC collection is: s3://prod-lads/MOD021KM/ To learn more, please refer to What is S3 access? |
Do I have to pay to access LAADS DAAC data stored in the Earthdata Cloud? As part of NASA's commitment to open science, data access will continue to be free to users. |
Do I have to pay for cloud computing resources? Yes, the user maintains responsibility for their own cloud computing resources. Cost is variable depending on which cloud compute types you choose. See guidance on Understanding and Managing Costs in the AWS Cloud. |
In which AWS region does LAADS DAAC data reside? LAADS DAAC data are stored in the AWS region US West (Oregon), or us-west-2. |
|
Can I access data in AWS regions outside of my geographic location? Yes. The AWS region refers to the location of servers, which are accessible via the internet regardless of a user's physical location. |
Can I perform direct S3 access outside of my account's default region? Yes, but first you will need to enable the region of interest in your AWS account. For example, to acquire LAADS DAAC data using direct S3 access, you will need to enable the us-west-2 region in your account and then use an AWS service in that region (e.g. EC2 instance, AWS Command Line Interface, Python, Javascript, Java etc. program software development kits (SDK) or libraries) to access the data. |
|
How do I obtain credentials to access LAADS DAAC data in S3 storage? You will need to pass credentials and configuration to AWS in order to interact with S3 objects. The How-To S3 Direct Access Section describes this process. |
|
I want to perform direct S3 access in a cloud environment, do I still need my Earthdata Login? Yes, you will still need to store your Earthdata Login credentials in your cloud environment. In order to access LAADS DAAC data, an Earthdata Login account is required. If you do not have an Earthdata Login, follow the steps listed here to obtain one. A short tutorial on how to store your Earthdata Login credentials is found in How to S3 Direct Access. |
|
With data hosting transitioning to the cloud, will I need to change the way I access data or services? No, existing data access methods and services continue uninterrupted during LAADS DAAC's cloud migration. |
|
With data hosting transitioning to the cloud, can I still download data from LAADS DAAC? Yes, existing on-premises download methods and services are still available throughout LAADS DAAC's cloud migration. |
How-To
How do I access LAADS DAAC data files in the cloud?
S3 Direct Access
Currently, AWS S3 Direct Access to LAADS DAAC data is accessible in cloud (Lambda, EC2, etc) and in region (US-WEST-2) only.
Instructions for direct access to LAADS DAAC AWS S3 datasets bucket can be found at:
https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentialsREADME
AWS credentials (in JSON format) are given by authenticating with Earthdata Login via the LAADS DAAC s3credentials
endpoint:
https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials
See the code examples in the S3 Credentials README (above).
HTTPS Download Access
HTTPS download access from the cloud is coming early 2023. Check back regularly for updates.
Last updated: November 2, 2022