No one processes and provides more Earth observation data than NASA. NASA provides access to more than 22 petabytes of Earth observation data across more than 11,000 unique data products and expects to deliver 247 petabytes of data by 2025.

Managing this data is a huge task and NASA is committed to keeping its data open. In 2016, NASA delivered more than 1.51 billion Earth science data products to more than 3 million data users around the world. Development Seed is working with NASA’s Earth Science Data Systems (ESDS) Program to build Cumulus, a cloud-based platform to ingest, process, catalog, archive, and distribute NASA’s Earth Data streams.

Cumulus is one of a suite of tools that NASA is developing to better leverage cloud computing for data processing, storage, and retrieval. We are working closely with the ESDS team as well as several of the Distributed Active Archive Centers (DAACs) to ensure Earth Data missions are available and accessible to users. We are working with these teams to build and refine a system that can handle data from MODIS as readily as lightning strike datasets.

Cumulus is much more than cloud-based storage: it is an open source framework for building a customizable data pipeline. The Cumulus API and dashboard support the scheduling and design of custom workflows. The Cumulus infrastructure is designed for scalability and reliability, using much of the AWS “Serverless” Platform. These services enable Cumulus to scale in real time in order to be performant under the largest expected workloads.

We are working with NASA to transfer ownership of Cumulus to NASA and to provide the code and documentation through their platforms. We hope that Cumulus will be useful to other organizations building imagery publishing pipelines.

Want to learn more about Cumulus? Cumulus Documentation provides high-level overview of architectural components and the Cumulus repository hosts and publishes npm packages for common Cumulus tasks and utilities, such as image processing and the Cumulus API.