Data Access and Analysis Tools

This page includes the various tools to access and analyze the data products in DP0.1.

RSP Hazards and User Responsibilities

All users are responsible for their own safe usage of the RSP and must consult this list of RSP Hazards.

Briefly, the DP0-era of the RSP is a shared-risk developmental version.

- future RSP safegaurds to protect against, e.g., data loss, are not yet present

- RSP performance during DP0 is not representative of the future RSP’s performance

- RSP compute resources during DP0 are not representative of the operations-era RSP

- documentation and support mechanisms are still in their very earliest versions

Rubin Science Platform (RSP)

The Rubin Science Platform (RSP) provides access to Rubin Observatory data products via two services: a Notebook Aspect and a Portal Aspect. These are complementary tools that can be used (separately or together) for data access and analysis. The Portal Aspect provides interactive query, data discovery, and visualization tools, while the Notebook Aspect enables programmatic interaction with, and analysis of, data products.

This is what you should see when you point your browser to https://data.lsst.cloud/. The Notebook and Portal Aspects can be accessed by clicking on their respective icons on this dashboard.

Notebook Aspect

The Notebook Aspect of the RSP provides an environment from which users can access and manipulate Rubin data products on the same machines where those data products reside (i.e., “next-to-the-data” – no downloading of data to your local machine). In particular, the Notebook Aspect offers Python-based access to DP0.1 data products via a custom implementation of web-based JupyterLab notebooks.

The Notebook Aspect is built on JupyterLab; see the extensive documentation of JupyterLab for details. Within the RSP Notebook Aspect, you will be able to query Rubin data, extract images and catalogs of many different types, manipulate and display images, interact with catalogs, and most other analyses you can imagine performing with Python on astronomical images and catalogs.

When you log into the Notebook Aspect, you will be taken to a JupyterLab environment from which you can create notebooks and execute Python code to perform actions on DP0.1 data products. The Notebook Aspect will provide some limited command-line access via a terminal, but most data access will be through notebooks. Jupyter notebooks provide “cells” within which you type either Python code or markdown. When you execute the cell (by either typing _Return_ while simultaneously holding down the _Shift_ key, or by clicking a button), the contents of the cell will be executed. If you typed Python code in the cell, then the code will be executed. If you entered markdown, then it will be rendered upon execution to yield nicely formatted text (for some handy markdown tips, see this blog post or the relevant section from the JupyterLab documentation).

In the RSP Notebook Aspect, your notebooks will be operating in a kernel that has access to the full Rubin Science Pipelines, including the “Butler” (ADD SOME LINK/REFERENCE?) that will be your main access tool to extract images and catalogs from the DP0.1 data. Many standard Python libraries and modules will be available, and users can install additional Python tools they wish to use. In DP0.1, the Notebook Aspect will not offer access to queries from the Portal Aspect.

To access DP0.1 data from the Notebook Aspect, users will need to use Python commands and code. We have provided many tutorial notebooks to help you get started. One feature of Jupyter notebooks is that these tutorials are not just text that you read, but contain executable examples of the commands required to access and analyze data. Thus they are a resource not only for learning how to use Rubin tools and science pipelines, but can serve as “seeds” from which you can borrow lines of code and alter them to suit your purposes. Because some facility with programming – and in particular with Python – will be necessary to make use of the Notebook Aspect, we recommend that users who are unfamiliar with Python learn some basics. There are countless resources on the internet to help you learn Python – here are a few that we recommend:

LINKS TO PYTHON BASICS

If you are not experienced at accessing data via Jupyter notebooks, or using a Science Platform more generally, you are not alone. The Rubin Observatory Community forum provides a searchable, community-based discussion platform that you can use as a resource to ask questions of the worldwide Rubin user community, to share your own tips and analyses, and discuss all aspects of Rubin Observatory, including using the RSP and the Rubin Science Pipelines, understanding the data products, and discussing Rubin-related science.

An example of what users see in the Notebook Aspect. The left panel is a file tree from which you can access your personal files, and which will come pre-populated with a handful of tutorial notebooks that users can execute on their own. The largest panel is the Jupyter notebook, where markdown and code can be entered and executed. You can also see that one can have multiple notebooks open in separate tabs.

For more information on the RSP Notebook Aspect, see https://nb.lsst.io/.

MLG notes: start with an overview of what is Jupyter Hub. For the Notebook aspect, only python notebooks and the terminal interface are supported, and RSP users will not be able to access their Portal queries from the Notebook aspect for DP0.1. In DP0.1 the Notebook Aspect of the RSP will offer image access via the Butler (a middleware component of the DMS for persisting and retrieving image datasets) through a python interface in the Notebook aspect.

See this page for a Introduction to the RSP Notebook Aspect.

Portal Aspect

This section includes information on using the portal for the RSP.

(MLG Note: For the Portal Aspect, only catalog queries will be available. Note that the Portal Aspect has not been under active development recently and it is expected to evolve significantly before the first LSST annual data release (DR1).)

See this page for a Introduction to the RSP Portal Aspect.

Table Access Protocol (TAP) service

This page includes information for accessing and basic utilization of the Table Access Protocol (TAP) service.

Data processing tools

This section includes information on tools used for data processing.

LSST Science Pipelines

This section includes information on the LSST Science Pipelines.

See https://pipelines.lsst.io/ for in-depth information.