GitHub

Data Model and Project Module

We adopt a data model inspired by the SPASE Model.

Data Model Overview

The data model organizes data in a hierarchical type structure:

  • Project/Observatory: Represents a mission or project (e.g., MMS, THEMIS)

    • Contains multiple instruments
    • Contains multiple datasets

  • Instrument: Represents a scientific instrument on a spacecraft

    • Associated with a specific project
    • Produces various types of data

  • LDataSet: A template for generating datasets with parameterized naming patterns

    • Defines the format for dataset names
    • Specifies variables and their naming patterns

  • DataSet: A concrete dataset created from an LDataSet template

    • Contains actual data parameters
    • Created with specific values (e.g., probe number, data rate)
SpaceDataModel.Project — Type
Project <: AbstractProject

A representation of a project or mission containing instruments and datasets.

Fields

  • name: The name of the project
  • metadata: Additional metadata
  • instruments: Collection of instruments
  • datasets: Collection of datasets
source
SpaceDataModel.Instrument — Type
Instrument <: AbstractInstrument

Fields

  • name: The name of the instrument
  • metadata: Additional metadata
  • datasets: Collection of datasets
source
SpaceDataModel.LDataSet — Type
LDataSet <: AbstractDataSet

A template for generating datasets with parameterized naming patterns.

Fields

  • format: Format string pattern for the dataset name
  • data: Dictionary of variable patterns
  • metadata: Additional metadata

Examples

using SPEDAS.MMS

# Access FPI dataset specification
lds = mms.datasets.fpi_moms

# Create a concrete dataset with specific parameters
ds = DataSet(lds; probe=1, data_rate="fast", data_type="des")

The format string and variable patterns use placeholders like {probe}, {data_rate}, which are replaced with actual values when creating a concrete DataSet.

source

Project Module and Configuration

We support TOML configuration files to define project-specific metadata. A typical configuration file includes:

  • Project metadata
  • Instrument definitions
  • Dataset template definitions

We use the configuration files to generate appropriate data structures in the project module. For example, the corresponding configuration file of Magnetospheric Multiscale (MMS) module src/projects/MMS/MMS.jl is config/MMS.toml.

Note

While configuration files provide a convenient way to structure project metadata, you can also create these data structures directly in code. The configuration approach is optional and primarily serves to separate configuration from implementation.

Usage Examples

Project-specific instruments and datasets are exported as global variables when importing a project module using SPEDAS.{project}, for example:

using SPEDAS
using SPEDAS.MMS

mms
Project: Magnetospheric Multiscale
  Metadata (Dict{Any, Any}):
    abbreviation: MMS
  Instruments (Dict{String, Instrument}):
    feeps: Fly's Eye Energetic Particle Sensor
    scm: Search-coil Magnetometer
    fgm: Fluxgate Magnetometer
    edp: Electric Field Double Probe
    hpca: Hot Plasma Composition Analyzer
    eis: Energetic Ion Spectrometer
    edi: Electron Drift Instrument
    fpi: Fast Plasma Investigation
  Datasets (Dict{String, LDataSet}):
    fpi_moms:

You can access instruments/datasets directly through the project or via exported variables:

@assert mms.instruments["feeps"] === feeps
feeps
Instrument: Fly's Eye Energetic Particle Sensor
  Metadata (Dict{Any, Any}):
  Datasets (Dict{String, LDataSet}):

Dataset templates can be used to create concrete datasets by providing specific parameter values:

# Multiple ways to access a dataset
@assert fpi_moms === mms.datasets["fpi_moms"] === mms.instruments["fpi"].datasets["fpi_moms"]
fpi_moms
LDataSet: 
  Data (Dict{String, String}):
    temppara: mms{probe}_{data_type}_temppara_{data_rate}
    bulkv_gse: mms{probe}_{data_type}_bulkv_gse_{data_rate}
    tempperp: mms{probe}_{data_type}_tempperp_{data_rate}
    numberdensity: mms{probe}_{data_type}_numberdensity_{data_rate}
    energyspectr_omni: mms{probe}_{data_type}_energyspectr_omni_{data_rate}
  Metadata (Dict{String, Any}):
    data_rates: ["fast", "brst"]
    data_types: ["des", "dis"]
    probes: [1, 2, 3, 4]

# Create a concrete dataset with specific parameters
dataset = DataSet(fpi_moms; probe=1, data_rate="fast", data_type="des")
DataSet: MMS1_FPI_FAST_L2_DES-MOMS
  Data (Dict{String, String}):
    temppara: mms1_des_temppara_fast
    bulkv_gse: mms1_des_bulkv_gse_fast
    tempperp: mms1_des_tempperp_fast
    numberdensity: mms1_des_numberdensity_fast
    energyspectr_omni: mms1_des_energyspectr_omni_fast
  Metadata (Dict{String, Any}):
    data_rates: ["fast", "brst"]
    data_types: ["des", "dis"]
    probes: [1, 2, 3, 4]

How Configuration Files Are Used

When you import a project module (e.g., using SPEDAS.MMS), the system:

  1. Reads the corresponding TOML configuration file
  2. Creates a Project object with the defined metadata
  3. Instantiates Instrument objects for each instrument definition
  4. Creates LDataSet (dataset template) objects for each dataset definition

These objects are then accessible through the project namespace (e.g., mms.instruments["fpi"] or mms.datasets["fpi_moms"]).

Adding a New Project

To add support for a new space physics mission:

  1. Create a new TOML file in the config/ directory (e.g., config/cluster.toml)
  2. Define the project metadata, instruments, and dataset templates
  3. Create a corresponding module in the src/projects/ directory (for example, MMS module is defined in src/projects/mms.jl)

The configuration file will be automatically loaded when the project module is imported.