JUNO Magnetic field data pipeline

JUNO Magnetic field data can be downloaded from PDS website.

File Naming Convention                                                        
==============================================================================
Convention:                                                                   
   fgm_jno_LL_CCYYDDDxx_vVV.ext                                               
Where:                                                                        
   fgm - Fluxgate Magnetometer three character instrument abbreviation        
   jno - Juno                                                                 
    LL - CODMAC Data level, for example, l3 for level 3                       
    CC - The century portion of a date, 20                                    
    YY - The year of century portion of a date, 00-99                         
   DDD - The day of year, 001-366                                             
    xx - Coordinate system of data (se = Solar equatorial, ser = Solar        
         equatorial resampled, pc = Planetocentric, ss = Sun-State,           
         pl = Payload)                                                        
     v - separator to denote Version number                                   
    VV - version                                                              
   ext - file extension (sts = Standard Time Series (ASCII) file, lbl = Label 
         file)                                                                
Example:                                                                      
   fgm_jno_l3_2014055se_v00.sts    

Downloading data

---

UnpackConvert

 UnpackConvert (members=None, extract_dir=None)

*Processor that unpacks a zip archive and returns a list of all files.

Use with :meth:pooch.Pooch.fetch or :func:pooch.retrieve to unzip a downloaded data file into a folder in the local data store. The method/function will return a list with the names of the unzipped files instead of the zip archive.

The output folder is {fname}.unzip.*

	Type	Default	Details
members	NoneType	None	If None, will unpack all files in the zip archive. Otherwise, members must be a list of file names to unpack from the archive. Only these files will be unpacked.
extract_dir	NoneType	None	If None, files will be unpacked to the default location (a folder in the same location as the downloaded zip file, with the suffix .unzip added). Otherwise, files will be unpacked to extract_dir, which is interpreted as a relative path (relative to the cache location provided by :func:pooch.retrieve or :meth:pooch.Pooch.fetch).

---

unpack_and_convert

 unpack_and_convert (fname, extract_dir, process_func=<function
                     unzip_convert_lbl>)

Post-processing hook to unzip a file and convert it to a different format in real-time. (Otherwise the files unzipped would take up too much space on the user’s computer.)

	Type	Default	Details
fname	str		Full path of the zipped file in local storage
extract_dir
process_func	function	unzip_convert_lbl

---

load_jno_lbl

 load_jno_lbl (file:str)

---

download_data

 download_data (dataset='JNO-SS-3-FGM-CAL-V1.0',
                phase:Literal['CRUISE','JUPITER']='CRUISE',
                coord:Literal['SE','SS','PL']='SE',
                datatype:Literal['1SEC','1MIN','FULL']='1SEC',
                processor:Callable=None, url_fmt='https://pds-ppi.igpp.ucl
                a.edu/ditdos/download?id=pds://PPI/{dataset}/DATA/{phase}/
                {coord}/{datatype}', fmt='arrow')

	Type	Default	Details
dataset	str	JNO-SS-3-FGM-CAL-V1.0
phase	Literal	CRUISE
coord	Literal	SE
datatype	Literal	1SEC	time resolution
processor	Callable	None
url_fmt	str	https://pds-ppi.igpp.ucla.edu/ditdos/download?id=pds://PPI/{dataset}/DATA/{phase}/{coord}/{datatype}
fmt	str	arrow
Returns	list

Dataset Overview

Index

pds_dir = "https://pds-ppi.igpp.ucla.edu/data"

possible_coords = ["se", "ser", "pc", "ss", "pl"]
possible_exts = ["sts", "lbl"]
possible_data_rates = ["1s", "1min", "1h"]

juno_ss_config = {
    "DATA_SET_ID": "JNO-SS-3-FGM-CAL-V1.0",
    "FILE_SPECIFICATION_NAME": "INDEX/INDEX.LBL",
}

juno_j_config = {
    "DATA_SET_ID": "JNO-J-3-FGM-CAL-V1.0",
    "FILE_SPECIFICATION_NAME": "INDEX/INDEX.LBL",
}

Process index

import pandas
import pdpipe as pdp

def process_jno_index(df: pandas.DataFrame):
    _index_time_format = "%Y-%jT%H:%M:%S.%f"

    df.columns = df.columns.str.replace(" ", "")
    jno_index_pipeline = pdp.PdPipeline(
        [
            pdp.ColDrop(["PRODUCT_ID", "CR_DATE", "PRODUCT_LABEL_MD5CHECKSUM"]),
            pdp.ApplyByCols("SID", str.rstrip),
            pdp.ApplyByCols("FILE_SPECIFICATION_NAME", str.rstrip),
            pdp.ColByFrameFunc(
                "START_TIME",
                lambda df: pandas.to_datetime(
                    df["START_TIME"], format=_index_time_format
                ),
            ),
            pdp.ColByFrameFunc(
                "STOP_TIME",
                lambda df: pandas.to_datetime(
                    df["STOP_TIME"], format=_index_time_format
                ),
            ),
            # pdp.ApplyByCols(['START_TIME', 'STOP_TIME'], pandas.to_datetime, format=_index_time_format), # NOTE: This is slow
        ]
    )

    return jno_index_pipeline(df)