PAS L3 CDF data structure and metadata
======================================
== Introduction
We will use as it is, the software provided by Teresa & Lubomir.
We have just added a main level IDL script to run procedure A and B for a given time period.
The software generates 2 kind of daily IDL .save files, containing the data described in their documentation:
- Ayyyy_ddd.sav : file generated using procedure A (no-imf) for year yyyy and day of year ddd
- Byyyy_ddd.sav : file generated using procedure B (using MAG data)
Then we are using a python script, to read these IDL files and create PAS L3 .cdf files.
We use a python library to read the IDL .save files:
data = scipy.io.readsav (<idl.save.filename>)
== PAS L3 datastets
We will produce 2 main datasets, with following pattern:
----
solo_L3_swa-pas-<dataset_id>_yyyymmdd_Vnn.cdf
----
* Procedure A
+
--
Dataset generated without use of MAG data
solo_L3_swa-pas-ground-fitted-moments-no-mag
--
* Procedure B
+
--
Dataset generated using PAS solo_L2 swa-pas-vdf and MAG solo_L2_mag_normal-srf.cdf
solo_L3_swa-pas-ground-fitted-moments-no-mag
--
== CDF variables
We have started to create CDFs with the following variables :
- Epoch (CDF_TT2000)
- Half_interval
- Info
- Np
- Na
- Vp_rtn
- Va_rtn
- Vsolo_rtn
- Tp_iso (no-mag)
- Tp_par (mag)
- Tp_perp (mag)
- Tp_tot (mag)
- Ta_iso (no-mag)
- Ta_par (mag)
- Ta_perp (mag)
- Ta_tot (mag)
- flag_1
- flag_2
- flag_3
- flag_4
(no-mag) theses variables will be only generated when using procedure A
(mag) these variable will be only generated when using MAG data (procedure B)
More details here:
http://solarorbiter.irap.omp.eu/documents/SCIENCE_DATA_PROCESSING/SWA_PAS_L3_processing.pdf
NOTE: Temperatures are ceomputed from IDL thermal speed variables (Wp_iso, Wp_par, Wp_perp,...), and converted from km/s to eV
== Metadata
Current content (text files):
* link:/documents/PAS_L3/solo_L3_swa-pas-ground-fitted-moments-no-mag_20220801_V01.info[]
* link:/documents/PAS_L3/solo_L3_swa-pas-ground-fitted-moments-mag_20220801_V01.info[]
Excel files:
* link:/documents/PAS_L3/solo_L3_swa-pas-ground-fitted-moments-no-mag.xls[]
* link:/documents/PAS_L3/solo_L3_swa-pas-ground-fitted-moments-mag.xls[]
These metadata have to be checked and completed.
If you want to make some changes in the metadata, you can download and update the corresponding Excel file,
adding your change in third column, and we will take in account your modifications.
== Test data
Some data have been generated from 2022/07/01 to 2022/08/31 (latest date where MAG data is available)
There are available on rosina1 under /DATA/SOLAR/DATA/L3, with a daily directory containing the 2 datasets.
They can be accessed from Amda (with a short ingestion delay) and soon from cl (or using generic ASCII/CDF pannels)
== General questions
==== Acquisition duration
I have added the Half_interval CDF variable, extracted from solo_L2_swa-pas-vdf dataset.
There will be a main time variable, epoch, that is set at the center of acquistion interval,
with a DELTA_PLUS_VAR = DELTA_MINUS_VAR = Half_interval.
This half_intervall has a values of 0.5s for most of PAS L2 vdf records, and sometimes 0.125s in the case there are K =4 sub-samplings per second.
==== Info
We can also provide a cdf variable Info that indicate the current PAS mode
(0 = static, 1 = normal, 2 = snapshot, 3 = burst, 4 = engineering, 5 = calib)
It can be useful as the acquisition duration and frequency change with the mode.
==== Proton and alpha velocities
They will be provided in RTN frame only
For RTN, we substract SC velocity (as for L2 moments)
==== Spacecraft velocity
We can extract the SC velocity in RTN frame (from solo_L2-swa-pas-grnd-mom dataset
or spice kernels) and copy it to PAS L3 datasets
==== MAG data
It was decided not to include MAG data in the PAS L3 CDFs
== Actions from previous meeting
http://solarorbiter.irap.omp.eu/software/page=PAS-L3-processing[]
==== Proposal naming
* solo_L3_swa-pas-ground-fitted-moments-mag
* solo_L3_swa-pas-ground-fitted-moments-no-mag
Do you prefer these dataset names? YES
NOTE: done
==== Acknowledgements
VG> Fournir une phrase "Please acknowledge..." pour remercier Teresa, Lubomir et IRAP (MSSL ?)
==== Documentation
Le document décrivant la méthode utilisée est actuellement sur le site solarorbiter.irap.omp.eu
http://solarorbiter.irap.omp.eu/documents/SCIENCE_DATA_PROCESSING/SWA_PAS_L3_processing.pdf
VG> se renseigner pour déposer document sur archive pérenne (doi ?)
==== Vitesse en RTN
A la demande d'Alexis Rouillard, EP a modifié son soft pour enlever (ajouter) la vitesse satellite .
AB> Doit prendre en compte cette option et ajouter dans les fichiers CDF:
- vitesse ions en RTN
- vitesse satellite en RTN
NOTE: done
== Various points
* Priority
+
--
The generation of PAS L3 cdf will take a while (~10H per month)
What are the priorities?
Reprocess from Launch to current date ?
--
* Validation of these data? Who?
* Delivery to SOAR? When?
* Access rights ? cl, AMDA, download...
== Issues
=== __2022/08/01__ L3 Np greater then L2 N data
A quick check in PAS L3 data for 2022/08/01 shows that L3 Np can be greater than L2 ground-moments N
Is it expected?
http://solarorbiter.irap.omp.eu/documents/PAS_L3/PAS_L3.zoom.png[]
Could be related to quality flag #4
=== __2022/08/22__ Gaps in MAG data but procedure B generates data
No MAG data from 2022/08/20 to 2022/08/22 07:00
=> the IDL software doesn't generate output file Byyyy_ddd.sav
PAS L3 data cannot be computed using procedure B on 2022/08/20 and 2022/08/21,
but data were generated on 2022/08/22 from 04:30/07:00, and also after 14:00,
whereas there are not available MAG data
NOTE: in her documentation, Teresa says there are no real gaps check when interpolating MAG data
----
MAG normal-mode RTN/SRF data are linearly interpolated to the PAS measurement times.
Time differences between two interpolating MAG points are not checked in the current version
and possible long MAG data gaps may impact the results.
----
http://solarorbiter.irap.omp.eu/documents/PAS_L3/PAS_L3.20220822.png[]
Should we improve the software and filter out these data?
=== __2022/08/28__ Spice kernels
Got an error message from spice kernels, when processing 2022/08/28 data.
Unable to compute SRF to RTN velocity, from 03:35 to 03:45
----
Toolkit version: CSPICE66
SPICE(NOFRAMECONNECT) --
At epoch 7.1492981338169E+08 TDB (2022 AUG 28 03:36:53.381 TDB), there is insufficient information available to transform from reference frame -144000 (SOLO_SRF) to reference frame 1 (J2000). SOLO_SRF is a CK frame; a CK file containing data for instrument or structure -144000 at the epoch shown above, as well as a corresponding SCLK kernel, must be loaded in order to use this frame. Failure to find required CK data could be due to one or more CK files not having been loaded, or to the epoch shown above lying within a coverage gap or beyond the coverage bounds of the loaded CK files. It is also possible that no loaded CK file has required angular velocity data for the input epoch, even if a loaded CK does have attitude data for that epoch. You can use CKBRIEF with the -dump option to display coverage intervals of a CK file.
pxform_c --> PXFORM --> REFCHG
----
TODO : Mail to Andrew Walsh to get more information.
== __2023/03/20__ Actions from the meeting
=== A01 : file nanimg
Change the datasets filename to:
* solo_L3_swa-pas-ground-fitted-moments-mag
* solo_L3_swa-pas-ground-fitted-moments-no-mag
NOTE: Done, 2023/03/21
=== A02 : Ion velocities
* Change CDF variable names (vVp, vVa) => (Vp, Va)
* Keep only velocities in RTN frame => Vp_rtn, Va_rtn
* Delete Vp_srf, Va srf
NOTE: Done, 2023/03/21
=== A03 : Thermal velocities
* Replace thermal velocities (km/s) with temperatures in eV
* Replace W[a/p]_[par/perp/iso] with T[a/p]_[par,perp,iso]
* Convert from km/s to eV (taking in account mass of proton and alpha)
* Add and compute total_temperatures:
Compute T[a/p]_tot = ( T[a/p]_par + 2 * T[a/p]_perp) / 3
NOTE: Done, 2023/03/21
=== A04 : Don't keep MAG data in PAS L3 CDFs
* Do not put vB_rtn, vB_srf in procedure B CDFs
NOTE: already done
=== A05 : Procedure B post-processing (filtering)
* Filter PAS L3 data with fill values, when no MAG data available in 10s window
CAUTION: to be done
=== A06 : Keep MAG data intermediate versions
* Rename the MAG data files in /DATA/SOLAR/DATA/MAG/L2 as they have the same filename version than
in CDF Logical_file_id attribute
CAUTION: to be done
