PAS Snapshot Full-Sampling problem
==================================
== Description
Previously, up to 2021/06/10, we were using PAS full-sampling (FS) at the begining and end of each snapshot period, with :
- START_ENERGY = 4, NB _ENERGY = 90
- START_ELEV = 0, NB_ELEV = 9
A typical PAS FS is : 90 x 9 x 11 x int16 = 17820 bytes
Starting on 2021/06/10 we tried another PAS configuration with:
- START_ENERGY = 14, NB _ENERGY = 80
- START_ELEV = 1, NB_ELEV = 7
That conduces to have smaller PAS data packets for FS : 80 x 7 x 11 x int16 = 12320 bytes
=== Problem encountered
From 2021/06/10 to 2021/01/13 we found some new errors in PAS data processing : some PAS FS were missing
The 2 columns indicates the number of leading (sample 1/9) or trailing (9/9) FS and their ratio (received/expected)
----
2021-06-10 ( 183 162) FS / 269 = ( 68.03% 60.22%) OK
2021-06-11 ( 165 158) FS / 280 = ( 58.93% 56.43%) OK
2021-06-12 ( 146 135) FS / 282 = ( 51.77% 47.87%) OK
2021-06-13 ( 5 3) FS / 20 = ( 25.00% 15.00%) OK
----
See link:documents/SNAPSHOT/snapshot.txt[]
=== 2021/06/10 Example of problem
Here is an extract of PAS FS header and data packet
----
1 | CCSDS.Read_CCSDS_packet | PACKET 1616
1 | STATS.Register_SID | SID = 203 count = 26
1 | TM.Get_TM_handler | APID ( 96,12) TM ( 21, 3) SEQ (00386,3) Size = 20 2021-06-10T00:51:25.029 : SID ( 203, 26) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_FIRST
1 | SCI.Parse_PAS_3D_C_snapshot | Sample 9/9
1 | SCI.Parse_PAS_3D_C_snapshot | SCET = 2021-06-10T00:51:14.153270144
1 | SCI.Parse_descriptor | K = 1 ENERGY = (14,80) ELEVATION = (1,7) CEM = (0,11)
1 | SCI.Parse_descriptor | Info = 2 K = 1 Sampling size = 12320 Total size = 12320 bytes
1 | SCI.Parse_PAS_3D_C_snapshot | LCI = 24
1 | CCSDS.Read_CCSDS_packet | --------------------------------------------------------------------------------
1 | CCSDS.Read_CCSDS_packet | PACKET 1617
1 | STATS.Register_SID | SID = 204 count = 26
1 | TM.Get_TM_handler | APID ( 96,12) TM ( 21, 6) SEQ (00387,3) Size = 462 2021-06-10T00:51:25.029 : SID ( 204, 26) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_DATA
1 | SCI.Parse_PAS_3D_C_snapshot | CDS = 40 Size = 454 bytes
1 | SCI.Uncompress_data | Compressed/expected/uncompressed sizes = 454 12320 12288 compression factor = 27.14
W | SCI.Uncompress_data | usize = 12288 < expected 12320 (missing 32 bytes)
1 | SCI.Parse_PAS_3D_C_snapshot | Total size = 12288/12320
E | SCI.Parse_PAS_3D_C_snapshot | Uncomplete data : 12288/12320 bytes
1 | SCI.Write_CDF_record | Record 00089 : Info = 2 K = 1/1 Sample = 9/9
----
After uncompressing the data packet, we were expecting to receive 12320 bytes, but we received only 12288 bytes (32 bytes missing)
* In a first processing, we were rejecting these packets, that leads to "Missing PAS snapshots" in our plots.
* In a second precessing, we completed these uncomplete packets with fill_values, to get the more complete plot possible.
NB : the problem doesn't occur for all PAS FS, but sometimes for more than 50% of FS
== Same problem on 2021/06/21 to 2021/06/23
We could find same kind of uncomplete uncompressed data for both snapshot or burst mode FS, with the same characteristics.
.Example in Burst mode
----
1 | CCSDS.Read_CCSDS_packet | PACKET 125711
1 | STATS.Register_SID | SID = 209 count = 356
1 | TM.Get_TM_handler | APID ( 97,12) TM ( 21, 3) SEQ (04925,3) Size = 20 2021-06-21T22:07:06.754 : SID ( 209, 356) : SWA_TM_SCI_PAS_TRIGG_OBURST_COMPR_FIRST
1 | SCI.Parse_PAS_3D_C_burst | Sample 61/296
1 | SCI.Parse_PAS_3D_C_burst | SCET = 2021-06-21T22:02:09.866040064
1 | SCI.Parse_descriptor | K = 1 ENERGY = (14,80) ELEVATION = (1,7) CEM = (0,11)
1 | SCI.Parse_descriptor | Info = 3 K = 1 Sampling size = 12320 Total size = 12320 bytes
1 | SCI.Parse_PAS_3D_C_burst | LCI = 24
1 | CCSDS.Read_CCSDS_packet | --------------------------------------------------------------------------------
1 | CCSDS.Read_CCSDS_packet | PACKET 125712
1 | STATS.Register_SID | SID = 210 count = 356
1 | TM.Get_TM_handler | APID ( 97,12) TM ( 21, 6) SEQ (04926,3) Size = 314 2021-06-21T22:07:06.755 : SID ( 210, 356) : SWA_TM_SCI_PAS_TRIGG_OBURST_COMPR_DATA
1 | SCI.Parse_PAS_3D_C_burst | CDS = 30 Size = 306 bytes
1 | SCI.Uncompress_data | Compressed/expected/uncompressed sizes = 306 12320 12288 compression factor = 40.26
W | SCI.Uncompress_data | usize = 12288 < expected 12320 (missing 32 bytes)
1 | SCI.Parse_PAS_3D_C_burst | Total size = 12288/12320
E | SCI.Parse_PAS_3D_C_burst | Uncomplete data : 12288/12320 bytes
1 | SCI.Write_CDF_record | Record 01370 : Info = 3 K = 1/1 Sample = 61/296
----
Same PAS configuration :
* 80 energy x 7 elevation x 11 CEM x int16 = 12320 bytes expected,
* but only 12288 bytes uncompressed for sample 61/296
