Bug uncompression algorithm
===========================
== __2020/05/31__ Core dump uncompressing data
We got a bug (core dump) while uncompressing 3D Normal data.
3D PAS normal data is constitued by two kind of TM packets :
* SID = 194 : SWA_TM_SCI_PAS_FULL3D_COMPR_HEAD
* SID = 195 : SWA_TM_SCI_PAS_FULL3D_COMPR_DATA
The first packet, SID = 194, contains a data_descriptor structure : dd,
that gives the number of CEM, elevation, energies, and K sub-samples.
From this data descriptor, we can compute the size of 3D data :
----
dd->sampling_size = dd->nb_energy * dd->nb_elevation * dd>nb_cem * sizeof (int16_t);
dd->total_size = dd->sampling_size * dd->k;
----
sampling_size gives the size of a normal sampling, and total_size taking in account when there are K sub-sampling per second.
The following packets, SID = 195, contains the 3D compressed data of these samplings.
We can find several DATA packets for a single HEAD packet, and we have to concatenate their data in a same buffer,
and then uncompress this data buffer, calling :
status = uncompress_data (input_buffer, input_size, lci, output_buffer, & uncompress_size);
The CCSDS 121 algorithms takes as input a byte buffer of input_size, and uncompress data to output_buffer,
and return uncompress_size.
=== Description of bug
We got a bug when uncompressing 2020/05/31 data from a stream data request to EDDS, whereas the bug doesn't appear later when
uncompressing data for the same data but coming from an EDDS batch request.
In the stream, a lot ot TM packet were missing that break a logical sequence for 3D normal compressesed data.
We could find a first packet SID = 194, with a data_descriptor that indicates a 3D product with a size of size 8156 bytes.
So we allocate a memory buffer of 8156 bytes to receive its uncompressed data.
Then we loose a sequence of 3D products packets, before receiving finally a packet SID = 195, with compressed 3D data but corresponding to another 3D product,
with an uncompressed size of 18216 bytes.
When the called the uncompression algorithm, it tried to uncompress 18216 bytes in a smaller buffer, that generates a buffer overflow,
and finally a core dump.
Later, when downloading data from batch request, we revover the full TM packets sequence, and this bug disappeared... but we cannot exclude it can appear again.
== How to fix
* Improve checking of packets sequences to avoid associating a data descriptor and a data packet that doesn't correspond.
* Adding the input_buffer size as argument in the uncompress_data function, to avoid such buffer overflows
=== SID count check
A possible solution to avoid such problems should be to check the value of SID_count.
Generally, each kind of data has a specifid SID number (Science ID) and a corresponding SID_count counter,
that indicates the sequence of data for a given SID.
There are 8 differents PAS 3D products, as a combination of mode [Normal, Snapshot, Burst, Trigger] x [Raw, Compressed]
Each kind of product are composed of a sequence of TM packet with a different SID.
For Normal mode, there are only 2 different types of packets : HEAD and DATA
* HEAD gives the 3D description : nb CEM, elevation, energies and sub-samplings per seconds(K)
* DATA contains the 3D counts for one or K sub_samplings
A single HEAD descriptor can be followed by a sequence of DATA packets for the same product.
For Burst, Snapshot and Trigger mode, there is another TM packet, called FIRST, that gives the number of sequential samples in this mode.
The following table gives the list of [FIRST, HEAD, DATA] SIDs for each kind of products
[options="header", cols="20,20,60"]
|===
| 3D type | SIDs | SID count evolution
| 3D_R_normal | 192, 193 | Not same SID_count
| 3D_C_normal | 194, 195 | Same SID_count
| 3D_R_snapshot | 205, 206, 207 |
| 3D_C_snapshot | 202, 203, 204 | SID 203, 204 : same SID count
| 3D_R_burst | 211, 212, 213 |
| 3D_C_burst | 208, 209, 210 | SID 209, 210 : same SID count
| 3D_R_trigger | 199, 200, 201 |
| 3D_C_trigger | 196, 197, 198 | To be checked
|===
We could notice that for compressed products, the HEAD and DATA packet of a given sequence have the same SID_count. The FIRST packet as a separate SID_count value.
=== Compressed 3D products
==== 3D Normal mode Compressed
HEAD (SID = 194) and DATA (SID = 195) share the same SID_count value
11422 for the first packet, 11423, etc...
----
APID ( 96,12) TM ( 21, 3) SEQ (10443,3) Size = 20 2020-05-31T00:00:09.150 : SID ( 194,11422) : SWA_TM_SCI_PAS_FULL3D_COMPR_HEAD
APID ( 96,12) TM ( 21, 6) SEQ (10450,3) Size = 304 2020-05-31T00:00:11.676 : SID ( 195,11422) : SWA_TM_SCI_PAS_FULL3D_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (10454,3) Size = 20 2020-05-31T00:00:13.148 : SID ( 194,11423) : SWA_TM_SCI_PAS_FULL3D_COMPR_HEAD
APID ( 96,12) TM ( 21, 6) SEQ (10461,3) Size = 312 2020-05-31T00:00:15.802 : SID ( 195,11423) : SWA_TM_SCI_PAS_FULL3D_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (10463,3) Size = 20 2020-05-31T00:00:17.148 : SID ( 194,11424) : SWA_TM_SCI_PAS_FULL3D_COMPR_HEAD
APID ( 96,12) TM ( 21, 6) SEQ (10464,3) Size = 288 2020-05-31T00:00:19.802 : SID ( 195,11424) : SWA_TM_SCI_PAS_FULL3D_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (10466,3) Size = 20 2020-05-31T00:00:21.148 : SID ( 194,11425) : SWA_TM_SCI_PAS_FULL3D_COMPR_HEAD
APID ( 96,12) TM ( 21, 6) SEQ (10467,3) Size = 310 2020-05-31T00:00:23.801 : SID ( 195,11425) : SWA_TM_SCI_PAS_FULL3D_COMPR_DATA
...
----
See link:documents/SID_count/3DCN.20200531.txt[] for full listing
==== 3D Snapshot Compressed
* SID = 202 = START +
contains the number of samples, and has a dedicated SID_count = 448
* SID = 203 = FIRST/HEAD and SID = 204 = DATA +
data descriptor and compressed packets share the same SID_count.
4033 for sample #1, 4034 for sample #2, etc...
----
APID ( 96,12) TM ( 21, 3) SEQ (01146,3) Size = 6 2020-06-01T00:10:07.050 : SID ( 202, 448) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_START
APID ( 96,12) TM ( 21, 3) SEQ (01147,3) Size = 20 2020-06-01T00:10:07.413 : SID ( 203, 4033) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (01148,3) Size = 334 2020-06-01T00:10:07.413 : SID ( 204, 4033) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (01154,3) Size = 20 2020-06-01T00:10:08.388 : SID ( 203, 4034) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (01155,3) Size = 760 2020-06-01T00:10:08.388 : SID ( 204, 4034) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (01160,3) Size = 20 2020-06-01T00:10:09.385 : SID ( 203, 4035) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (01161,3) Size = 756 2020-06-01T00:10:09.386 : SID ( 204, 4035) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (01162,3) Size = 20 2020-06-01T00:10:10.410 : SID ( 203, 4036) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (01163,3) Size = 694 2020-06-01T00:10:10.410 : SID ( 204, 4036) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (01164,3) Size = 20 2020-06-01T00:10:11.408 : SID ( 203, 4037) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (01165,3) Size = 748 2020-06-01T00:10:11.408 : SID ( 204, 4037) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (01168,3) Size = 20 2020-06-01T00:10:12.384 : SID ( 203, 4038) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (01169,3) Size = 750 2020-06-01T00:10:12.384 : SID ( 204, 4038) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (01171,3) Size = 20 2020-06-01T00:10:13.385 : SID ( 203, 4039) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (01172,3) Size = 722 2020-06-01T00:10:13.386 : SID ( 204, 4039) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (01173,3) Size = 20 2020-06-01T00:10:14.410 : SID ( 203, 4040) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (01174,3) Size = 742 2020-06-01T00:10:14.410 : SID ( 204, 4040) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (01175,3) Size = 20 2020-06-01T00:10:15.408 : SID ( 203, 4041) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (01176,3) Size = 370 2020-06-01T00:10:15.408 : SID ( 204, 4041) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_DATA
APID ( 96,12) TM ( 21, 3) SEQ (01448,3) Size = 6 2020-06-01T00:15:07.050 : SID ( 202, 449) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_START
APID ( 96,12) TM ( 21, 3) SEQ (01449,3) Size = 20 2020-06-01T00:15:07.413 : SID ( 203, 4042) : SWA_TM_SCI_PAS_TRIGG_SNAP_COMPR_FIRST
...
----
See link:documents/SID_count/3DCS.20200601.txt[] for full listing
=== 3D raw products
It seems they have a different behaviour
==== 3D Raw Normal mode
* SID = 192 = HEAD +
have a different SID_count value, but increasing one by one
* SID = 193 = DATA +
have a different SID_count value, but increasing one by one
Several DATA packet for a single HEAD, so SID_count increase separaterly.
It will be difficult to ensure the HEAD and DATA correspond to the same sequence...
----
APID ( 96,12) TM ( 21, 3) SEQ (00020,3) Size = 18 2020-04-14T12:35:13.266 : SID ( 192, 4) : SWA_TM_SCI_PAS_FULL3D_RAW_HEAD
APID ( 96,12) TM ( 21, 6) SEQ (00021,1) Size = 4096 2020-04-14T12:35:13.377 : SID ( 193, 16) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00022,0) Size = 4096 2020-04-14T12:35:13.502 : SID ( 193, 17) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00023,0) Size = 4096 2020-04-14T12:35:13.627 : SID ( 193, 18) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00024,2) Size = 408 2020-04-14T12:35:13.750 : SID ( 193, 19) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 3) SEQ (00025,3) Size = 18 2020-04-14T12:35:17.266 : SID ( 192, 5) : SWA_TM_SCI_PAS_FULL3D_RAW_HEAD
APID ( 96,12) TM ( 21, 6) SEQ (00026,1) Size = 4096 2020-04-14T12:35:17.377 : SID ( 193, 20) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00027,0) Size = 4096 2020-04-14T12:35:17.502 : SID ( 193, 21) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00028,0) Size = 4096 2020-04-14T12:35:17.627 : SID ( 193, 22) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00029,2) Size = 408 2020-04-14T12:35:17.750 : SID ( 193, 23) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 3) SEQ (00030,3) Size = 18 2020-04-14T12:35:21.266 : SID ( 192, 6) : SWA_TM_SCI_PAS_FULL3D_RAW_HEAD
APID ( 96,12) TM ( 21, 6) SEQ (00031,1) Size = 4096 2020-04-14T12:35:21.377 : SID ( 193, 24) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00032,0) Size = 4096 2020-04-14T12:35:21.502 : SID ( 193, 25) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00033,0) Size = 4096 2020-04-14T12:35:21.627 : SID ( 193, 26) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00034,2) Size = 408 2020-04-14T12:35:21.750 : SID ( 193, 27) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 3) SEQ (00035,3) Size = 18 2020-04-14T12:35:25.268 : SID ( 192, 7) : SWA_TM_SCI_PAS_FULL3D_RAW_HEAD
APID ( 96,12) TM ( 21, 6) SEQ (00036,1) Size = 4096 2020-04-14T12:35:25.377 : SID ( 193, 28) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00037,0) Size = 4096 2020-04-14T12:35:25.502 : SID ( 193, 29) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00038,0) Size = 4096 2020-04-14T12:35:25.627 : SID ( 193, 30) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (00039,2) Size = 408 2020-04-14T12:35:25.750 : SID ( 193, 31) : SWA_TM_SCI_PAS_FULL3D_RAW_DATA
...
----
See link:documents/SID_count/3DRN.20200414.txt[] for full listing
==== 3D Raw Snapshots
* SID = 205 = START : gives the number of samples, with a separate SID_ count (13, 14...)
* SID = 206 = HEAD : data decriptor, with separe SID_count (117, 118, ...)
* SID = 207 = DATA : snapshot data, with SID_count (494, 495...)
Several DATA packet for a same HEAD packet, so SID_count increasing differently
He also, difficult to ensure that DATA and HEAD correspond to the same product
----
APID ( 96,12) TM ( 21, 3) SEQ (03537,3) Size = 6 2020-04-24T11:40:07.000 : SID ( 205, 13) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_START
APID ( 96,12) TM ( 21, 3) SEQ (03538,3) Size = 18 2020-04-24T11:40:07.375 : SID ( 206, 117) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (03539,1) Size = 4096 2020-04-24T11:40:07.376 : SID ( 207, 494) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03540,0) Size = 4096 2020-04-24T11:40:07.501 : SID ( 207, 495) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03541,0) Size = 4096 2020-04-24T11:40:07.626 : SID ( 207, 496) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03542,0) Size = 4096 2020-04-24T11:40:07.751 : SID ( 207, 497) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03543,2) Size = 1862 2020-04-24T11:40:07.875 : SID ( 207, 498) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 3) SEQ (03544,3) Size = 18 2020-04-24T11:40:08.375 : SID ( 206, 118) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (03545,1) Size = 4096 2020-04-24T11:40:08.376 : SID ( 207, 499) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03546,0) Size = 4096 2020-04-24T11:40:08.501 : SID ( 207, 500) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03547,0) Size = 4096 2020-04-24T11:40:08.626 : SID ( 207, 501) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03548,2) Size = 408 2020-04-24T11:40:08.750 : SID ( 207, 502) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 3) SEQ (03552,3) Size = 18 2020-04-24T11:40:09.377 : SID ( 206, 119) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (03553,1) Size = 4096 2020-04-24T11:40:09.378 : SID ( 207, 503) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03554,0) Size = 4096 2020-04-24T11:40:09.502 : SID ( 207, 504) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03555,0) Size = 4096 2020-04-24T11:40:09.628 : SID ( 207, 505) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03556,2) Size = 408 2020-04-24T11:40:09.750 : SID ( 207, 506) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 3) SEQ (03557,3) Size = 18 2020-04-24T11:40:10.375 : SID ( 206, 120) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_FIRST
APID ( 96,12) TM ( 21, 6) SEQ (03558,1) Size = 4096 2020-04-24T11:40:10.376 : SID ( 207, 507) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03559,0) Size = 4096 2020-04-24T11:40:10.501 : SID ( 207, 508) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03560,0) Size = 4096 2020-04-24T11:40:10.626 : SID ( 207, 509) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
APID ( 96,12) TM ( 21, 6) SEQ (03561,2) Size = 408 2020-04-24T11:40:10.750 : SID ( 207, 510) : SWA_TM_SCI_PAS_TRIGG_SNAP_RAW_DATA
...
----
See link:documents/SID_count/3DRS.20200424.txt[] for full listing
=== Conclusion
It seems SID_count could be used to check if HEAD and DATA packet correspond to the same product....
solar
.
But, this is probably due to the good compression factor we have actually, so each 3D products data should be stored in a single DATA packet,
and so, HEAD and DATA SID_count packet increase in the same range, but if compression factor was not good enough, it could be necessary
to have 2 DATA packet for the same 3D distribution, and SID_count will be shifted....
This is the case for 3D raw products, where HEAD packets correspond to several DATA packets, and SID_count increase separately.
So, SID_count should not be used to check that HEAD and DATA correspond to a same 3D distribution.
=== Other candidates
There is another sequence counter, called seq_count, related not to SID, but for a given APID packets.
In case of these 3D data, they are all the same APID : (96,12) of (97,12) or 98(12), and so we can use the seq_count.
Look at SEQ (nnnnn,flag) in the listing :
* nnnn : is a sequence counter, increasing for each TM packet of a given APID
* flag : indicates the place ot TM packet in logical sequence (0 : first, 1 : median, 2 : last, 3 : single)
However, it's more tricky to use, as a given APID should be share by all SWA instruments.
In most of the previous example, PAS was operating alone, but during real mission time, both EAS1, EAS1, HIS and PAS will operate together,
and their APID sequence should interleave...
