CCSDS tools
===========
== CCSDS library
We created at IRAP a python library to parse CCSDS TM files.
See link:software/python/CCSDS.py[]
This tools is able to parse a sequence of CCSDS TM/TC packets, from binary or hexadecimal file.
=== Installation notes
This library needs a python 3.6.x or greater.
==== Spice kernels
It uses also another library, to handle Spice Kernels files to provide correct timetags from CCSDS coarse_time and fine_time fields.
See link:software/python/spice_manager.py[]
This tool depends on an external library, spiceypy that can be installed using:
----
pip instal spiceypy
----
==== MIB database
The CCSDS library needs also an access to Solar orbiter MIB database, to extract some useful informations.
This MIB database can be downloaded from MSSL or SOC website.
See link:software/python/mib.py[]
The various MIB database versions have to be installed in a directory, corresponding to MIB_DIR shell environmment variable.
----
export MIB_DIR=/DATA/SOLAR/MIB
----
And we need also to create a Linux logical link to the latest MIB available version
----
$ cd $MIB_DIR
$ ln -s MIB__20210330M08921PFMS001_SOL latest
----
== CCSDS packet attributes
For each packet TM/TC packet, we can have access to the following attributes
[cols="20%,80%"]
|====
| packet_type | Packet type (O: TC, 1: TM)
| pid | Identifier (95..99) for SWA instruments
| categ | Category of packets
| apid | APID = 16 PID + categ
| seq | Sequence counter (for each apid)
| service_type | PUS service type
| sub_service | PUS sub type
| coarse | coarse time (integer seconds)
| fine | fine time (integer x 1/512s)
| dt | timetag (python datetime) computed with SpiceKernels correction
| size | Size in bytes of paylod
| data | payload byte array
| length | CCSDS length field
| raw | byte array (including CCSDS headers)
| sid | Packet identifier (214 for PAS calibration)
| spid | Packet identifier in MID database
| descr | Packet description extracted from MIB
| tc_name | ZIAxxxx
| tc_descr | TC description from MIB
|===
== CCSDS file reader
This function allows to parse a binary or hexadecimal CCSDS file, as a sequence of TM/TC packets.
A first example, to parse the whole file, selecting TC_ACK TM packet (service_type = 1) and display default packet attributes
----
from CCSDS import CCSDS_reader
filename = "20210424_000000_20210425_000000.swa-batch-tm.bin"
for packet in CCSDS_reader (filename):
if packet.service_type == 1:
print (packet)
----
Produces the following output
----
APID( 99, 1) TM( 1, 1) # 1359 Size = 4 2021-04-24T05:27:30.701 SWA_TM_CMD_ACP
APID( 99, 1) TM( 1, 1) # 1359 Size = 4 2021-04-24T05:27:30.701 SWA_TM_CMD_ACP
APID( 99, 1) TM( 1, 7) # 1360 Size = 4 2021-04-24T05:27:33.951 SWA_TM_CMD_EXE
APID( 99, 1) TM( 1, 7) # 1360 Size = 4 2021-04-24T05:27:33.951 SWA_TM_CMD_EXE
APID( 99, 1) TM( 1, 1) # 1361 Size = 4 2021-04-24T05:27:34.094 SWA_TM_CMD_ACP
APID( 99, 1) TM( 1, 1) # 1361 Size = 4 2021-04-24T05:27:34.094 SWA_TM_CMD_ACP
APID( 99, 1) TM( 1, 7) # 1362 Size = 4 2021-04-24T05:27:34.244 SWA_TM_CMD_EXE
APID( 99, 1) TM( 1, 7) # 1362 Size = 4 2021-04-24T05:27:34.244 SWA_TM_CMD_EXE
APID( 99, 1) TM( 1, 1) # 1363 Size = 4 2021-04-24T05:27:34.244 SWA_TM_CMD_ACP
APID( 99, 1) TM( 1, 1) # 1363 Size = 4 2021-04-24T05:27:34.244 SWA_TM_CMD_ACP
APID( 99, 1) TM( 1, 7) # 1364 Size = 4 2021-04-24T05:27:34.368 SWA_TM_CMD_EXE
APID( 99, 1) TM( 1, 7) # 1364 Size = 4 2021-04-24T05:27:34.368 SWA_TM_CMD_EXE
APID( 99, 1) TM( 1, 1) # 1365 Size = 4 2021-04-24T12:00:00.777 SWA_TM_CMD_ACP
APID( 99, 1) TM( 1, 7) # 1366 Size = 4 2021-04-24T12:00:00.777 SWA_TM_CMD_EXE
APID( 99, 1) TM( 1, 1) # 1367 Size = 4 2021-04-24T12:00:10.786 SWA_TM_CMD_ACP
APID( 99, 1) TM( 1, 7) # 1368 Size = 4 2021-04-24T12:07:25.028 SWA_TM_CMD_EXE
----
We can modify the default behaviour, replacing the default output with:
----
if packet.service_type == 1:
print (packet.raw.hex())
----
Will display the whole TM packet, packet.raw, but converted to hexadecimal with the hex() function.
----
0e31c54f000d1001016e28166883c0a11dfccfcf
0e31c54f000d1001016e28166883c0a11dfccfcf
0e31c550000d1001076e2816688700991dfccfcf
0e31c550000d1001076e2816688700991dfccfcf
0e31c551000d100101fe2816688725611dfcc000
0e31c551000d100101fe2816688725611dfcc000
0e31c552000d100107fe281668874b981dfcc000
0e31c552000d100107fe281668874b981dfcc000
0e31c553000d100101fe281668874ba41dfcc000
0e31c553000d100101fe281668874ba41dfcc000
0e31c554000d100107fe281668876b641dfcc000
0e31c554000d100107fe281668876b641dfcc000
0e31c555000d1001016e2816c481c9e61dfccfd0
0e31c556000d1001076e2816c481ca071dfccfd0
0e31c557000d1001016e2816c48bcc4e1dfccfd1
0e31c558000d1001076e2816c63e0a2f1dfccfd1
----
We can also produce a binary output file, from a CCSDS hexadecimal file and select only SWA_EVENTS.
----
input_file = "20210424_000000_20210425_000000.swa-batch-tm.hex"
output_file = "20210424.event.bin"
with open (output_file, "wb") as output:
for packet in CCSDS_reader (input_file):
if packet.service_type == 5:
output.write (packet.raw)
----
== TM data filtering
It is possible to filter TM/TC data using dedicated filtering functions.
Filtering SWA EVENTS packet
----
def event_filter (packet):
return packet.service_type == 5 and packet.sub_service in [1,2,3,4]
----
Filtering TC acknowledgements
----
def ack_filter (packet):
return packet.service_type == 1 and packet.sub_service in [7,8]
----
Filtering PAS calibration data
----
def calib_filter (packet):
return packet.apid == 1532 and packet.sid == 214
----
A more complex example:
* the following function start to convert a string parameter to a internal python datetime.date
* then compute start and stop variables (python datetime objects) to cover from 00:00 to 24:00
* then creates a filtering function that embed these start/stop time interval to filter TM packets
* finally, return the filtering function
----
def make_daily_filter (str):
"""
Return a function that filters daily TM packets
"""
day = str_to_date (str)
start = datetime.datetime (day.year, day.month, day.day, 0, 0, 0)
stop = start + datetime.timedelta (days = 1)
def filter (packet):
return start <= packet.dt < stop
return filter
----
Then we can use these filtering functions to select the corresponding packets.
----
from CCSDS import CCSDS_reader
filename = "20210424_000000_20210425_000000.swa-batch-tm.bin"
# choose one from
my_filter = event_filter
my_filter = ack_filer
my_filter = make_daily_filter ("2021-04-24")
for packet in filter (my_filter, CCSDS_reader (filename)):
print (packet)
----
== Merge/select data from several input files
It should be possible to merge several input files, applying some filter, to produce an unique output file.
One example of use is to merge various CCSDS TM file, corresponding to daily files downloaded from EDDS server with storage_time parameter.
We can run every day a new request to EDDS to download all SWA data with storage_time in range 00:00/24:00.
That will create daily files with all SWA packets dumped during this period.
Then, we can merge all these files and filter CCSDS packet using their timetag (corresponding to CCSDS packet generation_time), in a given daily range.
So we can reconstruct a daily generation_time TM file.
----
def merge_TM (input_files):
"""
Produce a sequence of TM packet from a list of CCSDS input files
"""
for filename in input_files:
for packet in CCDS_reader(filename):
yield packet
def produce_TM_file (str_date, input_files):
"""
Produce daily TM file from a list of input files
"""
my_filter = make_daily_filter (str_date)
output_file = "%s.generation_time.bin" % str_date
with open (output_file, "wb") as output:
for packet in filter (my_filter, merge_TM (input_files)):
output.write (packet.raw)
print ("Generation", output_file)
if __name__ == "__main__":
from sys import argv
day = argv [1]
input_files = argv [2:]
produce_TM_file (day, input_files)
----
Example of use:
----
$ python produce_TM.py 2021-04-08 20210408.storage_time.bin 20210409.storage_time.bin 20210410.storage_time.bin 20210411.storage_time.bin ...
----
