IRAP EDDS tools
===============
:toc:
== Introduction
This page describe tools used by IRAP to recover data from EDDS server.
=== Modified Java Stream client
IRAP has modified a Java StreamClient application delivered by ESA, to be able to download
SolarOrbiter TM/TC in near real-time
Rather than creating a new file each N seconds (typically N=60), the new client will register
all TM/TC in a single file, increasing regularly as new data appears in the stream.
Data is registered in a single hexadecimal text file <requestId>.hex
This file contains separate lines for each CCSDS TM packet, consitued by hexadecimal dump of packets content.
It will increase regularly as soon as new TM packets are availble on the stream.
.Main script
----
StreamClientLauncher.sh
----
Shell script that runs an updated Java Stream Client to recover streaming data from EDDS server.
=== python EDDS tools
We wrote another set of python tools to make automated EDDS requests and download corresponding data :
* Launching a new stream request for TC or TM data, and recover data from the stream
* Connecting to an existing stream request (typically launched by MSSL for a wide time-range)
* Launching batch requests and donwload TM, TC or Parameters data
----
EDDS.py
----
This is a python3 script, using EDDS SOAP Web Services, able to:
* Read config.properties file, to extract user authentication parameters and EDDS server address
* Connect to EDDS server with user account
* Launch a batch request to recover automatically SWA TM/TC data or Parameters
* Launch a new data streaming request (TM or TC data) and +
run Java StreamClient to download streaming data in near real-time
* Launch Java StreamClient to recover data from an existing stream request
=== Links
* link:/documents/EDDS/software/STREAM_CLIENT.irap.tar.gz[Download STREAM_CLIENT.irap and tools]
* link:/documents/EDDS/software/STREAM_CLIENT.irap[Browse Software directory]
== Software requirements
* Access to EDDS server
+
--
Ask ESOC for an account on the EDDS server, with a valid Usename and password
Check your network configuration rules for your computer to access EDDS server
--
* python
+
--
You need python 3.x installed on your computer, with some extra libraries :
* zeep : to access SOAP Web services
* isodate : ISO 8601 library (ISO time)
* configobj : to parse config.properties
You can use pip to install those libraries
$ pip install zeep isodate configobj
--
* Java
+
--
Java probably 8.x is necessary (to be confirmed)
--
== Installation manual
Download and extract the link:/documents/EDDS/software/STREAM_CLIENT.irap.tar.gz[]
----
$ cd <install-directory>
$ tar xvzf STREAM_CLIENT.irap.tar.gz
$ cd STREAM_CLIENT.irap
----
You will find here:
* config.properties :
+
config file used by application to find username, password, EDDS server address
* EDDS.py
+
python EDDS library
* StreamCLientLauncher.sh
+
Shell script that run the Java StreamClient application
* streamclient.xml
+
Java beans XML config file, modified by IRAP to change ESA software behaviour
* lib
+
Original ESA Java library (.jar files)
* src
+
IRAP's added Java source files
* irap-streamedds.jar
+
IRAP modified library (build from src/*java)
* build
+
Shell script to build the IRAP Java modified software
=== Configuration
You have first to update the config.properties file with your usual text editor,
to fill your EDDS username and passord
.config.properties
----
# The username that the request submitter should log on as
username = <EDDS-username>
# The password that the request submitter should log on as
password = <your-EDDS-password>
# The EDDS web server that the request submitter should submit requests to
eddsServerAddress = https://mmaasweb01.esoc.ops.esa.int:8443/edds/EddsService?wsdl
----
Update also the content of streamclient.xml file, to set pathname of config.properties file
.streamclient.xml
----
<bean class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfigurer">
<property name="location"
value="file:///home/solar/EDDS/STREAM_CLIENT.irap/config.properties"/>
</bean>
----
The Java StreamClient will read these files, and any error in username, password of config.properties path
will generate an error.
You can also use the build script to reconstruct the IRAP Java library,
but it's not necessary if you dont't modify the corresponding java sources.
----
./build
----
Will create/update the irap-streamedds.jar library
== EDDS tools User Manual
Several kind of utilisation:
* Batch TM requests
* Batch TC requests
* Batch Param requests
* Connect to an existing data stream request
* Stream TM request
* Stream TC request (TC filtering with PID or APID not available)
=== Invoking request
After completing software installation and configuration, in order to use these software,
you must run the EDDS.py from the current installation directory.
You can run the python EDDS.py in two ways:
----
$ python EDDS.py <arguments>
$ python -m EDDS <arguments>
----
The first command line load and run EDDS.py that has to be in current working directory.
The second one search the EDDS.py using the PYTHONPATH environment variable, that gives a list of directories
where python script are located.
However, the python script must find config.properties and StreamClientLauncher.sh, so these 2 files
have to be in the current directory.
That should be improved in further releases.
=== Batch TM requests
Batch TM request are EDDS batch TM report requests, and use RAWSOURCEBINARY output format.
They are filtering TM data for PID in range 95..99, corresponding to whole SWA TM data.
The result files are constitued by a sequence of CCSDS TM binary packets, without any other header.
It could be possible to modify EDDS.py software:
* to get XML output data rather than binary data.
* to use different PID filtering, to get data from other experiments or platform data.
.Command line
----
$ python -m EDDS batch_TM <start-time> <duration>
----
Where <start-time> should be a valid ISO8601 datetime, like yyyy-mm-ddThh[:mm[:ss]]
----
2020-03-17T00
2020-03-17T12
2020-03-17T12:30
2020-03-17T12:30:15
----
And <duration> a valid ISO8601 duration
----
P1D a whole day
P1W a whole week (7 days)
PT8H 8 hours
PT90M 90 minutes
----
.Example
----
$ python -m EDDS batch_TM 2020-03-17T00 P1D
$ python -m EDDS batch_TM 2020-03-17T12:00:00 PT8H
----
The first command is used to download SWA TC data for the whole 2020-03-17
The second one can download data for 2020-03-17T12:00:00 to 2020-03-17T20:00:00
They will generate 2 binary files in the current directory:
----
solo_L0_swa-batch-tm_20200317000000_V00.bin
solo_L0_swa-batch-tm_20200317120000_V00.bin
----
.Example
----
$ python -m EDDS batch_TM 2020-03-17T00:00 P1D
Reading config from config.properties
Login EDDS D1D3593A150F914FB4CC44F29867487D
Send batch request : BatchRequest.PktTmReport.SOL.0.2020.086.16.49.03.503.fBsu
Save request file : BatchRequest.PktTmReport.SOL.0.2020.086.16.49.03.503.fBsu.xml
Send batch TM report : job_id = BatchRequest.PktTmReport.SOL.0.2020.086.16.49.03.503.fBsu
[17:49:03] Status = SUBMITTED
[17:49:08] Status = SERVER_COMPLETED
[17:49:13] Status = SERVER_COMPLETED
[17:49:18] Status = SERVER_COMPLETED
[17:49:24] Status = SERVER_COMPLETED
[17:49:29] Status = SERVER_COMPLETED
[17:49:34] Status = SERVER_COMPLETED
[17:49:39] Status = DELIVERED_PARTIAL_RESULTS
[17:49:39] Creation solo_L0_swa-batch-tm_20200317000000_V00.bin ...
[17:49:39] Done
Good bye...
----
=== Batch TC requests
Batch TC request are EDDS batch TC report requests.
They use XML output format, as RAWSOURCEBINARY format is not yet implemented in EDDS server for TC report requests.
They are filtering TC data with (PID >= 95 and PID <= 99), corresponding to whole SWA TC data.
The result files are XML files, constitued by a sequence of XML PktTcReportListElement,
that can be parsed to extract RawBodyData nodes that contains CCSDS packet content.
It could be possible to modify EDDS.py software:
* to get RAWSOURCEBINARY data, as soon as it will by implemented in EDDS
* to use different PID filtering, to get data from other experiments or platform data.
.Command line
----
$ python -m EDDS batch_TC <start-time> <duration>
----
Where <start-time> and <duration> should be valid ISO8601 datetime and duration
.Example
----
$ python -m EDDS batch_TC 2020-03-17T00 P1D
$ python -m EDDS batch_TC 2020-03-17T12:00:00 PT8H
----
The first command is used to download SWA TC data for the whole 2020-03-17
The second one can download TC for 2020-03-17T12:00:00 to 2020-03-17T20:00:00
They will generate 2 XML output files in current directory:
----
solo_L0_swa_batch-tc_20200317000000_V00.xml
solo_L0_swa_batch-tc_20200317120000_V00.xml
----
=== Batch Param requests
Batch Param request are EDDS batch Parameters requests
They are requesting data from PARC (Parameter Archive database) and produce XML output format.
The result files are XML files, constitued by a sequence of XML ParamSampleListElement
that can be parsed to extract time-tags and engineering values for each parameters.
A list of parameters is actually hardcoded in EDDS.py source code.
You can change this list in run_batch_param() function in EDDS.py
It could be possible to modify EDDS.py software:
* to add list of parameters in command line arguments
.Command line
----
$ python -m EDDS batch_Param <start-time> <duration>
----
Where <start-time> and <duration> should be valid ISO8601 datetime and duration
.Example
----
$ python -m EDDS batch_param 2020-03-17T00 P1D
$ python -m EDDS batch_param 2020-03-17T12:00:00 PT8H
----
The first command is requesting parameters data for while 2020-03-17
The second one is requesting parametres for 2020-03-17T12:00:00 to 2020-03-17T20:00:00
They will generate 2 XML output files in current directory:
----
solo_L0_swa_batch-param_20200317000000_V00.xml
solo_L0_swa_batch-param_20200317120000_V00.xml
----
=== Connect to an existing stream request
Must be used to download data from an existing stream request, using its requestId.
It will update requestId value in the config.properties file and run the Java StreamClient to download new data.
New packets data will be be added at the end of the file <requestId>.hex, in current directory.
MSSL should launch an unique stream TM data request and share its request ID, to all SWA users.
.Command line
----
$ python -m EDDS client <requestId>
----
Where <requestId> should be a valid request ID for an active stream request.
.Example
----
$ python -m EDDS client StreamRequest.PktTmStream.SOL.0.2020.078.18.20.15.589.vENe
----
Will append new TM data to:
StreamRequest.PktTmStream.SOL.0.2020.078.18.20.15.589.vENe.hex
The result file will contain a separate line for each CCSDS TM packet received in the stream.
Each line will correspond to the hexadecimal dump of CCSDS data packet (without any other header).
Every new packet wil be displayed in the console, so you can check new data arrival.
.Example of data
----
$ head StreamRequest.PktTmStream.SOL.0.2020.078.18.20.15.589.vENe.hex
0E37C000000B1005010080000003D8D5AB14
0E37C001000B100501002604C71B229CAA14
0E37C000000B100501002604C7229F85AA1A
0E37C0010011100502002604C722C004AA26000600050054
0E37C002000B100501002604C7250002AA0F
0E37C0030011100502002604C8020003AA26000000000000
0E37C004000B100501002604C8B5600FAA19
0E37C005000D100501002604CFD82015AA100000
0E37C006000D100501002604CFD8201EAA100001
0E37C007000D100501002604CFD82028AA100002
----
=== Stream TM requests
[CAUTION]
====
Please, do not use this functionnality to avoid saturation of EDDS server.
MSSL willl launch a unique stream TM request, with a long time period validity (severall day or weeks),
and share the corresponding request ID, so all SWA users can connect to the corresponding stream and recover SWA TM data.
====
It will launch a new stream TM request on EDDS server.
The corresponding requestId will be automatically registered in config.properties file, before it runs the Java StreamClient.
TM data will be stored in a single file, <requestId>.hex, in the current diretory.
This file will contain, for each CCSDS TM packet, separate lines with hexadecimal dump of CCSDS packet content, without any other header.
File will increase regularly, in near real-time, as soon as new TM packets are available on the stream.
.Command line
----
$ python -m EDDS stream_TM <duration>
----
Where <duration> should be a valid ISO8601 duration.
.Example
----
$ python -m EDDS stream_TM PT8H
$ python -m EDDS stream_TM P2W
----
The first command launch a new stream TM request with a duration of 8 hours.
The second one is for a duration of 2 weeks, 14 days.
=== Stream TC requests
This kind of stream request is not fully available, as it is not possible to filter TC,
for a given PID or APID.
However, it can be used as soon as such data filtering will be implemented in EDDS server.
.Command line
----
$ python -m EDDS stream_TC <duration>
----
Where <duration> should be a valid ISO8601 duration.
.Example
----
$ python -m EDDS stream_TC PT8H
$ python -m EDDS stream_TC P2W
----
The first command launch a new stream TC request with a duration of 8 hours, the second one is for 2 weeks.
== Possible improvements
It may be possible to improve the EDDS.py software to add new functionnalities:
* other kind of requests
* modifying output formats
* modifying data filtering
* etc...
You can also use the EDDS.py as a library and use its functionnalities to write automated tools,
corresponding to your needs.
Software developpers may have a look on:
* link:/documents/EDDS/software/STREAM_CLIENT.irap/EDDS.py[EDDS.py]
+
--
Main python script
--
* link:/documents/EDDS/software/STREAM_CLIENT.irap/EddsService.wsdl[EddsService.wsdl]
+
--
WSDL file that describes the EDDS Soap availables Web services
--
* link:/documents/EDDS/software/STREAM_CLIENT.irap/model.xsd[model.xsd]
+
--
XML Schema definition of XML entities used by EDDS Soap Web Services
--
== Acknowledgements
Please acknoledge Tom.barthe@gmail.com
* He has made available the EDDS Web services from remote python script, using SOAP protocol.
* He has also overloaded the ESA stream client software to modify the application behaviour,
in order to create a single result file per request, increasing regularly in near real-time.
