*Only RHEL 8 is supported not CentOS8

In the Fall 2024 Release, we will discontinue support for Windows Server 2016 and introduce support for Windows Server 2022.

*There is a compatibility issue with java 11.0.16. For more information see the ECCOSP-226 ticket.

**ECP 4.7.1/EDX 1.8.1 and older are not compatible with Java 1.8.0_272 and newer.

***Only JRE is supported for Windows installations. For Unix, the recommended option is to use Unix repositories – i.e. yum install java-17-openjdk (as the JRE package is not present).

MySQL

MS SQL

Oracle

PostgreSQL

Ticket: 9332633

There seems to be an issue with the usage of a SOCKS 5 – proxy in the ECP4 – environment, I’d like to report to you.

1. Setup

   a. ECP4 is installed on Windows Server 2016

   b. The connection to ecp.entsoe.eu is directed via SOCKS v5 – proxy (CISCO Ironport)

   c. The Proxy – Settings are configured in ecp.properties

1. Observed behaviour

   a. Synchronisation of Component Directory (HTTPS) works without issues

   b. The communication via AMQPS (Ports 5671, 5672) is not stable. That is, the connection gets lost and can’t be re-established for many hours. If the connection is established, it stay’s established for an arbitrary time (some hours up to 2-3 days).

   c. Even if we can’t upload files to broker 10V1001C—000438 and 10V1001C—000446, we still can receive data via these brokers (ENTSOE-PSD messages).

2. Changes necessary to get the connection stable

   a. Configure SOCKS Proxy to allow the complete server to access ecp.entsoe.eu on the given ports

   b. Omit ecp.security.proxy.username and ecp.security.proxy.password from ecp.properties

3. Conclusion

   a. In the setup shown above, the communication via SOCKS Proxy isn’t stable for AMQPS if the proxy – permissions are granted via User – Credentials (maybe especially, if this user is authenticated with Windows Active Directory or ECP4 / Tomcat is running on Windows Server)

   b. Maybe the documentation of how to use a SOCKS – Proxy in ECP-environment isn’t complete in the sense that OS-specific information is missing

Question 1: Does the “StatusDocument” include the original message metadata (The Pull message notification)? If it is not the case, is there any way to retrieve the original message metadata ?

Status document contains the following metadata:

• originalmessageID v=””/
• receiveTimestamp v=””/
• status v=””/
• changeTimestamp v=””/

Question 2: Where can we find the XSD schema/documentation of a “StatusDocument” ?

XSD schemas could be found in EDXInterfaces-1.7.0.834.zip, which should be stored to https://extra.entsoe.eu/

Question 3: Is it possible to change/re-configure the max retries to download a pull message content ?

The reconfiguration could be done by adding the following lines into edx.properties file:

• edx.toolbox.pull.downloadRetryCount= retry count
• edx.toolbox.pull.retryJobDelay= delay of retry job in seconds

Document created by Swissgrid about it : selfSigned_jks.docx

A guide for the generation process of root and integrated certificate for Windows and KeyStore Explorer:

1. Start KeyStore Explorer application
2. Select “Create a new KeyStore” and type “JKS”
3. Creation of root (global) certificate

Click on Tools -> Generate Keypair
Generate Key Pair screen 
  - Select RSA Algorithm with key size 4096
  - Add extensions
  - then click OK

4 Creation of integrated certificate (for ECP CD)

   Right-click on global certificate and select Sign -> Sign New Key Pair
   Generate Key Pair screen
      - Select RSA Algorithm with key size 4096, then click OK
   Generate Key Pair Certificate screen
      - Select Version 3, SHA-256 with RSA, validity period e.g. 15 years and click Apply
   Fill the name fields - email, common name, organization, locality name and country
      - CD vCode should be used as a common name
   Add Extensions
      - Authority Key Identifier - Key Identifier 160-bit hash
      - Basic Constraints with Critical flag - Mark "Subject is a CA"
      - Key Usage - Select "Digital Certificate", "Certificate Signing", "CRL Signing"
      - Subject Key Identifier : Key Identifier 160-bit hash
   Enter Alias "integrated"
   Enter new password "password"

5 Save the created keystore and place it on safe place - this is the keystore which can be used for generating certificates for another ECP CDs

6 Right click on global certificate and select Export -> Export Certificate Chain

 Head Only, X.509, PEM
 Save it as global.cer

7 Right-click on integrated certificate and select Export -> Export Key Pair

 Save it as integrated.p12

8 Compilation of keystore for ECP CD

 Click on File -> New and select JKS
 Click on Tools -> Import Trusted Certificate and select file global.cer
    - Fill Alias "global"

9 Click on Tools -> Import Key Pair, select PKCS #12 and select file integrated.p12

 Fill Alias "integrated" and password "password"

10 Save the created keystore as ecp_module_cd.jks

11 After installation of ECP CD, use ecp_module_cd.jks for registration of ECP CD

Please note that connection to the embedded Apache Derby database is not a common administrator task, so it is not described in ECP Administration/Installation Guide.

The connection via client DBeaver is described in ECP & EDX Migration guide, chapter ”Connect to embedded database” as follows:

1. Stop ECP Endpoint
2. Copy database from ECP server to location, where DBeaver client is installed.

Apache Derby database is stored in the folder db on path specified in ECP configuration file ecp.properties – dataDirectory value.

Default location of the embedded database:

• Linux: /var/lib/ecp-endpoint/db
• Windows: /db

1. Start DBeaver client, open ECP database

• Open DBeaver application, go to the top menu,
• select File > New > Database connection > Expand folder DBeaver > Database Connection > Expand folder Derby > Embedded >
• Click on the button “Browse” and traverse to “db” directory,
• fill in credentials ecp/ecp-password > Next > Finish​

ECP implements a message deleting functionality to free the Message Store from the old processed messages. The message retention period is configurable and can be adapted to the traffic on specific ECP installation. The ECP instances with high traffic will set the period to a lower interval (e.g., several hours) than the instances with low traffic (e.g., several days). By default, the retention period is configured to two weeks.

ECP implements a mechanism to prevent the contentions and slow performance. When the amount of the stored messages exceeds a threshold, ECP emits a warning message advising to reduce the message retention period. When the amount of the stored messages exceeds a critical threshold, the messaging functions of ECP are paused until the messages amount drops. In such case, the user is warned via an information message and is advised to immediately execute the deleting process. Both thresholds are configurable to adjust the limits in case of emergencies.

There are several configuration properties that influence message box behaviour. See descriptions below:

Alternative way

Another possibility is delete messages directly from database using following SQL commands:

DELETE FROM ECP.MESSAGE;

DELETE FROM ECP.SENT_MESSAGE_REGISTER;

By default, EDX Toolbox runs deleting job every two minutes, which deletes messages from the database and contents from DMS, which are older than 168 hours. Periods for deleting from the database and from DMS can be configured independently, so it is possible to delete DMS contents and keep the messages in the database for some time. Because it is not possible to find DMS content after the message record is removed from the database, removing of the database record also deletes DMS content if it is still present in DMS. Following properties can be used for deleting job configuration:

Alternative way

Another possibility is delete messages directly from database using following SQL commands:

DELETE FROM EDX.TOOLBOX_MESSAGE WHERE PARENT_MESSAGE_ID IS NOT NULL;

DELETE FROM EDX.TOOLBOX_MESSAGE;

DELETE FROM EDX.PULL_MESSAGE;

DELETE FROM EDX.INFLIGHT_EXTERNAL_PROCESSING;

DELETE FROM EDX.TOOLBOX_MESSAGE_ECP_DELIVERY;

DELETE FROM EDX.TOOLBOX_MESSAGE_LOG;

If necessary, files on DMS (/usr/share/edx-toolbox/temp/edx-dms/edx-01/) can be deleted by command: find /usr/share/edx-toolbox/temp/edx-dms/edx-01/ -mtime +3 -delete.

Please note that the parameter for age (3 days) should be equal to edx.toolbox.deleting.deleteOlderThan configuration (72h).

Files to this directory: /opt/opdm-data/edx-pull/in/ are incoming to your EDX Toolbox from RSC (Central components). These files are then processed by OPDM Client.

However, OPDM Client in version 2.7.1 does not clean up the files after being processed. This will be implemented in the new version of the OPDM Client.

You may use the “find /opt/opdm-data/edx-pull/in/ -mtime +3 -delete” command in cooperation with Cron, to handle the count of files in this directory.

To enable the AMQP set the following property to true in the runtime or system (ecp.properties) configuration:

• ecp.endpoint.amqpApiEnabled=true

Default AMQP receive handler will be configured by the following properties:

• ecp.endpoint.sendHandler[0].beanName=amqpApiSendHandler
• ecp.endpoint.sendHandler[0].typeName=* (receiver handler will be enabled for all message types)

Please note that the ecp.endpoint.sendHandler[0].className property should stay commented out. It is required only when some custom handler is used.

The propose of the document is to summarize issues during certificate renewal on ECP Endpoint through various application versions.

ECP 4.3.x

ISSUE

• In ECP 4.3.x there is an issue in selecting a proper certificate in case that more certificates of the same type exist (e.g. signing, encryption).
• The ECP Endpoint of this version selects as default the certificate with minimal “valid from” date and does not take into account the expiration date of the certificate.

This issue was resolved in ECP 4.4.x.

• In case there are ECP 4.3 and 4.4 in-network, which is valid for TP. Then it is necessary to delete old certificates on ECP 4.4 as well. (TP ECP Endpoint has 4.4 and after certificate renewal, the DP with ECP 4.3 was not able to communicate.)

WORKAROUND

• After certificate renewal, it is necessary to delete old certificates and push their configuration to ECP CD. These actions can be done from GUI.

-—————-

ECP 4.4.x

ISSUE

• During certificate renewal ECP Endpoint remains in registering state in some cases.
• Certificates are renewed and pushed, but the state of application causes the messaging issue

The other issue reported is that ECP Endpoint does not push renewed certificates into ECP CD. It can be revealed by finding the own ECP Endpoint in the Component list on ECP Endpoint GUI.

WORKAROUND

• Connect to database, in table “application_properties” change value of “ecp.internal.status” to ACTIVE.
• Manually push configuration from GUI (Settings -> Push Configuration)

-—————-

ECP 4.6.x + ECP 4.7.x

The issues from the previous version are resolved. There are no reported issues with certificate renewal.

During FAT the behaviour has been retested on ECP 4.7.2 and certificates were renewed properly.

ISSUE Having upgraded the ecp-endpoints to v4.6, you can no longer login using REST.

Performing the same procedure carried out on a v4.6 endpoint, returns:

{ “timestamp”: “2020-01-03T11:39:58.198+0000”, “status”: 403, “error”: “Forbidden”, “message”: “Invalid CSRF Token ‘’ was found on the request parameter ‘_csrf’ or header ‘X-XSRF-TOKEN’.”, “path”: “/ECP_MODULE/login” }

Why is the endpoint expecting a CSRF token on login?

RESPONSE ECP Endpoint does not have a separate REST interface, it was intended to access only by dashboard interface. That is the reason that CSRF protection is always enabled. But as a workaround for access by REST, you could try to call GET request on the login page, get the CSRF token from the response and use it for the POST request. 

______________________________________________________________________________________

ISSUE When trying to execute the messaging connectivity check, using the same CSRF token as in the first check, I get a failure:

status=403; error=Forbidden; message=Invalid CSRF Token ‘e22b2862-c21c-489f-a42c-99d7f77db8bd’ was found on the request parameter ‘_csrf’ or header ‘X-XSRF-TOKEN’.; path=/ECP_MODULE/settings/connectivityCheck

I’ve also tried to skip the CD connectivity check, but the messaging connectivity check still fails. So, how do I get the CSRF token for the messaging connectivity check, if not using a GET on the /login URL?

RESPONSE CSRF token changes after login, but then it will remain the same until logout. So after login you will need another token, which can be extracted from the response of the GET request to the application root (*/ECP_MODULE/).

I’ve a problem with edx-toolbox (V. 1.8.2). The toolbox has to be registered to the respective Service Catalog (10V1001C-00273W in this case), but the button “+Register Toolbox to SC” is deactivated (see screenshot).

-> Thank you for the question. I investigated a little bit and I realized that it is not straightforward. I think a pop-up is present when you put your mouse on the button.

The principle is that a toolbox needs to be first registered in the Service Catalogue manually (SC code is configured in the edx.properties). Without the initial registration in SC, the toolbox cannot be registered in other SCs through that button.

Please request to the Service Catalogue admin or project manager to add your toolbox in the SC.

--> Issue fixed by user: This issue has been resolved. It was caused by typo in the ServiceCatalog EIC-Code.

If you are running an old version (before 4.8) of ECP the automatic renewal doesn’t work correctly.

One way to overcome the issue is register the endpoint with a new Keystore provided by CD admin. Otherwise, register again the same V-code, once approved, it will over-write the previous entry in the CD.

Be aware that “configuration push” might be needed from the endpoint! Depending on the version used.

Error message:

• Failed to connect to remote at: <amqps://xxx>

This is usually causes by the endpoint being down (not running) and not accessible.

• ECP Endpoint is not running

• ECP Endpoint does not „trust“ the EDX Toolbox

  • ECP EP and EDX TB should have certificates with the same root CA. If the root CAs are different, ECP EP can reject the connection
  • Toolbox user and group must be defined on ECP EP in user.properties and groups.properties. For more information see the EDX Installation Guide, General prerequisites before installing EDX Toolbox

Possible causes:

• TTL of ECP Endpoint is expired - ECP EP has not been able to synchronize with CD for more than 28 days
• Receiver code doesn’t exist in the ECP Network - there is probably a typo in the receiver code or receiver EP has been revoked
• EDX Toolbox has not been able to find the pull message content – the content has not been copied to the correct pull.root.out directory or the content path in the notification document is invalid
• In case that embedded pull is used, message payload exceeded the maximum message size
• Receiver Toolbox is not in the same EDX Network as sender Toolbox – sender and receiver are not registered in the same Service Catalogue

ECP 4.14.0 / EDX 1.14.0 and newer

Please note, that the Hawtio console does not contain the information about queues anymore. Instead, the queues are available directly in the Internal Broker (Artemis Web Console). For more information about Artemis Web Console exposure, please see the How to allow connection to ECP Broker ActiveMQ GUI? section below. The Hawtio console is available as an actuator endpoint for ECP Endpoint and EDX Toolbox and it is disabled by default. To enable it, add hawtio,jolokia to the list in the management.endpoints.web.exposure.include property in the ECP/EDX configuration file. An access to the Hawtio console is protected by the same authentication as in ECP/EDX GUI. Please note, that in case of ECP Endpoint, the Howtio console is accessible only from localhost. To enable the remote access, follow the following procedure:

1. Go to the following path a. Linux: /usr/share/ecp-endpoint/webapps/ECP_MODULE/WEB-INF/classes b. Windows:\tomcat\webapps\ECP_MODULE\WEB-INF\classes
2. Update jolokia-access.xml file a. Remove, comment out, or specify host in the remote section

ECP 4.12.0 / EDX 1.13.0 and lower

This is how to deploy hawtio to your ECP Endpoint or EDX Toolbox:

1. Download Hawtio web war from MAVEN Central Repository (https://mvnrepository.com/artifact/io.hawt/hawtio-web/1.5.11)

   Please note, that some older version might not be compatible with this version of hawtio. In that case it is recommended to use some older version of hawtio-default or even hawtio-web.

2. Rename downloaded hawtio-web-1.5.11.war to hawtio.war

3. Drop the hawtio.war file to webapps directory

    a. On Linux: /usr/share/component/webapps

    b. On Windows:\tomcat\webapps

4. Set property spring.jmx.enabled in your properties to true (this step is optional, the property needs to be enabled only if you need to list the queues)
5. Restart the component – the plugin should be deployed
6. You can then access the hawtio GUI by opening address https://:/hawtio in your web browser.

1. Does the AMQPS messaging protocol natively support SOCKS or is a ‘Socksifier’ required to make this work?

   • AMQPS supports SOCKS natively.

2. Does traffic coming via socks support QoS?

   • QoS is not supported.

3. I’m assuming SOCKS 5 will be used for authentication? Where will authentication handoff reside? On the F5/LDAP?

   • SOCKS5 has possibility to fill in credentials (username/password) for connection to SOCKS5 server. Username and password is defined in the ECP configuration file.

4. How can we guarantee against nefarious use of SOCKS here?

   • SOCKS proxy is used only for outgoing connection, you can define which local server can connect to SOCKS server and which local server will be forwarded to remote host. The connection to SOCKS server can be secured by user/password.

5. Support for SSL on the AMQPS.

   • SSL is always enabled when AMPQS is used.

High Level Description

• No implementation necessary
• Services and Domains can be used for publication limitation.

Affected Components

• EDX Toolbox
• EDX Service Catalogue

Services and Domains can be used for publication limitation.

Toolbox will automatically consume all services in its domains, this means that toolbox will be able to subscribe all publications with defined service in its domain. Toolboxes from different domains will not be able to subscribe.

Note: When publication is created without service (only domain is defined), all toolboxes across all domains (within same SC) will be able to subscribe the publication.

Example scenario:

There are three Toolboxes registered in the same Service Catalogue and there will be two publications, Pub-TSO-only for TSO1 (as publication provider) and TSO2 only, and Pub-All for both TSOs and NEMO (as publication provider). The limitation can be done by following steps.

1. On Service Catalogue create domain for TSO1 and TSO2 - e. g. TSOs.
2. In domain TSOs create service e. g. TSOservice.
3. On TSO1 create publicaton in domain TSOs and service TSOservice. This publication will be available only for TSO1 and TSO2 because NEMO is in different domain which means it will not consume TSOservice.
4. Subscribe the publication on TSO1 and TSO2.
5. On NEMO create publicaton in any domain and without service. This publication will be available for all Toolboxes across all domains.
6. Subscribe the publication on TSOs and NEMO.

ECP 4.14.0 and newer

The credentials are located in the following path:

• ECP Broker o /opt/ecp-broker/artemis/broker/etc/artemis-users.properties
• Internal Broker o On Linux: /opt/eccosp-artemis/etc/artemis-users.properties o On windows:/broker/etc/artemis-users.properties

ECP 4.12.0 and lower

The credentials are located in the following path

• on Linux: /opt/ecp-broker/activemq/conf/jetty-realm.properties
• on Windows:\activemq\conf\jetty-realm.properties

ECP 4.14.0 and newer

To allow connection to GUI from other locations (not just from the server the broker is installed) it is necessary to change the following configuration files:

• ECP Broker

  • /opt/ecp-broker/broker/etc/jolokia-access.xml and bootstrap.xml

• Internal Broker

  • On Linux: /opt/eccosp-artemis/etc/jolokia-access.xml and bootstrap.xml
  • On Windows:\broker\etc\jolokia-access.xml and bootstrap.xml

In the jolokia-access.xml:

• Edit or add <allow-origin> element with an URL pattern to match URLs from where the access will be allowed
• Multiple <allow-origin> elements can be defined
• URL pattern supports a wildcard character: *

In the boostrap.xml:

• In the binding element, find and change the uri attribute:

  • E.g. uri=”http://0.0.0.0:8161”
    OR
  • uri=”http://<server IP address>:8161”

ECP 4.12.0 and lower

To allow connection to GUI from other locations (not just from the server where the ActiveMQ is installed) it is necessary to change the following configuration file:

• (for Linux) /opt/ecp-broker/activemq/conf/jetty.xml
• (for Windows)\activemq\conf\jetty.xml

Find element <property name=”host” value=”127.0.0.1”/> and change the IP address from localhost to the server’s IP address. Then restart ECP Broker.

After this, the ActiveMQ GUI will be available on http://<IP address>:8161

Some users have encountered an issue with EDX Toolbox while using an Oracle DB, where open cursors were not closed properly. This led to the EDX stopped working once the maximum cursor limit was reached.

This issue is caused by a slight change in the EDX 1.12.0 where the value of the property “spring.datasource.timeBetweenEvictionRunsMillis” has been set to -1 (same as for ECP). Because of that, the thread responsible for the eviction of the idle objects is not started, thus the connection remains idle in the database.

As a solution, the following property can be added to the edx.properties file:

• spring.datasource.timeBetweenEvictionRunsMillis=5000.

Upon restarting the application, the configuration change will be implemented, effectively resolving the issue related to open cursors.

A permanent solution was delivered in EDX 1.13.0 and ECP 4.12.0, part of the ECCo SP Fall release 2023.

It is possible to enable HTTPS for the Artemis Management Console. Open bootstrap.xml in /opt/ecp-broker/broker/etc folder and find the binding element. Change http:// to https:// in the uri attribute. Then configure the following attributes:

• keystorePath – A path to the keystore file.
• keystorePassword – A keystore password. Can be masked using ENC() syntax.
• sniHostCheck – Set it to false when the certificate in the keystore does not contain a host name in the Common Name or in the Subject Alternative Names. Default is true.
• includedTLSProtocols – TLSv1.3 is a recommended protocol.
• includedCipherSuites – Following ciphers are recommended: TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256,TLS_AES_128_GCM_SHA256

The Keycloak supports the User Federation feature, which allows the integration with LDAP or Active Directory server. To enable this feature and configure the User Federation, please refer to the Keycloak documentation.

Please note that the Active Directory will reject all password change requests when a non-SSL (ldap://) connection is used, therefore it is recommended to configure an SSL connection for this integration. For the certificate configuration, please refer to the Keycloak documentation.

For more information about LDAP authentication configuration, please refer to the ECP Administration Guide, chapter How to configure LDAP authentication.