Upgrade HA Nodes with Zero Downtime

EFT administrators can upgrade their HA nodes with "zero downtime," meaning you can upgrade a cluster without stopping all nodes at the same time.

NOTE: The warning prompt "Cluster is in upgrade mode" is thrown when a user makes a configuration change and then refreshes the EFT administration interface while the server is in "read only" mode.

The Server > HA tab is used to manage the cluster upgrade state

  • The button will be displayed as Begin when upgrades have not been started.

  • The button will be displayed as End when upgrades have been completed. Administrators can execute the End state at any point, but nodes that have not been upgraded will fail to start until they are upgraded.

  • When enabling the upgrade, EFT administrators will be prompted with a warning that the transition and change cannot be undone.

  • The warning prompt should reference details to review documentation.

To upgrade all of the HA nodes at that same time—without stopping any nodes

In the EFT administration interface, on the Server > High Availability tab, you must manually put the cluster into the Upgrade state, upgrade each node one by one, then after the upgrade has completed successfully, manually transition the cluster back to Normal state.

  • Upgrade State. Allows simultaneous updates to nodes that have an Outdated version.

    Upgrade state limitations:

    • Backup/Restore feature is prohibited

    • No configuration changes are allowed during the Upgrade state, except moving the cluster back into the Normal state

  • Normal State. Requires all nodes have the same version. Upon transition to Normal, the Upgrade State will display “Not Active”

    Normal state limitations:

    • Loading old nodes is prohibited. EFT will not start when the nodes are not upgraded.

    • You will be unable to transition from Upgrade to Normal when an upgrade has not been completed. This only affects nodes that have not been upgraded.

    • EFT will prevent COM API from transitioning from Upgrade to Normal state. A proper response code/error will be presented

    • EFT will prevent REST API from transitioning EFT from Upgrade to Normal state. A proper response code/error will be presented.

    • When upgrade has ended (at the global level or a node that was upgraded), for any nodes that were not upgraded, the EFT Server Service will stop and fail to start until they are upgraded.

    • Transition state from Upgrade to Normal are global, which affects all nodes within the cluster, so administrators should only end upgrades after all nodes have been upgraded.

Upon successful HA upgrade to a new version, the High Availability tab's Upgrade State will display “Active; Updated; x.x.x.x”, where x.x.x.x is the newer version of EFT (the version to which EFT was upgraded).

Normal state to Upgrade state

  • All nodes are Outdated. Admin logins to any of the nodes and enter the cluster into the "Upgrade" state (with GUI/COM/REST).

    • EFT admins shall be able to transition from Normal to the Upgrade state via our EFT administrator interface High Availability tab.

    • EFT admins shall be able to transition from Normal to the Upgrade state via COM. Appropriate error message(s) will be provided if enabling Upgrade state on any node when the cluster is already configured for Upgrade.

    • EFT admins shall be able to transition from Normal to the Upgrade state via REST API. Appropriate error message(s) will be provided if enabling Upgrade state on any node when the cluster is already configured for Upgrade.

    • The ability to start the Upgrade state is a global setting; once set at any node, all nodes will report the same state.

    • Upgrade state should be used ONLY when upgrading to a new version of EFT and not when an EFT administrator forgets to upgrade a node in the cluster from a previous upgrade.

  • Once configured for Upgrade, the cluster is in Upgrade state, and you can upgrade each node one by one.

  • Outdated nodes write audit data to local *.sql files only; Each node will have their own local *.sql file.

  • Updated nodes write into ARM database; If no SQL schema changes exist, there is a post process in the Normal state that will migrate the data from the *.sql file into the ARM database.

  • Uses Web folder appropriate to node's version.

    • The nodes' version should match the same version.

    • Shadowfax folder will be copied from \\<HASharedPath>\Web to \\<HASharedPath>\UpgradeFromx.x.xx (where x.x.x.x is the older EFT version); This …\UpgradeFromx.x.x.x will be used for Outdated nodes.

    • Shadowfax folder \\<HASharedPath>\Web will be used for Updated nodes and will be updated with the first upgraded node.

  • Outdated nodes allow admins to login in read-only mode. EFT administrators will not be able to make any configuration changes to EFT.

  • Updated nodes allow admins to login in read-only mode AND move the cluster to Normal state.

  • Configuration changes write in deltas only (there could be only internode mx messages).

    • Changes are not saved into the configuration files, ServerConfig.db and SiteConfigXX.db files. (Part of the advanced property HAFullConfigDumpIntervalMins.)

    • Internode mx messages are communications between EFT nodes (no QA validation).

  • Unknown internode messages are ignored; These are internal nodes for Outdated nodes that may not know about freshly added internode messages in Updated nodes.

  • Cluster Management subsystem doesn't send any messages and ignores all received messages.

  • ERLB subsystem doesn't delegate to run any event rules - all rules are processed with Master node only. Client connections to the server will be processed without any restrictions during the Upgrade state.

Upgrade state to Normal state

  • All nodes are Updated. Admin will need to log into any node and end the upgrade which will switch it back to Normal state (with GUI/COM/REST)

    • EFT admins shall be able to transition from Upgrade to Normal state via our EFT Admin UI

    • EFT admins shall be able to transition from Upgrade to Normal state via COM; Appropriate error message(s) will be provided if enabling Normal state on any node when the cluster is already configured for Normal

    • EFT admins shall be able to transition from Upgrade to Normal state via REST API; Appropriate error message(s) will be provided if enabling Upgrade state on any node when the cluster is already configured for Upgrade

    • The ability to end the upgrade state is a global setting; once set at any node, all nodes will report the same state. Nodes that were not upgraded and the upgrade state has ended, will fail to start.

    • Nodes that were not upgraded after the Upgrade state ends, now Normal, can be upgraded later and will be visible and continue to work in the cluster

  • Normal state limitations: Loading old nodes is prohibited. EFT will not start when the nodes are not upgraded

COM has several methods to manage cluster upgrade state

  • Enum: HAUpgradeNodeState

    • NotInUpgrade = 0,

    • OldNode,

    • NewNode

  • Method: ICIServer::GetHAUpgradeNodeState; Retrieves node upgrade state

  • REST has two endpoints to manage cluster upgrade state:

    • Get HA node upgrade state; GET /server/ha/upgrade

    • Changes entire HA cluster state to the upgrade or normal state: POST /server/ha/upgrade

Related Topics