High Availability configuration operations
The High Availability menu displays the current role of an Access Gateway node:
- Single: The node hasn't been configured as either a worker or an admin.
- Admin: The node has been configured as an administrator for High Availability.
- Worker: The node has been configured as a worker for High Availability.
- Reset: Reset the node keys.
- Prepare Admin: Prepare an administration node to connect to a new worker node.
- Prepare Worker: Prepare a node to become a worker node.
- List Nodes: List all worker nodes. This option only appears on the admin node.
- Remove Node: Remove a known worker node. This option only appears on the admin node.
- Check Status: Check high-availability status.
- Cluster Manager: Perform a rolling upgrade of a worker node to an admin node using the Cluster Manager.
You can perform the following high availability management operations:
- Reset the key associated with an Access Gateway node
- Add a worker node to an Access Gateway cluster
- List all worker nodes in an Access Gateway cluster
- Remove a worker node from an Access Gateway cluster
- Check the cluster configuration
- Perform a rolling upgrade of a worker node to an admin node
- Sync a worker node with an admin node
When you initially configure a high availability cluster, cluster members communicate using HTTPS over port 443. Use caution when configuring high availability where proxies are in use. See Proxy configuration in the Network section of Command Line Management Console reference for information on configuring proxies and proxy bypass lists.
Reset the key associated with an Access Gateway node
Access Gateway nodes use various keys to intercommunicate. You must regenerate keys if you want to use an instance as a part of an Access Gateway high availability cluster. You only need to regenerate keys once per instance.
- Connect to the Access Gateway Management console. ssh oag-mgmt@[admin or worker]
- Select 5 - System.
- Select 8 - High Availability.
- Select 1 - Reset Key.
- Enter y to reset the keys being used by the high availability sync process or N to end the reset process.
- Enter x to exit or any other menu item to continue.
Add a worker node to an Access Gateway cluster
When you add a worker node, both the administration and the worker node must meet the following conditions:
- The nodes have already been provisioned.
- The nodes are reachable using Secure Socket Shell (SSH).
- The nodes have had their keys reset. See Reset the key associated with an Access Gateway node
When you prepare workers, ensure that you're connected to a worker node and not an admin. Running the prepare worker operation on the cluster admin renders the Access Gateway Admin UI console inoperable. Access Gateway version 2021.11.2 and later prohibit this operation. Reset nodes previously used as admins before you reuse them as workers. See Reset Access Gateway: command line
-
Perform these tasks on the admin node:
- Connect to the Access Gateway Management console.ssh oag-mgmt@[admin.tld]
- Select 5 - System.
- Select 8 - High Availability.
- Select 2 - Prepare Admin.
When you configure an admin node for high availability for the first time, select 1 - Reset Keys to reset the instance's SSH keys. You only need to reset keys once per instance. See Command Line Management Console reference.
Access Gateway Replication uses the hostname setting from the command line console. Ensure that you update the hostname for both the admin and worker nodes using the menu items .
- The admin node generates and displays an authorization token, which you provide to the worker node. Copy the authorization token to a secure location, such as a secure notes app.
- The admin node waits for connections from worker nodes. Leave the window open until all worker nodes have been added. Entering X prematurely cause the admin node to end the process and stop listing worker node additions. Enter X only after all worker nodes have appeared in the window.
- Return to the command prompt on the worker node that you're attaching.
- Perform these tasks on each worker node:
- Connect to the Access Gateway Management console.ssh oag-mgmt@[worker.tld]
- Select 5 - System.
- Select 8 - High Availability.
- Select 3 - Prepare Worker.
When you configure a worker node for high availability for the first time, select 1 - Reset Keys to reset the instance's SSH keys. You only need to reset keys once per instance. See Command Line Management Console reference.
- Paste the token into the Access Gateway Management console window. The worker node connects to the admin node and completes the authorization.
- Press any key to continue. The worker instance is ready for use.
- Enter x to exit or any other menu item to continue.
- Perform these tasks on the admin node:
- Return to the admin instance Access Gateway Management console and view the results of adding the new worker node.
- Enter x to exit or any other menu item to continue.
List all worker nodes in an Access Gateway cluster
- Connect to the Access Gateway Management console.ssh oag-mgmt@[admin.tld]
- Select 8 - High Availability.
- Select 4 - List Nodes. A list of all currently enabled worker nodes appears.
- Enter x to exit or any other menu item to continue.
Remove a worker node from an Access Gateway cluster
When you remove a worker node from an Access Gateway cluster, the node still exists but no longer receives updates from the admin instance. In addition, the Access Gateway UI is disabled. Be sure to remove these nodes from any load balancer as well.
- Connect to the Access Gateway Management console. ssh oag-mgmt@[admin.tld]
- Select 5 - System.
- Select 8 - High Availability.
- Select 5 - Remove.
- Enter the name of the worker node that you want to remove and press Enter.
- Confirm the removal of the node.
- Enter x to exit or any other menu item to continue.
Check the cluster configuration
- Connect to the Access Gateway Management console.ssh oag-mgmt@[admin or worker]
- Select 5 - System.
- Select 8 - High Availability.
- Select 6 - Check Status. This option shows the latest status of the cluster only after there's a configuration change or when the NGINX engine is restarted. If newly added nodes don't appear, perform any Access Gateway Admin UI console function or restart the NGINX engine. See the NGINX sub-menu in the Access Gateway Management consoleServices section. A list of cluster instances appears. Pass indicates that the node is reachable and functioning. Fail indicates that the node is non-functional. See the node log for more information.
- Enter x to exit.
Perform a rolling upgrade of a worker node to an admin node
You can perform a rolling upgrade of a node from worker to admin using the Cluster Manager. See Configure and manage high availability for information on high-availability configurations.
- Connect to the Access Gateway Management console.ssh oag-mgmt@[admin or worker]
- Verify that you're in a worker node.
- Select 5 - System.
- Select 8 - High Availability.
- Press 7 - Cluster Manager.
- Press 2 - Authorize Node as Admin node. You can't select this option from an admin node or when another authorization process is in progress.
- Press Y to continue or N to stop in the confirmation dialog.
- The worker node generates an authorization code. Copy this code to a secure place, such as a secure note app.Copy
The authorization token required to initiate setup for nominated admin node: worker-c:8ba1c123-715d-4b70-ab5d-0e41493bef73
Copy the token and paste it on the admin node when prompted.
Press enter to continue - Press Enter to continue.
- Switch to an admin node.
- Press 1 - Authorize Node Nomination. You can't select this option from a worker node or when another authorization process is in progress.
- Copy and paste the authorization code from your secure note app into the terminal, and then press Enter. The renomination process starts. See Perform admin renomination.
- Delete the authorization code from your note app.
Sync a worker node with an admin node
If a worker node is out of sync because of downtime or network issues, you can synchronize the changes from an admin node to a specific worker node.
This option is only available on worker instances.
- Connect to the Access Gateway Management console on the worker that you want to sync.ssh oag-mgmt@[<worker>]
- Select 5 - System.
- Select 8 - High Availability.
- Select 8 - Synch from admin.
- Press Enter to continue.
- Enter x to exit.