Configure High Availability

Access Gateway high availability consists of:

  • A single administration instance of Access Gateway. The administration instance or admin node is used to maintain and propagate configuration changes to worker nodes. Additionally, you can use the admin node as a normal Access Gateway instance.
  • One or more worker instances bound to the admin node, which service requests.
  • A customer provided load balancer that routes requests to the Access Gateway high availability cluster.

Overview of the Access Gateway high availability instance life-cycle:

  • An instance of Access Gateway is provisioned. This instance is called the Admin node. This node is configured normally, including defining protected applications. You aren't required to configure applications or IDP support before configuring high availability.
  • Second and subsequent instances of Access Gateway are provisioned. These instances are called worker nodes. These nodes are not configured with applications, but obtain all configurations from the admin node.
  • Worker nodes are then specifically configured to use the admin node for all configuration. After you configure them, worker nodes don't expose the Access Gateway Admin UI console. You can only access then using the command line interface.

Access Gateway High Availability architecture

In this diagram the admin node is shown also acting as a worker node. If the admin is NOT intended to also service requests, it can be omitted from the load balancers configuration.

The admin user interface (UI) is only available on admin nodes and nodes that haven't been configured as worker nodes. After an instance is configured as a worker, you can't access it using the admin UI. All administration of an Access Gateway cluster is done through the admin node and its admin UI.

Overview

Configuring high availability includes the following overall process:

  1. Configuring an admin node.
    During this step, the administration node is configured normally.
  2. Configuring a worker node.
    During this step, worker nodes are configured without any applications.
  3. Preparing the admin node.
    During this step, the command line interface is used on the admin node to alert or prepare the admin node for the addition of one or more worker nodes.
  4. Preparing the worker node.
    During this step, the command line interface is used on the worker node to prepare the worker for becoming a part of the Access Gateway cluster.
  5. Worker integration into cluster.
    After the previous steps, the worker is automatically integrated into the cluster. During this phase, the worker admin UI is disabled and worker exchanges keys and the admin node provides the configuration.
Access Gateway High Availability add worker node sequence diagram

Operations

To configure high availability:

  1. Reset the key associated with an Access Gateway node - Reset the keys in both the admin instance and the worker instance.
  2. Add a worker node to an Access Gateway cluster - Add one or more worker nodes to the cluster.
  3. Check the cluster configuration - Review the final configuration.

Reset the key associated with an Access Gateway node

Access Gateway nodes use various keys to intercommunicate. You must regenerate keys if you want to use an instance as a part of an Access Gateway High Availability cluster.
You only need to regenerate keys once per instance.

  1. Connect to the Access Gateway Management console.
    ssh oag-mgmt@[admin or worker]
  2. Select 5 - System.
  3. Select 8 - High Availability.
  4. Select 1 - Reset Key node.
  5. Enter y to reset the keys being used by the high availability sync process or N to abort the reset process.
  6. Enter x to exit or any other menu item to continue.

Add a worker node to an Access Gateway cluster

When you add a worker node, both the administration and the worker node must meet the following conditions:

When you prepare workers, ensure that you're connected to a worker node and not an admin. Running the prepare worker operation on the cluster admin renders the Access Gateway Admin UI console inoperable. Access Gateway version 2021.11.2 and later prohibit this operation. Reset nodes previously used as admins before you reuse them as workers. See Reset Access Gateway command line

  1. Perform these tasks on the admin node:

    1. Connect to the Access Gateway Management console.ssh oag-mgmt@[admin.tld]
    2. Select 5 - System.
    3. Select 8 - High Availability.
    4. Select 2 - Prepare Admin.

      When you configure an admin node for high availability for the first time, select 1 - Reset Keys to reset the instance's SSH keys. You only need to reset keys once per instance. See Command Line Management Console reference.

      Access Gateway Replication uses the hostname setting from the command line console. Ensure that you update the hostname for both the admin and worker nodes using the menu items System (5)Change Hostname (1).

    5. The admin node generates and displays an authorization token, which you provide to the worker node. Copy the authorization token to a secure location, such as a secure notes app.
    6. The admin node waits for connections from worker nodes. Leave the window open until all worker nodes have been added. Entering X prematurely cause the admin node to end the process and stop listing worker node additions. Enter X only after all worker nodes have appeared in the window.
    1. Return to the command prompt on the worker node that you're attaching.
  2. Perform these tasks on each worker node:
    1. Connect to the Access Gateway Management console.ssh oag-mgmt@[worker.tld]
    2. Select 5 - System.
    3. Select 8 - High Availability.
    4. Select 3 - Prepare Worker.

      When you configure a worker node for high availability for the first time, select 1 - Reset Keys to reset the instance's SSH keys. You only need to reset keys once per instance. See Command Line Management Console reference.

    5. Paste the token into the Access Gateway Management console window. The worker node connects to the admin node and completes the authorization.
    6. Press any key to continue. The worker instance is ready for use.
    7. Enter X to exit or any other menu item to continue.
  1. Perform these tasks on the admin node:
    1. Return to the admin instance Access Gateway Management console. View the results of adding the new worker node, similar to: Authorization token required to initiate setup from worker nodes is given below. Copy the text below this line and paste it on worker node when prompted. <admin...com>:927da506-7efb-4520-bd32-dd03b86f2a9b Worker nodes available so far: <worker1...com> <worker2...com> <worker3...com>
    2. Enter X to exit or any other menu item to continue.

Check the cluster configuration

  1. Connect to the Access Gateway Management console.ssh oag-mgmt@[admin or worker]
  2. Select 5 - System.
  3. Select 8 - High Availability.
  4. Select 6 - Check Status. This option shows the latest status of the cluster only after there's a configuration change or when the NGINX engine is restarted. If newly added nodes don't appear, perform any Access Gateway Admin UI console function or restart the NGINX engine. See the NGINX sub-menu in the Access Gateway Management console Services section. A list of cluster instances appears. Pass indicates that the node is reachable and functioning. Fail indicates that the node is non-functional. See the node log for more information.
  5. Enter x to exit.