Admin renomination events

Admin renomination events found in the audit log.

Event fields

Access Gateway audit logs include information on the following events:

Field

Description

TIMESTAMP

Current system date and time

HOSTNAME

Hostname of node generating event

APPLICATION

One of:

  • ACCESS_GATEWAY
  • OAG
  • OAG_MONITOR
  • or a specific service such as check connection, log and others

SUB-PROCESS

One of:

  • ApplicationService
  • ACCESS
  • ADMIN_CONSOLE
  • EBS_SSOAGENT
  • HOST_IP_CHECK
  • MONITOR
  • SCRIPT
  • SERVICE
  • TrustedOriginUpdateScheduler
  • WEB_CONSOLE
  • Or a specific service such as check_connection

COMPONENT

Component of the sub-process such as:

  • AUTHN
  • AUTHZ
  • CLUSTER MANAGER ADMIN
  • ERROR
  • IDP
  • INFO
  • KRB5
  • LOG_DOWNLOAD_STATUS
  • LOG_PREPARE_OPERATION
  • LOG_PREPARE_STATUS
  • NGINX
  • SYSTEM
  • TRUSTED_DOMAINS

SUB-COMPONENT

Sub-component of the process such as:

  • ALERT
  • EBS_SSOAGENT
  • HOST
  • INFO
  • LOCAL
  • NETCAT
  • NOMINATION
  • POLICY
  • SESSION
  • STARTUP/SHUTDOWN

LOG_LEVEL

Log level, one of: TRACE, DEBUG, INFO, WARN, ERROR, or FATAL.

EVENT

Event type

STRUCTURED_DATA

Data related to the occurred event.

MESSAGE

Readable message.

Admin renomination events

Events logged when admin renomination takes place.

Nomination - Starting

Description: Events generated when the nomination starts on admin node.

Messages:

  • OAG Version - 2020.8.3, Cluster Manager Version - 2020.1.5.20200803.174755

    Starting authorized nomination process

  • OAG Version - 2020.6.3, Cluster Manager Version - 2020.1.5.20200803.174755
  • Sent nomination.authKey to admin node - existingadmin[.domain.tld]

Examples:

  • 2020-08-05T18:40:23.711-07:00 nodeB OAG ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Mgmt console Event [USER="oag-mgmt"] OAG Version - 2020.6.3, Cluster Manager Version - 2020.1.5.20200803.174755
  • 2020-08-05T18:40:23.711-07:00 nodeB OAG ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Mgmt console Event [USER="oag-mgmt"] Starting authorized nomination process - OAG Version - 2020.6.3, Cluster Manager Version - 2020.1.5.20200803.174755
  • 2020-08-05T18:40:23.905-07:00 nodeB ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Send auth key to admin node [USER="oag-mgmt"] Sent nomination.authKey to admin node - existingadmin[.domain.tld]
  • Structured data:
    • USER - User performing nomination actions, always oag-mgmt.
  • Corrective action :
    • N/A

    • Nomintation - initiated

    Description: Events generated when nomination starts on admin node.

    Messages:

    • Started nomination process with args: adminNode - [DNS name of existing admin ],

      nominatedNode - [DNS name of nominated worker],

      accessGatewayHostname - gw-admin.[domain.tld]

    Examples:

    • 2020-08-04T14:28:48.376-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Arguments [USER="root"] Started nomination process with args: adminNode - oag.nodeA.com,nominatedNode - oag.nodeC.com, accessGatewayHostname - gw-admin.[domain.tld]
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • N/A

    • Nomination - Worker nodes not at correct version

    Description: One or more nodes aren't at the required feature version.

    Messages:

    • Incompatible node list - [worker.[domain.tld]...] Update worker nodes to version - 2020.8.3 or higher.

    Examples:

    • 2020-08-04T14:28:48.380-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION ERROR Determine Version Compatibility [USER="root"] Incompatible Node List - oag.nodeB.com,oag.nodeD.com. Update worker nodes to version - 2020.8.3 or higher
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • Update the worker nodes identified to version 2020.8.3 or later, and then re-run the renomination process.

    • Nomination - Nominated worker detached from cluster

    Description: Events generated when the nominated worker has been detached from the cluster before becoming the new admin node.

    Messages:

    • Removed nominated admin node - existingadmin[.domain.tld] from HA configuration files

    • Failed to remove nominated admin node - existingadmin[.domain.tld] from HA configuration files.

    Examples:

    • 2020-08-04T14:28:48.380-05:00 existingadmin.[.domain.tld] ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION ERROR Remove nominated node from HA [USER="root"] Removed nominated admin node - name[.domain.tld] from HA configuration files.
    • 2020-08-04T14:28:48.380-05:00 existingadmin.[domain.tld] ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION ERROR Remove nominated node from HA [USER="root"] Failed to remove nominated admin node - name[.domain.tld] from HA configuration files.
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • No action required: If the operation fails, the admin node reverts to its original state. The nominated node is also reverted to its pre-renomination state.

    • Nomination - nominated node cloned

    Description: Events generated when current configuration has been transferred to nominated worker.

    Messages:

    • Transferred HA configuration files from current admin node - worker[.domain.tld].
    • Failed to transfer HA configuration files from current admin node - worker[.domain.tld].

    Examples:

    • 2020-08-04T14:27:42.345-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Copy HA configs [USER="root"] Transferred HA configuration files from current admin node - worker[.domain.tld].
    • 2020-08-04T14:27:42.345-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Copy HA configs [USER="root"] Failed to transfer HA configuration files from current admin node - worker[.domain.tld].
  • Structured data:
    • USER - User performing nomination actions, always oag-mgmt.
  • Corrective action :
    • No action required: If the process fails, the nominated worker node reverts to its original state.

    • Nomination - Key exchange

    Description: Events generated when the required keys are exchanged between existing admin and nominated admin.

    Messages:

    • Backup of ssh keys completed for nominated node - newadmin.[domain.tld].

    Examples:

    • 2020-08-04T14:27:42.362-05:00 newadmin[.domain.tld] ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Backup SSH Keys [USER="root"] Backup of ssh keys completed for nominated node - worker.[domain.tld]
  • Structured data:
    • USER - User performing nomination actions, always oag-mgmt.

    • Nomination - Key exchange

    Description: Events generated when the required keys are synchronized between admin and workers.

    Messages:

    • Synced known_hosts and authorized_keys of current admin node - worker[.domain.tld] with all worker nodes.

    • Failed to sync known_hosts and authorized_keys of current admin node - oag.nodeA.com with all worker nodes.

    Examples:

    • 2020-08-04T14:27:43.823-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Sync keys [USER="root"] Synced known_hosts and authorized_keys of current admin node - worker[.domain.tld] with all worker nodes
    • 2020-08-04T14:27:43.823-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Sync keys [USER="root"] Failed to sync known_hosts and authorized_keys of current admin node - worker[.domain.tld] with all worker nodes
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • No action required: If the operation fails, the nominated worker node reverts to its original state.

    • Nomination - Update worker high availability configuration

    Description: Events generated during exchange of high availability configuration between nominated admin and workers.

    Messages:

    • Prepared HA config file - /tmp/nolock.update_ha_configs.f76c8aea-073b-4241-8960-536fb26573d5.json for worker node - workerX[.domain.tld].
    • Sent nolock.update_ha_configs.f76c8aea-073b-4241-8960-536fb26573d5.json file to worker node - workerX[.domain.tld].
    • Prepared HA config file - /tmp/nolock.update_ha_configs.01880ce6-6ac3-4dd9-ac14-934a301490f3.json for worker node - workerX.[domain.tld]
    • Sent nolock.update_ha_configs.01880ce6-6ac3-4dd9-ac14-934a301490f3.json file to worker node - workerX.[.domain.tld]

    Examples:

    • 2020-08-04T14:27:43.883-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Prep HA config for worker node [USER="root"] Prepared HA config file - /tmp/nolock.update_ha_configs.f76c8aea-073b-4241-8960-536fb26573d5.json for worker node - workerX[.domain.tld]
    • 2020-08-04T14:27:44.086-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Send HA config to worker node [USER="root"] Sent nolock.update_ha_configs.f76c8aea-073b-4241-8960-536fb26573d5.json file to worker node - workerX[.domain.tld].
    • 2020-08-04T14:27:44.107-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Prep HA config for worker node [USER="root"] Prepared HA config file - /tmp/nolock.update_ha_configs.01880ce6-6ac3-4dd9-ac14-934a301490f3.json for worker node - workerX[.domain.tld]
    • 2020-08-04T14:27:44.307-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Send HA config to worker node [USER="root"] Sent nolock.update_ha_configs.01880ce6-6ac3-4dd9-ac14-934a301490f3.json file to worker node - workerX[.domain.tld].
    • 2020-08-04T14:27:44.307-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Send HA config to worker node [USER="root"] Failed to send nolock.update_ha_configs.01880ce6-6ac3-4dd9-ac14-934a301490f3.json file to worker node - workerX[.domain.tld].
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • No action required: If the operation fails, the nominated worker node and the impacted worker nodes revert to their original states.

    • Nomination - Update worker high availability configuration

    Description: Events generated as workers acknowledge receipt of new high availability configuration.

    Messages:

    • Received - 0/2 acknowledgements so far
    • eceived acknowledgment for Worker node - workerX[.domain.tld] with updated Admin Node - newadmin[.domain.tld]
    • Received - 2/2 acknowledgements so far
    • Newly nominated admin node - newadmin[.domain.tld] has been acknowledged by all worker nodes
    • Received acknowledgment 1/2 acknowledgements so far

    Examples:

    • 2020-08-04T14:27:44.346-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Received acknowledgment [USER="root"] Received - 0/2 acknowledgements so far
    • 8-04T14:27:54.383-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Received acknowledgment [USER="root"] Received acknowledgment for Worker node - workerX.[domain.tld] with updated Admin Node - newadmin[.domain.tld]
    • 2020-08-04T14:27:54.428-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Received acknowledgment [USER="root"] Received - 2/2 acknowledgements so far
    • 2020-08-04T14:28:04.448-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Received acknowledgment [USER="root"] Newly nominated admin node - oag.nodeC.com has been acknowledged by all worker nodes
    • 2020-08-04T14:28:04.448-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Received acknowledgment [USER="root"] Received acknowledgment 1/2 acknowledgements so far
  • Structured data:
    • USER - User performing nomination actions, always oag-mgmt.
  • Corrective action :
    • Contact support to determine why specific worker didn't return acknowledgment.

    • Nomination - Revert existing admin to standalone

    Description: Event generated at completion of renomination when the existing admin becomes standalone node.

    Messages:

    • Reset HA configs of older admin node - oldadmin[.domain.tld]

    Examples:

    • 2020-08-04T14:28:04.806-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Reset admin node HA setup [USER="root"] Reset HA configs of older admin node - oag.nodeA.com
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • N/A

    • Nomination - Confirmation of reset

    Description: Event generated to confirm the reset of an existing admin to standalone.

    Messages:

    • Admin nomination process completed successfully for new admin node - existingadmin[.domain.tld].

    • Reset HA configs for admin node - existingadmin[.domain.tld]

    Examples:

    • 2020-08-04T14:28:04.818-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Nomination completed [USER="root"] Admin nomination process completed successfully for new admin node - oldamin[.domain.tld].
    • 2020-08-04T14:28:04.818-05:00 oag.nodeC.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Nomination completed [USER="root"] Admin nomination process completed successfully for new admin node - oldamin[.domain.tld].
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • N/A

    • Nomination - Incorrect authorization code

    Description: Event generated when the admin user [in]correctly enters the renomination authorization code.

    Messages:

    • Authorization token is correct.
    • Authorization token is incorrect.
    • Initiating nomination process on Nominated node - newadmin[.domain.tld].

    • Sent nolock.start_admin_nomination.json to nominated node - oag.nodeC.com

    Examples:

    • 2020-08-06T12:15:23.136-05:00 oag.nodeA.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Auth Token Verification [USER="oag-mgmt"] Authorization token is correct
    • 2020-08-06T12:13:49.224-05:00 oag.nodeA.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION ERROR Auth Token Verification [USER="oag-mgmt"] Authorization token is incorrect
    • 2020-08-06T12:15:23.148-05:00 oag.nodeA.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Initiate nomination [USER="oag-mgmt"] Initiating nomination process on Nominated node - newadmin[.domain.tld].
    • 2020-08-06T12:15:23.423-05:00 oag.nodeA.com ADMIN_CONSOLE CLUSTER MANAGER ADMIN NOMINATION INFO Initiate nomination [USER="oag-mgmt"] Sent nolock.start_admin_nomination.json to nominated node - oag.nodeC.com
  • Structured data:
    • USER - User performing nomination actions, always oag-mgmt.
  • Corrective action :
    • Re-enter the correct code.

    • Nomination - Existing admin enters maintenance mode

    Description: Event generated when the existing admin enters maintenance mode.

    Messages:

    • Started Admin Node Nomination App on admin node - existingadmin.[domain.tld]

    Examples:

    • 2020-08-06T13:06:41.872-05:00 oag.nodeA.com ADMIN_CONSOLE CLUSTER MANAGER UPDATE ADMIN INFO Started Admin Node Nomination App [USER="oagha"] Started Admin Node Nomination App on admin node - existingadmin[.domain.tld]
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • N/A

    • Nomination - Existing admin exit maintenance mode

    Description: Event generated when admin exits maintenance mode.

    Messages:

    • Activated Admin App for admin node - existingadmin[.domain.tld].
    • Removed Admin Node Nomination Mode App for admin node - existingadmin[.domain.tld].

    • Reset process completed for admin node - existingadmin[.domain.tld].

    Examples:

    • 2020-08-06T13:08:05.086-05:00 oag.nodeA.com ADMIN_CONSOLE CLUSTER MANAGER UPDATE ADMIN INFO Activate Admin App [USER="oagha"] Activated Admin App for admin node - existingadmin[.domain.tld]
    • 2020-08-06T13:08:05.105-05:00 oag.nodeA.com ADMIN_CONSOLE CLUSTER MANAGER UPDATE ADMIN INFO Removed Nomination Mode App [USER="oagha"] Removed Admin Node Nomination Mode App for admin node - existingadmin[.domain.tld]
    • 2020-08-06T13:08:05.118-05:00 oag.nodeA.com ADMIN_CONSOLE CLUSTER MANAGER UPDATE ADMIN INFO Reset process completed [USER="oagha"] Reset process completed for admin node - existingadmin[.domain.tld]
  • Structured data:
    • USER - User performing nomination actions, always oagha.
  • Corrective action :
    • N/A

    • Nomination -Backup triggered on existing admin node

    Description: Event generated when a backup is performed on existing admin.

    Messages:

    • sudo ...

    Examples:

    • 2020-08-06T13:06:42.000-05:00 existingadmin[.domain.tld] sudo oagha : TTY=unknown ; PWD=/opt/oag/configs/ha/configs.install ; USER=root ; COMMAND=/opt/oag/scripts/oag_backup.sh
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • N/A

    • Nomination - Public and private keys transferred to new admin.

    Description: Event generated required public and private keys are transferred to new admin.

    Messages:

    • Copied private key of oagha user from node - existingadmin[.domain.tld to nominated node - newadmin[.domain.tld].
    • Copied public key oagha user from node - existingadmin[.domain.tld] to nominated node - existingadmin[.domain.tld].

    Examples:

    • 2020-08-06T13:07:43.331-05:00 oag.nodeA.com ADMIN_CONSOLE CLUSTER MANAGER UPDATE ADMIN INFO Copy user private key [USER="oagha"] Copied private key of oagha user from node - existingadmin[.domain.tld] to nominated node - newadmin.[domain.tld]
    • 2020-08-06T13:07:43.515-05:00 oag.nodeA.com ADMIN_CONSOLE CLUSTER MANAGER UPDATE ADMIN INFO Copy user public key [USER="oagha"] Copied public key oagha user from node - existing.[domain.tld] to nominated node - newadmin.[domain.tld]
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • N/A

    • Nomination - Worker node key updates

    Description: Event generated when a key is written to a worker node

    Messages:

    • Public host keys of nominated admin node - workerX[.domain.tld] has been added to known_hosts file of worker node - newadmin[.domain.tld]

    Examples:

    • 2020-08-06T13:07:43.228-05:00 oag.nodeB.com ADMIN_CONSOLE CLUSTER MANAGER UPDATE WORKER INFO Update keys [USER="root"] Public host keys of nominated admin node - workerX.[domain.tld] has been added to known_hosts file of worker node - newadmin[.domain.tld]
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • N/A

    • Nomination - Worker configuration updates

    Description: Event generated when a worker nodes keys are updated

    Messages:

    • Public host keys of nominated admin node - workerX[.domain.tld] has been added to known_hosts file of worker node - newadmin[.domain.tld]

    Examples:

    • 2020-08-06T13:07:44.505-05:00 oag.nodeB.com ADMIN_CONSOLE CLUSTER MANAGER UPDATE WORKER INFO Update HA Config [USER="root"] Updated HA config file - /opt/oag/configs/events/config/ha_configuration.config with nominated admin node as - newadmin[.domain.tld] and worker nodes as - workerX[.domain.tld], workerY[.domain.tld]
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • N/A

    • Nomination - Acknowledgements

    Description: Event generated as nominations are acknowledged.

    Messages:

    • Prepared acknowledgment file - /tmp/f76c8aea-073b-4241-8960-536fb26573d5.ack.prep.
    • Sent acknowledgment file - /tmp/f76c8aea-073b-4241-8960-536fb26573d5.ack.prep to nominated admin node - newadmin[.domain.tld]

    Examples:

    • 2020-08-06T13:07:44.520-05:00 oag.nodeB.com ADMIN_CONSOLE CLUSTER MANAGER UPDATE WORKER INFO Prepare Ack file [USER="root"] Prepared acknowledgment file - /tmp/f76c8aea-073b-4241-8960-536fb26573d5.ack.prep
    • 2020-08-06T13:07:44.734-05:00 oag.nodeB.com ADMIN_CONSOLE CLUSTER MANAGER UPDATE WORKER INFO Sent Ack file [USER="root"] Sent acknowledgment file - /tmp/f76c8aea-073b-4241-8960-536fb26573d5.ack.prep to nominated admin node - newadmin[.domain.tld]
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • N/A

    • Nomination - Successful nomination

    Description: Event generated when a nomination completes successfully.

    Messages:

    • Admin nomination process completed successfully on worker node - newadmin[.domain.tld].

    Examples:

    • 2020-08-06T13:07:44.746-05:00 oag.nodeB.com ADMIN_CONSOLE CLUSTER MANAGER UPDATE WORKER INFO Sent Ack file [USER="root"] Admin nomination process completed successfully on worker node - newadmin[.domain.tld]
  • Structured data:
    • USER - User performing nomination actions, always root.
  • Corrective action :
    • N/A

    • Related topics