Configure Oracle Audit Vault Server in High Availability

Version of Oracle Audit Vault Server to use in this article: 12.2.0.10.0

IP of the Primary Audit Vault Server: 192.168.56.20
IP of the Secondary Audit Vault Server: 192.168.56.40


Configuring the Secondary Oracle Audit Vault Server

Login in to Oracle Database Firewall web console of the Primary Oracle Audit Vault Server:

In the "Settings" tab of Primary Audit Vault Server, from the "Security" menu, click "Certificate".
Copy the certificate.

In another browser window, log in to Secondary Audit Vault Server as a super administrator.
In the Secondary Audit Vault Server web console, click the "Settings" tab.
From the "System" menu, select "High Availability".
Click in “Configure this server as: Secondary Server”
In the Peer System IP Address field, enter the IP address of Server1.
In the Peer System Certificate field, paste the certificate of Server1.
Click Save.

Configuring the Primary Oracle Audit Vault Server

Log in to Secondary Audit Vault Server as an administrator.

In the Settings tab of Server1, from the Security menu, click Certificate.
Copy the certificate.

In another browser window, log in to Primary Audit Vault Server web console as a super administrator.
In the Primary Audit Vault web console, click the "Settings" tab.
From the "System" menu, select "High Availability".
Select the checkbox "Configure this system as:  Primary server".
In the Peer System IP Address field, enter the IP address of Secondary Audit Vault Server.
In the Peer System Certificate field, paste the certificate of Secondary Audit Vault Server.

Click in "Initiate Pairing"
A message will appear asking if you are sure. Click in "OK".

A message will appear saying the configuration will be completed in around 10 minutes:

During the Pairing Process I took a look into the Primary Database to understand how the High Availability configuration is done. First I saw that the Primary database was in “No Archive log” mode.

SQL> archive log list

Database log mode              No Archive Mode

Automatic archival             Disabled

Archive destination            USE_DB_RECOVERY_FILE_DEST

Oldest online log sequence     6

Current log sequence           8

SQL>

SQL>

 There was not Data guard Configuration:

[oracle@avs08002778ad2b ~]$ dgmgrl /

DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.

Connected as SYSDG.

DGMGRL> show configuration;

ORA-16525: The Oracle Data Guard broker is not yet available.

Configuration details cannot be determined by DGMGRL

DGMGRL> exit

But after some minutes I saw the Primary Database was working in Archive Log Mode.

SQL> archive log list

Database log mode              Archive Mode

Automatic archival             Enabled

Archive destination            USE_DB_RECOVERY_FILE_DEST

Oldest online log sequence     7

Next log sequence to archive   9

Current log sequence           9

SQL>

That made me think that the High Availability configuration for Oracle Audit Vault servers is done with Oracle Data Guard, then I decided to check:

DGMGRL> show configuration;

Configuration - DBFWDB

  Protection Mode: MaxPerformance

  Members:

  DBFWDB_HA1 - Primary database

    DBFWDB_HA2 - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:

DISABLED

DGMGRL>

And after some minutes the Data Guard Configuration was working fine:

DGMGRL> show configuration;

Configuration - DBFWDB

  Protection Mode: MaxPerformance

  Members:

  DBFWDB_HA1 - Primary database

    DBFWDB_HA2 - (*) Physical standby database

Fast-Start Failover: ENABLED

Configuration Status:

SUCCESS   (status updated 13 seconds ago)

DGMGRL>

There is another nice script provided by oracle to check the status of the Audit Vault Server High Availability configuration:

[oracle@avs08002778ad2b ~]$ /usr/local/dbfw/bin/setup_ha.rb --status

HA mode:                   PRIMARY

HA server 1:               192.168.56.20

HA server 2:               192.168.56.40

Unique database name:      DBFWDB_HA1

Current database role:     PRIMARY

Data guard broker:         ENABLED

Data guard observer:       YES

Current log mode:          ARCHIVELOG

Logging forced:            YES

Flashback mode:            YES

Current open mode:         READ WRITE

Switchover status:         TO STANDBY

Automatic failover:        ENABLED

Failover status:           TARGET UNDER LAG LIMIT

Missing listener services: NONE

Archive destination:       ENABLE

Recovery mode:             MANAGED REAL TIME APPLY

FRA size:                  47,244,640,256

FRA used:                  2,189,426,688

FRA reclaimable:           1,113,587,712

FRA available:             46,168,801,280

Gap status:                NO GAP

Archived sequence:         13

Applied sequence:          12

Apply lag:                 0:00:21

[oracle@avs08002778ad2b ~]$

All the options of the script:

[oracle@avs08002778ad2b ~]$ /usr/local/dbfw/bin/setup_ha.rb -h

Usage: /usr/local/dbfw/bin/setup_ha.rb configure [options]

    -v, --verbose                    Verbose mode

    -R, --randompassword             Create random SYS password

        --syslog                     Send the output to the syslog instead of stdout

        --foreground                 Run all operations in foreground

        --standby                    Configure standby database (internal)

        --configure                  Configure HA on primary and standby systems

        --unconfigure                Unconfigure existing HA system

        --synchronize                Synchronize data not stored in the database

        --switchover                 Swap primary and standby roles

        --post_switchover            Execute actions on new primary after the database switchover completes (internal)

        --post_primary_upgrade       operations on standby after primary upgrade(Internal)

        --database_key_info          Collect database key info (internal)

        --system_properties          Collect system properties (internal)

        --storage_configuration      Export storage configuration (internal)

        --failover                   Failover to standby database

        --disable_failover           Disable automatic failover

        --enable_failover            Enable automatic failover if it was previously disabled. The automatic failover is enabled by default.

        --process_status             Return the status of setup_ha process as exit code

        --dg_status                  Print Data Guard Broker status

        --handle_role_change         Handle role change

        --status                     Print HA configuration status

        --ha_role                    Return HA role as exit code

        --partner_ha_role            Return partner HA role as exit code

        --check_cfg                  Check the AVS HA configuration (internal)

        --correct_ha                 Check the AVS HA settings and correct if necessary (internal)

        --test_settings              Test if the system settings are correctly configured for HA

        --update_partner_ip IP       Update the configuration file with the new IP of the other AVS

    -h, --help                       Show this message

[oracle@avs08002778ad2b ~]$

After to wait some minutes. The High Availability was completed, we can check the current status in the web console of the primary Oracle Audit Vault Server. If you try to access the web console of Secondary Oracle Audit Vault server you will be redirected automatically to the primary one. 

For Failover, take in consideration the following:

When failover is enabled, during normal operation, the system periodically checks the availability of the primary Audit Vault Server in the resilient pair.

Note the following scenarios:

• If the primary Audit Vault Server becomes unavailable, the system automatically fails over to the secondary Audit Vault Server after a 10 minute delay. The delay prevents a failover due to a reboot of the primary server.

I confirmed that after 10 minutes the Failover was performed:

DGMGRL> show configuration;

Configuration - DBFWDB

  Protection Mode: MaxPerformance
  Members:
  DBFWDB_HA2 - Primary database
    Warning: ORA-16829: fast-start failover configuration is lagging

    DBFWDB_HA1 - (*) Physical standby database (disabled)
      ORA-16661: the standby database needs to be reinstated

Fast-Start Failover: ENABLED

Configuration Status:
WARNING   (status updated 26 seconds ago)

• If the primary Audit Vault Server is manually shut down, the failover process is not triggered. If you bring the primary Audit Vault Server back online, then it continues in high availability mode.
• If the primary Audit Vault Server is manually shut down and reinstalled or replaced with another server, then you must perform the following procedure:

• Manually failover the current standby server by issuing the following command as the oracle user:  /usr/local/dbfw/bin/setup_ha.rb --failover
• Then log in to the Audit Vault console as the super administrative user so that you can unpair the two servers.
• Select Settings, and then select High Availability.
• In the High Availability status page, click the Unpair button.
• Copy the new certificates between the two Audit Vault servers.
• Initiate the high availability setup again by clicking the Initiate Pairing button.

• In the event of a failover, the secondary server becomes the new primary Audit Vault Server. You must do the following to configure this primary server, and repeat the high availability pairing:

• Log in to the Audit Vault Server console as a super administrator.
• Click on the Settings tab.
• Select Settings, and then select High Availability.
• In the High Availability Status page, unpair the new primary server to convert it to a standalone server by clicking on the Unpair button.
• On the standalone server, configure the network and services settings (for example DNS settings).
• On the standalone server, manually mount any remote filesystems (NFS shares) defined as archive locations, using this AVCLI command: ALTER REMOTE FILESYSTEM filesystem_name MOUNT
• Disconnect the failed server and replace it. The replacement server can now be configured as the new secondary server.
• Follow the configuration steps again to pair the two Audit Vault Servers.