Cloud console UI not in sync with actual inventory after patching

The behavior described below occurs when we use dbcli in Oracle Cloud Infrastructure or dbaascli in Oracle Cloud Infrastructure Classic for patching a Database Service. Please refer to Oracle Doc ID 2385424.1 for more information.

Symptom:

After applying the patch, the GUI console is not showing the updated version:

Oracle Cloud Infrastructure Classic case:

After I applied the patch 12.2.0.1.200414 the UI Cloud console is still showing the patch as available:

Oracle Cloud Infrastructure Classic:

After I applied the patch 18.5.0.0.19115 the UI Cloud console is still showing the patch as available:

Solution:

As you can see the DBCS Console UI still showed the patch as NOT APPLIED. This means, that the UI was not in sync with the actual inventory.

What we need to do is simple, click on the action button

 or  

and select the Precheck option.

The database will start checking the patch and when it finishes it will show an error saying that is not possible install the patch. And that is why the patch is already installed. But this simple process will sync the inventory with the UI Console.

The Oracle Note doesn’t mention this as an error or bug, in they words: “As per the details from internal team, the VM state is good. When next PSU is published and applied, the UI and VM state would be synced.” But, If you don’t want to wait, the Precheck action will fix the problem quickly and safely.

Applying Patch in a DB System with dbcli – Oracle Cloud

In the next article, I will show you how to apply the latest patch to an Oracle Cloud Database System using the command line tool dbcli.

We have two ways of patching the database:

1.     Using the Oracle Cloud GUI and a trough an Internet browser.

2.     Using the console tool dbcli

Boot methods are secure and valid, but I always prefer to use commands in the console tool because I have direct access to the status and logs in case any issue during the process.

During this article I will use the Cloud GUI console for some tasks, so we need access to the cloud tenancy and access to the Database resource. So, let’s begin.

1.     First step, not always needed but strongly recommended. Create a backup!

Go to the compartment where the database reside and click the database name to access the administration console:

Inside the admin console, click on the Create Backup button. A confirmation windows will arise, type the backup name, and click on Create Backup again.

Wait until the backup finished. The time needed for the backup will depend on the database size. The database status will change to Updating during the process and will change to Available when the backup finished.

2.     This is the only action executed by the GUI console. Now, we access the server residing the database using the OPC user and the we change to the root user. All the actions performed now must be executed using the root user.

ssh opc@130.XXX.100.20 [opc@prd-repo-db1 ~]$ [opc@prd-repo-db1 ~]$ sudo su - [root@prd-repo-db1 ~]#

3.     The next step is optional, we will check our environment. All the databases in the “new” Oracle Cloud infrastructure use ASM by default so we need to know that for the patching process we are going to patch the Grid Infrastructure first, and the database home or homes. The command to check the environment is dcli list- or dbcli describe- followed by the component we want to list:

[root@prd-repo-db1 ~]# dbcli describe-dbsystem -d DbSystem Information ----------------------------------------------------------------                      ID: aa9f8619-aa0a-4a4b-ad8a-ecc6e27b41bb                Platform: Vmdb         Data Disk Count: 8          CPU Core Count: 2                 Created: April 17, 2019 1:12:46 AM UTC System Information ----------------------------------------------------------------                    Name: erxkyoca             Domain Name: agentspubsub.prdskavcn.oraclevcn.com               Time Zone: UTC              DB Edition: Se             DNS Servers:             NTP Servers: Disk Group Information ---------------------------------------------------------------- DG Name                   Redundancy                Percentage ------------------------- ------------------------- ------------ Data                      External                  100 Reco                      External                  100 DcsCli Details ----------------------------------------------------------------                 Version: 19.4.3.0.0-SNAPSHOT             BuildNumber: jenkins-dcs-cli-19.4.3.0.0-11               GitNumber: 8c1cc781413e47134131fd7c25db3e2b073912b7               BuildTime: 2020-02-05_1836 UTC DcsAgent Details ----------------------------------------------------------------                 Version: 20.1.1.0.0-SNAPSHOT             BuildNumber: jenkins-dcs-agent-20.1.1.0.0-48               GitNumber: f61d8940aff561e666bb96d6084f6bc0aca33104               BuildTime: 2020-03-28_0856 UTC DcsAdmin Details ----------------------------------------------------------------                 Version: 19.3.1.2.0-SNAPSHOT             BuildNumber: jenkins-dcs-admin-19.3.1.2.0-2               GitNumber: f70905ab1f4aef09295e5206cf2408acf18d7927               BuildTime: 2019-10-03_1011 UTC

[root@prd-odi-repo-db1 ~]# dbcli list-dbhomes ID                                       Name                 DB Version                               Home Location                                 Status   ---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ---------- 149ab232-YYYY-40ae-8ce1-fae2d551d961     OraDB18000_home1     18.1.0.0.0                          /u01/app/oracle/product/18.0.0.0/dbhome_1     Configured

We can see, as I mentioned before, we have two components: DB and GI and we also can se that there are patches available for both components.

4.     The fist operative step will be uprade the dbcli tool, this step is like update the Opatch component when we work with On Premise databases.

[root@prd-repo-db1 ~]# cliadm update-dbcli Job details ---------------------------------------------------------------- ID: 0424a409-YYYY-41e1-8bb4-71ed0d5d5df1 Description: DcsCli patching Status: Created Created: Apr 21, 2020 8:28:13 PM UTC Message: Dcs cli will be updated Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- -------------------------------- --- ----------

The upgrade process will take some minutes, to check the status of the job we use the list-jobs command, re run the command and wait until the job finished successfully.

5.     Check for available patches, once you have updated the dcli tool, we can check the list of actual version and available patches in the components.

[root@prd-odi-repo-db1 ~]# dbcli describe-component System Version 18.1.1.4.0 Component                                Installed Version    Available Version ---------------------------------------- -------------------- ----------- GI                                        18.1.0.0.0       18.2.0.0.188417 DB                                        18.1.0.0.0       18.2.0.0.188417

As you can see, the actual version is 18.1.0.0.0 (base) and the available path is the next RU for 18c.

6.     When we are using Oracle Cloud Infrastructure, the first step is patching GI and then patching the Database. We cannot unify the patch as we do usually in On Premise servers. So, lets start with GI. The command is update-server

[root@prd-repo-db1 ~]# dbcli update-server {   “jobId” : “fa0996a6-7f99-48b2-92dc-5c1fb8e20481”,   “status” : “Created”,   “message” : null,   “reports” : [ ],   “createTimestamp” : “ April 21, 2020 8:50:30 PM UTC”,   “resourceList” : [ ],   “description” : “Server Patching”,   “updatedTime” : “April 21, 2020 8:50:30 PM UTC”,   “percentageProgress” : null }

Please pay attention to the jobId returned by the command, we will use that code to check the status of the Job. The jobs runs in background so we don’t have to worry about losing the ssh connection or another network related issues.

7.     After launching the job, we can check the status using the describe-job command and the Job Id we got from the last step.  Repeat the step until the job finished successfully.

[root@prd-repo-db1 ~]# dbcli describe-job -i fa0996a6-7f99-48b2-92dc-5c1fb8e20481 Job details ----------------------------------------------------------------                      ID:  fa0996a6-7f99-48b2-92dc-5c1fb8e20481             Description:  Server Patching                  Status:  Running                 Created:  April 21, 2020 8:50:30 PM UTC                 Message: Task Name                                                                Start Time                          End Time                            Status ------------------------------------------------------------------------ ----------------------------------- ----------------------------------- ---------- Create Patching Repository Directories[skantdr1]                         April 21, 2020 8:50:24 AM UTC    April 21, 2020 8:50:34 AM UTC    Success Create Patching Repository Directories[skantdr2]                         April 21, 2020 8:50:24 AM UTC    April 21, 2020 8:50:34 AM UTC    Success Download latest patch metadata[skantdr1]                                 April 21, 2020 8:50:24 AM UTC    April 21, 2020 8:50:45 AM UTC    Success Download latest patch metadata[skantdr2]                                 April 21, 2020 8:50:45 AM UTC    April 21, 2020 8:51:05 AM UTC    Success Update Patching Repository[skantdr1]                                     April 21, 2020 8:51:06 AM UTC    April 21, 2020 8:56:40 AM UTC    Success Update Patching Repository[skantdr2]                                     April 21, 2020 8:57:40 AM UTC    April 21, 2020 8:58:29 AM UTC    Success Opatch updation[skantdr2]                                                April 21, 2020 8:59:32 AM UTC    April 21, 2020 9:08:36 AM UTC    Success Opatch updation[skantdr1]                                                April 21, 2020 9:08:32 AM UTC    April 21, 2020 9:08:36 AM UTC    Success Patch conflict check[skantdr1]                                           April 21, 2020 9:08:36 AM UTC    April 21, 2020 9:10:30 AM UTC    Success Patch conflict check[skantdr2]                                           April 21, 2020 9:10:30 AM UTC    April 21, 2020 9:12:22 AM UTC    Success apply clusterware patch[skantdr1]                                        April 21, 2020 9:12:22 AM UTC    April 21, 2020 9:21:58 AM UTC    Success apply clusterware patch[skantdr2]                                        April 21, 2020 9:21:58 AM UTC    April 21, 2020 9:22:57 AM UTC    Success Updating GiHome version[skantdr1]                                        April 21, 2020 9:22:58 AM UTC    April 21, 2020 9:23:22 AM UTC    Success Updating GiHome version[skantdr2]                                        April 21, 2020 9:23:27 AM UTC    April 21, 2020 9:23:27 AM UTC    Running

8.     Once the job finished, we can run the describe-component again to check the applied patch.

[root@prd-odi-repo-db1 ~]# dbcli describe-component System Version 18.1.1.4.0 Component                                Installed Version    Available Version ---------------------------------------- -------------------- ----------- GI                                        18.2.0.0.188417       up-to-date DB                                        18.1.0.0.0            18.2.0.0.188417

9.     Next step, patching the database. First, we have to query the db home id and then patch the result id.

[root@prd-odi-repo-db1 ~]# dbcli list-dbhomes ID                                       Name                 DB Version                               Home Location                                 Status   ---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ---------- 149ab232-7ff6-40ae-8ce1-fae2d551d961     OraDB18000_home1     18.1.0.0.0                          /u01/app/oracle/product/18.0.0.0/dbhome_1     Configured

[root@prd-repo-db1 ~]# dbcli update-dbhome -i 149ab232-7ff6-40ae-8ce1-fae2d551d961 {   "jobId" : "53aecb87-33a5-4e0d-b430-db563402a8d8",   "status" : "Created",   "message" : null,   "reports" : [ ],   "createTimestamp" : "April 21, 2020 09:29:36 AM UTC",   "resourceList" : [ ],   "description" : "DB Home Prechecks",   "updatedTime" : "April 21, 2020 09:29:36 AM UTC",   "percentageProgress" : null }

10.  Check the job status until the job finish successfully.

[root@prd-repo-db1 ~]#  dbcli describe-job -i 53aecb87-33a5-4e0d-b430-db563402a8d8 Job details ----------------------------------------------------------------                      ID:  53aecb87-33a5-4e0d-b430-db563402a8d8             Description:  DB Home Patching: Home Id is 149ab232-7ff6-40ae-8ce1-fae2d551d961                  Status:  Running                 Created:  April 21, 2020 09:29:36 AM UTC                 Message: Task Name                                                                Start Time                          End Time                            Status ------------------------------------------------------------------------ ----------------------------------- ----------------------------------- ---------- Create Patching Repository Directories                                          April 21, 2020 9:29:36 AM UTC    April 21, 2020 9:29:36 AM UTC    Success Download latest patch metadata                                                   April 21, 2020 9:29:36 AM UTC    April 21, 2020 9:29:56 AM UTC    Success checking GiHome version                                                           April 21, 2020 9:30:17 AM UTC    April 21, 2020 9:31:21 AM UTC    Success Update System version                                                          April 21, 2020 9:31:24 AM UTC    April 21, 2020 9:31:24 AM UTC    Success Update Patching Repository                                                    April 21, 2020 9:31:24 AM UTC    April 21, 2020 9:32:52 AM UTC    Success Validating dbHome available space                                         April 21, 2020 9:34:01 AM UTC    April 21, 2020 9:34:01 AM UTC    Success Opatch updation                                                                     April 21, 2020 9:34:04 AM UTC    April 21, 2020 9:34:07 AM UTC    Success Patch conflict check                                                           April 21, 2020 9:34:07 AM UTC    April 21, 2020 9:39:55 AM UTC    Success db upgrade  to version                                                            April 21, 2020 9:39:45 AM UTC    April 21, 2020 8:17:08 AM UTC    Success SqlPatch upgrade                                                         April 21, 2020 9:00:19 AM UTC    April 21, 2020 9:04:09 AM UTC    Success updating the Database version                                            April 21, 2020 9:04:41 AM UTC    April 21, 2020 9:05:09 AM UTC    Running

11.  When the process finished, describe the components once again. Both components should be up-to-date

[root@prd-repo-db1 ~]# dbcli describe-component System Version 18.1.1.4.0 Component                                Installed Version    Available Version ---------------------------------------- -------------------- ----------- GI                                        18.2.0.0.188417       up-to-date DB                                        18.2.0.0.188417       up-to-date

12.  As final step, we should guarantee that all our pluggable database are working fine. You can use many queries to verify the actual status, here some examples:

SQL> select con_id, dbid, name, open_mode from v$pdbs; SQL> select owner,object_name,object_type, status, con_id from cdb_objects where status='INVALID' union select owner, index_name,table_name, status, con_id from cdb_indexes where status ='UNUSABLE'; SQL> select * from registry$history; SQL> select * from dba_registry_sqlpatch;

Summary:

You can get the same result using the Oracle Cloud GUI Console, this is just my preferred method. Remember always to keep your databases updated. The Oracle Cloud Database service greatly facilitates this process with minimum downtime.