Oracle’s ACFS clustered file system installation


Often in Oracle clustered database environments, a challenge can be finding somewhere to put common files – things like backups, scripts, logs etc and anything you might want to make accessible to all nodes in the cluster.  This can also apply to non-RAC database environments (RAC one node or any single node installations of Grid Infrastructure) for convenience too.

This challenge gets extended when Oracle wants to perform backups and ASM appears to be the only viable (logically) place to dump backups – however of course you may have some utility (Netbackup for example) that doesn’t do well with ASM disk groups. A good solution is needed – one that leverages the existing high availability and clustered facilities of Oracle’s ASM, but can be exposed to the operating system and treated like any other mount point.

Enter Oracle Automatic Storage Management Cluster File System (ACFS for short) designed to fill this gap.  Depending on the version you are using, it can support everything other than database files (11gR2) or even database files (12.1) with the exception of Oracle Restart (single node) datafiles.  The list of things it can and cannot do by version are available here:

The architecture of ACFS looks like this (regardless of version):

Oracle ACFS

Oracle ACFS

So, enough of the marketing – how do we install it?    For my example, I’m using a RedHat Linux 6.5 server with the UEK3 kernel (actually a VirtualBox machine) that is running an existing 11.2.0.4 database on filesystem and for the purpose of this exercise, I have just installed Oracle Grid Infrastructure 11.2.0.4 to go with it.  In order to get this to work with Oracle ASM, I’ve used VBoxManage to add disks and formatted them with parted to become ASM candidates.

 


Prerequisites

First things first.  You can go through this exercise and get to the point when you attempt to initialize ACFS only to find it won’t work because it’s not compatible….  What?  Well, it’s only compatible after you patch Oracle Grid Infrastructure specifically to address this.  For this you will need the following patches:

  • 6880880
  • 16318126

I hope you will recognize the number for that first one.  If so, great.  If not, you might not enjoy this part.  Install the OPatch update as usual.  Generate the response file and use opatch auto as root to install the UEK compatibility patch.

 


Oracle ASM setup

Create an ASM disk group to hold the ACFS volume you wish to create.  Bear in mind you must specify the compatible attributes.

[oracle@ora-node-2 ~]$ sqlplus / as sysasm
 
 SQL*Plus: Release 11.2.0.4.0 Production on Wed Oct 22 13:17:40 2014
 
 Copyright (c) 1982, 2013, Oracle.  All rights reserved.
 
 Connected to:
 Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
 With the Automatic Storage Management option
 
 SQL>  create diskgroup backup
   2  external redundancy
   3  disk '/dev/asm-disk2' name backup
   4  ATTRIBUTE
   5  'compatible.asm' = '11.2',
   6* 'compatible.rdbms' = '11.2'
 SQL> /
 
 Diskgroup created.


ACFS Preparation

As root,  run the ACFS initialization command.  This is the part that will fail without patching.

[oracle@ora-node-2 ~]$ su -
 Password:
 [root@ora-node-2 ~]#
 [root@ora-node-2 ~]# cd /opt/app/oracle/product/11.2.0/grid/bin
 [root@ora-node-2 bin]# ./acfsroot install
 ACFS-9300: ADVM/ACFS distribution files found.
 ACFS-9312: Existing ADVM/ACFS installation detected.
 ACFS-9314: Removing previous ADVM/ACFS installation.
 ACFS-9315: Previous ADVM/ACFS components successfully removed.
 ACFS-9307: Installing requested ADVM/ACFS software.
 ACFS-9308: Loading installed ADVM/ACFS drivers.
 ACFS-9321: Creating udev for ADVM/ACFS.
 ACFS-9323: Creating module dependencies - this may take some time.
 ACFS-9154: Loading 'oracleoks.ko' driver.
 ACFS-9154: Loading 'oracleadvm.ko' driver.
 ACFS-9154: Loading 'oracleacfs.ko' driver.
 ACFS-9327: Verifying ADVM/ACFS devices.
 ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
 ACFS-9156: Detecting control device '/dev/ofsctl'.
 ACFS-9309: ADVM/ACFS installation correctness verified.

For grins, make sure the ACFS processes have started.

[root@ora-node-2 bin]# ps -ef|grep acfs
 root     19410     2  0 18:22 ?        00:00:00 [acfsioerrlog]
 root     19411     2  0 18:22 ?        00:00:00 [acfs_bast0]
 root     19412     2  0 18:22 ?        00:00:00 [acfs_bast1]
 root     19413     2  0 18:22 ?        00:00:00 [acfs_bast2]
 root     19414     2  0 18:22 ?        00:00:00 [acfs_bast3]
 root     19415     2  0 18:22 ?        00:00:00 [acfs_bast4]
 root     19416     2  0 18:22 ?        00:00:00 [acfs_bast5]
 root     19417     2  0 18:22 ?        00:00:00 [acfs_bast6]
 root     19418     2  0 18:22 ?        00:00:00 [acfs_bast7]
 root     19504 19356  1 18:26 pts/2    00:00:00 grep acfs

Next, load the ACFS driver.

[root@ora-node-2 bin]# ./acfsload start
 ACFS-9391: Checking for existing ADVM/ACFS installation.
 ACFS-9392: Validating ADVM/ACFS installation files for operating system.
 ACFS-9393: Verifying ASM Administrator setup.
 ACFS-9308: Loading installed ADVM/ACFS drivers.
 ACFS-9327: Verifying ADVM/ACFS devices.
 ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
 ACFS-9156: Detecting control device '/dev/ofsctl'.
 ACFS-9322: completed

Create your destination mount point directory.

[root@ora-node-2 bin]# mkdir /opt/app/rman_backups
 [root@ora-node-2 bin]# chown oracle:oinstall /opt/app/rman_backups
 [root@ora-node-2 bin]# chmod 775 /opt/app/rman_backups


Oracle ASM-ACFS configuration

Back as Oracle, create the ACFS volume within the disk group.

[root@ora-node-2 bin]# su - oracle
 [oracle@ora-node-2 ~]$ . oraenv
 ORACLE_SID = [SECU] ? +ASM
 The Oracle base remains unchanged with value /orab/app/oracle
 [oracle@ora-node-2 ~]$ sqlplus / as sysasm
 SQL*Plus: Release 11.2.0.4.0 Production on Wed Oct 22 19:37:45 2014
 Copyright (c) 1982, 2013, Oracle.  All rights reserved.
 Connected to:
 Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
 With the Automatic Storage Management option
 SQL> alter diskgroup backup add volume acfsvol1 size 3G;
 Diskgroup altered.
 SQL> exit
 Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
 With the Automatic Storage Management option

So what did that do?  Let’s take a look.

[oracle@ora-node-2 ~]$ ls -ltr /dev/asm/*
 brwxrwx---. 1 root oinstall 251, 40449 Oct 22 19:38 /dev/asm/acfsvol1-79
[oracle@ora-node-2 ~]$ exit
 logout


ACFS Volume Preparation

In order to use this volume we just created, it has to have a filesystem created on it and added to the ACFS cluster registry.  Note that the user is now root.

[root@ora-node-2 bin]# /sbin/mkfs -t acfs -b 4k /dev/asm/acfsvol1-79 -n "acfsvol1"
 mkfs.acfs: version                   = 11.2.0.4.0
 mkfs.acfs: on-disk version           = 39.0
 mkfs.acfs: volume                    = /dev/asm/acfsvol1-79
 mkfs.acfs: volume size               = 3221225472
 mkfs.acfs: Format complete.
 
 [root@ora-node-2 bin]# /sbin/acfsutil registry -f -a \
                        /dev/asm/acfsvol1-79 /opt/app/rman_backups
 acfsutil registry: mount point /opt/app/rman_backups successfully 
          added to Oracle Registry

ACFS Volume Mount

So, let’s mount that shiny new volume.

[root@ora-node-2 bin]# mount -t acfs /dev/asm/acfsvol1-79 /opt/app/rman_backups/
[root@ora-node-2 bin]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_oranode1-lv_root
                        45G   40G  2.9G  94% /
 tmpfs                 498M   91M  407M  19% /dev/shm
 /dev/sda1             477M  108M  341M  24% /boot
 /dev/sdb1             112G   72G   35G  68% /opt/app
 /dev/asm/acfsvol1-79  3.0G   45M  3.0G   2% /opt/app/rman_backups


Startup Preparation

May as well add this to our server startup scripts.

# echo "/opt/app/oracle/product/11.2.0/grid/bin/acfsload start" >> /etc/rc.local
# echo "/sbin/mount.acfs -o all" >> /etc/rc.local
#


Reboot and….

You should now see the ACFS processes are running – and that your filesystem has been mounted.

[root@ora-node-2 ~]# ps -ef|grep acf
 root      2422     2  0 19:49 ?        00:00:00 [acfsioerrlog]
 root      2423     2  0 19:49 ?        00:00:00 [acfs_bast0]
 root      2424     2  0 19:49 ?        00:00:00 [acfs_bast1]
 root      2425     2  0 19:49 ?        00:00:00 [acfs_bast2]
 root      2426     2  0 19:49 ?        00:00:00 [acfs_bast3]
 root      2427     2  0 19:49 ?        00:00:00 [acfs_bast4]
 root      2428     2  0 19:49 ?        00:00:00 [acfs_bast5]
 root      2429     2  0 19:49 ?        00:00:00 [acfs_bast6]
 root      2430     2  0 19:49 ?        00:00:00 [acfs_bast7]
 root      2875  2852  1 19:55 pts/0    00:00:00 grep acf
 
 [root@ora-node-2 ~]# df -h
 Filesystem                      Size  Used Avail Use% Mounted on
 /dev/mapper/vg_oranode1-lv_root  45G   40G  2.9G  94% /
 tmpfs                           498M   90M  409M  18% /dev/shm
 /dev/sda1                       477M  108M  341M  24% /boot
 /dev/sdb1                       112G   72G   35G  68% /opt/app
 /dev/asm/acfsvol1-79            3.0G   45M  3.0G   2% /opt/app/rman_backups


Finally

For a RAC cluster, you have to do some of the following on every other node:

  • Mount the ASM disk group
  • Enable the ACFS Volume
  • Start the ACFS processes
  • Create your destination mount points and change their ownership/permissions
  • Mount the ACFS volume.

 


Trivia Question

Why don’t we just add the ACFS filesystem definition to /etc/fstab so it mounts at boot time automatically?

Answers on a post card.

 

 

Categories: Computing, DatabaseTags: , , , , , , , , , , , , , , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

dbaonTap

My Experiences with Oracle Technology

globetrottingteen

If travel is an addiction, I'm afflicted.

The Black Veil Ones

My Diary Travel, Photography, My Sight About Somethings

Off to Ireland

Trotz Knacks den Schritt ins Ausland wagen

Randy Johnson's Oracle Blog

Oracle DBA Topics Including Unix, Scripting, & Automation

Model For {Change

Creating a positive change.

Globe-T.

Le Bonnet voyageur • The travelling Winter Hat

Another Photography Enthusiast

CLICKS THROUGH MY EYES

corkandspoon

Adventures in food and drink

Break Your Cage

Embracing The Unknown - Travel is not a pause from life, merely an extension of it.

My Girls On Tour

Fashion & Travel Blog by Abigail Royston

NYCQuirk

Two college girls trying to make the most out of living in NYC without breaking the bank!

Don Charisma

because anything is possible with Charisma

Bunny Kitchen

Exploring the possibilities of cruelty free food

Angelart Star

The beautiful picture of angels makes you happy.

%d bloggers like this: