Setting up Oracle on Linux

The following documents the installation and setup of Oracle at the University of Michigan for use by the ATLAS Muon Calibration and Alignment centers.

Select a machine to host the database

The important aspects are making sure the machine has sufficient memory and disk space to host Oracle. In general more is better. I would also recommend a system with a fast processor (or better: dual or dual-core processors).

At Michigan we used the following machine to install Oracle on: * A 1U Supermicro system with an H8DAR-T motherboard * Dual dual-core Opteron 280 processors (2.4 GHz, 1MB cache per core) * 8 GB of DDR 400 RAM * Four SATA-II 250GB hard-disks * Dual gigabit network interfaces

Make sure your processor supports 64 bit installs: either an Opteron or Intel EM64T processor will work. The details from Oracle are here:

Certify - Product Version and Other Selections: Oracle Database - Enterprise Edition On Linux x86-64 (AMD64/EMT64)

Critical Notes For Oracle Database - Enterprise Edition

General Notes For Oracle Database - Enterprise Edition On Linux x86-64 (AMD64/EM64T):

Processor Chip Information:

• Linux x86-64, support on AMD64/EM64T and Intel Processor Chips that adhere to the x86-64 architecture with supported Linux releases. Running Oracle 32-bit on AMD64/EM64T (64-bit OS) is not supported .
• Oracle 32-bit running on AMD64/EM64T with a 32-bit OS is supported, refer to platform Linux x86 for more information on product availability.

O/S Information:

• Unbreakable Linux: Support for Red Hat Enterprise Linux AS, Red Hat Enterprise Linux ES and SuSE.
• Oracle will support Red Hat Enterprise Linux AS/ES on any platform or driver(s) that Red Hat supports, for customers with, Standard or Premium Red Hat subscription agreement contracts. It is a requirement that the OS (binary) has not been modified. A script has been provided to Oracle Support Services to run against the customers environment in order to assure the binary has not been modified. If the binary proves to have been modified, the customer will need to support it themselves or call Red Hat for technical support. Again, OSS is supporting all versions of Red Hat Enterprise Linux AS/ES that Red Hat supports.
• Oracle does not provide any support if products are installed on Developer's Release of an Operating System.
• Red Hat Enterprise Linux : Red Hat Enterprise Linux family includes Red Hat Enterprise Linux AS, Red Hat Enterprise Linux ES and Red Hat Enterprise Linux WS. Oracle products are supported on Red Hat Enterprise Linux AS, Red Hat Enterprise Linux ES ONLY.

Patches and Linux Kernel Information:

• Customers can take the latest patches/kernels from Red Hat for Red Hat Enterprise Linux AS/ES, from Novell for SLES or from Asianux, and they will be supported by Oracle for Oracle products running with these operating systems.
• It is important for customers and 3rd party vendors to NOT Modify or add anything with the latest kernel/glibc version, in order to receive Unbreakable Support.
• If the customer decides to deploy a 3rd party kernel module, which is not shipped with Oracle supported distribution, Oracle will provide full OS support with the exception that if a problem is determined to be caused by this module, Oracle will defer the customer to the system vendor.
• If a generic kernel bug fix can cause the 3rd party no longer to load after applying the fix, a bug number made available by Oracle, will allow the customer to follow up with the 3rd party vendor to fix the module.

Install RedHat AS V4 Update 3 (64 bit, x86_64)

We had access via our University to RHEL AS 4 U3 for $50/year. We burned a CDROM containing a boot image so we could install via the network (a service provided by our University). Others should determine the best way to acquire either RHEL AS 4 U3 or RHEL ES 4 U3 (AS is more expensive and Oracle wants at least ES). Make sure your hardware supports 64 bits (either an Intel EM64T or Opteron system will work).

Since we could install via a single boot CD it was rather easy. The anaconda_ks.cfg file is here. I am also including the install log.

For our system we also had to create a driver disk since the Marvell SATA controller on the H8DAR-T is NOT supported by RHEL4 (it is supported by 2.6.17+ kernels).

Total Operating System install time was about 2 hours

Note that we chose to use only two of the four disks during the install. There may be optimizations that would help the Oracle setup but I haven't explored this very deeply. I choose to create a RAID-1 (mirror) using the two disks. I made an 18 GB /boot area and a 2 GB swap area on the first disk (since /boot cannot be mirrored). I then added a 20GB /tmp area on the second disk. The rest of both disks were mirrored and mounted as '/'. This was done via software (Linux) RAID rather than trying to use hardware RAID.

New note: You must follow the Oracle install instructions below (from the Oracle web site). For example, the swap space needs to be the size of the physical memory (approximately) according to the table in section 3 of the Oracle Quick installation. I am modifying my partitioning above to follow this guideline (/tmp-> swap (8GB) and /tmp (12 GB)).

Two prepare for using ASM I created "whole" disk partitions on the third and fourth disks: /dev/sdc1 and /dev/sdd1. Later I can use these to hold the databases and flash recovery under ASM.

Prepare for Oracle installation

• Download Oracle software from Oracle on Linux
• Read the appropriate documentation at Oracle documentation
• Follow the install guide

I have a shell script to check some of the RPM pre-requisites here.

Make sure the correct accounts and groups are setup:

The following local operating system groups and users are required if you are installing Oracle Database:

    *      The Oracle Inventory group (oinstall)
    *      The OSDBA group (dba)
    *      The Oracle software owner (oracle)
    *      An unprivileged user (nobody)

To determine whether these groups and users already exist, and if necessary, to create them, follow these steps:

   1. To determine whether the oinstall group exists, enter the following command:

      # more /etc/oraInst.loc

      If the output of this command shows the oinstall group name, then the group already exists.
      If the oraInst.loc file exists, then the output from this command is similar to the following:

      inventory_loc=/u01/app/oracle/oraInventory
      inst_group=oinstall

      The inst_group parameter shows the name of the Oracle Inventory group, oinstall.

   2. To determine whether the dba group exists, enter the following command:

      # grep dba /etc/group

      If the output from this commands shows the dba group name, then the group already exists.

   3. If necessary, enter the following commands to create the oinstall and dba groups:

      # /usr/sbin/groupadd oinstall
      # /usr/sbin/groupadd dba

   4. To determine whether the oracle user exists and belongs to the correct groups, enter the following command:

      # id oracle

      If the oracle user exists, this command displays information about the groups to which the user belongs. The output should be similar to the following, indicating that oinstall is the primary group and dba is a secondary group:

      uid=440(oracle) gid=200(oinstall) groups=201(dba),202(oper)

   5. If necessary, complete one of the following actions:
          * If the oracle user exists, but its primary group is not oinstall or it is not a member of the dba group, then enter the following command:

            # /usr/sbin/usermod -g oinstall -G dba oracle

          * If the oracle user does not exist, enter the following command to create it:

            # /usr/sbin/useradd -g oinstall -G dba oracle

            

*NOTE* _I am not sure about it but on my system this creates a user oracle with home /home/oracle. I don't think this is what you want. See below for the /app/oracle directory_

            This command creates the oracle user and specifies oinstall as the primary group and dba as the secondary group.
   6. Enter the following command to set the password of the oracle user:

      # passwd oracle

   7. To determine whether the nobody user exists, enter the following command:

      # id nobody

      If this command displays information about the nobody user, then you do not have to create the user.
      If the nobody user does not exist, then enter the following command to create it:

      # /usr/sbin/useradd nobody

Note you will need some RPMS from Oracle.

You must also have the libaio rpms installed. I thought I had them installed (and I did) but I had the 32 bit version and NOT the x86_64 bit one. You should have both. Because I didn't I had a problem that took me a day to resolve! To make sure you have the x86_64 bit version installed do:

up2date --arch=x86_64 libaio

The ASM rpms are at ASMLIB. Here is what I installed:

[root@umors Oracle]# rpm -Uvh oracleasm-support-2.0.2-1.x86_64.rpm oracleasmlib-2.0.2-1.x86_64.rpm oracleasm-2.6.9-34.ELsmp-2.0.2-1.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:oracleasm-support      ########################################### [ 33%]
   2:oracleasm-2.6.9-34.ELsm########################################### [ 67%]
   3:oracleasmlib           ########################################### [100%]

Make sure you "configure" the oracleasm via service oracleasm configure:

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [oracle]: 
Default group to own the driver interface [dba]: 
Start Oracle ASM library driver on boot (y/n) [y]: 
Fix permissions of Oracle ASM disks on boot (y/n) [y]: 
Writing Oracle ASM library driver configuration:           [  OK  ]
Scanning system for ASM disks:                             [  OK  ]

You must also insure the system settings are proper. According to the install guide I modifed /etc/sysctl.conf, /etc/security/limits.conf, /etc/pam.d/login and /etc/profile:

sysctl.conf:

# Modifications for Oracle SPM July 12, 2006
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
# fs.file-max = 65536  ...this is already larger...
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 262144

limits.conf:

# Changes for Oracle SPM July 12, 2006
oracle           soft    nproc   2047
oracle           hard    nproc   16384
oracle           soft    nofile  1024
oracle           hard    nofile  65536

login:

# Added for Oracle SPM July 12, 2006
session    required     pam_limits.so

profile:

# Added for Oracle SPM  July 12, 2006
if [ $USER = "oracle" ]; then
        if [ $SHELL = "/bin/ksh" ]; then
              ulimit -p 16384
              ulimit -n 65536
        else
              ulimit -u 16384 -n 65536
        fi
fi

Last step before installing is to ready the disk space. On our system we are using the '/' mount point for the Oracle install so I did:

[umors:~]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md0             220202596   4530696 204486248   3% /
/dev/sda1             18145092     85268  17138100   1% /boot
none                   3989976         0   3989976   0% /dev/shm
/dev/sdb3             12278552     63272  11591560   1% /tmp
[umors:~]# mkdir -p /app/oracle
[umors:~]# chown -R oracle:oinstall /app/oracle
[umors:~]# chmod -R 775 /app/oracle

I think this directory should(?) be the oracle users home directory so I moved it there by editing the /etc/passwd file and by moving all the contents of /home/oracle to /app/oracle.

I then used the remaining two disks in our system for the Oracle Automated Storage Management (ASM). First I created whole disk partitions on /dev/sdc1 and /dev/sdd1 using fdisk. Then I formatted them with an EXT3 filesystem:

mke2fs -j /dev/sdc1
mke2fs -j /dev/sdd1

I needed to reboot to make sure the oracleasm service was running. Then I added them under Oracle's control:

[umors:Oracle]# service oracleasm createdisk DISK1 /dev/sdc1
Marking disk "/dev/sdc1" as an ASM disk:                   [  OK  ]
[umors:Oracle]# service oracleasm createdisk DISK2 /dev/sdd1
Marking disk "/dev/sdd1" as an ASM disk:                   [  OK  ]

For the install you need to unpack the software:

• gunzip 10201_database_linux_x86_64.cpio.gz
• cpio -idmv < 10201_database_linux_x86_64.cpio

This will create a database directory whevever you unpack it. I did the following:

=cd /app/oracle/=
=mkdir OraDb10g=
=cd /app/oracle/OraDb10g=
=cpio -idmv < /root/10201_database_linux_x86_64.cpio=

I made sure the oracle:oinstall owned all files and that I ran chmod -R 775 /app/oracle/OraDb10g/database on the results.

The time taken to get to this point, including reading from the install instructions and creating this document was about 4 hours.

Do the installation

Login as the oracle user. Make sure you have X-windows running or are on the X-windows console at the host. 'cd' to the database directory where you unpacked the downloaded software.

You can then run the installer: ./runInstaller

To install the ASM software you need to select the Advanced mode (which will restart the installer in Advanced mode).

Follow the instructions in the Oracle Install guide in section 3.5.2 "Creating the Automatic Storage Management and Disk Groups".

I changed the default name to OraDb10g_ASM and the ASM name from DATA to DATAFLASH. I accepted the other default values. I picked the two ASM disks I setup in the previous section above and then clicked "Install".

In the middle of this installation you need to open another window (as 'root') and run two scripts 'orainstRoot.sh' and 'root.sh'.

Total time to do this section was about 20 minutes (plus 1 day for the problem)

You can verify things are working (if you used ASM) by doing the following:

[oracle@umors admin]$ sqlplus "SYS/<Password> as SYSDBA"

Use the real password instead of =<Password>=

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Jul 13 15:23:55 2006

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

SQL> STARTUP
ORA-01081: cannot start already-running ORACLE - shut it down first
SQL> SELECT NAME,TYPE,TOTAL_MB,FREE_MB FROM V$ASM_DISKGROUP;

NAME                           TYPE     TOTAL_MB    FREE_MB
------------------------------ ------ ---------- ----------
DATAFLASH                      NORMAL     476944     476838

Post Install details

After installation you need to take care of a few more details.

First is making sure you are up-to-date on patches:

Downloading and Installing Patches

Check the OracleMetaLink Web site for required patches for the installation.

To download required patches:

1. Use a Web browser to view the OracleMetaLink Web site: http://metalink.oracle.com
2. Log in to OracleMetaLink. Note: If you are not an OracleMetaLink registered user, click Register for MetaLink! and follow the registration instructions.
3. On the main OracleMetaLink page, click Patches.
4. Select Simple Search.
5. Specify the following information, then click Go:
   • In the Search By field, choose Product or Family, then specify RDBMS Server.
   • In the Release field, specify the current release number.
   • In the Patch Type field, specify Patchset/Minipack.
   • In the Platform or Language field, select your platform.

When I looked I found no patches for 10.2.0.1.

I was wrong! There are three patches that need to be applied:

• The first patch is to update Oracle to 10.2.0.2 from 10.2.0.1: patch #4547817
• The second patch fixes a problem that shows up in streams: patch #5089630
• Last patch is critical update: #5225799

All these patches are for Linux AMD 64. Make sure to read the README files and instructions. You must shutdown all Oracle services and DBs before applying the patches. You can you Enterprise Manager (em) to find and download the patches into a local directory. I created /app/oracle/patchsets for this purpose and make sure oracle.oinstall owned the directory. Then I used the em interface at http://:1158/em to find and download the three patches above. Make sure you don't check the box which tries to install the patches. The patches should be installed manually later after you shutdown all Oracle DBs and services. See the section below about Oracle DB Tips and the tar file which includes the 'dbora' service. Note that the 1158/em login should use user 'SYS' and the "special" password.

In working with Gancho at CERN there were a number of additional steps I needed to get our DB ready for streams replication. I will cover them in the "Setup for Michigan" section below.

Configuring Oracle products

You need to take care of some configuration items.

1. Backup the root.sh file (I copied mine to root.sh.10.2.0.1)
2. Run the utlrp.sql script after creating or upgrading a database. (See install guide...this was a no-op for me)
3. Create users (see Managing Users)
4. Set the NLS_LANG (See NLS_LANG in Appendix E)
5. There are more product specific instructions.

Setup at Michigan

For the Michigan setup we also followed some of the instructions at: Oracle DB Tips Corner. A modified version of the tar file is available here (edited for the Michigan setup).

We needed to modify our /etc/inittab as per the DB Tips article above. We also setup a new 'dbora' service to control startup and shutdown of Oracle (DB and services). Last thing was to add oracle.sh to /etc/profile.d (a copy of the oracle_profile.sh file in the tar file above). The oracle.sh file makes sure the correct environment is running on the system (for sh/bash/ksh shells).

To prepare for streams replication I worked with Gancho Dimitrov, Florbela Tique Aires Viegas (both CERN/ATLAS DB?) and Eva Dafonte Perez. In addition to the patches mentioned above there were a few additional requirements to get things working which are listed here.

-- ShawnMcKee - 12 Jul 2006