We had a piece of PL/SQL code that copies data from remote view to a local table. This code executes once a day and took about 4 hours. As if this is not bad enough, we received a request to run this piece of code 3-4 times daily which will total up to about 16 hours per day.

The original code to copy data from a remote view consists of 3 steps:

1. Copy over employee numbers (one column only) from remote view to local temp table. This step takes few seconds only.
2. Use employee numbers copied locally to do a lookup serially (lookup row by row) from remote view. This step takes about 4 hours to copy 20,000 rows. The reason it takes that long is one column from the view is an actual function. Most of the time is spent executing this function and for so many reasons (not all are technical) we can’t get to function to tune it.
3. Commit one time at the end.

Here how the original code looks like:

– Step 1:
– truncate local temp table so we can copy employee numbers
– copy over employees numbers to local temp table from the remote view

execute immediate 'truncate table temp_emp_no';
execute immediate 'truncate table employees';

insert into temp_emp_no select distinct EMPLOYEE_NUMBER from remote_employee_view@remote_link;
commit;

– Step 2:
– Loop through copied over employee numbers and do a remote lookup 1 row at a time

cursor c1 is
   select emp_no from temp_emp_no;

for i in c1 loop
   exit when c1%notfound;
   insert into employees
   select EMPLOYEE_NUMBER, first_name, last_name, RATE, PERIOD, total_rate
     from remote_employee_view@remote_link;
   where EMPLOYEE_NUMBER = i.emp_no;
end loop;

commit;

At this point we were left with one option only which is run the PL/SQL code in parallel (multithreading), but as you might now, PL/SQL doesn’t support multithreading natively,

To simulate multithreading we need to accomplish 2 steps:

1. Break the job in multiple pieces (threads).
2. Schedule every thread to run concurrently.

Remember first step which is coping employee numbers only from remote view to local table. We hash-partitioned this local table into 4 partitions. Each of these partitions will translate into a thread as you will see.

CREATE TABLE vb_emp_no
   (emp_no  VARCHAR2(30))
    PARTITION BY HASH (EMP_NO)
     PARTITIONS 4
/

This is the thread code (stored procedure) which takes in a partition name as a parameter and copies employee data from remote view for that partition only.

CREATE procedure refresh_employee_data_part (p_name in varchar2 )
  authid definer
is
  TYPE EmpCurTyp  IS REF CURSOR;
  v_emp_cursor    EmpCurTyp;
  sql_stmt varchar2(2048);
  v_emp_no varchar2(30);

begin
  sql_stmt := 'select emp_no from vb_emp_no partition(' || p_name || ')';
  open v_emp_cursor for sql_stmt ;

loop
  fetch v_emp_cursor into v_emp_no;
  EXIT WHEN v_emp_cursor%NOTFOUND;
  insert into employees
          (EMPLOYEE_NUMBER,
           VACATION_BALANCE,
           RATE, PERIOD, TOTAL_RATE)
  select distinct employee_number, vacation_balance, rate, period, total_rate
    from   remote_employee_view@remote_link;
  where  EMPLOYEE_NUMBER = v_emp_no;
end loop;

commit;
close v_emp_cursor;
end;
/

This procedure glues all parts together. We schedule the previous procedure 4 times and achieve concurrency.

CREATE procedure refresh_employees
  authid definer
is
begin
  execute immediate 'truncate table vb_emp_no';
  insert into vb_emp_no
  select distinct employee_number from remote_view@db_link;
  commit;
  execute immediate 'truncate table employees';

dbms_scheduler.create_job(job_name => dbms_scheduler.generate_job_name('VB_P1_'),
   job_type => 'PLSQL_BLOCK',
   job_action => 'begin refresh_employee_data_part(''VB_EMP_NO_P1''); end;',
   comments => 'Thread 1 to refresh employees',
   enabled => true,
   auto_drop => true);

dbms_scheduler.create_job(job_name => dbms_scheduler.generate_job_name('VB_P2_'),
   job_type => 'PLSQL_BLOCK',
   job_action => 'begin refresh_employee_data_part(''VB_EMP_NO_P2''); end;',
   comments => 'Thread 2 to refresh employees',
   enabled => true,
   auto_drop => true);

dbms_scheduler.create_job(job_name => dbms_scheduler.generate_job_name('VB_P3_'),
   job_type => 'PLSQL_BLOCK',
   job_action => 'begin refresh_employee_data_part(''VB_EMP_NO_P3''); end;',
   comments => 'Thread 3 to refresh employees',
   enabled => true,
   auto_drop = true);

dbms_scheduler.create_job(job_name => dbms_scheduler.generate_job_name('VB_P4_'),
   job_type => 'PLSQL_BLOCK',
   job_action => 'begin refresh_employee_data_part(''VB_EMP_NO_P4''); end;',
   comments => 'Thread 4 to refresh employees,
   enabled => true,
   auto_drop => true);

end;
/

The procedure would finish immediately even if there are errors. To check the execution status look under DBA_SCHEDULER_JOB_RUN_DETAILS to look for errors and running time.

End Result the job finished in 50 minutes more than 4x faster.

Hazem Ameen
Senior Oracle DBA

Column masking is a simple way to hide you valuable data from certain users without having to apply encrypt/decrypt techniques and increase the column width to accommodate the new string like the old times. Through some simple configuration you can create policies to show your important columns as null without rewriting a single line of code on your application side.

There are 3 steps for accomplish column masking:

1. A function to be used by the policy (function policy) created in next step.
2. Use dbms_rls package to create the policy.
3. Assign “exempt access policy” to users to be excluded from the policy. These users can see all data with no masking.

Step 1: Create Function Policy

This function will be called be the policy to create the column masking. Function name can be any name you select. In my case I called vpd_function. If predicate evaluated to true, then data will to show to all users. In my case I made sure that function will always evaluate to false by looking for (rowid = 0) which will never be true. I suggest creating the function under system account.

CREATE OR REPLACE
FUNCTION vpd_function (obj_owner IN VARCHAR2, obj_name IN VARCHAR2)
RETURN VARCHAR2
AS
BEGIN
RETURN 'rowid = ''0''';
END vpd_function;
/

Step 2: Create Policy

BEGIN
DBMS_RLS.ADD_POLICY(object_schema=> 'SCOTT',
object_name=> 'EMP',
policy_name=> 'scott_emp_policy',
function_schema=> 'SYSTEM',
policy_function=> 'vpd_function',
sec_relevant_cols=> 'JOB',
policy_type => DBMS_RLS.SHARED_STATIC,
sec_relevant_cols_opt=> dbms_rls.ALL_ROWS);

END;
/

Step 3: Exclude Some Users from Policy

Users who need to see all the data without any masking need to be granted “exempt access policy”

Notes:

• Dropping a table that has a policy will drop the policy but not the function policy.
• Renaming the table will NOT drop the policy or disable it. The policy will remain active.
• Export/Import will also export/import the policy along with the table, but will not export/import the function policy.
• When exporting a table under a policy, make sure the user exporting the table has “exempt access policy” grant, otherwise, the column under the policy will always be null and column data will be lost.

Some Important Views

dba_policies
v$vpd_policy

Hazem Ameen
Senior Oracle DBA

I’m always amazed as how a little file as the SPFILE can complicate situations “when you can really use a break”. For example, you shutdown your database gracefully just to find out that you can’t bring it up again due to the fact that you have bad parameters in your SPFILE or the file somehow got deleted and you just came to find out. So you want to recover a backup copy of SPFILE, that’s when you discover that you don’t really know how Oracle is backing up your SPFILE, where the file is (disk, tape or both) and how to recover it.

I hope this post will answer all these questions.

SPFILE Backup

Our design goal is to have 2 backups of SPFILE: 1 on disk and the other backup on tape. This way you can first attempt to recover from disk which is faster, if you can’t then try the tape.

Here are the steps to accomplish our design (Tested in 10.2.0.4 Linux 64-bit):

1) Enable control file auto backup to disk.

Every time a change is made to control file, an auto backup will take place of both control file and SPFILE. Unfortunately SPFILE changes by itself don’t constitute an auto backup. In addition to auto backups when control file changes, every time you have a backup statement in your RMAN script (ex. backup database plus archive), an auto backup will be generated.

To enable control file auto backup:

rman> CONFIGURE CONTROLFILE AUTOBACKUP ON;

You can send your auto backups to anywhere on disk. The default is $ORACLE_HOME/dbs. I prefer sending auto backups to FRA (especially in a RAC environment). You can read about how to set up FRA and send auto backups to it here.

Here is an example of sending auto backups to a specific location on disk:

CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO ‘/u04/backups/auto/%F’;

Now you accomplished first design goal which is backup SPFILE to disk.

2) Include SPFILE in your RMAN backup sets to tape.

The default behavior when you enable auto backup is NOT to backup SPFILE in RMAN backup pieces. To overwrite this behavior, this RMAN statement will create backup piece for SPFILE and implicitly create an auto backup on disk as configured in previous step.

backup
incremental level 0
spfile format ’spfile_%d_%s_%T.bak’  tag ’spfile backup’
database include current controlfile format ‘data_%d_%s_%T.bak’  tag ‘data backup’
archivelog all format ‘arc_%d_%s_%T.bak’  tag ‘archive backup’ delete input;

Now we accomplished second step which is backing up SPFILE to tape with your daily rman backup.

Recovery Scenarios

Here are the most common recovery scenarios I came across, remember Oracle will not recover an SPFILE file to its original location while the database is up even if SPFILE is deleted. You will get “RMAN-06564 must use the TO clause when the instance is started with SPFILE”, error message is a little confusing. So simply recover to a different location and do “cp”.

A) Recover latest SPFILE while database down

set dbid <your DBID>

startup nomount;

restore spfile from autobackup;

shutdown immediate

startup

B) Recover from auto backup in Flashback Recovery Area, database up

This is RAC/ASM with FRA placed in ASM

RMAN> restore spfile to ‘<path with filename>‘ from autobackup;
Starting restore at 11-MAR-09
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=128 instance=raca1 devtype=DISK
recovery area destination: +ARCDG
database name (or database unique name) used for search: RACA
channel ORA_DISK_1: autobackup found in the recovery area
channel ORA_DISK_1: autobackup found: +arcdg/RACA/AUTOBACKUP/2009_03_11/s_681250817.257.681250819

channel ORA_DISK_1: SPFILE restore from autobackup complete
Finished restore at 11-MAR-09

Another syntax is to accomplish the same thing:

restore spfile to pfile ‘<path and filename>’ from autobackup db_recovery_file_dest=’+ARCDG’ db_name=’<your db name>’;

C) Recover a specific backup of SPFILE

This is useful if you have bad parameters in your SPFILE and want to recover an older cleaner copy (SPFILE point in time recovery, now how about that )

list backup of spfile;

# select proper backup piece

restore spfile to ‘<path and filename except original location when DB started> from ‘<backup piece or auto backup>’;

D) Recover an SPFILE to PFILE

Always useful if you want to read content of your parameter file.

restore spfile to pfile ‘<your path and filename>‘  from autobackup;

Hazem Ameen
Senior Oracle DBA

If you ever tried to use dbms_streams.set_tag function to avoid replicating DDL or DML in your Streams environment, we’ll you unpleasantly surprised that it doesn’t work (I hope you didn’t try it on production right way ).

This post deals with steps you have to implement to get it working.

First, every entry in the redo log files has a tag associated with it. The default value for this tag is null. The capture process default behavior is not to check for tags at all, in other words, it will read and process every entry in the redo log regardless of it’s tag value. But what we want is for the capture process to read and process null tags only and ignore non-null tags (these are the DML or DDL that will be skipped).

So the question, how to change the capture process behavior so it will ignore non-null tags?
The answer is by adjusting the capture rules for DML & DDL (depends on whether you need to ignore both operations).

The following scripts assume you have the default Streams configuration without adding any additional custom rules.

Get Capture Rule name and rule condition.
select rule_name, rule_condition
from dba_streams_rules
where rule_set_owner = ‘STRMADMIN’
and streams_type = ‘CAPTURE’;

It should return 2 rows, 1 rule for DML and the other for DDL.

Here how the DDL capture rule looks like before changing:

((((:ddl.get_object_owner() = ’schema name’ or :ddl.get_base_table_owner() = ‘schema name’) and :ddl.get_source_database_name() = ‘database name’ )) and (:ddl.get_compatible() <= dbms_streams.compatible_10_2))

Now add condition to capture null tags only

BEGIN
DBMS_RULE_ADM.ALTER_RULE (
rule_name => ‘CWSAPP66′,
condition => ‘((((:ddl.get_object_owner() = ”CWSAPP” or :ddl.get_base_table_owner() = ”CWSAPP”) and :ddl.get_source_database_name() = ”CWSRD” ))
and (:ddl.get_compatible() <= dbms_streams.compatible_10_2) and (:ddl.is_null_tag() = ”Y”))’,
evaluation_context => NULL);
END;
/

Now from a new session

dbms_streams.set_tag(‘01’);

DDL or DML changes will not get replicated

dbms_streams.set_tag(null); — go back to replicating everything

Remember if using bi-directional replication, you have to adjust capture rules in every site.

Hazem Ameen
Senior Oracle DBA

Not so long ago, I was driving my van and sitting right next to me one of the mildest tempered people I knew in my life, my wife. However, this time, she was yelling at me saying, FOLLOW HIM, GET HIM.

The person that made my wife yell was not a thief or an attacker, but your common average driver in KSA. The driving in Saudi is a combination of rude, aggressive, and in some cases flat out suicidal. It definitely gets under your skin and you respond rather erratically like.

Back in USA people are scared of drunk drivers, but in Saudi, we are fearsome of most drivers and especially of teen drivers. There are young teens that barely can see over the wheel and driving an 8-15 passenger 4×4 SUV. It came no surprise that KSA has one of the highest car accident death rates in the world.

Knowing all risks involved, would you let your wife or daughter drive (if it is ever allowed)? Even though is none of my business, but I don’t think so, not unless KSA fixes this domestic problem. Why would anybody want to take a risk on his wife’s or daughter’s life!

If you ever in KSA, here are some tips to help you cope with this issue:

Few expatriates elect not to own a car but rather relying on company’s transportation and taxis.

Some co-workers take longer and easier routes just to elude traffic and come to the job calmer.

Know the problematic routes and times of traffic congestion to avoid them.

Remind yourself to always remain calm. This is one fight you can’t really win.

Most expatriates buy 4×4 SUVs; they are safer in case car accidents (God forbid).

Drive defensively, can’t stress this point enough.

Hazem Ameen

After years of changes made to an in-house system, it became very hard for our developers to identify which tables are no longer used. Going through hundreds of thousands of line of code and cross referencing hundreds of tables manually is not a simple task. So, and like usual, send it over to the DBA team.

I’m must admit it wasn’t a stimulating task in the beginning, but once we figured we can use fine grained auditing (FGA), it became more exciting.

Here is the procedure we used, we have only 1 main schema:

1)      Identify tables with DML. This is easily done by checking user_tab_modifications.

2)      Create a table unused_tables containing all tables not in user_tab_modifications. With this step, you would eliminate some of the tables currently being used (hopefully most of them):

create table unused_tables as
Select table_name from user_tables
Minus
Select table_name from user_tab_modifications

3)      Enable FGA for “selects” only on the tables in previous step. FGA puts overhead on your system so don’t enable it on all your tables at once if you have thousands of tables. Enable few say 50 or use you best judgment at a time.

# generate script to enable auditing

‘exec dbms_fga.add_policy(object_schema => ”your_schema”, object_name => ”’ || table_name || ”’, policy_name => ”’ ||
table_name || ‘_P”);’
from unused_tables;

4)  Look under DBA_FGA_AUDIT_TRAIL and disable policies for tables that have a trail. Here is script we use to disable policies, we run about once a day:

# generate script to disable auditing

select ‘exec dbms_fga.disable_policy(object_schema => ”’ || object_schema || ”’ , object_name => ”’ || object_name ||
”’, policy_name => ”’ || policy_name || ”’);’
from
(
select distinct a.object_schema, a.object_name, a.policy_name
from DBA_FGA_AUDIT_TRAIL a, DBA_AUDIT_POLICIES b
where a.object_schema = b.object_schema
and a.object_name = b.object_name
and a.policy_name = b.policy_name
and b.enabled = ‘YES’
)

5)  Repeat steps 3 & 4 until you enable auditing for all tables

6)  Tables with enabled policies and no audit trail are these tables you can safely drop. Off course you have to wait an X amount of time (weeks most probably) to confirm.

7)  Be aware that if you do a nightly dump (export), it will show in the audit trail. We normally delete these entries from the audit trail. Our export is done with a different user then the schema owner:

delete from sys.fga_log$ where dbuid = ‘export user’;

commit;

Hazem Ameen
Senior Oracle DBA

ORA-15042:  ASM disk “n” is missing
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “diskgroup name”

These 2 errors are related to missing disk(s) from ASM disk group. After you go through the initial inspection which is basically checking if the disk does exist and correct permissions assigned to Oracle user, if you are not using ASM disk mirroring, then what to do?

In my case, the system administrator took 7 ASM disks belonging to 2 different disk groups. One of these groups was the archive destination while the other was data. I also came to find out that this happened 2 other times in other production systems in addition to 1 time in a development system. All happened by different system administrators on HP and Solaris. Goes to show you how difficult to fit new Oracle system tools in big organizations.

If you have a disk group with 100 disks and you lose 1 disk, then the entire disk group will be dismounted.  Attempts to mount the disk group will render the following error:

ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk “n” is missing

What I find out – the hard way unfortunately – taking a disk from ASM is the mother of all mistakes. The only solution at this point is to use dd command to wipe the headers of all of the disks participating in this disk group, recreate the disk group with same name, and then restore from backup. In my case we had to do point-in-time recovery because we also lost the archive files. Data loss was imminent.

In a nutshell, losing 1 disk from a disk group = losing entire disk group, but why is this?

It is because ASM is designed to take a file and spread (stripe) it across multiple disks in your disk group for performance benefits.  You can find out the stripe size or allocation unit (AU) from asmcmd -> lsdg. So if you lose 1 disk, you lost parts or units of every file in this disk group.

This got me thinking about a recommendation I read a while back which said create 2 disk groups only to simply maintenance; 1 for archive destination and the other for data. But if you come across a scenario like this and you have a large database, you would lose all you control files and also you would have a prolonged downtime to recover your entire database.

If you feel you need to design your environment to handle such a scenario, then here are some recommendations:

1. Disk group total size should be recovered within the time limit allowed for recovery. That means you will have more disk groups, not just 1 or 2.
2. Multiplex your control files in different disk group. In my case, I lost all control files.
3. If possible, multiplex your archive destination to a file system (not ASM); 1 destination is ASM while the other is file system outside ASM. This will allow you to recover to point of failure even if you loss your entire ASM disks.

Hazem Ameen
Senior Oracle DBA

When I first started experimenting with Oracle Streams, I found that removing Streams from our environment was challenging. Oracle documentation just says use dbms_streams_adm.remove_streams_configurations, which didn’t clean the Streams completely, and as such, trying to reconfigure Streams again was unsuccessful.

Here are 2 ways I use to remove bi-directional schema replication from our environment. Both have its pros and cons.

These steps have been tested in 10.2.0.4 environment.

1)      Use dbms_streams_adm package
2)      Drop strmadmin user

Using dbms_streams_adm

Here are the steps, execute on every replication site:

A)  Disable propagation schedule

exec dbms_aqadm.disable_propagation_schedule(
queue_name => ‘capture name’,
destination => ‘destination’,
destination_queue => ‘apply name’);

B) Drop propagation

exec dbms_propagation_adm.drop_propagation(
propagation_name => ‘propagation name’,
drop_unused_rule_sets => true);

C) Remove Streams

exec  dbms_streams_adm.remove_streams_configuration;

D) Drop Queues for capture and apply

– drop capture queue
exec  DBMS_STREAMS_ADM.REMOVE_QUEUE(‘<capture queue name>’,TRUE);

– drop apply queue
Exec DBMS_STREAMS_ADM.REMOVE_QUEUE(‘<apply queue name>’,TRUE);

When finished executing these steps, you should see only be left with 1 object only (database link) under strmadmin.

Again remember to execute on every site.

Pros

1)   It drops the replication configuration without dropping strmadmin or database link which makes it simpler to rebuild your Streams environment.

Cons

1)      Too many steps.
2)      It has been my experience if your replication is hung with “Pause for flow control” these steps might not work.

Dropping strmadmin

Just drop Streams admin user  (strmadmin) on every site.

drop user strmadmin cascade;

Pros

1)      Just 1 step.
2)      It has been my experience when replication is hung that is the only way to clean it up.

Cons

1)      User strmadmin must be disconnected. You might have to kill few sessions before dropping the user.
2)      If your intention is to rebuild the Streams environment, then you must recreate the user on every site and grant proper privileges.

Caution

If you are completely removing replication from your environment, then remember to disable supplemental logging. Supplemental log does use some CPU and additional disk space. Both of methods mentioned in this post don’t disable the supplemental logging.

alter table <schema.object_name> drop supplemental log group <group name>;

You can get object name and group name from dba_log_groups.

Hazem Ameen
Senior Oracle DBA

This post is just a delightful recap of “not so bright” actions I’ve seen done unintentionally that wind up causing a reboot of a node or more in CRS. Some were done by DBAs while others were done by system administrators.

Environment HP & Solaris 10gR2

Kill ocssd.bin Process
This instantly crashed the node.

Delete Content under /tmp/.oracle
This hanged CRS. Commands such as crsctl check crs won’t even come back. You have to reboot the node to recreate the socket file under tmp.

Change System Date
This rebooted CRS flaging a “Cluster Integrity” issue. Poor system administrator almost had a heart attack.

Change Physical Hostname
System administrator accidentally changed the physical hostname then changed it back. The host file was changed as well. This hung CRS. A reboot took care of it.

These 2 tips for Enterprise Manager database control have been tested in 10.2.0.4

Adjusting Collection Interval for Alert Log
 
The default collection interval for scanning alert logs is every 15 minutes which might me too long for some sites.

Here are the steps to adjust the time down to 5 minutes:
 

1. cd $ORACLE_HOME/<hostname_instancename>/sysman/emd/collection
2. vi oracle_database_<sid>.xml
3. Look for “alert_log_rollup”
4. Adjust “ScheduleInterval=5″
5. Restart your DB Control.
6. If RAC, then do the same on rest of the nodes.

Adjusting Logout Time  Interval

If your session remains idle for more than 45 minutes, then DB Control will log you out

Here are the steps to increase the logout time:

1. cd $ORACLE_HOME/<hostname_instancename>/sysman/config/emoms.properties
2. vi emoms.properties
3. add this line to bottom: oracle.sysman.eml.maxInactiveTime=<time in minutes> #1440 for 24 hours
4. Restart your DB Control.

Hazem Ameen
Senior Oracle DBA