5
Real Application Clusters Resource Coordination

This chapter provides an overview of Real Application Clusters resource coordination and resource management. The topics in this chapter include:

Overview of Real Application Clusters Resource Coordination

Cluster operations require synchronization among all instances to control shared access to resources. Real Application Clusters uses the Global Resource Directory to record information about how resources are used within a cluster database. The Global Cache Service (GCS) and Global Enqueue Service (GES) manage the information in this directory. Each instance maintains part of the global resource directory in the System Global Area (SGA).

Resource coordination within Real Application Clusters occurs at both an instance level and at a cluster database level. Instance level resource coordination within Real Application Clusters is referred to as local resource coordination. Cluster level coordination is referred to as global resource coordination.

The processes that manage local resource coordination in a cluster database are identical to the local resource coordination processes in single instance Oracle. This means that row and block level access, space management, system change number (SCN) creation, and data dictionary cache and library cache management are the same in Real Application Clusters as in single instance Oracle. Global resource coordination, however, is somewhat more complex as described later in this chapter.

The Contents of the Global Resource Directory

Each instance's portion of the Global Resource Directory contains information about the current statuses of all shared resources. Oracle also uses information in the Global Resource Directory during instance failure recovery or cluster reconfigurations.

The types of shared resource information in the Global Resource Directory include:

Data block identifiers, such as a data block addresses
Locations of the most current versions of data blocks if they have been read into buffer caches on multiple nodes in the cluster
The modes in which data blocks are being held by instances: (N) Null, (S) Shared, or (X) Exclusive
The roles in which data blocks are being held by each instance: local or global

Real Application Clusters Synchronization Processes

When an instance requests a resource, such as a data block, Real Application Clusters locally manages the instance's acquisition of the block. If this block is later modified by one or more instances, then Oracle performs further synchronization on a global level to permit shared access to this block across the cluster. Synchronization in this case requires intranode messaging as well as the preparation of consistent read versions of the block and the transmission of copies of the block between memory caches within the cluster database.

The Global Cache Service Processes (LMSn) locate, prepare, and transmit the most current block images when one or more instances request a data block that is being updated by another instance. To coordinate GCS processing, the GES (GES) transmits intranode messages over the interconnect.

Note:

Oracle uses shared memory for cluster database processing within a node. Therefore, messaging is not necessary.

Therefore, in Real Application Clusters some elements of local coordination become global when remote instances require locally held resources. This is especially true for enqueues and past images of data blocks which are described in the following sub-sections.

Enqueues

An enqueue is a memory structure that serializes updates to a particular row. Coordination of enqueues is not needed until another instance requires the resource to which an enqueue is assigned.

You do not have to configure global enqueues. Their number is calculated automatically at startup and Oracle records the calculated values in the alert.log file.

Past Images

Past images are copies of dirty blocks that Oracle maintains until writes affecting the versions of information in the dirty blocks are complete. The GCS manages past images and the GCS also uses them in failure recovery.

Resource Modes and Roles

As a result of interinstance block transfers, the same data block can exist in multiple caches. The block can also be held by multiple instances in different modes depending on whether a holding instance modifies the data or merely reads the contents of the block. The modes and roles that the GCS uses for resource coordination within a cluster, such as the sharing of data blocks, are described under the following headings:

Resource Modes

A resource mode determines whether the holder of the resource can modify it. Table 5-1 compares the three modes, null (N) mode, shared (S) mode, and exclusive (X) mode.

Table 5-1 Global Cache Service Resource Modes

Resource Mode	Identifier	Description
Null	N	Holding a resource at this level does not convey access rights.
Shared	S	A protected read. When a resource is held at this level, an instance cannot modify it. Multiple instances can read the resource.
Exclusive	X	When a resource is held at this level, it grants the holding instance exclusive access. Other instances cannot write to the resource. Consistent reads of older resources may still be available.

Resource Roles

Oracle assigns a resource role to the resource on the holding instance. The roles are either local or global and are therefore mutually exclusive. The evolution of local-to-global roles is as follows:

When a block is first read into an instance's cache and other instances have not read the block, the block is locally managed. The GCS therefore assigns a local role to the block.
If the block has been modified by the local instance and transmitted to another instance, then it is considered globally managed and the GCS assigns a global role to the block.

All resources have the local role if they only exist in only one cache. If a data block was changed in one instance and subsequently transferred to another instance, then the buffer containing the data block is considered globally dirty and the resource therefore has a global role.

Global Cache Service Operations

The GCS tracks the locations, modes, and roles of data blocks. The GCS therefore also manages the access privileges of various instances in relation to resources. Oracle uses the GCS for cache coherency when the current version of a data block is in one instance's buffer cache and another instance requests that block for modification.

If an instance reads a block in exclusive mode, then in subsequent operations multiple transactions within the instance can share access to a set of data blocks without using the GCS. This is only true, however, if the block is not transferred out of the local cache. If the block is transferred out of the local cache, then the GCS updates the Global Resource Directory that the resource has a global role; whether the resource's mode converts from exclusive to another mode depends on how other instances use the resource.

Global Cache Service Processing Example

Assume that an instance in a cluster database needs to update a data block. At the same time, another instance needs to update the same block. Without the cache coherency provided by the GCS, both instances could simultaneously update the same block. However, due to the synchronization controls of the GCS, only one instance can update the block; the other instance must wait.

By ensuring that only one instance can update a block at any time, the GCS maintains cache coherency. The blocks do not have to be held by an instance in exclusive mode until the instance's transaction completes. For example, one instance can modify a row in a block while a transaction from another instance modifying a different row in the same block has not committed its changes.

Cache Coherency and the Global Cache Service

The GCS ensures cache coherency by requiring instances to acquire a resource at a cluster-wide level before modifying a database block. Thus, the GCS synchronizes global cache access, allowing only one instance at a time to modify a block.

Oracle's multi-versioning architecture distinguishes between current data blocks and one or more consistent read (CR) versions of a block. The current block contains changes for all committed and yet-to-be-committed transactions.

A consistent read (CR) version of a block represents a consistent snapshot of the data at a point in time. The LMSn processes produce consistent read versions by applying rollback segment information to past images. Both current and consistent read blocks are managed by the GCS.

If one instance holds a block that it has modified and another instance requests it, then the holding instance maintains a past image (PI) of the block. In the event of failure, Oracle can reconstruct current and consistent read versions of the block by reading PIs.

System Change Number Processing

To synchronize resources within a cluster, especially data blocks, Oracle must track successive generations of data block changes. That is, Oracle records each change made to a data block by assigning a numeric identifier to each version of a block. This enables Oracle to generate redo logs in an orderly manner and to accommodate subsequent recovery processing.

Oracle uses a logical timestamp known as a system change number (SCN) to order data block change events within each instance and across all instances. The main purpose of SCNs is to mark redo log entries to synchronize recovery processing.

Oracle assigns an SCN to each transaction. Conceptually, there is a global serial point that generates SCNs. In practice, however, SCNs can be read and generated in parallel. One of the SCN generation schemes is called Lamport SCN Generation.

In single instance Oracle, the System Global Area (SGA) maintains and increments SCNs from an instance that has mounted the database in exclusive mode. In Real Application Clusters, the SCN must be maintained globally and its implementation can vary by platform. The SCN can be managed by the GCS, by the Lamport SCN generation scheme, or by using a hardware clock or dedicated SCN server.

Lamport SCN Generation

The Lamport SCN generation scheme is efficient and scalable because it generates SCNs in parallel on all instances. In this scheme, all messages between instances carry SCNs. Multiple instances can generate SCNs in parallel without additional communication among these instances.

On most platforms, Oracle uses the Lamport SCN generation scheme when the value for MAX_COMMIT_PROPAGATION_DELAY is larger than a platform-specific threshold. This is generally the default and this value is typically set to 7 seconds. Note that by changing this value you change the SCN generator used by your cluster database.

Examine the alert.log after instance startup to determine whether the Lamport SCN generation scheme is in use. If the value for MAX_COMMIT_PROPAGATION_DELAY is smaller than the threshold value, then Oracle uses the hardware clock SCN generation scheme.

Chapter 6 describes Cache Fusion processing in more detail.

5 Real Application Clusters Resource Coordination