Message Passing

CDS offers message-passing functions, defined in terms of the previously-defined primitives, and there are cases when these functions will be easier to use than the underlying primitives, especially in cases where the communicated data must either originate or end up in the private process area of one (or both) processes (rather than in the comm heap). In those cases, using the message-passing functions may also yield a peformance increase over the underlying primitives, especially if the processes communicating do not have hardware shared memory between them and hardware DMA/NIC functionality is available for performing packing/unpacking and conversion.

Logical Model

Functionality virtually identical to a traditional message-passing "send" can be modeled in CDS by allocating a region in the Comm Heap, copying the data from the process space into the region, then enq'ing the region into a cell in the receiver and rgfreeing the region. Here, the cell name acts like a tag in message-passing, and the cell context ID acts like a context in MPI. A traditional message-passing "receive" can similarly be modeled by deq'ing a region from the cell, copying the data out, and rgfreeing the region.

These constructions of send and receive functionality are actually more general than their corresponding traditional message passing counterparts, in that they also support a demand-driven style. For example, a process may "send" regions to one of its own cells, leaving many other processes to "receive" regions from that cell as they are needed.

CDS provides composite functions that are defined to have precisely these semantics, though not necessarily these implementations. Specifically

cds_send(buffer,btype,cnvrsn,proc,cntxt,cell,blktime)

is defined to have identical external behavior as

cds_rgalloc(&rgid, cds_typlen(btype,cnvrsn,0));
cds_copyto(rgid, 0, buffer, btype, cnvrsn);
cds_enq(rgid,proc,cntxt,cell,CDS_PNONE,blktime);

with the possible exception that certain intermediate resource shortage conditions (such as lack of available space in a comm heap) may not actually result in erroneous results (in which case they will not be reported as errors), and other standard error conditions will be reported in a logical manner. Likewise,

cds_recv(buffer,btype,cnvrsn,proc,cntxt,cell,timeout)

is defined to have identical external behavior as

cds_deq(&rgid,proc,cntxt,cell,perm,timeout);
cds_copyfm(rgid, 0, buffer, btype, cnvrsn);
cds_rgfree(rgid);

with the same caveats.

Additional operations, called bsend, sendx and recvx, are also provided, which are identical to send and recv, but instead of using enq and deq, respectively, these use benq, write, and read, respectively. In other words, bsend blocks until the cell is empty and there is a recv or deq waiting; sendx is like a send but destroys any other regions in the cell; and recvx is like recv but leaves the region in the cell to be accessed again by a subsequent recv, recvx, deq, or read.

Split transaction modes (namely ibsend, irecv, and irecvx) are available for bsend, recv, and recvx, using the same approach as described for benq, deq, and read.

Physical Model

Significant optimizations, identical to or (in some cases) surpassing those used in other message-passing packages, are possible when internally implementing the send and recv functions (and therefore to users of those functions), due to the fact that the CDS regions used to define the semantics of these operations do not persist long enough for the user to detect their presence, and therefore do not have to exist at all. Specifically, where possible, data can be copied directly from the sender's address space to the receiver's, without going through an intermediate region at all. This is even more likely if the sender uses a bsend or advisory blocking time on a send. The packing and umpacking functionality in the copy operation itself, in some cases, can be performed directly by DMA/NIC hardware.

[Index] On to Shared Memory