PWC (put-with-completion) RDMA

Functions that perform put/get with completion (PWC, GWC) and probing for completion events. More...

Files
file	photon_wc.h

Functions
int	photon_put_with_completion (int proc, uint64_t size, photonBuffer lbuf, photonBuffer rbuf, photon_cid local, photon_cid remote, int flags)
int	photon_get_with_completion (int proc, uint64_t size, photonBuffer lbuf, photonBuffer rbuf, photon_cid local, photon_cid remote, int flags)
int	photon_probe_completion (int proc, int *flag, int *remaining, photon_cid *comp, int *src, void(*cb)(photon_cid), int flags)

Detailed Description

Functions that perform put/get with completion (PWC, GWC) and probing for completion events.

The with-completion interface is a one-sided, asynchronous, and nonblocking approach for to exchanging user buffers with RDMA between peers. Buffer metadata exchange must have been completed before using these functions as they all require the specification of remote buffer structs. A single probing call is used to determine if operations have completed, both locally (sending side) and on the target (receiving side).

The distinguishing feature of the with-completion interface is the ability to specify local and remote completion identifiers (CIDs) encoded in photon_cid structs. These CIDs may be arbitrarily sized and the max value is controlled by the photon_config_t::max_cid_size parameter. The encoded values in the CIDs are opaque to the Photon library and are the responsibility of the user to interpret. The local and remote CID values passed in via the photon_put_with_completion() or photon_get_with_completion() calls are returned by subsequent polling calls to photon_probe_completion() on both the sender and receiver sides when their respective operations complete.

The with-completion interface may be used with the rendezvous interface as the internal request tracking is managed independently. Note that polling and probing for both rendezvous and with-completion events may introduce unexpected overheads in an application and it is advisable to pick a single approach in latency-sensitive scenarios.

Photon 2-PUT PWC

Photon Packed PWC

Photon GWC

A number of other configuration thresholds control the backend behavior of this interface. See photon_config_t::pwc_buf_size, photon_config_t::small_pwc_size, and photon_config_t::ledger_entries.

Function Documentation

photon_get_with_completion()

int photon_get_with_completion	(	int	proc,
		uint64_t	size,
		photonBuffer	lbuf,
		photonBuffer	rbuf,
		photon_cid	local,
		photon_cid	remote,
		int	flags
	)

This function performs an RDMA GET from peer proc of size size. The lbuf and rbuf parameters specify the local (initiating) and remote (target) buffers, respectively.

The remote (target) buffer must be specified in rbuf along with the remote key (private buffer metadata). Local AND remote keys can be specified in lbuf and rbuf. As a convenience, if lbuf keys are zeroed, Photon will attempt to find the matching local registered buffer using photon_buffer_t::addr and photon_buffer_t::size.

The flags parameter allows for the control of the CID notification behavior. The default behavior is to enable all CQ events and local and remote CIDs from probe_completion(). This is equivalent to flags set to PHOTON_REQ_NIL. flags may be logically OR'ed with PHOTON_REQ_PWC_NO_LCE and/or PHOTON_REQ_PWC_NO_RCE to disable the return of local and remote CIDs, respectively.

Parameters

proc	The target peer of the GET
size	The size of the GET in bytes
lbuf	A pointer to a photon_buffer_t struct containing the local buffer metadata
rbuf	A pointer to a photon_buffer_t struct containing the remote buffer metadata
local	The local completion ID to return from probing at the initiator
remote	The remote completion ID to return from probing at the target
flags	Flags that control the completion ID return behavior

Returns

PHOTON_OK on success or an error code

Definition at line 526 of file libphoton.c.

photon_probe_completion()

int photon_probe_completion	(	int	proc,
		int *	flag,
		int *	remaining,
		photon_cid *	comp,
		int *	src,
		void(*)(photon_cid)	cb,
		int	flags
	)

This functions probes for completion identifiers (CIDs) from the specified peer proc. PHOTON_ANY_SOURCE can be given to check for completions from any source peer, but given a specific proc, the probe call will only poll the backend completion queue (if available) and completion ledger associated with that rank.

The flag parameter is set to 1 if a completion is found. The comp parameter is filled with the dequeued CID data on success and the remaining paramter returns the number of requests still pending. The src parameter is filled with the source (rank) of the CID probe on success.

A callback may be passed to the probe call using the cb parameter. Note that the callback should not block probing progress.

The parameter controls the probing behavior in terms of which event type to query. PHOTON_PROBE_ANY probes both local event queue as well as the ledger for remote completions. Setting flags to either PHOTON_PROBE_EVQ or PHOTON_PROBE_LEDGER will force probing of only the local event queue or only the ledger, respectively.

Parameters

proc	The peer to probe completions for, can be PHOTON_ANY_SOURCE
flag	A pointer to an integer that will be set to 1 on successful probe of a CID
comp	A pointer to a photon_cid struct that will be filled with the CID on success
src	A pointer to an integer that will be filled with the source peer (rank) on success
cb	A callback function pointer that will be called on success with the CID as argument
flags	Flags that control the completion ID probing behavior

Returns

PHOTON_OK on success or an error code

Definition at line 533 of file libphoton.c.

photon_put_with_completion()

int photon_put_with_completion	(	int	proc,
		uint64_t	size,
		photonBuffer	lbuf,
		photonBuffer	rbuf,
		photon_cid	local,
		photon_cid	remote,
		int	flags
	)

This function performs an RDMA PUT to peer proc of size size. The lbuf and rbuf parameters specify the local (sending) and remote (receiving) buffers, respectively.

If lbuf is NULL and size is 0, then only the completion value in remote is sent.

The remote (target) buffer must be specified in rbuf along with the remote key (private buffer metadata). Local AND remote keys can be specified in lbuf and rbuf. As a convenience, if lbuf keys are zeroed, Photon will attempt to find the matching local registered buffer using photon_buffer_t::addr and photon_buffer_t::size.

The flags parameter allows for the control of the CID notification behavior. The default behavior is to enable all CQ events and local and remote CIDs from probe_completion(). This is equivalent to flags set to PHOTON_REQ_NIL. flags may be logically OR'ed with PHOTON_REQ_PWC_NO_LCE and/or PHOTON_REQ_PWC_NO_RCE to disable the return of local and remote CIDs, respectively.

Parameters

proc	The target peer of the PUT
size	The size of the PUT in bytes
lbuf	A pointer to a photon_buffer_t struct containing the local buffer metadata
rbuf	A pointer to a photon_buffer_t struct containing the remote buffer metadata
local	The local completion ID to return from probing at the sender
remote	The remote completion ID to return from probing at the receiver
flags	Flags that control the completion ID return behavior

Returns

PHOTON_OK on success or an error code

Definition at line 519 of file libphoton.c.