This section describes how data is submitted to a service using direct data transfer. But first, it is helpful to know how data is submitted in Symphony’s default model.
Here is the sequence of events when a client wants to send task data to the service:
Client sends a task input message containing the data to the API layer, which serializes the message and submits it to the underlying communication layer.
The message is transferred by the communication layer to the session manager on the management host. The session manager replies to the client with an acknowledgement upon successful receipt of the message.
The session manager routes the message to the service instance manager and service instance on the compute host.
The service performs calculations on the input data within the message and returns the result to the client via the service instance manager and session manager.
The following diagram shows the data flow between the client and the service instance in Symphony’s default model.
When direct data transfer is enabled for task input messages, the messages are sent to the service in the same manner as in Symphony’s default model. The difference is when direct data transfer is enabled, the application data is not included in the task input message itself. Only metadata is actually sent with the task input message. To better understand the data flow between the client and the service when using direct data transfer for input/output messages, let’s look at the sequence of events.
The client formulates a task input message encoded with the URL of the client (metadata). This is the URL the client will listen on.
The message is propagated to the service in the same manner as in Symphony’s default model.
The service driver extracts the client URL and uses it to retrieve the data from the client.
The service performs calculations on the data and sends the resulting data directly to the client.
The client waits for acknowledgement from the session manager about the success of the task before accessing the output data locally.
The following diagram shows the data flow between the client and the service instance with direct data transfer enabled for input/output messages.
The direct data transfer feature can only be enabled through the client API at the session level.
You can enable direct data transfer for all tasks associated with a session, and optionally, for common data and common data updates. To enable direct data transfer, the client application must do the following:
The following code sample shows how direct data transfer is enabled using the SessionCreationsAttribute class in each supported language. For more information about the SessionCreationAttributes and SessionOpenAttributes classes, refer to the API reference documentation.
The direct data transfer flags allow greater control over Symphony behavior when the direct data transfer feature is enabled. By default, when direct data transfer is enabled, only the task data is sent directly between the client and service. This means that common data and common data updates are still sent to the service via the session manager. To override this behavior, it is necessary to set the appropriate direct data transfer flag.
For example, to set the direct data transfer flag for all tasks including common data and common data updates in a new session, the client application must do the following:
The following code sample shows how a direct data transfer flag is set with a SessionCreationAttributes object in each supported language. For more information about the SessionCreationAttributes class, refer to the API reference documentation.
You can define a port or port range for the client to listen for connections from the service. You may want to do this if your client is running behind a firewall. The SOAM_DIRECT_DATA_PORT environment variable is used to define the port or port range; e.g., SOAM_DIRECT_DATA_PORT="25000" or SOAM_DIRECT_DATA_PORT="25000-25100".
Since direct data transfer will most likely be used in situations where an application needs to transfer large amounts of data, the client’s memory usage can become an issue. To conserve memory, Symphony can write the cached input/output data to disk and restore the data to memory only when it is required. Once the session is completed, Symphony removes all the input/output data from the file system.
For clients that have access to larger address space, for example, 64-bit clients, they have the ability to be optimized by keeping all the data in memory and relying on O/S paging instead of file caching.
The SOAM_DIRECT_DATA_STORAGE environment variable with possible values of StoreInMemory or StoreOnDisk is used to define whether the data resides in client memory or is written to disk. The default behavior when this variable is not defined is StoreOnDisk.