Expression queue tuning

The expression engine might need tuning based on the server resources, expression count and complexity for each installation. Generally, the process involves tuning and measuring the performance for one parameter at a time repeatedly until the performance cannot be increased further. A track of records must be maintained. A new tuning is required for change of hardware, usage and setup.
As an administrator, the following application must be checked for each installation:
Table 1. Expression queue tuning application setting
Setting Unit Default Runtime* Description Tuning tip
formula.queue.maxage h 24 Yes The maximum number of hours the expression queue history is retained. Only the expressions that are run are pruned. This is done using formula. The purpose of retaining the entries that are run is for expression statistics (Advanced > Expression Queue) and checking for evaluation loops. On evaluating the loops, this number might be lowered in high-throughput setups to keep the table size down.
formula.queue.maxsize entries 500000 Yes The maximum number of entries in the expression queue. The cleanup task is to keep the entry count below this number by pruning history. Only the expressions that are run are pruned. This overrides the maxage setting. For better performance, this number might be tuned to keep the queue history size down during high load periods and to save history otherwise. A good approximation of this number is to keep it a little below the average throughput during the maxage period.
formula.execution.batch.size entries 30 Yes The number of expression queue entries that are allocated for one thread at each allocation run. The entries are marked as "started" and the thread run and completed before allocating a new batch. Expressions are run concurrently on several threads on several nodes simultaneously, but always in a dependency-aware chronological order. If two expressions on the queue are not related, they can be run in another order than they were added. Expression entries might depend on the value of another entry. The expression entry might not be run if the instance of that entry (the value of which the expression entry depends) is not run before the expression. This, and the concurrency, requires allocation to avoid collisions, inconsistent order and redundant work. The allocation is a bit expensive, so several expressions are allocated at once with the above requirement and the condition that they are run in the given order. If a lot of expressions are allocated, the allocation overhead is saved, but that "locks up" concurrency, and vice versa. The general rule is to keep this number down if the expressions form long dependency chains, and increase it if they are short, or dependencies are few. You can change this value at runtime and therefore should tune it until a reasonable optimum is found. Values less than 10 or more than 200 are rarely optimal.
formula.cleanup.interval msecs 1800000 No The frequency of cleaning the expression queue. Here maxage and maxsize settings are enforced. Only the queue pruning is performed. The first cleaning is when the server starts, at which point special maintenance is done. Also, every two weeks a timer expression integrity check is done (at night). Most of the time the default is reasonable. If the throughput is high, cleaning takes longer, but should be performed more often, and vice versa. Therefore beware of increasing this time too much, as this may cause execution outages or bad performance if the load suddenly increases significantly (for example, after an import or added attribute).
formula.background.interval msecs 10000 No The frequency of checking the expression queue for new unallocated expressions to run. When a batch is finished, a new check takes place immediately without waiting for the interval. If the queue is not "done" when the next interval occurs, a new worker thread is spawned to double the effort. In earlier versions of Rational® Focal Point™, the execution was asynchronous and immediate. When expressions moved to the database, this was not possible without the risk of deadlocks. This is the interval at which the expression queue is checked for new expressions to run, that is, the maximum time (plus execution time) that the user must wait for an expression to be run. If the interval is too short, it just consumes resources unnecessarily. This also affects the ramping of concurrency: If the worker thread that was started at the previous interval is not finished within the interval, another thread (till thread.per.nodes) is also started. If ramping is slow, you might lower this number. If expressions are rarely used, you can increase it slightly.
formula.max.background.threads.per.node count 2 No The maximum number of concurrent threads running expressions for each node. The actual maximum number dynamically depends on whether there are free threads in the application thread pool. The concurrency ramping described in background.interval increases the number of expression evaluation threads on the node at each interval if there is more work to be done. Note that this is a maximum. This requires free threads in the application thread pool, which is automatically tuned with regard to the number of cores (virtual or otherwise). It is also capped to not consume the last thread in the pool. You can experiment with this number as it will not break any operation. Better performance is not guraranteed though, as it depends largely on the concurrency ability of the database to keep up. A sign that this is too low is that the queue most often has expressions that are not evaluated.
formula.cleanup.enabled boolean true Yes The expression queue cleanup is enabled by default. This must never be disabled except for debugging purposes or when running performance measurements where cleanup has to be ruled out. There is no tuning to be done with this setting, never change it, keep it at the default "true".

Feedback