Runtime Options

Number of Threads

The API allows adjusting the number of worker threads in the code: see CnC::debug::set_num_threads. You can also set the number of threads at runtime by setting the environment variable CNC_NUM_THREADS to the desired number of threads.

Note:

The given number must include the environment thread, e.g. n-1 worker threads will be created if CNC_NUM_THREADS is set to n.

Depending on the scheduling strategy, the runtime might actually spawn one or more extra internal helper threads. These threads will mostly be idle and should not have any significant effect on the other threads.

Scheduler

The CnC runtime comes with a selection of different scheduling strategies from which you can chose at runtime. At start-up time, the runtime selects the scheduler upon the evaluation of the environment variable CNC_SCHEDULER. Allowable values are

• TBB_TASK uses the TBB task-stealing scheduler (default)
• FIFO_SINGLE uses a single FIFO task-queue
• FIFO_STEAL implements task-stealing on thread-local FIFO task-queues
• FIFO_AFFINITY is the same as FIFO_STEAL but additionally evaluates step_er::affinity()

Note:

: tuner::affinity() is only evaluated by FIFO_AFFINITY. If your tuner provides affinity hints, you need to use FIFO_AFFINITY to actually make use of it.

The default scheduler is TBB_TASK. Depending on your application characteristics you might observe significant performance variations with different schedulers. Our experience is that in many cases either TBB_TASK or TBB_FIFO shows best (or best-equivalent) performance.

Scheduler Control

The CnC runtime reacts on a few more flags that can be controlled through environment variables:

• CNC_USE_PRIORITY: by default, the step_tuner::priority() hint is not evaluated. The FIFO schedulers (but not TBB_TASK) will evaluate step_tuner::priority() if CNC_USE_PRIORITY is set to 1
• CNC_PIN_THREADS: The runtime can pin threads to the available cores. The value specified in CNC_PIN_THREADS will be used as the stride to distribute threads in a strided round-robin fashion. This allows adjusting the thread-to-core mapping to the HW hierarchy (sockets, caches etc). Examples for 8 cores (c0-c7) CNC_PIN_THREADS=1: t0->c0, t1->c1, t2->c2, t3->c3, ... CNC_PIN_THREADS=2: t0->c0, t1->c2, t2->c4, t3->c6, t4->c1, t5->c3, t6->c5, t7->c7, t8->c0, ... CNC_PIN_THREADS=4: t0->c0, t1->c4, t2->c1, t3->c5, t4->c2, t5->c6, t6->c3, t7->c7, t8->c0, ...
  By default, pinning is off.

  Note:

  On Intel(R) Xeon Phi(TM) (MIC) pinning is on by default with stride 4.

Bypassing the Scheduler

The CnC runtime can optionally execute the first step-instance that is created by the current step (e.g. by putting a tag) right after the currently executing step. This successor step-instance will not go through the normal scheduling procedure and thus avoids overhead. The trade-off is that it can also limit parallelism. Hence this feature is disabled by default. To enable it set CNC_SCHEDULER_BYPASS to a number which is unequal to 0.

Next: Running CnC applications on distributed memory