Threading --------- .. versionchanged:: 1.5 The threading model changed substantially. In 1.4 the processing thread was spawned per operation (``com.thread``). On master there is **one persistent worker thread** for the whole application, fed by a FIFO job queue, with an explicit cancellation signal. This chapter documents that model. The authoritative source is the header comment in :file:`src/core/processing_thread.h` — read it alongside this page. This chapter covers *who runs on which thread* and *how jobs are controlled*. For the synchronisation primitives (which lock guards what, and the deadlock rules) see :ref:`locking:locking`. Thread topology =============== - **GTK main thread** — runs GApplication, widget callbacks, idle functions and the renderer. The **only** thread allowed to touch GTK. - **The persistent processing worker thread** (``worker_thread_main`` in :file:`processing_thread.c`) — one thread for the application lifetime. Jobs are serialised through a FIFO queue; **never more than one job runs at a time**. Every heavy operation runs here. - **OpenMP teams** — spun up *inside* a job for data-parallel loops (splitting an image across cores, or processing several sequence frames at once). - **The script/CLI thread** (``com.script_thread``) — reads a script and issues its commands, which in turn use the worker thread. - **Per-Python-script worker threads** — one C thread per running Python script, servicing that script's IPC connection (see :ref:`pythonintegration:python integration`). Submitting a job ================ The entry point is ``start_in_new_thread`` (and its aliases ``start_in_reserved_thread`` and ``start_and_wait_from_main_thread``). Submission is **context-aware**, decided by ``context_requires_wait()`` which is true when ``com.script`` or ``com.python_command`` is set: - **From a script or Python-command thread** — the call **blocks** until the job completes. This is what makes "one command finishes before the next starts" automatic in scripts, with no explicit join. - **From any other thread (GUI main thread included)** — the job is queued **fire-and-forget**; the caller returns immediately. - **Re-entrant submission** — if called from the worker thread itself (the ``already_in_a_thread`` pattern), the function is invoked **directly and synchronously** rather than re-queued. ``processing_in_worker_thread()`` reports whether you are on the worker thread. ``waiting_for_thread`` retrieves a job's return value (blocking for a background caller, an instant cached fetch in the script/Python context, and an immediate ``0`` on the GTK main thread to avoid deadlocking the idle dispatcher). .. tip:: For a GUI-thread caller that must wait for its result, use ``start_and_wait_from_main_thread``, which pumps the main loop until the job finishes. The job it submits **must not** call ``execute_idle_and_wait_for_it`` internally, or the two mutually wait. Cancellation ============ Cancellation is a **separate signal** from "is a job active?". - ``processing_request_cancel`` sets a sticky ``cancel_flag``. - Worker hooks and long loops must poll ``processing_should_continue`` (declared in :file:`processing.h`, worker-facing only) and bail out when it returns FALSE. The generic sequence worker already polls it each frame. - ``cancel_flag`` is cleared automatically when the next queued job starts, and also by ``reserve_thread``. .. warning:: **Stale cancel flag on synchronous embedding.** A caller that runs ``generic_sequence_worker`` directly with ``already_in_a_thread = TRUE`` (e.g. compositing alignment) polls ``processing_should_continue()`` but does **not** go through the queue that clears the flag. If a previous ``stop_processing_thread`` left ``cancel_flag`` set, the embedded worker aborts on frame 0. The fix — and the contract — is to call ``reserve_thread`` first: it atomically claims the slot **and** clears ``cancel_flag``. This exact bug has shipped. See the comment on ``reserve_thread`` in :file:`processing_thread.c`. Reserving the thread ==================== ``reserve_thread`` / ``unreserve_thread`` claim and release the "active slot" for short synchronous operations that must block concurrent submissions without actually running on the worker thread. ``reserve_thread`` is an atomic test-and-set returning FALSE if the slot is already busy. ``processing_is_job_active`` reports TRUE while a job runs *or* the slot is reserved — use it (not ``processing_should_continue``) for GUI button-sensitivity checks and pre-command guards. The Python thread claim ======================= A running Python script that needs exclusive access claims the worker slot: - ``claim_thread_for_python`` drains the queue, waits for the active job to finish, then gates all further submissions. It returns ``0`` on success, ``1`` if busy/already reserved, and ``2`` if an image-processing dialog is open. - ``python_releases_thread`` releases the claim. - While claimed, GUI-initiated submissions are rejected. .. warning:: ``claim_thread_for_python()`` waits on the queue condition while a job is active. Never call it from code that the active job can itself be waiting on — that is a circular wait. Preview cancellation ==================== ``cancel_and_wait_for_preview`` sets the cancel flag and then **pumps the GTK main loop** until the active job finishes. It is safe **only** for preview jobs that clean up with an asynchronous idle (``siril_add_idle``) and never call ``execute_idle_and_wait_for_it``. A preview job that uses a synchronous idle would deadlock against the pumping loop. .. important:: **Preview-style jobs must use async idles only.** This rule is stated in :file:`processing_thread.h` and echoed in the deadlock catalogue in :ref:`locking:locking`. The thread budget ================= The number of threads a job may use is a budget, not a free-for-all: - ``com.max_thread`` is the total budget; ``com.max_images`` / ``com.fftw_max_thread`` are related caps. - ``threading_type`` values (``MULTI_THREADED`` / ``SINGLE_THREADED`` / an explicit N) express intent. ``check_threading`` resolves ``MULTI_THREADED`` to ``com.max_thread``; ``limit_threading`` additionally caps the count to the number of work chunks (``total_iterations / min_iterations_per_thread``). - For sequences, ``compute_thread_distribution`` splits ``com.max_thread`` between the images processed in parallel (``max_parallel_images``) and the OpenMP subthreads each image's ``image_hook`` receives (``nb_subthreads``). .. warning:: An ``image_hook`` that is handed ``nb_subthreads`` (sequence) or ``max_threads`` (single image) **must not** spawn more than that. A whole class of oversubscription bugs came from legacy single-image operations reading ``com.max_thread`` directly instead of obeying the worker's ``generic_img_args->max_threads``. When you convert an operation to the generic worker, make it honour the passed-in count. Per-format reentrancy decides whether OpenMP parallelism is enabled at all for a sequence: SER and fitseq are reentrant (parallel), film formats via FFMS2 are not (serial), and single-FITS depends on ``fits_is_reentrant()``. See :ref:`genericworkers:generic_sequence_worker`. The GTK rule ============ **Only the main thread may touch GTK.** GTK is not thread-safe, and Siril runs headless where GTK is absent entirely. Therefore: - Gather all GUI-derived parameters **before** submitting the job. - During processing, use only the thread-safe bridges: logging (``siril_log_message``), progress (via ``gui_iface.set_progress``), and other ``gui_iface`` slots — never GTK directly. - After processing, GUI updates run on the main thread via **idle functions** registered with ``siril_add_idle`` (which no-ops headless). The generic workers handle this for you through ``args->idle_function``. .. warning:: Never call ``execute_idle_sync`` (or ``execute_idle_and_wait_for_it``) from the GTK main thread — it waits on the main thread to run the idle, which is itself, so it deadlocks. The ``gui_iface`` implementations guard against this; see :ref:`locking:locking`. Sequence parallelism philosophy =============================== Siril's preferred mode of sequence parallelism is to work on **several images at once**. When there are not enough images to keep every thread busy (live stacking is the motivating case), spare threads are handed to each image's processing instead, as ``nb_subthreads`` passed to the ``image_hook``. Prefer frame-parallelism; reach for per-frame parallelism only when frames are too big or too few to fill the thread pool one-per-thread.