Python integration ------------------ .. versionadded:: 1.5 Siril can drive, and be driven by, Python scripts. This chapter is for contributors who add Python-reachable functionality or who touch the IPC layer itself. If instead you are *writing* a script, you want the ``sirilpy`` package API documentation, which is maintained separately; nothing here is needed to author a script. The Python code that ships with Siril lives in two places. The C-side transport and command handlers are in :file:`src/io/siril_pythonmodule.c`, :file:`src/io/siril_pythonmodule.h` and :file:`src/io/siril_pythoncommands.c`. The Python-side client library, ``sirilpy``, lives under :file:`python_module/sirilpy`. Architecture ============ Scripts are **never embedded**. Siril does not host a Python interpreter in its own process; instead ``execute_python_script()`` (in :file:`siril_pythonmodule.c`) spawns the venv's ``python`` as a **subprocess** and talks to it over a local socket. This keeps a misbehaving or crashing script out of Siril's address space. The transport is: - an ``AF_UNIX`` stream socket on Unix (created in ``create_connection()``), or - a Windows named pipe (created in the ``#ifdef _WIN32`` branch of the same setup code). The path to the endpoint is handed to the child in an environment variable: ``MY_SOCKET`` on Unix, ``MY_PIPE`` on Windows (both set in ``execute_python_script()``). ``sirilpy`` reads these back in :file:`python_module/sirilpy/connection.py`. For each running script there is **one C worker thread**, created as ``"python-comm"`` with ``g_thread_new()``. That thread owns the ``Connection`` object (``struct _Connection`` in :file:`siril_pythonmodule.h`) and loops in ``handle_client_communication()``, reading requests and dispatching them. The wire protocol ***************** Every message begins with a fixed **5-byte header**. On the C side this is: .. code-block:: c typedef struct { uint8_t command; uint32_t length; } __attribute__((packed)) CommandHeader; so the header is one byte of command id followed by a four-byte payload length. The length is carried in **network (big-endian) byte order**: ``process_connection()`` reads it back with ``GINT32_FROM_BE()``, and ``sirilpy`` packs it with ``struct.pack('!Bi', command, data_length)``. The response header (``ResponseHeader``) has the same 5-byte layout — a one-byte ``status`` plus a big-endian ``uint32_t length`` — and ``sirilpy`` unpacks it with ``struct.unpack('!BI', ...)``. The status byte is one of the ``StatusCode`` values: ``STATUS_OK`` (0), ``STATUS_NONE`` (1, for calls that may legitimately return no data) or ``STATUS_ERROR`` (``0xFF``). Small payloads travel **inline**, in the same buffer as the header. The C read buffer, ``BUFFER_SIZE``, is **65536** bytes (``#define BUFFER_SIZE 65536`` in :file:`siril_pythonmodule.c`), so the inline path is capped at roughly 64 KB. Anything larger — pixel data above all, but also FITS headers, history, ICC profiles and unknown-key blocks — is transferred through **shared memory** instead (see `Shared memory lifecycle`_). Command dispatch ================ ``process_connection()`` in :file:`siril_pythoncommands.c` is the heart of the C side. It validates that the buffer is at least ``sizeof(CommandHeader)`` long, reads the big-endian payload length, checks the buffer actually holds the whole payload, and then runs a single big ``switch (header->command)`` over the ``CommandType`` enum. Each ``case`` is one command handler; it does its work and calls ``send_response()`` to reply. The command ids themselves are the ``CommandType`` enum. It is defined **twice**, once on each side, and the two copies must stay identical: - C side: ``CommandType`` in :file:`src/io/siril_pythonmodule.h` (values run from ``CMD_SEND_COMMAND = 1`` up, with ``CMD_ERROR = 0xFF``). - Python side: the ``_Command`` ``IntEnum`` in :file:`python_module/sirilpy/enums.py` (its docstring already says it *MUST* match the header). There is **no protocol negotiation or version handshake** at connection time. The two enums are kept in sync **by hand**, and a Siril release must ship a matching pair. If they drift, a command id means one thing on one side and another on the other, silently. .. tip:: **Adding a new Python command — the checklist.** Because the enum is duplicated and unnegotiated, a new command touches four things and they must all agree: 1. Add the new value at the **end** of ``CommandType`` in :file:`src/io/siril_pythonmodule.h` (do not reuse or renumber existing ids — that breaks every other script in flight against an old build). 2. Add the identical value to ``_Command`` in :file:`python_module/sirilpy/enums.py`. 3. Implement the handler as a new ``case`` in the ``process_connection()`` switch in :file:`src/io/siril_pythoncommands.c`, ending in a ``send_response()`` call. 4. Add the ``sirilpy`` method that sends the command and interprets the response, in :file:`python_module/sirilpy/connection.py`. Ship the C and Python halves together in the same release. Thread-safety rules for handlers ================================ Every handler in ``process_connection()`` runs on the **python worker thread**, not on the GTK main thread and not on the processing worker. That has three hard consequences. **Take the fits rwlock for** ``gfit``. ``gfit`` is a public ``fits`` shared with the processing worker, so a handler that reads or writes it must hold the per-fits lock for the duration. Take the **reader** lock for reads and the **writer** lock for writes, and always release with the matching flavour. ``CMD_GET_DIMENSIONS`` shows the read pattern (``g_rw_lock_reader_lock(&gfit->rwlock)`` around the ``rx``/``ry``/``naxes`` copy); the header- and metadata-mutating handlers take ``g_rw_lock_writer_lock(&gfit->rwlock)``. See :ref:`globalvariables:the fits rwlock` and :ref:`locking:locking`. **Route every GUI effect through** ``gui_iface``. A handler that needs a display side effect — showing a plot, marking Siril busy, refreshing a view — calls through the ``gui_iface`` function table (for example ``gui_iface.set_busy()``, ``gui_iface.show_siril_plot()``, ``gui_iface.clear_roi()``). This is what keeps the Python code compiling and running in ``siril-cli`` with no GUI at all. **Never call GTK directly** from a handler. The worker thread is not the GTK main thread; touching widgets from it is undefined. ``gui_iface`` exists precisely so that core and IPC code never has to. The thread-claim model ====================== Reading ``gfit`` only needs the rwlock. **Modifying** ``gfit`` additionally requires that Python has *claimed the processing thread*, so that a script's edits cannot race a Siril processing job over the same image. A script claims the thread with ``CMD_CLAIM_THREAD``, which calls ``claim_thread_for_python()`` (in :file:`src/core/processing_thread.c`). That function can refuse in two distinct ways, and the handler maps each to a message: - If an image-processing **dialog is open** (``gui_iface.is_dialog_open()``), it returns ``2`` — the dialog holds the image lock and must be closed first. - If the thread is already reserved by Python it returns ``1``. Otherwise it **waits** until any running job has finished and the pending job queue has drained (it blocks on ``queue_cond``, which the worker broadcasts as each job completes), then sets ``python_reserved`` and marks Siril busy via ``gui_iface.set_busy(TRUE)``. On success the handler sets ``conn->thread_claimed = TRUE``. The write handlers enforce this. ``handle_set_pixeldata_request()`` and ``handle_set_image_mask_request()`` both refuse outright — with an explicit "processing thread is not claimed … this is a script error" message — if the target is ``gfit`` and ``conn->thread_claimed`` is false. Releasing is symmetric: ``CMD_RELEASE_THREAD`` calls ``python_releases_thread()``, which clears ``python_reserved``, broadcasts ``queue_cond`` to wake any blocked job submissions, and clears the busy state. The handler only releases if this connection actually holds the claim, and otherwise succeeds silently — ``sirilpy`` is expected to call release from a ``finally:`` block, so a double release must not error. ``processing_is_reserved_for_python()`` reports the current state to the rest of the core. Commands issued **from** Python — via ``CMD_SEND_COMMAND`` — run the other way round. The handler sets ``com.python_command``, calls ``processcommand()`` **synchronously** (it blocks the worker thread until the command finishes), clears the flag, and returns the command's integer return code as the payload so that ``sirilpy`` can raise on failure. See :ref:`threading:threading` for how the processing thread and job queue behave. Shared memory lifecycle ======================= Bulk data uses a named shared-memory region: ``shm_open`` + ``mmap`` on Unix, ``CreateFileMapping`` + ``MapViewOfFile`` on Windows, both wrapped by ``siril_allocate_shm()``. The region name is random and returned to Python inside a ``shared_memory_info_t``. **Tracking and release.** Every region Siril allocates for a script is recorded against the connection: ``track_shm_allocation()`` appends a ``shm_allocation_t`` to ``conn->g_shm_allocations``, guarded by ``conn->g_shm_mutex`` (initialised lazily by ``init_shm_tracking()``). Siril cannot know when the Python side has finished mapping the region, so it does not free it immediately. Instead the script sends ``CMD_RELEASE_SHM`` with the region name; the handler calls ``cleanup_shm_allocation()``, which finds the tracked allocation by name and unmaps/closes/unlinks it on the C side. **Both sides must unlink.** As the comments in ``handle_pixeldata_request()`` note, the region is only fully torn down once *both* processes have unmapped, closed and unlinked it. ``sirilpy`` does its half (``close`` then ``unlink`` in :file:`python_module/sirilpy/shm.py`) in a ``finally:`` block. If a script **crashes** before it releases, its half is skipped and the region can be orphaned — this is the main leak risk in the design, and a reason the subprocess model matters: an orphan is bounded to that script's regions rather than corrupting Siril. **Region validation is security-relevant.** The dimensions and offsets in an incoming request come from a possibly untrusted script, so every handler that consumes a region validates them before computing any ``memcpy``: - ``handle_pixeldata_request()`` rejects any region that is empty or steps outside the image, and deliberately writes the comparisons with subtraction (``region.x > fit->rx - region.w``) so a crafted width or height cannot overflow past the bounds check and read arbitrary heap into the region. - ``handle_set_pixeldata_request()`` re-derives the expected byte count in 64-bit (``(size_t)width * height * channels``), caps width and height well below any overflow point, requires the declared ``size`` to match exactly, and rejects anything over half of available memory. - Before mapping, the incoming region is checked to be **at least** the declared size (via ``fstat`` on Unix / ``VirtualQuery`` on Windows) so a short region cannot be read past its end. If you add a handler that maps a script-supplied region, copy this pattern: validate first, compute sizes in a width big enough not to wrap, compare with subtraction, and confirm the mapped object is large enough. Virtual environment management ============================== Siril runs scripts inside a dedicated Python virtual environment under the user data directory: ``g_get_user_data_dir()/siril/venv`` (built in several places, e.g. ``rebuild_venv()`` and ``check_or_create_venv()``). ``initialize_python_venv_in_thread()`` kicks off setup on a background thread, guarded by a ``static GMutex init_mutex`` and a ``com.python_init_thread`` check so two initialisations cannot run at once. ``check_or_create_venv()`` looks for the venv's ``python`` executable, and if it exists runs ``validate_venv_health()`` — which confirms the interpreter is present and that the required modules (``pip``, ``venv``, ``ensurepip``) import and that ``pip`` is functional. An unhealthy venv is deleted and recreated. At launch, ``prepare_venv_environment()`` sets ``VIRTUAL_ENV`` and the ``PATH`` / ``PYTHONPATH`` entries for the child, installs or version-checks the bundled ``sirilpy`` module with pip, and (in the non-CLI case) can run any scripts marked to execute at startup via ``execute_startup_scripts()``. The user can force a clean rebuild from the preferences GUI: the reset-venv button handler ``on_button_python_reset_venv_clicked()`` (after a confirmation dialog) calls ``rebuild_venv()``, which deletes the venv directory and calls ``initialize_python_venv_in_thread()`` again to build a fresh one, discarding any modules that scripts had installed. .. note:: The packaging internals here (how the interpreter is located, how modules are provisioned) are evolving. Treat the function names above as the stable entry points and read the current source for the details of a given release. Debug and CLI modes =================== Two environment variables, both set by ``execute_python_script()`` and read by ``sirilpy`` in :file:`python_module/sirilpy/connection.py`, change the child's behaviour: - ``SIRIL_PYTHON_DEBUG`` (set to ``"1"`` when the script is launched with debug mode on — from the GUI this comes from the Python debug preference passed into ``execute_python_script()``). ``sirilpy`` exposes it as ``self.debug`` and emits extra diagnostics. - ``SIRIL_PYTHON_CLI`` (set to ``"1"`` when the script was launched from the command line). ``sirilpy`` reads it as ``self._is_cli``. The connection loop is also written to tolerate a client that disconnects and reconnects on the same endpoint — as an early CLI-mode probe does. In ``handle_client_communication()`` a client disconnect returns control to ``wait_for_client()`` rather than tearing everything down; the worker only stops for good when ``should_stop`` is set on the real teardown path.