Python integration

Added in version 1.5.

Siril can drive, and be driven by, Python scripts. This chapter is for contributors who add Python-reachable functionality or who touch the IPC layer itself. If instead you are writing a script, you want the sirilpy package API documentation, which is maintained separately; nothing here is needed to author a script.

The Python code that ships with Siril lives in two places. The C-side transport and command handlers are in src/io/siril_pythonmodule.c, src/io/siril_pythonmodule.h and src/io/siril_pythoncommands.c. The Python-side client library, sirilpy, lives under python_module/sirilpy.

Architecture

Scripts are never embedded. Siril does not host a Python interpreter in its own process; instead execute_python_script() (in siril_pythonmodule.c) spawns the venv's python as a subprocess and talks to it over a local socket. This keeps a misbehaving or crashing script out of Siril's address space.

The transport is:

  • an AF_UNIX stream socket on Unix (created in create_connection()), or

  • a Windows named pipe (created in the #ifdef _WIN32 branch of the same setup code).

The path to the endpoint is handed to the child in an environment variable: MY_SOCKET on Unix, MY_PIPE on Windows (both set in execute_python_script()). sirilpy reads these back in python_module/sirilpy/connection.py.

For each running script there is one C worker thread, created as "python-comm" with g_thread_new(). That thread owns the Connection object (struct _Connection in siril_pythonmodule.h) and loops in handle_client_communication(), reading requests and dispatching them.

The wire protocol

Every message begins with a fixed 5-byte header. On the C side this is:

typedef struct {
    uint8_t  command;
    uint32_t length;
} __attribute__((packed)) CommandHeader;

so the header is one byte of command id followed by a four-byte payload length. The length is carried in network (big-endian) byte order: process_connection() reads it back with GINT32_FROM_BE(), and sirilpy packs it with struct.pack('!Bi', command, data_length). The response header (ResponseHeader) has the same 5-byte layout — a one-byte status plus a big-endian uint32_t length — and sirilpy unpacks it with struct.unpack('!BI', ...). The status byte is one of the StatusCode values: STATUS_OK (0), STATUS_NONE (1, for calls that may legitimately return no data) or STATUS_ERROR (0xFF).

Small payloads travel inline, in the same buffer as the header. The C read buffer, BUFFER_SIZE, is 65536 bytes (#define BUFFER_SIZE 65536 in siril_pythonmodule.c), so the inline path is capped at roughly 64 KB. Anything larger — pixel data above all, but also FITS headers, history, ICC profiles and unknown-key blocks — is transferred through shared memory instead (see Shared memory lifecycle).

Command dispatch

process_connection() in siril_pythoncommands.c is the heart of the C side. It validates that the buffer is at least sizeof(CommandHeader) long, reads the big-endian payload length, checks the buffer actually holds the whole payload, and then runs a single big switch (header->command) over the CommandType enum. Each case is one command handler; it does its work and calls send_response() to reply.

The command ids themselves are the CommandType enum. It is defined twice, once on each side, and the two copies must stay identical:

  • C side: CommandType in src/io/siril_pythonmodule.h (values run from CMD_SEND_COMMAND = 1 up, with CMD_ERROR = 0xFF).

  • Python side: the _Command IntEnum in python_module/sirilpy/enums.py (its docstring already says it MUST match the header).

There is no protocol negotiation or version handshake at connection time. The two enums are kept in sync by hand, and a Siril release must ship a matching pair. If they drift, a command id means one thing on one side and another on the other, silently.

Tip

Adding a new Python command — the checklist. Because the enum is duplicated and unnegotiated, a new command touches four things and they must all agree:

  1. Add the new value at the end of CommandType in src/io/siril_pythonmodule.h (do not reuse or renumber existing ids — that breaks every other script in flight against an old build).

  2. Add the identical value to _Command in python_module/sirilpy/enums.py.

  3. Implement the handler as a new case in the process_connection() switch in src/io/siril_pythoncommands.c, ending in a send_response() call.

  4. Add the sirilpy method that sends the command and interprets the response, in python_module/sirilpy/connection.py.

Ship the C and Python halves together in the same release.

Thread-safety rules for handlers

Every handler in process_connection() runs on the python worker thread, not on the GTK main thread and not on the processing worker. That has three hard consequences.

Take the fits rwlock for gfit. gfit is a public fits shared with the processing worker, so a handler that reads or writes it must hold the per-fits lock for the duration. Take the reader lock for reads and the writer lock for writes, and always release with the matching flavour. CMD_GET_DIMENSIONS shows the read pattern (g_rw_lock_reader_lock(&gfit->rwlock) around the rx/ry/naxes copy); the header- and metadata-mutating handlers take g_rw_lock_writer_lock(&gfit->rwlock). See The fits rwlock and Locking.

Route every GUI effect through gui_iface. A handler that needs a display side effect — showing a plot, marking Siril busy, refreshing a view — calls through the gui_iface function table (for example gui_iface.set_busy(), gui_iface.show_siril_plot(), gui_iface.clear_roi()). This is what keeps the Python code compiling and running in siril-cli with no GUI at all.

Never call GTK directly from a handler. The worker thread is not the GTK main thread; touching widgets from it is undefined. gui_iface exists precisely so that core and IPC code never has to.

The thread-claim model

Reading gfit only needs the rwlock. Modifying gfit additionally requires that Python has claimed the processing thread, so that a script's edits cannot race a Siril processing job over the same image.

A script claims the thread with CMD_CLAIM_THREAD, which calls claim_thread_for_python() (in src/core/processing_thread.c). That function can refuse in two distinct ways, and the handler maps each to a message:

  • If an image-processing dialog is open (gui_iface.is_dialog_open()), it returns 2 — the dialog holds the image lock and must be closed first.

  • If the thread is already reserved by Python it returns 1.

Otherwise it waits until any running job has finished and the pending job queue has drained (it blocks on queue_cond, which the worker broadcasts as each job completes), then sets python_reserved and marks Siril busy via gui_iface.set_busy(TRUE). On success the handler sets conn->thread_claimed = TRUE.

The write handlers enforce this. handle_set_pixeldata_request() and handle_set_image_mask_request() both refuse outright — with an explicit "processing thread is not claimed … this is a script error" message — if the target is gfit and conn->thread_claimed is false.

Releasing is symmetric: CMD_RELEASE_THREAD calls python_releases_thread(), which clears python_reserved, broadcasts queue_cond to wake any blocked job submissions, and clears the busy state. The handler only releases if this connection actually holds the claim, and otherwise succeeds silently — sirilpy is expected to call release from a finally: block, so a double release must not error. processing_is_reserved_for_python() reports the current state to the rest of the core.

Commands issued from Python — via CMD_SEND_COMMAND — run the other way round. The handler sets com.python_command, calls processcommand() synchronously (it blocks the worker thread until the command finishes), clears the flag, and returns the command's integer return code as the payload so that sirilpy can raise on failure. See Threading for how the processing thread and job queue behave.

Shared memory lifecycle

Bulk data uses a named shared-memory region: shm_open + mmap on Unix, CreateFileMapping + MapViewOfFile on Windows, both wrapped by siril_allocate_shm(). The region name is random and returned to Python inside a shared_memory_info_t.

Tracking and release. Every region Siril allocates for a script is recorded against the connection: track_shm_allocation() appends a shm_allocation_t to conn->g_shm_allocations, guarded by conn->g_shm_mutex (initialised lazily by init_shm_tracking()). Siril cannot know when the Python side has finished mapping the region, so it does not free it immediately. Instead the script sends CMD_RELEASE_SHM with the region name; the handler calls cleanup_shm_allocation(), which finds the tracked allocation by name and unmaps/closes/unlinks it on the C side.

Both sides must unlink. As the comments in handle_pixeldata_request() note, the region is only fully torn down once both processes have unmapped, closed and unlinked it. sirilpy does its half (close then unlink in python_module/sirilpy/shm.py) in a finally: block. If a script crashes before it releases, its half is skipped and the region can be orphaned — this is the main leak risk in the design, and a reason the subprocess model matters: an orphan is bounded to that script's regions rather than corrupting Siril.

Region validation is security-relevant. The dimensions and offsets in an incoming request come from a possibly untrusted script, so every handler that consumes a region validates them before computing any memcpy:

  • handle_pixeldata_request() rejects any region that is empty or steps outside the image, and deliberately writes the comparisons with subtraction (region.x > fit->rx - region.w) so a crafted width or height cannot overflow past the bounds check and read arbitrary heap into the region.

  • handle_set_pixeldata_request() re-derives the expected byte count in 64-bit ((size_t)width * height * channels), caps width and height well below any overflow point, requires the declared size to match exactly, and rejects anything over half of available memory.

  • Before mapping, the incoming region is checked to be at least the declared size (via fstat on Unix / VirtualQuery on Windows) so a short region cannot be read past its end.

If you add a handler that maps a script-supplied region, copy this pattern: validate first, compute sizes in a width big enough not to wrap, compare with subtraction, and confirm the mapped object is large enough.

Virtual environment management

Siril runs scripts inside a dedicated Python virtual environment under the user data directory: g_get_user_data_dir()/siril/venv (built in several places, e.g. rebuild_venv() and check_or_create_venv()).

initialize_python_venv_in_thread() kicks off setup on a background thread, guarded by a static GMutex init_mutex and a com.python_init_thread check so two initialisations cannot run at once. check_or_create_venv() looks for the venv's python executable, and if it exists runs validate_venv_health() — which confirms the interpreter is present and that the required modules (pip, venv, ensurepip) import and that pip is functional. An unhealthy venv is deleted and recreated. At launch, prepare_venv_environment() sets VIRTUAL_ENV and the PATH / PYTHONPATH entries for the child, installs or version-checks the bundled sirilpy module with pip, and (in the non-CLI case) can run any scripts marked to execute at startup via execute_startup_scripts().

The user can force a clean rebuild from the preferences GUI: the reset-venv button handler on_button_python_reset_venv_clicked() (after a confirmation dialog) calls rebuild_venv(), which deletes the venv directory and calls initialize_python_venv_in_thread() again to build a fresh one, discarding any modules that scripts had installed.

Note

The packaging internals here (how the interpreter is located, how modules are provisioned) are evolving. Treat the function names above as the stable entry points and read the current source for the details of a given release.

Debug and CLI modes

Two environment variables, both set by execute_python_script() and read by sirilpy in python_module/sirilpy/connection.py, change the child's behaviour:

  • SIRIL_PYTHON_DEBUG (set to "1" when the script is launched with debug mode on — from the GUI this comes from the Python debug preference passed into execute_python_script()). sirilpy exposes it as self.debug and emits extra diagnostics.

  • SIRIL_PYTHON_CLI (set to "1" when the script was launched from the command line). sirilpy reads it as self._is_cli.

The connection loop is also written to tolerate a client that disconnects and reconnects on the same endpoint — as an early CLI-mode probe does. In handle_client_communication() a client disconnect returns control to wait_for_client() rather than tearing everything down; the worker only stops for good when should_stop is set on the real teardown path.