Memory management ----------------- .. versionchanged:: 1.5 Siril routinely processes sequences whose combined size dwarfs the machine's RAM, so almost every operation has to answer the same question up front: *how much memory may I use, and therefore how many images may I hold at once?* This chapter describes how Siril measures available memory on each OS, how it turns that into a budget, how the sequence workers split that budget into a degree of parallelism, and the allocation and ``fits``-lifecycle conventions you are expected to follow. For *who runs on which thread* see :ref:`threading:threading`; for the workers that call the memory machinery on your behalf see :ref:`genericworkers:generic workers`; for the per-OS quirks referenced below see :ref:`osdifferences:os differences`. The budget model ================ The user-facing budget is one number, in megabytes, returned by ``get_max_memory_in_MB()`` in :file:`src/core/OS_utils.c`. It is driven by a single preference, ``com.pref.mem_mode``, an anonymous enum defined in :file:`src/core/settings.h` with exactly two members: - ``RATIO`` (the default) — the budget is ``com.pref.memory_ratio`` times the *currently available* memory reported by ``get_available_memory()``. Because it tracks live availability, this mode adapts to whatever else the machine is doing. - ``AMOUNT`` — the budget is a fixed ``com.pref.memory_amount`` gigabytes, converted to MB (``memory_amount * 1024``). This ignores what is actually free, so it is the user's promise that the memory exists. On a 32-bit build the address space, not the RAM, is the hard limit. After computing the budget, ``get_max_memory_in_MB()`` checks ``sizeof(void *) == 4`` and, if so, clamps the result to **1900** MiB, logging *"Limiting processing to 1900 MiB allocations (32-bit system)"*. Treat this as the ceiling on any single run's working set on those platforms. Everything downstream — the sequence parallelism formula, the writer queue sizing, the per-image pre-flight checks — is expressed against this one budget, so a single preference change scales the whole application at once. Measuring available memory per OS ================================= ``get_available_memory()`` returns *bytes currently available* and is the most platform-dependent function in the codebase. Each branch is a separate implementation with its own caveats; when you touch it, read the branch for your platform, because "available" means something different on each. Linux (and Cygwin) ****************** Two sources, checked in order: #. **cgroups limits.** ``get_available_mem_cgroups()`` first resolves the process's cgroup via :file:`/proc/self/cgroup` (``find_cgroups_path()`` distinguishes cgroups v1 from v2 by the empty controller field of the single v2 line), then reads the memory limit. For v1 it tries ``memory.soft_limit_in_bytes`` then ``memory.limit_in_bytes``; for v2 it tries ``memory.low``, ``memory.high``, ``memory.max`` under :file:`/sys/fs/cgroup`. The "available" figure it returns is the limit minus the process's own current usage (from ``get_used_RAM_memory()``, which reads :file:`/proc/self/statm`). This is what makes Siril behave inside a container or a systemd slice that would otherwise let it read the host's full RAM. #. **/proc/meminfo.** If no cgroup memory controller is found, Siril parses :file:`/proc/meminfo` for ``MemAvailable:``, falling back to ``MemFree:`` on kernels older than 3.14 or on WSL2 where ``MemAvailable`` is absent. The value is cached and re-read at most once per second. macOS ***** ``host_statistics64(HOST_VM_INFO64)`` plus ``sysctl(HW_MEMSIZE)``. Siril computes availability two ways and returns the smaller: (1) free + **purgeable** + external (file-cache) pages, and (2) physical memory minus genuinely used pages (internal minus purgeable, plus wired and compressor pages). Counting purgeable memory as available is deliberate — the kernel will reclaim it under pressure — and taking the smaller of the two estimates is the conservative choice. Windows ******* ``GlobalMemoryStatusEx()``, and here the subtlety matters. Siril starts from ``ullAvailPhys`` (free physical RAM) but then takes the **minimum** of that and ``ullAvailPageFile`` (remaining commit charge). Windows does not overcommit: an allocation fails once the commit charge reaches RAM plus pagefile, so the binding constraint is the commit headroom, not free physical memory. On a machine with a small or disabled pagefile the commit headroom can be *lower* than free RAM, and a budget derived from ``ullAvailPhys`` alone would promise memory the OS then refuses to hand over — an allocation failing mid-processing instead of an honest upfront "not enough memory" decision. This one line is the fix for a whole class of Windows-only out-of-memory reports. BSD *** ``get_available_memory()`` parses ``avail memory`` out of :file:`/var/run/dmesg.boot` **once** and caches it. As the source comment flags, this is the amount available *at boot*, not at runtime, so on BSD the budget is effectively static. Treat it as a known limitation rather than a measurement. The sequence parallelism formula ================================ Sequences are processed frame-by-frame, and the budget decides how many frames may be in flight at once. The core computation lives in ``compute_nb_images_fit_memory()`` in :file:`src/io/sequence.c` (delegating to ``compute_nb_images_fit_memory_from_dimensions()``), with a ``compute_nb_images_fit_memory_from_fit()`` variant for a single ``fits``. The per-image footprint is ``rx * ry * layers * bytes-per-sample``, where bytes-per-sample is ``sizeof(float)`` when the frame is float or ``force_float`` was requested, else ``sizeof(WORD)``. A ``factor`` argument scales the dimensions for operations that enlarge their output (registration upscaling, drizzle), so the function reports both an *original* and a *scaled* MB figure. The answer is simply ``max_memory_MB / MB_per_scaled_image`` — the budget divided by the scaled footprint. For a **variable-size** sequence, ``get_max_seq_dimension()`` picks the largest included frame's dimensions, so the estimate is safe for every frame. ``seq_compute_mem_limits()`` in :file:`src/core/processing.c` wraps this for the generic sequence worker. Without OpenMP the parallelism is 1. With OpenMP the result is capped at ``com.max_thread``. If the footprint does not fit even once, it returns 0, and the operation is rejected with a *"not enough memory"* error naming the per-image requirement and the memory considered available. When the default is wrong: ``compute_mem_limits_hook`` ****************************************************** The default formula assumes the only memory in play is the input plus the scaled output. That is false for any operation whose *peak* footprint is larger — typically because it allocates intermediate buffers. When that is the case you must supply ``args->compute_mem_limits_hook`` on your ``struct generic_seq_args`` (see :ref:`genericworkers:generic workers`); the worker calls it in preference to ``seq_compute_mem_limits()``. Two worked examples, both of which enumerate their intermediates explicitly: - **Registration**, ``apply_reg_compute_mem_limits()`` (with its accounting in ``apply_reg_compute_mem_consumption()``) in :file:`src/registration/applyreg.c`. Beyond input + scaled output it adds the two 32-bit remap maps used for undistortion or drizzle, the drizzle output counts image, and — the interesting part — the transient OpenCV overhead of colour interpolation, where ``fits_to_bgrbgr`` and ``Mat_to_image`` briefly hold roughly ``2 × orig`` or ``2 × dest − orig`` instead of the naive ``orig + dest``, plus a guide image and an 8-bit mask copy when clamping is on. - **Blending / mask creation**, ``compute_mask_compute_mem_limits()`` in :file:`src/stacking/blending.c`, which likewise adds the 8-bit and 32-bit layer copies that flow in and out of OpenCV to the base image cost. If your operation transforms via OpenCV, upscales, or keeps more than one working copy alive at the peak, write a hook and account for every buffer that is live simultaneously. Underestimating here is exactly how you produce an OOM that the pre-flight was supposed to prevent. The writer side (``for_writer``) ******************************** Both ``seq_compute_mem_limits()`` and every ``compute_mem_limits_hook`` take a ``for_writer`` boolean. When ``FALSE`` the return value is the number of frames processed in parallel (the *threads*). When ``TRUE`` it is instead the number of finished frames that may sit queued in the sequence writer waiting to be flushed to disk — computed from the budget *left over* after the processing threads have taken their share, and capped at ``com.max_thread * 3``. ``seq_prepare_writer()`` feeds this number to ``seqwriter_set_max_active_blocks()`` in :file:`src/io/seqwriter.c`, which blocks producers once that many output blocks are active. This back-pressure is what keeps a fast multi-thread stage from filling RAM with results faster than the writer can drain them. Single-image operations: ``mem_ratio`` ======================================= For one-shot operations run through ``generic_image_worker`` there is no frame count to compute; instead ``struct generic_img_args`` carries a ``float mem_ratio`` — the operation's peak memory expressed as a **multiple of the image size**. A simple in-place filter is ~1; something that keeps a full extra working copy is ~2, and so on. When ``mem_ratio > 0`` the worker runs a pre-flight before doing any work, in ``default_img_mem_hook()`` (:file:`src/core/processing.c`): it computes ``mem_ratio * rx * ry * channels * bytes-per-sample`` and compares it against ``get_available_memory()``, aborting with *"Not enough memory for operation"* if it does not fit. ``generic_mask_worker`` has the analogous ``default_mask_mem_hook()``, sized on the mask's ``bitpix`` (falling back to 4 bytes per pixel as a worst case when no mask is attached yet). Setting ``mem_ratio`` to 0 skips the check entirely — do that only for operations whose footprint is genuinely negligible. Allocation conventions ====================== **Pair your allocator and freer.** Memory from ``malloc``/``calloc`` is released with ``free``; memory from GLib's ``g_malloc``/``g_new`` is released with ``g_free``. Never cross the streams. Most string and GLib-object handling uses the ``g_*`` family; raw pixel and scratch buffers typically use plain ``malloc``/``calloc``. ``siril_malloc`` / ``siril_calloc`` / ``siril_free`` (:file:`src/core/siril_alloc.c`, header :file:`src/core/siril_alloc.h`) are a third family used specifically for large buffers. On Windows they are backed by ``VirtualAlloc`` (``MEM_COMMIT | MEM_RESERVE``, ``PAGE_READWRITE``) and freed with ``VirtualFree``/``MEM_RELEASE``; ``siril_calloc`` relies on ``VirtualAlloc`` returning zeroed pages and also guards against a multiplication overflow in ``num * size``. On every other platform they are thin wrappers over ``malloc``/``calloc``/``free``. The intent is to route big single allocations around the Windows CRT heap, which can fail or fragment on large requests where ``VirtualAlloc`` succeeds. In the tree today they back exactly the large transform buffers that motivated them — the stacking data pool and scaled-mask buffers in :file:`src/stacking/median_and_mean.c`, the drizzle and undistortion remap maps in :file:`src/drizzle/cdrizzlemap.c` and :file:`src/registration/distorsion.c`, and the drizzle output buffers in :file:`src/registration/applyreg.c`. A block obtained from ``siril_malloc`` **must** be released with ``siril_free`` and no other freer, because on Windows the two use ``VirtualAlloc``/``VirtualFree`` rather than the CRT heap. .. note:: ``siril_realloc`` does not exist — it is commented out in :file:`src/core/siril_alloc.c` because ``VirtualAlloc`` has no in-place grow. If you need to resize a ``siril_malloc`` block, allocate a new one and copy. **Report allocation failures with** ``PRINT_ALLOC_ERR`` — the macro from :file:`src/core/siril.h` that prints ``Out of memory in (:)``. Use it on the failure path of any allocation you check. **Give worker-argument structs a destructor.** By convention the ``user`` pointer on ``struct generic_img_args`` (and the sequence and mask equivalents) must have a *destructor function pointer as its first member*, which the worker calls when it frees the args. This is the standard cleanup contract for the generic workers — see :ref:`genericworkers:generic workers`. ``fits`` memory and the statistics cache ========================================= A ``fits`` (defined in :file:`src/core/siril.h`) holds pixels in **one** of two mutually exclusive buffers, selected by its ``type`` field: ``data`` (with per-layer pointers ``pdata[3]``) for ``DATA_USHORT`` 16-bit ``WORD`` pixels, or ``fdata`` (with ``fpdata[3]``) for ``DATA_FLOAT`` pixels. Only one is allocated at a time; the ``pdata``/``fpdata`` arrays are not separate allocations but offsets into the single buffer, one per layer. Do not assume both are present. Converting between the two — for example honouring ``com.pref.force_16bit`` to avoid 32-bit pixel depth — goes through ``fit_replace_buffer()`` in :file:`src/io/image_format_fits.c`, which swaps the buffer, updates ``type``, and crucially calls ``invalidate_stats_from_fit()``. That brings us to the classic bug. A ``fits`` caches per-layer statistics in its ``stats`` array (``imstats``). Any code that **modifies pixels in place must call** ``invalidate_stats_from_fit()`` (:file:`src/algos/statistics.c`) afterwards, or later consumers will read stale mean/median/noise values from the cache and silently do the wrong thing. Forgetting this is a recurring class of subtle regression; if you write to a buffer, invalidate the stats. Finally, an image may carry a ``mask`` (``mask_t`` in :file:`src/core/siril.h`), a separate ``data`` buffer whose per-pixel size follows its own ``bitpix``. It is an additional allocation to account for in any memory hook that runs on mask-active images. Disk space ========== Sequence-producing operations write intermediate and output files, so they must also check *disk*, not just RAM. ``test_available_space()`` (:file:`src/core/OS_utils.c`) compares a requested byte count against the free space in the working directory ``com.wd`` (via ``find_space()``, itself an OS-specific ``statvfs``/``statfs``/``GetDiskFreeSpaceExW`` implementation). Call it before writing a sequence. Two behaviours are worth knowing: - If FITS compression is enabled (``com.pref.comp.fits_enabled``) the check is softened: because compressed output is smaller than the raw estimate, a shortfall is downgraded from a hard error to a warning, and only escalates in tone once the requested size exceeds free space by more than ``MAX_COMP_FREESPACE_RATIO`` (3×). Without compression, insufficient space is a hard error that stops the operation. - The swap/temporary directory used for sequence work is a separate preference, ``com.pref.swap_dir``; ``update_displayed_memory()`` reports free space for both ``com.wd`` and ``swap_dir`` in the GUI. See also ======== - :ref:`genericworkers:generic workers` — the workers that call the budget, the hooks, and the args-destructor contract. - :ref:`threading:threading` — how ``com.max_thread`` and the parallelism figure relate. - :ref:`osdifferences:os differences` — the per-OS behaviour behind ``get_available_memory()`` and ``find_space()``.