Memory management

Changed in version 1.5.

Siril routinely processes sequences whose combined size dwarfs the machine's RAM, so almost every operation has to answer the same question up front: how much memory may I use, and therefore how many images may I hold at once? This chapter describes how Siril measures available memory on each OS, how it turns that into a budget, how the sequence workers split that budget into a degree of parallelism, and the allocation and fits-lifecycle conventions you are expected to follow.

For who runs on which thread see Threading; for the workers that call the memory machinery on your behalf see Generic workers; for the per-OS quirks referenced below see OS differences.

The budget model

The user-facing budget is one number, in megabytes, returned by get_max_memory_in_MB() in src/core/OS_utils.c. It is driven by a single preference, com.pref.mem_mode, an anonymous enum defined in src/core/settings.h with exactly two members:

RATIO (the default) — the budget is com.pref.memory_ratio times the currently available memory reported by get_available_memory(). Because it tracks live availability, this mode adapts to whatever else the machine is doing.
AMOUNT — the budget is a fixed com.pref.memory_amount gigabytes, converted to MB (memory_amount * 1024). This ignores what is actually free, so it is the user's promise that the memory exists.

On a 32-bit build the address space, not the RAM, is the hard limit. After computing the budget, get_max_memory_in_MB() checks sizeof(void *) == 4 and, if so, clamps the result to 1900 MiB, logging "Limiting processing to 1900 MiB allocations (32-bit system)". Treat this as the ceiling on any single run's working set on those platforms.

Everything downstream — the sequence parallelism formula, the writer queue sizing, the per-image pre-flight checks — is expressed against this one budget, so a single preference change scales the whole application at once.

Measuring available memory per OS

get_available_memory() returns bytes currently available and is the most platform-dependent function in the codebase. Each branch is a separate implementation with its own caveats; when you touch it, read the branch for your platform, because "available" means something different on each.

Linux (and Cygwin)

Two sources, checked in order:

cgroups limits. get_available_mem_cgroups() first resolves the process's cgroup via /proc/self/cgroup (find_cgroups_path() distinguishes cgroups v1 from v2 by the empty controller field of the single v2 line), then reads the memory limit. For v1 it tries memory.soft_limit_in_bytes then memory.limit_in_bytes; for v2 it tries memory.low, memory.high, memory.max under /sys/fs/cgroup. The "available" figure it returns is the limit minus the process's own current usage (from get_used_RAM_memory(), which reads /proc/self/statm). This is what makes Siril behave inside a container or a systemd slice that would otherwise let it read the host's full RAM.
/proc/meminfo. If no cgroup memory controller is found, Siril parses /proc/meminfo for MemAvailable:, falling back to MemFree: on kernels older than 3.14 or on WSL2 where MemAvailable is absent. The value is cached and re-read at most once per second.

macOS

host_statistics64(HOST_VM_INFO64) plus sysctl(HW_MEMSIZE). Siril computes availability two ways and returns the smaller: (1) free + purgeable + external (file-cache) pages, and (2) physical memory minus genuinely used pages (internal minus purgeable, plus wired and compressor pages). Counting purgeable memory as available is deliberate — the kernel will reclaim it under pressure — and taking the smaller of the two estimates is the conservative choice.

Windows

GlobalMemoryStatusEx(), and here the subtlety matters. Siril starts from ullAvailPhys (free physical RAM) but then takes the minimum of that and ullAvailPageFile (remaining commit charge). Windows does not overcommit: an allocation fails once the commit charge reaches RAM plus pagefile, so the binding constraint is the commit headroom, not free physical memory. On a machine with a small or disabled pagefile the commit headroom can be lower than free RAM, and a budget derived from ullAvailPhys alone would promise memory the OS then refuses to hand over — an allocation failing mid-processing instead of an honest upfront "not enough memory" decision. This one line is the fix for a whole class of Windows-only out-of-memory reports.

BSD

get_available_memory() parses avail memory out of /var/run/dmesg.boot once and caches it. As the source comment flags, this is the amount available at boot, not at runtime, so on BSD the budget is effectively static. Treat it as a known limitation rather than a measurement.

The sequence parallelism formula

Sequences are processed frame-by-frame, and the budget decides how many frames may be in flight at once. The core computation lives in compute_nb_images_fit_memory() in src/io/sequence.c (delegating to compute_nb_images_fit_memory_from_dimensions()), with a compute_nb_images_fit_memory_from_fit() variant for a single fits.

The per-image footprint is rx * ry * layers * bytes-per-sample, where bytes-per-sample is sizeof(float) when the frame is float or force_float was requested, else sizeof(WORD). A factor argument scales the dimensions for operations that enlarge their output (registration upscaling, drizzle), so the function reports both an original and a scaled MB figure. The answer is simply max_memory_MB / MB_per_scaled_image — the budget divided by the scaled footprint. For a variable-size sequence, get_max_seq_dimension() picks the largest included frame's dimensions, so the estimate is safe for every frame.

seq_compute_mem_limits() in src/core/processing.c wraps this for the generic sequence worker. Without OpenMP the parallelism is 1. With OpenMP the result is capped at com.max_thread. If the footprint does not fit even once, it returns 0, and the operation is rejected with a "not enough memory" error naming the per-image requirement and the memory considered available.

When the default is wrong: `compute_mem_limits_hook`

The default formula assumes the only memory in play is the input plus the scaled output. That is false for any operation whose peak footprint is larger — typically because it allocates intermediate buffers. When that is the case you must supply args->compute_mem_limits_hook on your struct generic_seq_args (see Generic workers); the worker calls it in preference to seq_compute_mem_limits().

Two worked examples, both of which enumerate their intermediates explicitly:

Registration, apply_reg_compute_mem_limits() (with its accounting in apply_reg_compute_mem_consumption()) in src/registration/applyreg.c. Beyond input + scaled output it adds the two 32-bit remap maps used for undistortion or drizzle, the drizzle output counts image, and — the interesting part — the transient OpenCV overhead of colour interpolation, where fits_to_bgrbgr and Mat_to_image briefly hold roughly 2 × orig or 2 × dest − orig instead of the naive orig + dest, plus a guide image and an 8-bit mask copy when clamping is on.
Blending / mask creation, compute_mask_compute_mem_limits() in src/stacking/blending.c, which likewise adds the 8-bit and 32-bit layer copies that flow in and out of OpenCV to the base image cost.

If your operation transforms via OpenCV, upscales, or keeps more than one working copy alive at the peak, write a hook and account for every buffer that is live simultaneously. Underestimating here is exactly how you produce an OOM that the pre-flight was supposed to prevent.

The writer side (`for_writer`)

Both seq_compute_mem_limits() and every compute_mem_limits_hook take a for_writer boolean. When FALSE the return value is the number of frames processed in parallel (the threads). When TRUE it is instead the number of finished frames that may sit queued in the sequence writer waiting to be flushed to disk — computed from the budget left over after the processing threads have taken their share, and capped at com.max_thread * 3. seq_prepare_writer() feeds this number to seqwriter_set_max_active_blocks() in src/io/seqwriter.c, which blocks producers once that many output blocks are active. This back-pressure is what keeps a fast multi-thread stage from filling RAM with results faster than the writer can drain them.

Single-image operations: `mem_ratio`

For one-shot operations run through generic_image_worker there is no frame count to compute; instead struct generic_img_args carries a float mem_ratio — the operation's peak memory expressed as a multiple of the image size. A simple in-place filter is ~1; something that keeps a full extra working copy is ~2, and so on.

When mem_ratio > 0 the worker runs a pre-flight before doing any work, in default_img_mem_hook() (src/core/processing.c): it computes mem_ratio * rx * ry * channels * bytes-per-sample and compares it against get_available_memory(), aborting with "Not enough memory for operation" if it does not fit. generic_mask_worker has the analogous default_mask_mem_hook(), sized on the mask's bitpix (falling back to 4 bytes per pixel as a worst case when no mask is attached yet). Setting mem_ratio to 0 skips the check entirely — do that only for operations whose footprint is genuinely negligible.

Allocation conventions

Pair your allocator and freer. Memory from malloc/calloc is released with free; memory from GLib's g_malloc/g_new is released with g_free. Never cross the streams. Most string and GLib-object handling uses the g_* family; raw pixel and scratch buffers typically use plain malloc/calloc.

siril_malloc / siril_calloc / siril_free (src/core/siril_alloc.c, header src/core/siril_alloc.h) are a third family used specifically for large buffers. On Windows they are backed by VirtualAlloc (MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE) and freed with VirtualFree/MEM_RELEASE; siril_calloc relies on VirtualAlloc returning zeroed pages and also guards against a multiplication overflow in num * size. On every other platform they are thin wrappers over malloc/calloc/free. The intent is to route big single allocations around the Windows CRT heap, which can fail or fragment on large requests where VirtualAlloc succeeds. In the tree today they back exactly the large transform buffers that motivated them — the stacking data pool and scaled-mask buffers in src/stacking/median_and_mean.c, the drizzle and undistortion remap maps in src/drizzle/cdrizzlemap.c and src/registration/distorsion.c, and the drizzle output buffers in src/registration/applyreg.c. A block obtained from siril_malloc must be released with siril_free and no other freer, because on Windows the two use VirtualAlloc/VirtualFree rather than the CRT heap.

Note

siril_realloc does not exist — it is commented out in src/core/siril_alloc.c because VirtualAlloc has no in-place grow. If you need to resize a siril_malloc block, allocate a new one and copy.

Report allocation failures with PRINT_ALLOC_ERR — the macro from src/core/siril.h that prints Out of memory in <func> (<file>:<line>). Use it on the failure path of any allocation you check.

Give worker-argument structs a destructor. By convention the user pointer on struct generic_img_args (and the sequence and mask equivalents) must have a destructor function pointer as its first member, which the worker calls when it frees the args. This is the standard cleanup contract for the generic workers — see Generic workers.

`fits` memory and the statistics cache

A fits (defined in src/core/siril.h) holds pixels in one of two mutually exclusive buffers, selected by its type field: data (with per-layer pointers pdata[3]) for DATA_USHORT 16-bit WORD pixels, or fdata (with fpdata[3]) for DATA_FLOAT pixels. Only one is allocated at a time; the pdata/fpdata arrays are not separate allocations but offsets into the single buffer, one per layer. Do not assume both are present.

Converting between the two — for example honouring com.pref.force_16bit to avoid 32-bit pixel depth — goes through fit_replace_buffer() in src/io/image_format_fits.c, which swaps the buffer, updates type, and crucially calls invalidate_stats_from_fit().

That brings us to the classic bug. A fits caches per-layer statistics in its stats array (imstats). Any code that modifies pixels in place must call invalidate_stats_from_fit() (src/algos/statistics.c) afterwards, or later consumers will read stale mean/median/noise values from the cache and silently do the wrong thing. Forgetting this is a recurring class of subtle regression; if you write to a buffer, invalidate the stats.

Finally, an image may carry a mask (mask_t in src/core/siril.h), a separate data buffer whose per-pixel size follows its own bitpix. It is an additional allocation to account for in any memory hook that runs on mask-active images.

Disk space

Sequence-producing operations write intermediate and output files, so they must also check disk, not just RAM. test_available_space() (src/core/OS_utils.c) compares a requested byte count against the free space in the working directory com.wd (via find_space(), itself an OS-specific statvfs/statfs/GetDiskFreeSpaceExW implementation).

Call it before writing a sequence. Two behaviours are worth knowing:

If FITS compression is enabled (com.pref.comp.fits_enabled) the check is softened: because compressed output is smaller than the raw estimate, a shortfall is downgraded from a hard error to a warning, and only escalates in tone once the requested size exceeds free space by more than MAX_COMP_FREESPACE_RATIO (3×). Without compression, insufficient space is a hard error that stops the operation.
The swap/temporary directory used for sequence work is a separate preference, com.pref.swap_dir; update_displayed_memory() reports free space for both com.wd and swap_dir in the GUI.