VNG demosaicer OpenCL performance regression on 5.4

[agat114]

Is there an existing issue for this?

• I checked and did not find my issue in the already reported ones

Describe the bug

After upgrade to 5.4 and 5.5. I am experiencing short freezes when using OpenCL accelerated modules, such as exposure, raw chromatic aberrations, demosaic. Some of those are applied during opening a file, so the freeze occurs right of the start.
What's strange, it didn't happen with DT 5.2.
There was similar issue #20050 Although hardware of previous reporter is much more powerful, so the freezes may be less pronounced.

Short freeze appears along with following lines in the output
8.4097 [opencl copy_host_to_device_constant] could not allocate oversize buffer on device ‘NVIDIA CUDA Quadro K2000’ id=0: CL_SUCCESS
** 8.4097 [opencl copy_host_to_device_constant] could not allocate oversize buffer on device ‘NVIDIA CUDA Quadro K2000’ id=0: CL_SUCCESS**
** 8.4098 [opencl copy_host_to_device_constant] could not allocate oversize buffer on device ‘NVIDIA CUDA Quadro K2000’ id=0: CL_SUCCESS**
** 8.4099 [opencl copy_host_to_device_constant] could not allocate oversize buffer on device ‘NVIDIA CUDA Quadro K2000’ id=0: CL_SUCCESS**

Could you help interpret the above?
Is it a problem with GPU memory usage by OpenCL?
Is it possible to tweak by modifying some limits?
Could this be due to changes in newest DT releases?
Conflict of new DT with OpenCL packages for kernel 6.18?
It is important to understand nature of the issue before taking action - buy new GPUs, downgrade DT version or change configuration.

Steps to reproduce

1. Get the same setup
2. Launch DT in CLI in debug mode with darktable -d opencl
3. While looking at stdout, open RAW file with DT and perform operation with one of OpenCL accelerated modules

Expected behavior

No freezes

Logfile | Screenshot | Screencast

Debug output:

darktable -d opencl
darktable 5.5.0~git41.fa8b49d6-1+13547.1
Copyright (C) 2012-2025 Johannes Hanika and other contributors.

Compile options:
Bit depth → 64 bit
Exiv2 → 0.27.5
Lensfun → 0.3.2
Debug → DISABLED
SSE2 optimizations → ENABLED
OpenMP → ENABLED
OpenCL → ENABLED
Lua → ENABLED - API version 9.6.0
Colord → ENABLED
gPhoto2 → ENABLED
OSMGpsMap → ENABLED - map view is available
GMIC → ENABLED - Compressed LUTs are supported
GraphicsMagick → ENABLED
ImageMagick → DISABLED
libavif → DISABLED
libheif → DISABLED
libjxl → DISABLED
LibRaw → ENABLED - Version 0.22.0-PreRC1
OpenJPEG → ENABLED
OpenEXR → ENABLED
WebP → ENABLED

See resources | darktable for detailed documentation.
See Sign in to GitHub · GitHub to report bugs.

0.0001 [dt starting]

darktable -d opencl
0.2614 [dt_dlopencl_init] could not find default opencl runtime library ‘libOpenCL’
0.2615 [dt_dlopencl_init] could not find default opencl runtime library ‘libOpenCL.so’
0.2618 [opencl_init] opencl library ‘libOpenCL.so.1’ found on your system and loaded, preference ‘default path’
0.2958 [opencl_init] found 1 platform
[opencl_init] found 2 devices

[dt_opencl_device_init]
DEVICE: 0: ‘Quadro K2000’
CONF KEY: cldevice_v5_nvidiacudaquadrok2000
PLATFORM, VENDOR & ID: NVIDIA CUDA, NVIDIA Corporation, ID=4318
CANONICAL NAME: nvidiacudaquadrok2000
DRIVER VERSION: 470.256.02
DEVICE VERSION: OpenCL 3.0 CUDA, SM_20 SUPPORT
DEVICE_TYPE: GPU, dedicated mem
GLOBAL MEM SIZE: 1991 MB
MAX MEM ALLOC: 498 MB
MAX IMAGE SIZE: 16384 x 16384
MAX CONSTANT BUFFER: 64 KB
ADDRESS ALIGN: 512
MAX WORK GROUP SIZE: 1024
MAX WORK ITEM DIMENSIONS: 3
MAX WORK ITEM SIZES: [ 1024 1024 64 ]
ASYNC PIXELPIPE: NO
PINNED MEMORY TRANSFER: NO
AVOID ATOMICS: NO
MICRO NAP: 250
ROUNDUP WIDTH & HEIGHT 16x16
CHECK EVENT HANDLES: 128
TILING ADVANTAGE: 61427584.000
DEFAULT DEVICE: NO
KERNEL BUILD DIRECTORY: /usr/share/darktable/kernels
KERNEL DIRECTORY: /home/alex/.cache/darktable/cached_v5_kernels_for_NVIDIACUDAQuadroK2000_47025602
CL COMPILER OPTION: -cl-fast-relaxed-math
CL COMPILER COMMAND: -w -cl-fast-relaxed-math -DNVIDIA_SM_20=1 -DNVIDIA=1 -I"/usr/share/darktable/kernels"
CL EXCEPTION: DT_OPENCL_ONLY_CUDA
KERNEL LOADING TIME: 0.0404 sec

[dt_opencl_device_init]
DEVICE: 1: ‘Quadro K2000’
CONF KEY: cldevice_v5_nvidiacudaquadrok2000
PLATFORM, VENDOR & ID: NVIDIA CUDA, NVIDIA Corporation, ID=4318
CANONICAL NAME: nvidiacudaquadrok2000
DRIVER VERSION: 470.256.02
DEVICE VERSION: OpenCL 3.0 CUDA, SM_20 SUPPORT
DEVICE_TYPE: GPU, dedicated mem
GLOBAL MEM SIZE: 2000 MB
MAX MEM ALLOC: 500 MB
MAX IMAGE SIZE: 16384 x 16384
MAX CONSTANT BUFFER: 64 KB
ADDRESS ALIGN: 512
MAX WORK GROUP SIZE: 1024
MAX WORK ITEM DIMENSIONS: 3
MAX WORK ITEM SIZES: [ 1024 1024 64 ]
ASYNC PIXELPIPE: NO
PINNED MEMORY TRANSFER: NO
AVOID ATOMICS: NO
MICRO NAP: 250
ROUNDUP WIDTH & HEIGHT 16x16
CHECK EVENT HANDLES: 128
TILING ADVANTAGE: 61427584.000
DEFAULT DEVICE: NO
KERNEL BUILD DIRECTORY: /usr/share/darktable/kernels
KERNEL DIRECTORY: /home/alex/.cache/darktable/cached_v5_kernels_for_NVIDIACUDAQuadroK2000_47025602
CL COMPILER OPTION: -cl-fast-relaxed-math
CL COMPILER COMMAND: -w -cl-fast-relaxed-math -DNVIDIA_SM_20=1 -DNVIDIA=1 -I"/usr/share/darktable/kernels"
CL EXCEPTION: DT_OPENCL_ONLY_CUDA
KERNEL LOADING TIME: 0.0351 sec
[opencl_init] OpenCL successfully initialized. internal numbers and names of available devices:
[opencl_init] 0 ‘NVIDIA CUDA Quadro K2000’
[opencl_init] 1 ‘NVIDIA CUDA Quadro K2000’
0.4540 [opencl_init] FINALLY: opencl PREFERENCE=ON is AVAILABLE and ENABLED.
[opencl_init] opencl_scheduling_profile: ‘multiple GPUs’
[opencl_init] opencl_device_priority: ‘/!0,///!0,*’
[opencl_init] opencl_mandatory_timeout: 1000
[opencl_update_priorities] these are your device priorities:
[opencl_update_priorities] image preview export thumbs preview2
[dt_opencl_update_priorities] 0 0 0 0 0
[dt_opencl_update_priorities] 1 1 1 1 1
[opencl_update_priorities] show if opencl use is mandatory for a given pixelpipe:
[opencl_update_priorities] image preview export thumbs preview2
[opencl_update_priorities] 0 0 0 0 0
[opencl_synchronization_timeout] synchronization timeout set to 20
[opencl_update_priorities] these are your device priorities:
[opencl_update_priorities] image preview export thumbs preview2
[dt_opencl_update_priorities] 0 0 0 0 0
[dt_opencl_update_priorities] 1 1 1 1 1
[opencl_update_priorities] show if opencl use is mandatory for a given pixelpipe:
[opencl_update_priorities] image preview export thumbs preview2
[opencl_update_priorities] 0 0 0 0 0
[opencl_synchronization_timeout] synchronization timeout set to 20

darktablerc configuration:
cat $HOME/.config/darktable/darktablerc | grep -e opencl -e nvid
cldevice_v5_nvidiacudaquadrok2000=0 250 0 16 16 128 0 0 0.000 61427584.000 0.250
cldevice_v5_nvidiacudaquadrok2000_building=-cl-fast-relaxed-math
cldevice_v5_nvidiacudaquadrok2000_id0=600
cldevice_v5_nvidiacudaquadrok2000_id1=600
clplatform_intelropenclhdgraphics=FALSE
clplatform_nvidiacuda=TRUE
clplatform_openclon12=FALSE
opencl=TRUE
opencl_checksum=1654065287
opencl_device_priority=/!0,///!0,*
opencl_library=
opencl_mandatory_timeout=1000
opencl_scheduling_profile=multiple GPUs
opencl_tune_headroom=TRUE

Settings:
darktable resources: large
Activate OpenCL support: ON
OpenCL scheduling profile: multiple GPUs
tuned GPU memory: ON

Commit

No response

Where did you obtain darktable from?

darktable.org / GitHub release

darktable version

darktable 5.5.0~git41.fa8b49d6-1+13547.1

What OS are you using?

Linux

What is the version of your OS?

Ubuntu 22.04

Describe your system

My setup is as follows:
Linux kernel 6.8.0-90-generic
Ubuntu 22.04
2x NVIDIA CUDA Quadro K2000
CPU: Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
~ cat /etc/OpenCL/vendors/nvidia.icd
libnvidia-opencl.so.1

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

2x NVIDIA CUDA Quadro K2000, driver Nvidia 470

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

[agat114]

Please find the log of DT run with -d perf and -d opencl options. In this log I had opened RAW file and started clicking initial processing modules - raw chromatic aberrations, lens correction, sharpen. Freezes were pronounced ~1-2sec.
dt_perf_opencl_stdout.txt

I keep on testing various options, not covered by above run. So, good news I got rid of the freezes. They happened when I used VNG4 demosaic, as default DT algorithm. After I switched to AMAZE, RCD or PPG, there were no more freezes. With RCD (dual) the freezes were back, but less pronounced.

[jenshannoschwalm]

You mean like in these patterns?

    38.8505 [dev_pixelpipe] took 0.421 secs (3.281 CPU) [full] processed `cacorrect' on CPU, blended on CPU
    40.3171 [opencl copy_host_to_device_constant] could not allocate oversize buffer on device 'NVIDIA CUDA Quadro K2000' id=1: CL_SUCCESS
    40.3172 [opencl copy_host_to_device_constant] could not allocate oversize buffer on device 'NVIDIA CUDA Quadro K2000' id=1: CL_SUCCESS
    40.3173 [opencl copy_host_to_device_constant] could not allocate oversize buffer on device 'NVIDIA CUDA Quadro K2000' id=1: CL_SUCCESS
    40.3173 [opencl copy_host_to_device_constant] could not allocate oversize buffer on device 'NVIDIA CUDA Quadro K2000' id=1: CL_SUCCESS
    40.3182 [resample_cl] plan 0.000 secs (0.000 CPU) resample 0.001 secs (0.001 CPU)
    40.4236 [dev_pixelpipe] took 1.573 secs (1.607 CPU) [full] processed `demosaic' on GPU, blended on GPU

1. As mentioned in a closed issue the "could not allocate oversize ..." reports are internal and nothing to worry about.
2. I am pretty sure you are running with capture sharpen active, here it autocalculates the radius - that's a) done once per image b) is currently processed on CPU so pretty slow
   3, always report with -d pipe

[jenshannoschwalm]

BTW most of that has already been explained before in the issue you mentioned.

[agat114]

@jenshannoschwalm The message is not a problem. The freezing is. Could you answer my questions about the freeze?
Is it a problem with GPU memory usage by OpenCL?
Is it possible to tweak by modifying some limits?
Could this be due to changes in newest DT releases?
Conflict of new DT with OpenCL packages for kernel 6.18?
It is important to understand nature of the issue before taking action - buy new GPUs, downgrade DT version or change configuration.

[jenshannoschwalm]

1. I told you what is likely the reason for "stuttering": the radius calculation
2. I told you also about required -d pipe option to confirm.
3. So you provide a log and i can tell you.

[agat114]

1. I told you what is likely the reason for "stuttering": the radius calculation

   2. I told you also about required -d pipe option to confirm.

   3. So you provide a log and i can tell you.

Thanks, I agree with your suggestions.
I am providing logs generated with -pipe option.
In the meantime I tried various options and it turs out these freezes happen when VNG4 and RCD (dual) algorithms are used. Once I selected different demosaicing algorithm (e.g AMAZE), they were gone.
Attaching two files where one shows using VNG4 and other AMAZE with rest of the algorithms.

dt_perf_opencl_pipe_demosaic_amaze_and_other_stdout.txt
dt_perf_opencl_pipe_demosaic_vng_stdout.txt

[agat114]

@jenshannoschwalm Feel free to ask for other tests.

[jenshannoschwalm]

@agat114 it's a bit of wasting my time reading your logs, i asked for -d pipe option when logging :-(

And we have the very good manual :-)

[agat114]

@agat114 it's a bit of wasting my time reading your logs, i asked for -d pipe option when logging :-(

And we have the very good manual :-)

It's my fault. I was tired and missed the option. Now with pipe.

dt_perf_opencl_pipe_demosaic_amaze_and_other_stdout.txt
dt_perf_opencl_pipe_demosaic_vng_stdout.txt

[jenshannoschwalm]

Nothing unusual there. demosaic does tiling code i suppose, you could confirm that by adding the -d verbose option. BTW dual demosaicing is only rarely helping, mostly if you find maze patterns like on snow, VNG is more stable for this. Otherwise, RCD is your friend.