View Issue Details

IDProjectCategoryView StatusLast Update
0000215Cinelerra-GG[All Projects] Featurepublic2020-10-05 01:17
ReporterAndrew-R Assigned To 
PrioritynormalSeverityminorReproducibilityhave not tried
Status newResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0000215: OpenCL support via ffmpeg
DescriptionIt seems ffmpeg already have some filters utilizing openCL (even running arbitrary CL program).
See my mail https://lists.cinelerra-gg.org/pipermail/cin/2019-May/000650.html
Steps To Reproduceadd two hacks in usual places (thirdparty/Makefile and global_config)

system_libs += -lOpenCL for global_config
and
--enable-opencl for ffmpeg.cfg_params in thirdparty/Makefile

I don't have OpenCL-capable drivers for my again NVIDIA Corporation G92 [GeForce 8800 GS] (nv50-generation card), but new-ish AMD and Intel should work to some degree ....?

Do not forgot to install opencl-headers. I used icd loader coming with Mesa, but thing like ROCm and Beignet may require their own loader files.

https://github.com/RadeonOpenCompute/ROCm - overview advanced OpenCL for AMD (integrated in mesa clover state tracker limited to OpenCL 1.1 at the moment)

https://github.com/intel/beignet - Intel thing.
TagsNo tags attached.

Activities

Andrew-R

Andrew-R

2020-10-05 01:17

reporter   ~0004176

Here is bug for Image support in Mesa/clover:
https://gitlab.freedesktop.org/mesa/mesa/-/issues/130

and from there linked git repo with _some_ support for radeonsi driver( new-ish AMD cards - RX-something)
https://gitlab.freedesktop.org/awatry/mesa/-/commits/clover-image-support/
Andrew-R

Andrew-R

2020-10-01 06:09

reporter   ~0004095

Oh, well, there is now OpenCL support via llvmpipe (but you need llvm (I use llvm-10), mesa git, libclc git (currently lives in llvm.git), SPIRV-LLVM-Translator (branch depend on llvm-version in use) and spirv-tools (for me on Slackware this provided by vulkan-sdk package), all compiled in correct order from correct branches.)

Mesa 20.3.0-devel (git-8dc8922af2) , llvmpipe (LLVM 10.0.1, 256 bits)
plus slightly edited patches:
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4974
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6040

Unfortunately, images support still not finished, so ffmpeg (4.3+git) fails with error message.

uest@slax:/dev/shm/ffmpeg$ LP_CL=1 CLOVER_DEVICE_VERSION_OVERRIDE=1.2 CLOVER_DEVICE_CLC_VERSION_OVERRIDE=1.2 ./ffmpeg -hwaccel opencl -init_hw_device opencl=gpu -filter_hw_device gpu -i ~/6029.mp4 -vf "format=nv12, hwupload, avgblur_opencl=10, hwdownload" -f null /dev/null
ffmpeg version N-98258-g3bf5cc9c58 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 5.5.0 (GCC)
  configuration: --enable-opencl
  libavutil 56. 60.100 / 56. 60.100
  libavcodec 58.108.100 / 58.108.100
  libavformat 58. 59.100 / 58. 59.100
  libavdevice 58. 11.102 / 58. 11.102
  libavfilter 7. 87.100 / 7. 87.100
  libswscale 5. 8.100 / 5. 8.100
  libswresample 3. 8.100 / 3. 8.100
[AVHWDeviceContext @ 0xb9b2480] QSV to OpenCL mapping not usable.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/home/guest/6029.mp4':
  Metadata:
    major_brand : mp42
    minor_version : 0
    compatible_brands: mp42isomavc1
    creation_time : 2011-06-24T09:52:37.000000Z
    encoder : HandBrake 0.9.5 2011043000
  Duration: 01:28:31.60, start: 0.000000, bitrate: 789 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, smpte170m/smpte170m/bt709), 720x400, 657 kb/s, 25 fps, 25 tbr, 90k tbn, 180k tbc (default)
    Metadata:
      creation_time : 2011-06-24T09:52:37.000000Z
      encoder : JVT/AVC Coding
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default)
    Metadata:
      creation_time : 2011-06-24T09:52:37.000000Z
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
[AVHWDeviceContext @ 0xb9b2480] Failed to query supported image formats: -30.
[Parsed_hwupload_1 @ 0xbd257c0] Query format failed for 'Parsed_hwupload_1': Invalid argument
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!

I looked for this error message online and I can see it here:
https://lists.ffmpeg.org/pipermail/ffmpeg-cvslog/2017-November/111296.html

clinfo says:
 Image support No

There is interesting blogpost about adding openCL support to ffmpeg's filtering framework:
https://cldfire.dev/blog/writing-an-opencl-filter-for-ffmpeg/

But right now I'll wait for more OpenCL work to land in Mesa.
Andrew-R

Andrew-R

2020-09-17 12:23

reporter   ~0004029

Ah, obviously OpenGL/OpenCL sharing from 11 years old spec STILL not implemented on Linux :}

https://github.com/intel/compute-runtime/issues/166
"PiotrRozenfeld commented Sep 14, 2020
While we did lay some groundwork for cl_khr_gl_sharing with MESA on Linux, we also had to increase our initial effort estimates, as additional work to be done in MESA and Compute before we can proceed with implementation.
 This does not mean we abandon the effort, but we can no longer accommodate that in the near future.
We'll post updates when the work will resume."

so, not yet on (open) driver side ....

Simple OpenCL example ...
https://www.eriksmistad.no/gaussian-blur-using-opencl-and-the-built-in-images-textures/
Andrew-R

Andrew-R

2020-09-17 11:52

reporter   ~0004028

Well, it seems ffmpeg filters require OpenCL 1.2, while Mesa's state tracker only provide 1.1 ?

https://trac.ffmpeg.org/wiki/HWAccelIntro
" At run-time, an OpenCL 1.2 driver is required - most GPU manufacturers will provide one as part of their standard drivers. CPU implementations are also usable, but may be slower than using native filters in ffmpeg directly."
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4974 - not yet merged (and my nv50-era cards only will have at best 1.1, it seems, due to hw limitations)

Also, found interesting archived article on Final Cut Pro X , (Mac OS NLE), from 2011:
https://web.archive.org/web/20110715000927/http://www.cnet.com.au/apple-final-cut-pro-x-339318183.htm
It lists OpenCL as part of then-new engine features ..... on cards/drivers from that era

I also downloaded OpenCL 1.0 specification, it talks about improved accuracy in openCL as compared toOpenGL, and interoperability between two.
https://www.khronos.org/registry/OpenCL/specs/opencl-1.0.pdf
[also done on Mac OS X, according to document info :}]

"7.4 Relative Error as ULPs
In this section we discuss the maximum relative error defined as ulp (units in the last place).
Addition, subtraction, multiplication, fused multiply-add and conversion between integer and a
floating-point format are IEEE 754 compliant and are therefore correctly rounded. Conversion
between floating-point formats and explicit conversions specified in section 6.2.3 must be
correctly rounded."

"9.11 Creating CL context from a GL context or share group" and "9.12 Sharing Memory Objects with OpenGL / OpenGL ES Buffer, Texture and Renderbuffer Objects"

So, in theory one can reuse OpenGL infrastructure in cinGG, just add OpenCL kernels and switches into codepaths. IIUC compute units still separate from GL pipe units, so potentially they can be not just more accurate, but faster.
MatN

MatN

2019-06-25 19:32

reporter   ~0001781

Thanks Andrew. It turned out the ffmpeg on my Mint 19.1 XFCE was too old (version 3.4.6). The
¨ffmpeg -hide_banner -v verbose -init_hw_device list¨ output was
Supported hardware device types: vdpau, cuda, vaapi, drm

When I used Fedora 30 workstation with ffmpeg installed via RPM Fusion, the ffmpeg version was 4.1.3 , and the supported hardware devices vdpau, cuda, vaapi, qsv, drm, opencl.

¨ffmpeg -hide_banner -v verbose -init_hw_device opencl¨ output:
[AVHWDeviceContext @ 0x56010e3fd440] 0.0: Clover / AMD RAVEN (DRM 3.30.0, 5.1.6-300.fc30.x86_64, LLVM 8.0.0)
[AVHWDeviceContext @ 0x56010e3fd440] 1.0: Portable Computing Language / pthread-AMD Ryzen 5 2400G with Radeon
and a third one which fails for the i915 driver.

This matches the output of ¨clinfo¨, don´t know why the i915 driver is here on this AMD machine.
¨ffmpeg -hide_banner -v verbose -init_hw_device opencl:0:0¨ ,
or the equivalent 1:0 for the POCL software implementation gives an good reply.
So any support for hardware acceleration in cin-gg should allow selecting one of possible multiple openCL implementations present, similar to what is done for playback acceleration ¨HW device¨ .

Now the ¨ffmpeg -filters | grep opencl¨ worked, and gave me:
... avgblur_opencl V->V Apply average blur filter
 ... boxblur_opencl V->V Apply boxblur filter to input video
 ... convolution_opencl V->V Apply convolution mask to input video
 ... dilation_opencl V->V Apply dilation effect
 ... erosion_opencl V->V Apply erosion effect
 ... overlay_opencl VV->V Overlay one video on top of another
 ... prewitt_opencl V->V Apply prewitt operator
 ... program_opencl |->V Filter video using an OpenCL program
 ... roberts_opencl V->V Apply roberts operator
 ... sobel_opencl V->V Apply sobel operator
 ... tonemap_opencl V->V perform HDR to SDR conversion with tonemapping
 ... unsharp_opencl V->V Apply unsharp mask to input video
 ... openclsrc |->V Generate video using an OpenCL program

There are less than in your latest. I don´t know if any of these are in frequent use for video editing.

I noticed that clinfo shows ¨max work item sizes¨ of 256x256x256 for this AMD OpenCL, and much larger for the POCL one. I have no idea if that is of consequence for video editing as done by Cin-gg.
Andrew-R

Andrew-R

2019-06-25 17:56

reporter   ~0001778

ffmpeg -filters | grep opencl

... avgblur_opencl V->V Apply average blur filter
 ... boxblur_opencl V->V Apply boxblur filter to input video
 ... colorkey_opencl V->V Turns a certain color into transparency. Operates on RGB colors.
 ... convolution_opencl V->V Apply convolution mask to input video
 ... dilation_opencl V->V Apply dilation effect
 ... erosion_opencl V->V Apply erosion effect
 ... nlmeans_opencl V->V Non-local means denoiser through OpenCL
 ... overlay_opencl VV->V Overlay one video on top of another
 ... prewitt_opencl V->V Apply prewitt operator
 ... program_opencl |->V Filter video using an OpenCL program
 ... roberts_opencl V->V Apply roberts operator
 ... sobel_opencl V->V Apply sobel operator
 ... tonemap_opencl V->V perform HDR to SDR conversion with tonemapping
 ... transpose_opencl V->V Transpose input video
 ... unsharp_opencl V->V Apply unsharp mask to input video
 ... openclsrc |->V Generate video using an OpenCL program

this is for ffmpeg-git-N-93994-gd81913e680
MatN

MatN

2019-06-25 17:39

reporter   ~0001777

I think this would be very nice to have. I tried getting a list of ffmpeg effects that can use OpenCL, but StackOverflow instructions didn´t work. Does anyone know how to get a list of effect that would potentially benefit?
I do not think we should go the proprietary Cuda route, that is useless for people with Intel or AMD hardware.
On my AMD 2400G cpu, OpenCL is version 1.1 . Version 2 should do away with the need to copy memory between CPU and GPU, which should speed up where those are on one chip (Intel and AMD).
PhyllisSmith

PhyllisSmith

2019-05-15 02:46

manager   ~0001530

This sounds like it should be at least given a test; GG should be able to eventually come up with some configuration to test this now that you have done the groundwork of what has to be done to compile it in. But no time schedule yet.

Issue History

Date Modified Username Field Change
2019-05-14 04:36 Andrew-R New Issue
2019-05-15 02:46 PhyllisSmith Note Added: 0001530
2019-06-25 17:39 MatN Note Added: 0001777
2019-06-25 17:56 Andrew-R Note Added: 0001778
2019-06-25 19:32 MatN Note Added: 0001781
2020-09-17 11:52 Andrew-R Note Added: 0004028
2020-09-17 12:23 Andrew-R Note Added: 0004029
2020-10-01 06:09 Andrew-R Note Added: 0004095
2020-10-05 01:17 Andrew-R Note Added: 0004176