View Issue Details

IDProjectCategoryView StatusLast Update
0000575Cinelerra-GG[All Projects] Featurepublic2021-06-30 06:55
Reporterfary54 Assigned To 
PrioritynormalSeverityminorReproducibilityhave not tried
Status newResolutionopen 
PlatformX86_64OSDebianOS Version10.4
Product Version 
Target VersionFixed in Version 
Summary0000575: How to optimize the rendering speed of Cinelerra GG ?
DescriptionIn accordance with @Andrea_Paz's last post on BT574 I have decided to continue the discussion by opening this new BT.
The objective is to allow all potential users to benefit from the particular interest of this process that I use permanently.

You will find attached the following documents:
- The procedure
- The Bash script
TagsNo tags attached.

Activities

fary54

fary54

2021-06-30 06:55

reporter   ~0004871

@Andrea_Paz

"Do you use CinGG installed from binary for your distro?"
Personally, I don't compile CinelerraGG, I use the cinelerra appimage version 20210531x86_64-older_distos provided by the CinelerraGG website.

"I then had the problem of not finding the /tmp/fab-render folder; I think due to not being root. I then tried changing the WORKING DIRECTORY to a path pointing to my /home. Everything seems OK but there is still an action pointing to /tmp/fab-render that I can't find. As a result the render farm still fails. I'll do more testing"
About /tmp/fab-render, it only appears in 2 places:
-on the 13th line of the fab-render.py script
-on the 241st line of the fab-render.sh script

"no need to waste your time on my inability to get the script to work. I can't thank you enough."
You don't need to thank me, because I'm happy to do it. I have always tried, at my level, to exploit the different possibilities of the software, and this to adapt it to my needs. If on top of that it allows other users in the community to use it, that's the icing on the cake...

"I would like others to try the script as well to see if it works for them.
You're right, feedback from other users would be useful.
Andrea_Paz

Andrea_Paz

2021-06-29 12:48

manager   ~0004870

@fary54
Do you use CinGG installed from binary for your distro? I compile the source without being root. I had to put the path to the command "cin -d $n" otherwise it wouldn't recognize the command.
I then had the problem of not finding the /tmp/fab-render folder; I think due to not being root. I then tried changing the WORKING DIRECTORY to a path pointing to my /home. Everything seems OK but there is still an action pointing to /tmp/fab-render that I can't find. As a result the render farm still fails. I'll do more testing; no need to waste your time on my inability to get the script to work. I can't thank you enough. I would like others to try the script as well to see if it works for them.
fary54

fary54

2021-06-29 06:54

reporter   ~0004869

@Andrea_Paz

It's good that you figured out how to adapt the script to use avidemux3_cli.
Personally, knowing only Debian, I didn't even imagine that other distributions could package Avidemux with a different name. I really thought there was a standard for this.

"render farm start failed".
I also got this error message several times and even now it happens to me.
For me, in all cases, it was a simple data encoding error on my part.
In fact, there are several reasons for this message:
- You forget to check or uncheck renderfarm in CinelerraGG.
- Not respecting the render name, path or extension. The name, path and extension must match exactly the name of the first pipe created by my script.
- Other reasons, such as the declaration of nodes in cinelerraGG, etc.
For these reasons my script constantly reminds the data that needs to be changed in cinelerraGG.

"I remember when I tried the render farm I had to manually create every single "cin -d ..." because putting it in a script didn't work for me"
I don't understand why. To be sure, as soon as you run my script, type this command without quotes in a terminal.
 "lsof -nP -i tcp:1030-$((1030 + 1130))"
If you get the five lines containing "cin", like my attached print screen, it's ok.

I hope my answers will help you solve the problems you encountered. Let me know if it works or if you need more explanation.

Cin -d listing.jpg (64,408 bytes)
Cin -d listing.jpg (64,408 bytes)
fary54

fary54

2021-06-29 06:43

reporter   ~0004866

@Andrea_Paz

First of all a big thank you for your compliments on my video. Let's move on to your questions.

"I was not setting the nodes in CinGG/renderfarm because I thought (stupidly) that the script was doing it."
Indeed, you have to create the renderfarm in cinelerraGG because my script only creates a link between the CinelerraGG renderfarm and the ffmpeg renderfarm, which is created by the script. That's kind of the secret of the rendering speed I get.

"The thing is that in Arch linux the program is called avidemux-cli and so the links in your script don't fit. I think especially in fab-render.py, where I don't know where to look."
From your second message, I see that you have solved this problem.

"Does DIRECTORY_OF_THIS_SCRIPT have to be set by me (by replacing dirname)? How does it work?"
Normally, you don't have to do anything, it's automatic.
$(dirname $0) is simply a bash command that returns the directory address of the active script. [...]

I hope I've answered your questions.
Andrea_Paz

Andrea_Paz

2021-06-28 17:55

manager   ~0004865

After finding that in Arch the binary of avidemux is: avidemux3_cli, the script starts without errors. But I get stuck at the start of the render farm with the error
"render farm start failed".
I remember when I tried the render farm I had to manually create every single "cin -d ..." because putting it in a script didn't work for me. I think that problem has remained.
Andrea_Paz

Andrea_Paz

2021-06-28 12:26

manager   ~0004864

Hi,
sorry for the delay of the answer. Thanks to your video I realized where I was wrong previously with your first script: I was not setting the nodes in CinGG/renderfarm because I thought (stupidly) that the script was doing it.
Also the new script doesn't work for me because it doesn't find avidemux. The thing is that in Arch linux the program is called avidemux-cli and so the links in your script don't fit. I think especially in fab-render.py, where I don't know where to look.
One question: does DIRECTORY_OF_THIS_SCRIPT have to be set by me (by replacing dirname)? How does it work?

Anyway I thank you for all the explanations and for the video that is very clear and well done.
fary54

fary54

2021-06-26 07:20

reporter   ~0004863

Video tutorial on how to use my script.

You will find attached the script and its video tutorial to show you how it works. It wasn't that easy, because I had to respect the Cinelerra GG site's limit of 4.77 MB maximum per file, but I still managed to do it. For the same reason, as you will see, the rendering time of the video part had to be accelerated.

The final result is composed of two video files, one mixed by avidemux and the other by ffmpeg:
    -final_via_avidemux.mkv
    -final_via_ffmpeg(en_test).mkv
The official final version is the avidemux version because the ffmpeg version is currently under test.
Of course, if the test is conclusive, I will keep both processing systems for safety. You never know.

@Andrea_Paz
As promised, I am sending you the video tutorial which I hope will make it easier for you to understand my script.
I dare to hope in any case, that you will be able to use it for your next projects.
Of course, I remain at your disposal if you need me.

fab-render-2.sh (8,972 bytes)
fab-render-2.py (639 bytes)
fab-render_tuto.webm (4,691,189 bytes)
fary54

fary54

2021-06-21 13:27

reporter   ~0004851

@Andrea_Paz

First of all, many thanks for your information about ffmpeg. I will definitely use it on a case by case basis.

"mkv slows down my playback and rendering so much. Have you tried using .mov instead of .mkv? Are there any differences?
Interesting comment. I modified my script to try both ( mkv and mov). Personally I didn't notice any difference, the result is the same. The processing is not slower or faster.

About the video tutorial, I haven't started it yet but I will post it as soon as possible.
Andrea_Paz

Andrea_Paz

2021-06-21 11:40

manager   ~0004850

Last edited: 2021-06-21 11:44

View 2 revisions

Thank you very much for the scripts and explanations.
Unfortunately your scripts are beyond my ability to understand (which is close to 0). I will gladly wait for the video tutorial.
A small note: with ffmpeg you can merge a video track with an audio track using the -map option. e.g.:

ffmpeg -i video.mp4 -i audio.aac -c:v copy -c:a copy -map 0:v:0 -map 1:a:0 output.mp4

or

ffmpeg -i video.mp4 -i audio.wav -c:v copy -c:a aac -map 0:v:0 -map 1:a:0 output.mp4

This way ffmpeg uses the first video track (v:0) of the first stream 0 (excluding the rest) and the first audio track (a:0) of the second stream 1.

Another small curiosity: mkv slows down my playback and rendering so much. Have you tried using .mov instead of .mkv? Are there any differences?

fary54

fary54

2021-06-21 07:21

reporter   ~0004849

New version of my script.

I discovered a bug in the display of a zenity box and fixed it.

I also took advantage of this bug resolution to make some improvements to my script:
    - Cleaning up the code and limiting the length of the lines to make it more understandable.
    - Giving more detailed names to variables and functions. This reduces unnecessary comments and makes the code more accessible.
    - Added a function to test the presence of the packages needed for my script to work.
    - Added an automatic audio/video mixing function. Indeed, as @Andrea_Paz pointed out to me, my script did not manage this mixing automatically. So I looked for and modified the script so that this mixing is done automatically and in two different ways, either via avidemux and/or via ffmpeg.

This script greatly improves the ease of rendering and, for my personal use, is indispensable. Indeed, it brings automation to all my films which all have to keep a well defined format.
 
As soon as I have some time, I will post a video tutorial to show how this script works.

fab-render.py (639 bytes)
fab-render.sh (8,893 bytes)
fary54

fary54

2021-06-11 07:12

reporter   ~0004806

First of all, many thanks for your positive comments.

"The mknod command"
The mknod command does not create a file but a pipe. This pipe creates a link between ffmpeg and the cinelerra nodes. You can see that the size of the pipe in question does not increase.

"Is cin -r never used"
Yes, "cin -r" is never used. But I do use "cin -d" and I keep a Cinelerra as a master always open. I use it as a server to feed the Cinelerra nodes (cin -d). These nodes in turn feed the ffmpeg(s) via the pipes. I did try to use the "cin -r" command, which should improve performance.
But this command does not transfer the data correctly to ffmpeg. The system hangs at the pipe which is mistakenly considered as a file.
I don't know if this is a bug or if it was never implemented.

"Avidemux when is it used"
Indeed, avidemux is not in my script.
My script works in two steps
-First, it encodes the video alone in a file, using the render farm.
-Secondly, it encodes only the audio in a file, without using the render farm.


The mixing is then to be done manually with Avidemux because until now I still haven't found how to do an automatic mixing with ffmpeg. Indeed, I always have an audio/video lag in playback on a home cinema.
If anyone knows the answer? More work to do...

"There is probably a huge difference in power between Ryzen and Threadripper"
I don't know about your processor but this script greatly improves the rendering speed with the disadvantage of increasing the temperature of the said processor. Moreover, my movies are at least 60 minutes long (finished) and I often use a lot of plugins. These two elements together mean that my processor does not have time to breathe.

"So I will have to make several adaptations to your script, but I don't know if I will be able to"
Let me know if you need more information.

"This script could be the subject of an additional section of the manual, following yours already in the appendix."
I am not against the principle of sharing my method with all users.
Andrea_Paz

Andrea_Paz

2021-06-10 13:36

manager   ~0004805

Awesome work, my compliments!
It will take me a long time to study your script, given my ignorance. At first glance I don't really understand the mknod command, it creates a file of what type? Is "cin -r" never used? Avidemux when is it used?

There is probably a huge difference in power between Ryzen and Threadripper. Using the renderfarm with 16 threads active on 16 (at 100%) for about 20 min, the temperature always stays below 70°C with only a few spikes towards 73-74°C. I use stock air cooling, however I must say I have taken good care of the airflow inside the case.
So I will have to make several adaptations to your script, but I don't know if I will be able to. I will let you know. Anyway thank you and congratulations for the great solution you found.

PS: This script could be the subject of an additional section of the manual, following yours already in the appendix.
fary54

fary54

2021-06-10 06:42

reporter  

rendu_en_h264_via_pipes_parallélisés.sh (6,357 bytes)
Procedure.pdf (41,433 bytes)

Issue History

Date Modified Username Field Change
2021-06-10 06:42 fary54 New Issue
2021-06-10 06:42 fary54 File Added: rendu_en_h264_via_pipes_parallélisés.sh
2021-06-10 06:42 fary54 File Added: Procedure.pdf
2021-06-10 13:36 Andrea_Paz Note Added: 0004805
2021-06-11 07:12 fary54 Note Added: 0004806
2021-06-21 07:21 fary54 File Added: fab-render.py
2021-06-21 07:21 fary54 File Added: fab-render.sh
2021-06-21 07:21 fary54 Note Added: 0004849
2021-06-21 11:40 Andrea_Paz Note Added: 0004850
2021-06-21 11:44 Andrea_Paz Note Edited: 0004850 View Revisions
2021-06-21 13:27 fary54 Note Added: 0004851
2021-06-26 07:20 fary54 File Added: fab-render_tuto.webm
2021-06-26 07:20 fary54 File Added: fab-render-2.sh
2021-06-26 07:20 fary54 File Added: fab-render-2.py
2021-06-26 07:20 fary54 Note Added: 0004863
2021-06-28 12:26 Andrea_Paz Note Added: 0004864
2021-06-28 17:55 Andrea_Paz Note Added: 0004865
2021-06-29 06:43 fary54 Note Added: 0004866
2021-06-29 06:54 fary54 File Added: Cin -d listing.jpg
2021-06-29 06:54 fary54 Note Added: 0004869
2021-06-29 12:48 Andrea_Paz Note Added: 0004870
2021-06-30 06:55 fary54 Note Added: 0004871