View Issue Details

IDProjectCategoryView StatusLast Update
0000509Cinelerra-GG[All Projects] Featurepublic2020-10-07 21:16
Reportersge Assigned Toadministrator  
PrioritynormalSeverityfeatureReproducibilityN/A
Status closedResolutionfixed 
Product Version 
Target VersionFixed in Version2020-09 
Summary0000509: Enhanced Motion plugin
DescriptionHere I suggest a new feature: enhanced Motion plugin. The archive with modified source files of this plugin, another archive to test the new features and compare with the old plugin behavior, description of code changes, and changes for the manual are attached as separate files.

As there are too many diffs, it would not be convenient to make a patch, so I packed in the archive motion-src.tar.gz the complete motion plugin directory. It is to be unpacked in cinelerra-5.1/plugins. If one uses the new version with all the new parameters switched off (or zeroed), the plugin will function exactly as the old one. Therefore I expect, that creation of one more plugin (there exist already a few!) leaving the older one intact would be redundant.
Additional InformationThe old version of plugin "Motion" is subject to the following problem.

When tracking previous frame during a long clip, tracking may be inaccurate because of error propagation between frames. Such accumulation of errors can become significant.

But when tracking single frame, both translation and rotation, a large rotation can induce significant inaccuracies while comparing match blocks in the translation calculation. It can happen that the strongly tilted block can give the least difference from the reference one at the wrong coordinates. And the following rotation calculation will be also inaccurate
because the translation vector has been calculated incorrectly.

Ideally, the rotation search should be carried out at every checked translation position, but such an algorithm would be inacceptably slow.

I implemented another way of solving this problem.

Motion tracking calculates translation, then rotation. Both can be inaccurate because of a large rotation of the picture, but both are better than no calculation at all.

Then the motion tracking pass (translation followed by rotation) is repeated once again, starting with the motion vector and rotation angle calculated at the first pass. I call this the refinement pass.

If motion tracker (during the first pass) tends to wildly spring away because of interference of strong rotation of the picture, the new parameters "Motion noise level" and "Rotation noise level" can help.

If the user does not activate the two pass tracking feature via the new controls in the effect GUI, and leaves both noise levels at zero percent, the motion tracker will function in just the same manner as before.

How to test the new feature.

Unpack attached archive motion-test.tar.gz, run cin and open project tm1.xml from this archive (with the video clip test1.m4v, also from this archive). Open the Motion plugin dialog. Most parameters are already preset with the
values listed below.

Track translation ON, Track rotation ON
Translation search radius: 50x50%, Rotation search radius: 25
Translation block size: 52x65%
With Draw vectors ON the block should cover all the geometry objects
Translation search steps: 256, Rotation search steps: 16
Translation direction: Both
Block X: 50%, Block Y: 55%
Both Maximum offsets set to maximum
Both Settling speeds set to 0
Track single frame, Frame number: 0
Add offset from tracked frame OFF
Tracking file: any name; this file should not exist
Action: Do Nothing
Calculation: Save coords to tracking file

Set both Noise levels to zero (when testing under the old unmodified plugin version - these new parameters did not exist)
Set Two pass tracking to OFF (when testing under the old unmodified plugin version - this new parameter did not exist)

Now start playback and watch on the match box appearance. With the standard behavior (noise level zero, two pass tracking off, or under the old unmodified plugin version where these new features did not exist) you can notice that after the picture becomes strongly tilted, the geometry figures will be not exactly in the center of the drawn box, and at some time match box will even loose the right position totally. This behavior demonstrates the Motion plugin problem.

Now build the new version of the plugin (from motion-src.tar.gz), start cin, open the same project. Leave all the Motion parameters as listed above, with two exceptions:

Motion noise level: 18%
Two pass tracking ON

Rotation noise level is set to 2% in the project's xml file, but it can be zeroed equally well. Anyway, it must not be big.

Ensure deleting the tracking file from the previous run (now you can do it via the new handy pushbutton "Clear tracking file contents").

Start playback and watch on the match box appearance. Now you can notice that the geometry figures remain correctly centered in the drawn box whole time, even when the picture is tilted. This difference demonstrates the working solution.

You can also try the project tm2.xml. Test video is the same, but the match box is defined to cover the left one of the two yellow stars. Without two pass tracking and with zeroed noise level in some time the match box will catch the second star instead of the first one. The good Motion noise level not to loose the correct object is here 9-10%.

Despite somehow more complicated algorithm, the new plugin version is not slower than the older one. Actually, the new two-pass tracking is almost 2 times faster than single pass tracking with the old plugin, and the new single pass tracking - 2.5 times faster than the old one. And under the new plugin two-pass tracking is about 40% slower (not twice slower) than the single pass tracking with the same other parameter settings. It is due to certain optimizations in the calculation of the search grid.

The list of code changes I attach in the file "changes.txt".

The necessary changes for the manual I attach in the file "manual.txt".
TagsNo tags attached.

Activities

PhyllisSmith

PhyllisSmith

2020-09-30 23:53

manager   ~0004090

Georgy mods are have been included in the latest September builds. Thank you very much. I will close in a day or 2 so users can see the resolved status.
PhyllisSmith

PhyllisSmith

2020-09-17 19:33

manager   ~0004034

@sge
Thank you for your meticulous and hard work. GG has tested with your test and one he always uses as a standard, that is:
    https://www.cinelerra-gg.org/media/demos/stabilize/shaker1-orig.mp4
The improved motion plugin has been checked into GIT as provided.

@andrea
I hope you have time to do the manual updates as suggested in the email from Georgy.
Sam

Sam

2020-09-17 19:32

administrator   ~0004033

Just WOW! What a great work!
PhyllisSmith

PhyllisSmith

2020-09-17 14:54

manager   ~0004030

WOW !! GG is reading this all now.
sge

sge

2020-09-17 14:33

reporter  

motion-src.tar.gz (30,165 bytes)
motion-test.tar.gz (2,527,233 bytes)
changes.txt (8,400 bytes)
Changes in the Motion plugin source code.
=========================================

The following 2 new MotionMain class methods added.

refine_global()
refine_rotation()

The following 3 new effect parameters added. Corresponding controls also
added to the effect dialog.

NOISE_LEVEL
NOISE_ROTATION
TWOPASS

The following 3 new handy buttons added to the effect dialog.

Get current - to set tracked frame number from the current cursor position

Generate tracking file name - to suggest tracking file name based on the
current asset name (like in the Motion51 plugin)

Clear tracking file contents - to explicitly clear current tracking file
contents (before clearing the contents is saved in the file with the suffix
".old")

==================================================

Tracking translation (two pass tracking on).
--------------------------------------------

The first pass is done as before. It will do only approximate search anyway,
so subpixel search is skipped.

The differences between translated and reference frame calculated for each
match block position are compared to get the minimum and maximum difference. 
Then the new motion noise level parameter is applied to calculate noise
level as percent between min and max difference.  From the block positions
whose differences are under noise level, that motion vector is taken which
defines the smallest displacement from the best position found on the
preceding cycle.

The second pass is similar to the first one, with the following exceptions.

As reference a temporary frame is used which is produced from that frame
which was used in the first pass, but after applying translation and
rotation calculated in the first pass.

Searching is started not from Block X, Y coords set in the effect
parameters, but from X = Y = 0 (the displacements found in the first pass
have been compensated by pre-applying them to the reference frame).

Translation search radius is reduced and placed symmetrically aroung zero
coords.

Subpixel search step is performed.

Noise level is ignored.

The displacements found in the second pass are added to the first ones to
give the final refined translation vector.

If two pass tracking is off, then translation search is done just as before,
including subpixel search.  If valid displacements have been loaded from the
tracking file, then second pass is not called at all, calcuation skipped,
and the loaded motion vector used as is.

Improvement: the translation search grid is divided taking into account
translation direction (horizontal/vertical/both), not to waste CPU time for
unneeded repeated calculation for identical displacements (especially in
subpixel search where cache is not used).

Minor improvement: an additional translation vector defining zero
displacement from the best position found in the preceding cycle is added to
the set of tested positions.  Such emphasizing the probably best position
helps to utilize motion noise level efficiently.

Minor improvement: the current translation search range is reduced on each
cycle not twice, but to include +/- translation search grid cell size used
in the preceding cycle around the best position found so far.  So the
translation search can be accelerated without loosing accuracy.

Minor bugfix: in subpixel search the 2x2 pixel block was placed in such a
way that the best pixel position was not in the 2x2 block center, but on its
top left corner.  Therefore not the whole area around the best pixel was
checked with subpixel accuracy.  This behavior is corrected now.

Tracking rotation (two pass tracking on).
-----------------------------------------

The first pass is done as before. It will do only approximate search anyway,
so it is performed up to ~1 degree accuracy only.

The differences between rotated and reference frame calculated for each
angle are compared to get the minimum and maximum difference.  Then the new
rotation noise level parameter is applied to calculate noise level as
percent between min and max difference.  From the angles whose differences
are under noise level, that angle is taken which has smallest absolute
deviation from the best angle found on the preceding cycle.

The second pass is similar to the first one, with the following exceptions.

As reference a temporary frame is used which is produced from that frame
which was used in the first pass, but after applying the complete
translation and the rotation calculated in the first pass.

Searching is started not from the rotation center which was set in the
effect parameters, but from zero angle (the rotation found in the first pass
has been compensated by pre-applying it to the reference frame).

Rotation search radius is reduced and placed symmetrically aroung zero
angle.

Rotation search is performed with full accuracy.

Noise level is ignored.

The rotation angle found in the second pass is added to the first one to
give the final refined rotation.

If two pass tracking is off, then rotation search is done just as before. 
If valid angle has been loaded from the tracking file, then second pass is
not called at all, calcuation skipped, and the loaded rotation angle used as
is.

==================================================

There exist an additional source of inaccuracy. Although the translation
displacements are calculated to subpixel precision, the rotation pivot can
be defined with entire pixel accuracy only.  To compensate this discrepancy,
the "previous" and "current" frames are interchanged in comparison: the
current frame with applied translation and rotation is compared with the
unmodified previous frame, to which the rotation pivot is assigned, and the
obtained coords and angle are taken with opposite sign.  For translation, it
was in the old plugin already exactly so, but now such trick is used also
for rotation.

Minor improvement: some integer values which were calculated by integer
division (and truncated), now are calculated in floating point followed by
rounding.

Minor improvement: if the tested rotation angle is exactly zero, a tiny
angle is used instead for the difference calculation.  It is necessary to
trigger interpolation in the affine transformation engine, otherwise the
difference calculation will produce a small spike in this place.

Minor bugfix: the minimum angle limits (min_angle and MIN_ANGLE) were
defined in radian, but then used in degrees instead.  This bug lead to a lot
of excess rotation calculations wasting CPU time.

Minor improvement: an odd number of rotation search steps is used (for
example, 17 instead of 16), in order to emphasize the angle in the symmetry
center, so that it be always tested first.  It helps to utilize rotation
noise level efficiently, and also to get rid of spurious tiny rotation
angles where actually there would be no rotation at all.

Minor improvement: the current rotation range is reduced on each cycle not
twice, but to include +/- rotation step size used in the preceding cycle. 
So the rotation search can be accelerated.

Minor improvement/bugfix: the feature of adding loaded offset from tracked
frame could be used before only for translation tracking, and only once
during each instance of the motion effect (moreover, it did not work
correctly).  Now it is used also for rotation, and adding loaded offset can
be requested any times in the same effect instance.

Minor improvement/bugfix: the logics of clearing tracking file contents was
doubtful.  From other point, there may be a lot of diverse conditions under
which this file is or is not to be cleared.  Now clearing tracking file
contents is done only manually (via a special pushputton in the plugin GUI),
and never implicitly.  The old tracking file contents is saved.  But, as the
suffix ".bak" is already in use in the caching system, another suffix ".old"
is taken for this purpose.

Minor improvement: the new GUI pushbutton "Generate tracking file name" can
be used to suggest an automatic tracking file name based on the asset name
(like in the Motion51 plugin).

Minor bugfix: the .bak file left on disk after previous unsuccessful
tracking could interfere with the new tracking pass.  Now such file is
always removed before the new one comes in use.

Minor improvement: the new GUI pushbutton "Get current" reads the address of
the current timeline cursor position and sets to it the value of the
tracking frame number parameter.
changes.txt (8,400 bytes)
manual.txt (31,622 bytes)
10.9.50 Motion

The motion tracker is almost a complete application in itself. The motion
tracker tracks two types of motion: translation and rotation.  It can track
both simultaneously or one only.  It can do 1/4 pixel tracking or single
pixel tracking.  It can stabilize motion or cause one track to follow the
motion of another track.  Although the motion tracker is applied as a
realtime effect, it usually must be rendered to see useful results.  The
effect takes a long time to precisely detect motion so it is very slow.

Motion tracker works by using one region of the frame as the region to track
(Match Box).  It compares this region between 2 frames to calculate the
motion.  This region can be defined anywhere on the screen.  Once the motion
between 2 frames has been calculated, a number of things can be done with
that motion vector.  It can be scaled by a user value and clamped to a
maximum range.  It can be thrown away or accumulated with all the motion
vectors leading up to the current position.

To save time the motion result can be saved in a file for later reuse,
recalled from a previous calculation, or discarded.  The motion tracker has
a notion of 2 tracks, the master layer and the target layer.  The master
layer is where the comparison between 2 frames takes place.  The target
layer is where motion is applied either to track or compensate for the
motion in the master layer.

Motion tracking parameters:

Track translation Enables translation operations. The motion tracker tracks
X and Y motion in the master layer and adjusts X and Y motion in the target
layer.

Translation block size For the translation operations, a block is compared
to a number of neighboring blocks to find the one with the least difference. 
The size of the Match Box to search for is given by this parameter.

Translation search radius The size of the area to scan for the translation
block.

Translation search steps Ideally the search operation would compare the
translation block with every other pixel in the translation search radius. 
To speed this operation up, a subset of the total positions is searched. 
Then the search area is narrowed and rescanned by the same number of search
steps until the motion is known to 1/4 pixel accuracy.

=======*ADD*=======
Translation direction Tracking translation is possible in any direction, or
in horizontal or vertical direction only, depending on this selection.
===================

Block X, Y These coordinates determine the center of the translation block
based on percentages of the width and height of the image.  The center of
the block should be part of the image which is visible at all times.

Maximum absolute offset The amount of motion detected by the motion tracker
is unlimited if this is 100.  If it is under 100 the amount of motion is
limited by that percentage of the image size.

Motion settling speed The motion detected between every frame can be
accumulated to form an absolute motion vector.  Settling speed determines
how fast the accumulated translation fades away, and the image resettles to
its actual appearance.  If the settling speed is 0% the total absolute
vector is added to the next frame.  So every frame that is processed
accumulates the whole motion of the past.  If the settling speed is 100% the
absolute vector is cancelled completely, adding no past translation to the
next frame.  If the settling speed is intermediate between 0% and 100% the
absolute vector is downscaled by (100 - settling amount) before being added
to the next frame.

Track rotation Enables rotation operations. The motion tracker tracks
rotation in the master layer and adjusts rotation in the target layer.

======*REMOVE*=====
Rotation block size For rotation operations a single block is compared to
equally sized blocks, each rotated by a different amount.  This is the size
of the rotation block.
=======*ADD*=======
Rotation block size For rotation operations a single block is compared to
equally sized blocks, each rotated by a different amount.  This is the size
of the rotation block.  In this implementation of the Motion plugin the same
block is used to track both translation and rotation, and this parameter is
not used.  However, in some other implementations, like MotionCV, rotation
block size is specified explicitly via this extra parameter.
===================

Rotation search radius This is the maximum angle of rotation from the
starting frame the rotation scanner can detect.  The rotation scan is from
this angle counterclockwise to this angle clockwise.  Thus the rotation
search radius is half the total range scanned.

Rotation search steps Ideally every possible angle would be tested to get
the rotation.  To speed up the rotation search, the rotation search radius
is divided into a finite number of angles and only those angles compared to
the starting frame.  Then the search radius is narrowed and an equal number
of angles is compared in the smaller radius until the highest possible
accuracy is achieved.  Normally you need one search step for every degree
scanned.  Since the rotation scanner scans the rotation search radius in two
directions, you need two steps for every degree in the search radius to
search the complete range.

=======*ADD*=======
Rotation center Usually this parameter is zero, and the rotation search
range is +/- <rotation search radius>.  Otherwise, it shifts the rotation
search range to the area of angles between <rotation center> - <rotation
search radius> and <rotation center> + <rotation search radius>.

Maximum angle offset The total accumulated amount of rotation is unlimited
if this is 90 degrees.  If it is under 90, the total amount of rotation is
limited by the given maximum angle.

Rotation settling speed This parameter determines how fast the accumulated
rotation fades away.  Analogously to the motion settling speed, if it is 0%,
the total accumulated rotation is added to the next frame, and every frame
accumulates the whole rotation of the past.  If the settling speed is 100%,
no past rotation is added to the next frame.  If the settling speed is
intermediate between 0% and 100%, the amount of accumulated rotation is
downscaled by (100 - settling amount) before being added to the next frame.

Motion noise level This parameter allows to compensate such undesirable
behavior when motion tracker wildly springs away because some outlying area
of a noisy picture becomes accidentally more "similar" to the reference
picture than the right one.  If noise level is set to 0% (its normal value),
then it is not taken into account at all.  When it is set to 100%, then
similarity of the match block contents is totally ignored, and the current
motion vector will be zero.  An intermediate value between 0% and 100%
defines the proportional noise level between minimum and maximum difference
of all examined match block positions.  From all such positions whose
difference is found below noise level, that one is chosen which has the
shortest translation relative to the reference frame.

Rotation noise level This parameter has similar meaning to the motion noise
level.  It influences rotation tracking.  If set to 0% (its normal value),
it is not taken into account.  The 100% setting completely suppresses
rotation detection.  An intermediate value between 0% and 100% defines the
proportional noise level between minimum and maximum difference of all
examined rotated match blocks from that of the reference picture.  From all
rotation angles whose match block differences are found below noise level,
that one is chosen which has the minimum rotation angle relative to the
reference frame.
===================

Draw vectors When translation is enabled, 2 boxes are drawn on the frame. 
One box represents the translation block.  Another box outside the
translation block represents the extent of the translation search radius. 
In the center of these boxes is an arrow showing the translation between the
2 master frames.  When rotation is enabled, a single box the size of the
rotation block is drawn rotated by the amount of rotation detected.

=======*ADD*=======
Two pass tracking Although this parameter can be enabled at any time, it is
useful only while simultaneous tracking both translation and rotation, when
the rotation amount is large.  The normal tracking algorithm firstly
examines translation, then rotation.  However, comparison of a strongly
tilted picture with the reference one can lead to rather inaccurate
determination of the translation vector.  The subsequent determination of
rotation will also inexact because of the preceding translation error.  The
second tracking pass, if enabled, refines translation and rotation.

If motion and/or rotation noise level is set above 0% and two pass tracking
is enabled, then both noise levels affect the first tracking pass only. 
During the second tracking pass of two pass tracking both noise levels are
ignored.

It should be noted that two pass tracking is not twice slower than the usual
single pass tracking.  Actually it is only about 40% slower.  As the second
pass serves only for motion refinement, it starts from the reduced search
range for block positions and rotation angles.  And, as the first pass only
yields an approximate position, which will be later refined during the
second pass, it skips subpixel position search and examines rotation angles
with the accuracy reduced to 1 degree.
===================

======*REMOVE*=====
Track single frame When this option is used the motion between a single
starting frame and the frame currently under the insertion point is
calculated.  The starting frame is specified in the Frame number box.  The
motion calculated this way is taken as the absolute motion vector.  The
absolute motion vector for each frame replaces the absolute motion vector
for the previous frame.  Settling speed has no effect on it since it does
not contain any previous motion vectors.  Playback can start anywhere on the
timeline since there is no dependence on previous results.  We talk about
Keep shape and it is the most precise way to calculate the motion vector;
but it only works well when the object to be traced does not change along
the clip, remaining identical in shape, size and without rotation.

Track previous frame Causes only the motion between the previous frame and
the current frame to be calculated (Follow shape).  This is added to an
absolute motion vector to get the new motion from the start of the sequence
to the current position.  After every frame processed this way, the block
position is shifted to always cover the same region of the image.  Playback
must be started from the start of the motion effect in order to accumulate
all the necessary motion vectors.  This method is less precise because you
have error propagation between frames.  However, it is essential when the
object changes shape or size or rotates.
=======*ADD*=======
Track single frame When this option is used the motion between a single
starting frame and the frame currently under the insertion point is
calculated.  The starting frame is specified in the Frame number box.  The
motion calculated this way is taken as the absolute motion vector.  The
absolute motion vector for each frame replaces the absolute motion vector
for the previous frame.  Settling speed has no effect on it since it does
not contain any previous motion vectors.  Playback can start anywhere on the
timeline since there is no dependence on previous results.  We talk about
Keep shape and it is the most precise way to calculate the motion vector;
but it only works well when the object to be traced does not change along
the clip, remaining identical in shape, size and luminance.

Strong rotation of the picture can also have a bad impact on the tracking
accuracy.  If it is the case, accuracy can be significantly improved by
enabling two pass tracking.

Frame number The number of the reference frame used for motion tracking in
the "Track single frame" mode is specified in this input field.  Frame
number is set as the 0-based absolute number starting from the very
beginning of the whole timeline.  Instead of manual calculating, it is
possible to get frame number directly from the current cursor position by
pressing the "Get current" button.

Get current Pressing this handy button provides a convenient way to set the
Frame number parameter to the value of the current cursor position.

Add (loaded) offset from tracked frame This toggle can help when the tracked
object changes shape, leaves the screen area or becomes obscured by other
objects somewhere in the middle of the motion effect.  Sometimes it can help
also if a strong rotation causes the motion tracker to miss the right
tracked object and wildly spring away.

To overcome this problem, one has to define two keyframes: the first one at
the start of the motion effect, the second shortly before the problematic
location in the clip.  At the first keyframe one defines the motion effect
parameters as usual, the "Add offset" toggle being switched off.  At the
second keyframe the match block can be redefined to cover another region of
the picture which remains visible during the following part of the effect. 
The "Frame number" parameter should be set several frames before position of
this second keyframe, and the "Add offset" toggle switched on.

After defining the two keyframes (even more can be added in the similar way)
one switches at all the defined keyframes Calculation mode to "Save coords
to tracking file", Action mode to "Do Nothing", rewinds the timeline cursor
to the beginning of the motion effect, and starts playback.  The determined
motion vectors will be written to the tracking file, the "Add offset"
parameter having no effect during the save stage.

After playback finishes and all the relevant motion vectors are saved in the
tracking file, one switches at all the defined keyframes Calculation mode to
"Load coords from tracking file", Action mode to whatever one needs, rewinds
again to the beginning of the motion effect and starts playback or
rendering.  After reaching the second keyframe, where "Add offset" was
switched on, the program continues to follow the second object adding the
total motion and rotation, accumulated so far.

Of course, the "Add offset" toggle can be used in the "Track single frame"
mode only.  In the other tracking modi such a technics is unnecessary.

Track previous frame Causes only the motion between the previous frame and
the current frame to be calculated (Follow shape).  This is added to an
absolute motion vector to get the new motion from the start of the sequence
to the current position.  After every frame processed this way, the block
position is shifted to always cover the same region of the image.  Playback
must be started from the start of the motion effect in order to accumulate
all the necessary motion vectors.  This method is less precise because you
have error propagation between frames.  However, it is essential when the
object changes shape, size or luminance.  Possible inaccuracies caused by
rotation of the picture can partly be reduced by enabling two pass tracking.
===================

Previous frame same block This is useful for stabilizing jerky camcorder
footage.  In this mode the motion between the previous frame and the current
frame is calculated.  Instead of adjusting the block position to reflect the
new location of the image, like Track Previous Frame does, the block
position is unchanged between each frame.  Thus a new region is compared for
each frame.

======*REMOVE*=====
Master layer This determines the track which supplies the starting frame and
ending frame for the motion calculation.  If it is Bottom the bottom track
of all the tracks sharing this effect is the master layer.  The top track of
all the tracks is the target layer.

Calculation This determines whether to calculate the motion at all and
whether to save it to disk.  If it is Don’t Calculate the motion calculation
is skipped.  If it is Recalculate the motion calculation is performed every
time each frame is rendered.  If it is Save the motion calculation is always
performed but a copy is also saved.  If it is Load, the motion calculation
is loaded from a previous save calculation.  If there is no previous save
calculation on disk, a new motion calculation is performed.
=======*ADD*=======
Master layer This determines the track which supplies the frames for the
motion calculation.  If it is Bottom the bottom track of all the tracks
sharing this effect is the master layer.  Then the top track of all the
tracks is the target layer.  And vice versa, if the master layer is Top,
then the target layer is bottom.

Calculation This determines whether to calculate the motion at all and
whether to save it to disk.  If it is Don’t Calculate the motion calculation
is skipped.  If it is Recalculate the motion calculation is performed every
time each frame is rendered.  If it is Load, the motion vectors are loaded
from a previous save calculation.  If the previous save calculation does not
provide the data for the currently processed frames, a new motion
calculation is performed for that frames.  If the Calculation mode is Save
the motion vectors can be still loaded from disk if the previously
calculated data exist.  Otherwise, a new motion calculation is performed,
and the calculated motion vectors are saved.

Tracking file This parameter determines the name of the file where the
motion data are saved and/or loaded from.  By default it is "/tmp/motion"
but it can have any name and reside anywhere on the disk.  Tracking file is
a text file where each line contains four numbers: the absolute frame number
(0-based integer), the X and Y translation displacements of subpixel
precision (a subpixel is 1/4 pixel in size, and to give integer values of X
and Y these subpixel coords are multiplied by 4), and the rotation angle in
degrees (as a floating point value).  The user may have different tracking
files for different timeline regions, it is possible to do any manipulations
on them if necessary and even manually edit in a text editor.

Generate tracking file name This handy pushbutton suggests a tracking file
name based on the name of the current asset.  It helps to maintain more
different tracking files with unique but predictable names used in the same
project.  When tracking file name is changed (either via the pushbutton or
after entering it in the corresponding input field), the contents of that
files (both the old and the new ones) remains unchanged.

Clear tracking file contents The values and meaning of the saved motion data
depend on such a bunch of parameters, they can be even edited manually, so
one can hardly predict whether the outdated motion vectors are to be reset
or left untouched.  Therefore the existing motion data are never altered
implicitly (only newly calculated data may be added).  And, should the user
need to reset them here and right now, it is possible to clear the given
tracking file contents explicitly by pressing this pushbutton.

As protection against accident clearing tracking file contents, right before
clearing the program ensures closing the current tracking file and renames
it to the same name with the suffix ".old".  This way, the user can undo an
erroneous clearing action by manual restoring the saved file.
===================

Action Once the motion vector is known this determines whether to move the
target layer opposing the motion vector or following the motion vector.  If
it is Do Nothing the target layer is untouched.  If it is Track...  the
target layer is moved by the same amount as the master layer.  This is
useful for matching titles to objects in the frame.  If it is Stabilize... 
the target layer is moved opposite to the motion vector.  This is useful for
stabilizing an object in the frame.  The motion operations can be accurate
to single pixels or subpixels by changing the action setting.

=======*ADD*=======
As motion tracking works consistently only in the "Play every frame" mode,
the corresponding info on the current playback mode is shown in the bottom
right edge of the effect dialog.  The needed playback mode is defined in the
Preferences dialog of Cinelerra-GG main window.
===================

Secrets of motion tracking

Since it is a very slow effect, there is a method to applying the motion
tracker to get the most out of it.  First disable playback for the track to
do motion tracking on.  Then drop the effect on a region of video with some
motion to track.  Then rewind the insertion point to the start of the
region.  Set Action → Do Nothing; Set Calculation → Don’t calculate; Enable
Draw vectors.  Then enable playback of the track to see the motion tracking
areas.

Enable which of translation motion or rotation motion vectors you want to
track.  By watching the compositor window and adjusting the Block x,y
settings, center the block on the part of the image you want to track.  It
is advisable to choose elements that have evident edges in the x and y
directions because the calculations are made on these coordinates.  Then set
search radius, block size and block coordinates for translation and
rotation.

Once this is configured, set the calculation to Save coords and do test runs
through the sequence to see if the motion tracker works and to save the
motion vectors.  Next, disable playback for the track, disable Draw vectors,
set the motion action to perform on the target layer and change the
calculation to Load coords.  Finally enable playback for the track.

When using a single starting frame to calculate the motion of a sequence
(Keep Shape), the starting frame should be a single frame with the least
motion to any of the other frames.  This is rarely frame 0.  Usually it is a
frame near the middle of the sequence.  This way the search radius need only
reach halfway to the full extent of the motion in the sequence.
=======*ADD*=======
The suitable reference frame number can be defined conveniently via the "Get
current" pushbutton.
===================

If the motion tracker is used on a render farm, Save coords and previous
frame mode will not work.  The results of the save coords operation are
saved to the hard drives on the render nodes, not the master node.  Future
rendering operations on these nodes will process different frames and read
the wrong coordinates from the node filesystems.  The fact that render nodes
only visualize a portion of the timeline also prevents previous frame from
working since it depends on calculating an absolute motion vector starting
on frame 0.

====*ATTENTION*====
Here and in the following subsections I'd suggest to change the word "pass"
to the word "step" (two-pass -> two-step) in order not to mix this with the
quite other meaning of the new two pass tracking mode described above.
===================

2 pass motion tracking

The method described above is two-pass motion tracking. One pass is used
just to calculate the motion vectors.  A second pass is used to apply the
motion vectors to the footage.  This is faster than a single pass because
errors in the motion vector calculation can be discovered quickly.  This
also allows the motion tracking to use a less demanding colormodel like
RGB888 in the scanning step and a more demanding colormodel like RGB Float
in the action step.  The scanning step takes much longer than action.  This
has the disadvantage of not being practical for extremely long sequences
where some error is acceptable and the picture quality is low to begin with,
like stabilizing camcorder footage.

The slower method is to calculate the motion vectors and apply them
simultaneously.  This method can use one track as the motion vector
calculation track and another track as the target track for motion vector
actions.  This is useful for long sequences where some error is acceptable.

Preprocessing the shot

1. The motion plugin uses luminance to do its own calculations, so we can
edit the clip to enhance contrast and make it easier to calculate motion
vectors.  You can even create a copy of the monochrome clip and optimize it
for the plugin.  It lengthens the time but minimizes errors.  The saved file
can then be used for the original clip.

2. Correct lens distortion, especially if the object to be tracked moves to
the edges of the frame.

3. Study the entire shot well: if necessary, divide it into many edits, each
with its own Motion plugin.  For example, if the object to be tracked leaves
the frame or is covered by some other element or changes in shape, size or
rotation.  You can try to use the Offset Tracking technique described below.

Using blur to improve motion tracking

With extremely noisy or interlaced footage, applying a blur effect before
the motion tracking can improve accuracy.  Either save the motion vectors in
a tracking pass and disable the blur for the action pass or apply the blur
just to the master layer.  You can even use a copy of the track formed only
by the channels of Red + Green, because the channel of Blue is the noisiest. 
Another trick is to enlarge the Match Box to minimize the effect of noise.

Using histogram to improve motion tracking

A histogram is almost always applied before motion tracking to clamp out
noise in the darker pixels.  Either save the motion vectors in a tracking
pass and disable the histogram for the action pass or apply the histogram
just to the master layer.  Finally, you can use the histogram to increase
contrast.

=======*ADD*=======
Using two pass tracking to improve accuracy and stability of the algorithm

A common source of inaccuracy when tracking single frame is rotation of the
picture.  Ideally, the motion vectors should be determined by checking all
possible rotation angles at every possible match block location.  But such
an algorithm would be inacceptably slow.  Therefore the program finds for
each frame the translation vector first, and then the rotation angle.  A
strong rotation can prevent motion tracker from finding accurate translation
displacements.  Then the subsequent rotation search will be also inaccurate
because translation has been estimated not sufficiently well.  In the two
pass tracking algorithm the first, coarse estimation of translation and
rotation is followed by the second, refinement step yielding more exact
motion vectors.

Sometimes, particularly if the amount of rotation is large, the translation
tracker can lose its right object and wildly spring away.  Then the
following strategy may help.

Firstly switch two pass tracking off, Track rotation off, set Motion noise
level to zero.  Switch Draw vectors on, Action: Do Nothing, Calculation:
Save coords to tracking file.  Start playback and watch if the motion
tracker has problems finding the right displacements (you can also examine
the tracking file afterwards on the existence of discontinuities in
coordinates).  If there are such, try to increase Motion noise level, let's
say, to 10% or 20%, clear tracking file contents, repeat playback and look
if tracker does not spring away any more.  A single pass translation
tracking can be relatively fast, while tracking rotation is expensive.

When a sufficient motion noise level is found, switch Track rotation on, Two
pass tracking on, clear tracking file contents and perform the complete
translation + rotation tracking pass.  As the refinement step uses
restricted search area and ignores the noise level parameters, the result
should be more stable and accurate.
===================

Possible sources of errors

Search radius too small: the traced object moves too fast with respect to
the size of the search box set.

Search radius too large: The search box is so large that it also collects
other similar items in the frame.

Occlusions: the traced object is temporarily hidden by some other element. 
Offset tracking or splitting of the video into several homogeneous clips is
required.

Focus change: you may get errors if the object changes its focus. The video
must be divided into several homogeneous clips.

Motion Blur: blurs the object making the calculation of the motion vector
less precise.  Very little can be done.

Shape change: you can use Track previous frame or the subdivision of the
video in more homogeneous clips.

Lighting change: Contrast change can produce errors. Track previous frame or
color correction can be used to return to the initial illumination.

Tracking stabilization in action

This is an explanation of how to stabilize a video as in the case of a video
taken from a vehicle.

First select on the timeline the part of the footage you want to stabilize,
using the in and out points.  Then apply the motion effect on that part of
the video.  Select the Previous frame same block option.  That option is
recommended for stabilizing jerky camcorder footage.  Its goal is not to
follow an object.  The block stays exactly at the same place during all the
effect length.

Enlarge the block and select almost half the size of the video. Select the
Stabilize subpixel option as it will give a finer stabilization.  Reduce the
Maximum absolute offset value to limit the stabilization amplitude.  You
probably prefer to get a nonperfect stabilization on some places on the
video rather than having a very large black border on one side of the
picture during big shakes.  Set the Translation search steps value to 128. 
Increasing that value will not give a better result and only considerably
increases the rendering time.  Make sure the Draw vectors option is
selected, and render the part of the video where the motion effect is
applied.

If the result is good, deselect the Draw vectors option so that the block
and vectors are not drawn anymore on the video.  Then, render your video to
a .dv file, and import it into your project.  You will notice the video is
stabilized but there are black borders which appear on sides of the frame. 
You have to zoom in and define projector keyframes to move the projector
around the screen, in order to remove those black borders.  The more your
footage is jerky, the more you have to zoom in to discard the black borders. 
That is why the result is better with HDV footage than with DV footage.

An interesting side note about add offset usage is explained next.

To stabilize video, the motion plugin uses a tracking frame to which to
track to and a region within that frame to track (generally an object in the
background) in the current frame.  When the region is obscured, often by
something in the foreground or by leaving the screen, then the motion
compensation would fail, and the video jumps all over the place.

You set a second region to track, and then add offset.

This shows how it is used. It works very well:

------k-------k-----------------
^      ^^
A      BC

A - object1 is visible in the background up until C
B - (the frame before C) has both object1 and object2 visible
C - has only object2 visible

1. Make a keyframe and set to track object1.
2. Make a keyframe at C and track frame at B, set to track object2.
3. Set keyframe at C to add offsets that were calculated at B.

Tips

1. The motion vector is a text file located in /tmp. We can open it with a
plain editor and modify the wrong X Y coordinates, i.e.  those that deviate
from the linearity, to correct the errors that always happen when we perform
a motion tracking (jumps).  It can be a long and tedious job, but it leads
to good results.

2. You can try tracking using reverse playback of the track. Sometimes it
may lead to a better calculation.
manual.txt (31,622 bytes)

Issue History

Date Modified Username Field Change
2020-09-17 14:33 sge New Issue
2020-09-17 14:33 sge File Added: changes.txt
2020-09-17 14:33 sge File Added: manual.txt
2020-09-17 14:33 sge File Added: motion-src.tar.gz
2020-09-17 14:33 sge File Added: motion-test.tar.gz
2020-09-17 14:54 PhyllisSmith Assigned To => goodguy
2020-09-17 14:54 PhyllisSmith Status new => assigned
2020-09-17 14:54 PhyllisSmith Status assigned => acknowledged
2020-09-17 14:54 PhyllisSmith Note Added: 0004030
2020-09-17 19:32 Sam Note Added: 0004033
2020-09-17 19:33 PhyllisSmith Status acknowledged => confirmed
2020-09-17 19:33 PhyllisSmith Note Added: 0004034
2020-09-30 23:53 PhyllisSmith Assigned To goodguy => administrator
2020-09-30 23:53 PhyllisSmith Status confirmed => resolved
2020-09-30 23:53 PhyllisSmith Resolution open => fixed
2020-09-30 23:53 PhyllisSmith Fixed in Version => 2020-09
2020-09-30 23:53 PhyllisSmith Note Added: 0004090
2020-10-07 21:16 PhyllisSmith Status resolved => closed