View source for DeDup

{{FilterCat5|External_filters|Plugins|Plugins_x64|Adjustment_filters|Duplicate_Frame_Detectors}}
{{Filter3
|1={{Author/akupenguin}}
|2=v0.17-20210227
|3=[https://drive.google.com/file/d/1L2vWFgMaGYnrOMV9mhxg7T8NIVnZZtVE/view DeDup-0.17-20210227.7z]
|4=Duplicate frame detectors
|5=[http://www.gnu.org/licenses/gpl-2.0.html GPLv2]
|6=}}
<br>
== Description ==
[[DeDup]] is an Avisynth, intended to remove duplicate frames in the interest of compression quality and speed. In some cases, it can also reduce noise or compression artifacts in the source.

DeDup differs from Dup in that it completely drops the duplicate frames, rather than keeping several copies of each in the output stream. Then, to compensate for the dropped frames, DeDup outputs a Matroska-compatible timecode file to put the remaining frames back in the right places.

This is a two pass filter. See [[#Examples|Examples]] for usage.

<br>

== Requirements ==
* [x86]: [[AviSynth+]] or [https://sourceforge.net/projects/avisynth2/ AviSynth 2.6]
* [x64]: [[AviSynth+]]
*Supported color formats: [[YUY2]], [[YV12]]
<br>

== [[Script variables|Syntax and Parameters]] ==
===DupMC===
<pre>
First Pass Options:

DupMC(clip c, bool chroma, string log)

chroma:
	default: true
	Use chroma planes when calculating differences.
	(This is the same as in Dup)

log:
	default: none. required.
	File to save dup metrics for the second pass.

DupMC does not modify the frames passed through it. It only generates the log.
There is no harm in seeking during the first pass: The log will be out of order,
but DeDup doesn't care.
Unlike Dup, which compares the last frame in a run to the first frame,
DupMC always compares adjacent frames. You don't decide on a threshold until
the second pass.
</pre>

===DeDup===
<pre>
Second Pass Options:

DeDup(clip c, float threshold, float threshold2, int range2, float trigger2,
      bool show, bool dec, int maxcopies, int maxdrops, bool blend,
      string log, string times, string timesin, string ovr, string debug)

threshold:
	range: 0 - 100
	default: 0.3
	Consecutive frames that differ by less than or equal to this much will be
	merged into a single output frame, taken from the end of the merged sequence.
	(This uses the same units as in Dup)

threshold2:
	range: 0 - 100
	default: 0.5 (but disabled due to range2)
	Use this instead of threshold if there are nearby high-motion frames in
	both directions. (Actually, linearly interpolate between threshold and
	threshold2 based on the amount of motion.)

range2:
	range: 0 - lots (sane values: 0 - maxdrops)
	default: 0 (disabled)
	Search range for threshold2.

trigger2:
	range: 0 - 100
	default: 5.0
	What counts as "high-motion" for threshold2.

show:
	default: false
	Print diagnostic information into each frame.

dec:
	default: true
	Drop frames. Without this, DeDup won't do much. But you might want to
	disable this with show=true, to see the frames that would be dropped.

maxcopies:
	range: 1 - 20
	default: 12
	Break any string of duplicates longer than this.
	(This is the same as in Dup.)

maxdrops:
	range: 1 - maxcopies
	default: 1 (no dropped frames (like the original Dup))
	Max input to output frame ratio.
	(So maxdrops=3 means a max of 2 consecutive dropped frames.)
	Don't set this above 3-4 or so if you want to display subtitles, because many
	media players can't update the subtitles until they get a new video frame.

decwhich:
	Which frame from a run of duplicates do we keep?
	0: first frame
	1: middle, round down
	2: middle, round up (default)
	3: last

log:
	default: none.
	The file created in the first pass with DupMC().
	Required, unless you want to specify every frame with ovr.

times:
	default: none.
	File to write matroska timecodes to.
	Not strictly required, but you'll need it for muxing unless maxdrops=1.

timesin:
	default: none.
	Read timestamps for a VFR input.
	Matroska timecode formats v1 and v2 accepted.

ovr:
	default: none.
	File to override settings on a per-frame basis. See below for format.

debug:
	default: none.
	File to write the same information as with show=true.



Override File Format:
Example:

  0-350 threshold=.3
350-900 threshold=.7 threshold2=1.5 trigger2=8.0
100 k
200-220 ddk


Any line not beginning with a frame number is ignored as a comment.

There is no notation for 'last frame', aside from using it's real frame
number. However, it's OK for ranges to extend past the end of the movie.

If two lines override the same property of the same frame, the one later in
the ovr file takes precedence.

Each line is of the format:
	startframe-endframe option [more options]
or
	frame option [more options]

Option overrides accepted:

A string matching /[kd]+/:
	This forces the frame(s) to be (k) kept or (d) considered dups.
	The string of [kd]s will be repeated as necessary to fill the frame range. 
	Forcing a dup does not necessarily cause that frame to be dropped;
	it's still subject to maxdrops.
	This option modifies thresholds, so put any [kd] option after any relevant
	threshold override.

threshold=%f
threshold2=%f
trigger2=%f
	These are the same as the global options.

range2=%d
	The range2 parameter is taken from the frame being considered from dropping,
not from the frames being searched over.

maxdrops=%d
maxcopies=%d
	These values are taken from the frame at the end of the clump of dups, but
it's better to just set them the same over the whole clump.
</pre>
<br>

== Examples ==
<pre>
--- first pass ---

LoadPlugin("DeDup.dll")
AVISource("blah.avi")
DupMC(log="blah.dup.txt")

-------------------
--- second pass ---

LoadPlugin("DeDup.dll")
AVISource("blah.avi")
DeDup(threshold=0.3, maxcopies=10, maxdrops=3, log="blah.dup.txt", times="blah.times.txt")
</pre>

===Tips===
<pre>
Tips:
	If you're doing other CPU-intensive filtering, you can save time
(at the cost of space) by saving the first pass (with DupMC) to a lossless 
video format, and then run the second pass (with DeDup) from that, instead 
of repeating the other filters.
	Don't set threshold too high (especially if maxcopies is large):
Unlike the original Dup, I don't do sanity checking on slow zooms/pans/fades.
	The second pass options 'times' and 'debug' take effect as soon as you
open the avs. You don't have to play/encode the whole video in order to produce
those files.
</pre>
<br>
== Changelog ==
<pre>
No version change (2021-02-27):
Avisynth 2.6 interface.
x64 build.
Added version.

DeDup 0.17 (2004-11-7):
decwhich<3 was keeping the wrong input frame.
Improved neighborhood display.
Changed default maxdrops to 1 (no VFR).

DeDup 0.16 (2004-9-16):
Now keeps the middle frame instead of the last frame in a run of length>2.
(Set decwhich=3 for the old behaviour.)
Reduced default thresh to 0.3

DeDup 0.15 (2004-7-18):
Added a neighborhood display when show=true.

DeDup 0.14 (2004-6-5):
Fixed crash on show=true.
Added overrides for range2, maxdrops, and maxcopies.

DeDup 0.13 (2004-6-3):
Fixed crash when logfile is longer than second pass's input video.
Flush logfile when done.

DeDup 0.12 (2004-6-2):
Allow no or partial first pass if you override the k/d status of the missing frames.
Matroska timecode v1 is now compatible with Yatta.

DeDup 0.11 (2004-5-26):
Added a second threshold for better noise resilience in periodic low-fps material.
Support VFR input.
Misc fixes.
Blend is still broken.

DeDup 0.10 (2004-5-23):
Initial release.
</pre>
<br>
== Archived Downloads ==
{| class="wikitable" border="1"; width="600px"
|-
!!width="100px"| Version
!!width="150px"| Download
!!width="150px"| Mirror
|-
!v0.17-20210227
|[https://drive.google.com/file/d/1L2vWFgMaGYnrOMV9mhxg7T8NIVnZZtVE/view DeDup-0.17-20210227.7z]
|[https://web.archive.org/web/20210228193434if_/https://files.videohelp.com/u/223002/DeDup-0.17-20210227.7z DeDup-0.17-20210227.7z]
|-
!v0.17
|[http://akuvian.org/src/avisynth/dedup/dedup-0.17.zip dedup-0.17.zip]
|[https://web.archive.org/web/20181011152748if_/http://akuvian.org/src/avisynth/dedup/dedup-0.17.zip dedup-0.17.zip]
|}
*For older versions see: http://akuvian.org/src/avisynth/dedup/
<br>
== External Links ==
<br>
<br>
-----------------------------------------------
'''Back to [[External_filters#Duplicate_Frame_Detectors|External Filters]] &larr;'''