Thoughts of ffmpeg and whisper filters

January 2026

Thoughts of ffmpeg and whisper filters

January 2026

I’ve been experimenting with ffmpeg and the whisper filters. In general I think its awesome that such functionality exists, but at the same time, I don’t believe it addresses the particular painpoints when you go beyond the ‘obvious’ thing.

Installation

On macos to install ffmpeg with the whisper filters, the easiest way is via brew:

brew tap homebrew-ffmpeg/ffmpeg
brew install homebrew-ffmpeg/ffmpeg/ffmpeg --with-whisper-cpp

Considerations

One of the cool functionalities of whisper-cpp is the ability to integrate voice activity detection (VAD). This works out of the box with the whisper filter. Unfortunately what does not work is integrating translations, instead it is expected you create the translation yourself.

Anecdotally, the performance of the setup is lacking. In fact it is slower than:

extracting the raw audio out of the base media file
using whisper-cpp to generate the srt

In code it will look something like:

1ffmpeg -i '{{input_path}}' -vn -acodec libvorbis -q:a 0 '{{output_path}}'
2whisper-cli -m path/to/ggml-large-v3-turbo-q5_0.bin -sns -osrt --vad -vm path/to/ggml-silero-v6.2.0.bin -of '{{output_path}}' -f '{{input_path}}' -l en

As of right now, I’ll probably stick with running it manually in this way.

Further Notes

If you need translations don’t use large-v3-turbo you’ll need a non-turbo model otherwise translations won’t work.

Sometimes when using VAD it screws up the srt timings, I have a hacky fix which:

calculates the median time
finds timings which are more than double the median time
tries to fix it

It also tries to fix timestamps which are very short, and can be extended (due to there not being a subtitle in the frame before). Using median also means the outputs will be idempotent.

 1import srt
 2import numpy as np
 3from pathlib import Path
 4
 5
 6def fix_srt_timestamp(input_srt_path: str | Path):
 7    """
 8    Sometimes whispers generates really long timestamps.
 9    This calculates the median duration, clipping timestamps that are double the length
10    and replacing with the median duration.
11    """
12    data = Path(input_srt_path).read_text()
13    if data.strip() == "":
14        Path(input_srt_path).unlink()
15    subs = list(srt.parse(data))
16    if len(subs) < 2:
17        return
18    # median
19    median_time = np.median([sub.end - sub.start for sub in subs])
20    fixed_subs = []
21    counter = 0
22    for sub in subs:
23        duration = sub.end - sub.start
24        if duration > median_time * 2:
25            sub.start = sub.end - median_time
26            counter += 1
27
28        # other fixes
29        sub.content = sub.content.strip()
30
31        # remove if single word
32        if len(sub.content.split()) <= 1:
33            counter += 1
34            continue
35
36        # if the time between current and previous one is < median but > median/2 make it median/2
37        if len(fixed_subs) > 0:
38            sub_time_diff = sub.start - fixed_subs[-1].end
39            if median_time / 2 < sub_time_diff < median_time:
40                prev_sub = fixed_subs[-1]
41                prev_sub.end = prev_sub.end + sub_time_diff / 2
42                sub.start = sub.start - sub_time_diff / 2
43                fixed_subs[-1] = prev_sub
44                counter += 1
45
46            elif (
47                sub.start - fixed_subs[-1].end > median_time
48                and sub.end - sub.start < median_time
49            ):
50                sub.start = sub.end - median_time
51                counter += 1
52        fixed_subs.append(sub)
53
54    fixed_srt = srt.compose(fixed_subs)
55    Path(input_srt_path).write_text(fixed_srt)
56    if counter > 0:
57        print(f"Fixed {counter} timestamps in {input_srt_path}")