[oefrha]

Video timestamps are weird. Years ago I routinely pulled event VODs from an HLS source and re-uploaded to YouTube. To speed up downloading, I downloaded the MPEG-TS segments in parallel and assembled them with FFmpeg. Initially I used the basic and familiar concat demuxer during assembly. The results were fine locally. Months in a visitor told me that all my VODs had subtle yet frequent stutters. Turned out the videos played perfectly fine in any libavcodec-based (i.e. FFmpeg-based) video player, and still played fine even after libavcodec re-encoding, yet once they went through YouTube’s encoder, which AFIAK was also libavcodec-derived, subtle stutters appeared at segment boundaries. I then switched to the hls demuxer during assembly and the YouTube problem went away. I never got to the bottom of this, so to this day it’s still a mystery to me.

[thiscatis]

This is what I love about tech. I have 15 years experience as a software developer but I have no clue what any of these words mean. Amazing you can have such specialised knowledge about something.

[Omniusaspirer]

Your comment gave me perspective on how far down the rabbit hole my media server has taken me, as I was nodding along to everything the parent poster said having encountered similar issues with FFmpeg in the past.

Niche knowledge really can creep up on you over the years as you gradually encounter problems and work to solve them a few hours at a time.

[diftraku]

While I have not been this deep into the inner workins, I personally have skimmed that rabbit hole when I started a side-project that was essentially grep but for (mainly) MKV files.

Idea was that you could "grep" by specific text in the subtitles and automatically create a clip of every occurrence of the text (by looking at the subtitle timing and padding that in both directions).

The biggest source of my frustration was that I was unable to get the clipping to work exactly as I wanted, where the start or end of the clip would seemingly drift back and forth. That was until I realized it boiled down to how the different seek modes in ffmpeg handled keyframes.

I still haven't gotten the clipping to work exactly as I want but I figured doing two passes might be the way to go: first pass would do a fuzzy match and ensure there is enough extra on both ends of the desired clip and the second pass could re-encode the fuzzy-matched clip to shuffle the keyframes around, allowing more accurate clipping.

[IanCal]

It's tricky. Subs aren't always brilliantly timed either.

I did something similar many moons ago to auto create summary videos based on changing sentiment in the subtitles for the BBC. Worked... interestingly.

If you want to nail scene changes, one thing you can do is look for sudden changes to the histogram frame by frame. It'll change pretty smoothly as people move, cameras move etc but there's a discontinuity when there's a cut. One issue though then is that there are a lot of camera cuts! Surprising how many there are that you don't really notice.