"I-frame first" fast seeking for inter frame compressed video files
When you are "seeking," (which means QUICKLY looking through a long video by clicking and dragging the playhead with the cursor,) there is sometimes very high "prefetch latency," meaning that frames will load slowly, resulting in very few frames being shown. This can make it very difficult to find what you're looking for.
But when seeking, perfect frame accuracy is not nearly as important as seeing what you're doing. Therefore, to reduce the "prefetch latency," I suggest that the nearest I-frame is loaded and displayed FIRST, (because I-frames are always the fastest to load) while the exact B or P frame is still being slowly decoded. As soon as it's decoded, it can be displayed, replacing the displayed I-frame.
If the user is seeking quickly, there might not be enough time to load the P and B frames before the user has already moved the playhead nearer to another I-frame. This is fine! The I-frames should always take priority.
In VLC, there is a similar feature called "fast seek."
Combined with my "ghost playhead" feature suggestion, it would be made clear to the user exactly what is going on. (Linked below)
However, it would probably be important for fast seeking to be optional, and perhaps even off by default, as it may confuse some users.
NOTE: This works best if only ONE video source is visible. With more than one video on the timeline, the I-frames will probably not be in the same spots. Nevertheless, it COULD still work, but there would have to be multiple ghost playheads. This might be too difficult to code, and confusing and frustrating for users, who won't understand why premiere is showing video frames out of lockstep, especially in cases where one video has high prefetch latency, and the other is low.
LINK TO FULL EXPLANATORY VIDEO: https://youtu.be/XeFezzmG22c
LINK TO GHOST PLAYHEAD USERVOICE ENTRY: https://adobe-video.uservoice.com/forums/911233-premiere-pro/suggestions/42186274-a-ghost-playhead-which-always-marks-the-timeline
TaranVH
shared this idea
Hi Taran and everyone subscribed to the thread,
first of all: thanks for the incredibly detailed suggestion and the great conversation in the comments. While making such a change for the playback engine has a ton of implications and therefore isn’t done ad hoc, we’re certainly going to create a POC for it and will evaluate if there’s a way to put this into a production build for a test drive in an upcoming beta version.
I’ll keep you posted.
Cheers,
Patrick
22 comments
-
Waldorf & Statler
commented
In my opinion it is important to know the different GOP structures. In my opinion, Long-GOP is for consumers and I-Frame is only for editors. We recommend our editors to convert long GOP files in the ADOBE media encoder into e.g. ProRes I-Frame only and to cut them with it.
This is how it works with the fast search, e.g. with JKL.
If you have previously generated a watch folder for such tasks, the job is very easy to do next to. -
Ojan Seyedashrafi
commented
I don't know if this is exactly the cause for being relatively slow and laggy timeline scrubbing to left and right. because in my experience it doesn't matter what kind of video you have in your timeline even a ProRes Proxy file on a relatively fast machine with 40 min long timeline is not smooth like it is in a avid timeline for example.
more shocking fact is that a fast raid usb c hard drive doesn't make any difference -
Ariel Brener
commented
Wow! this is an amazing idea! it can be a real game-changer in many user's workflows.
I have lots of footage that premiere can handle in simple playback (including sony alpha and DJI's hi compressed formats) but when I try to SKIM it or play it back in x8 and above it becomes unusable. - So I use PROXY workflow which has a lot of downsides.
If this genius suggestion will be implemented it will be a huge step above other NLEs! -
Chris Spiegl
commented
Yes, yes, yes! Thank you Patrick for taking this under review. ❤ This is awesome 🌸.
-
David Johnson
commented
SUCCESS. At least Taran ... at least via Patrick, Adobe is listening, and the olive branch is out there.
-
Pierre Louis Beranek
commented
@TaranVH
I see what you're saying. I made an official FR for those 2 additional ideas in case it helps Adobe solve this issue.You can check it out and vote here: https://adobe-video.uservoice.com/forums/911233-premiere-pro/suggestions/42773999-improve-scrubbing-fast-forwarding-performance-with
What I most often have to do is quickly scrub through lots footage since even though I put it in my Timeline, I don't know exactly where individual clips from my first rough cut are. Once I start assembling clips I select from my first rough cut, then I have a better idea of what is where in the final cut. ;)
-
TaranVH
commented
@Pierre Louis Beranek
Those are some really interesting ideas that I had not considered. They could definitely help in some situations.
But, they would not fix the problems that I pose in the explanatory video. In my example, the "where gaming begins.m4v" AMD video is in the source monitor. It has no effects on it at all. But it still has very high (slow) prefetch latency, because of the way it was encoded. The "downloading high sierra (mac).mp4" is even worse. Again, it had no effects, and was in the source monitor at 1/2 resolution.I don't really need to scrub through footage quickly to find stuff once it's already been placed onto the timeline. I already know where everything is on my timeline... I put it there!
You might think that automatically lowering the quality to 1/4 or 1/8 or even 1/16 would help, but in my experience (all 8+ years of it) that only helps with all-I-frame video, like cineform, .r3d, and .braw. For H.264, I believe that a decoder still needs to load the entire frame BEFORE it can determine what it would look like at a lower resolution. Like, go look up "DCT zigzag pattern" and you'll see why. Each of those 8x8 blocks of pixels is a "macroblock." The decoder can't just SKIP 3/4 of the pixels. It needs to go through ALL of them or it'll get the values wrong.
It is possible that only loading and displaying the luminance data would be faster than loading and displaying the luminance + chrominances, but I don't know enough about H.264, H.265, etc, to say for sure.
-
Pierre Louis Beranek
commented
Hi Taran,
I did watch the video you link to in your post, but if your brilliant idea in this FR to display the I-frame first were applied, wouldn't it speed up video scrubbing to the point that displaying a ghost frame would no longer be necessary in most cases? If anything, I'd rather Pr do the following: anytime the user's system can't keep up because the footage has too many effects, Pr could show the footage without any (or without certain processing heavy) effects while scrubbing and then update to the frame under the playhead with all effects as soon as processing catches up. Whenever this happens, the playhead could change to another color, perhaps red. When the frame is properly rendered with all effects, the playhead would turn back to its current blue color. To best suit all users, this feature could be added to the Timeline's hamburger menu so users can turn it on/off.Another (additional or alternative) solution might be to provide adaptive quality playback while scrubbing. Pr already gives us options for Playback and Paused resolution, as well as a wrench menu on/off option for "High Quality Playback". What if there was an option for adaptive quality scrubbing (possibly also available when playing video back at 2x, 4x or faster), that would allow users to set the variable quality anywhere from Full only all the way down to 1/16th?
Perhaps one or both of these solutions would take care of the delays you were seeing in your video?
What do you think?
-
TaranVH
commented
@Pierre Louis Beranek
You clearly did not watch the accompanying video. I recommend that you watch the whole thing.
The delay from prefetch latency can be enormous -- multiple seconds, in some cases.
https://youtu.be/XeFezzmG22c?t=378
In those cases, it is impossible to know if the frame you're looking at, is /actually/ the frame that you have selected. Especially with screen capture footage. Thus, the ghost playhead would show which frame IS being shown. Premiere already knows which frame it's showing... it just has no way of telling the user. -
Pierre Louis Beranek
commented
@Antoine Seeing the wrong frame for a fraction of a second while scrubbing wouldn't be a big deal. Probably no one would ever even notice the difference. As soon as the user stops scrubbing, the correct frame would show up perhaps 1/10th of a second later?
Personally I think there's zero need for a 'ghost' playhead. Trying to program that into Pr would be utter nonsense, and the 'with and without fast seek.mp4' video the OP posted himself proves it. Just look at the delay when he stops scrubbing the VLC video between the last static image and the image just before it. The gap is so short, eyeballing it I'd say it's just 1 frame long (1/30th of a second). Why would anyone need to see a 'ghost' playhead for a 1-3 frame delay?
-
Antoine Autokroma
commented
Just to be clear Taran, you would be OK for Premiere Pro to show you the wrong frame ? Showing you the previous closest I frame instead of the real frame
-
Sander Skjegstad
commented
A simple solution for the multiple playheads issue would be to have the lines draw inside each media item. That way you automatically get one for each.
-
Antoine Autokroma
commented
Very good idea indeed
-
John Pooley
commented
Yes this is desired, but the FramePrefetchLatency bug is a bug and let's call it what it is. You shouldn't get multiple seconds of Prefetch lag before clearing the Media Cache, and 1ms afterwards. This wouldn't fix that entirely, just as hardware decoding didn't fix it. Adobe needs to get to the root of the problem.
-
Thomas Berglund
commented
@TaranVH Cool. Let us know if you notice a difference. I have tested this on both Windows 10 and macOS 10.15.7, with Premiere Pro 14.6, and even with a 2 hour long i-frame only based video the amount of frames displayed while scrubbing, when "Play audio while scrubbing" is turned off compared to on, is substantial.
The Adobe Premiere team also changed the way Premiere Pro handles pre-roll for playback of audio, which dramatically speeds up start of playback for audio when using clip based audio effects. This changed was introduced in Premiere Pro 14.5.
"Faster audio pre-roll in Premiere Pro provides responsive playback for large projects or projects that use a lot of audio effects. No more waiting for playback to start on macOS and Windows."
This new feature can also be turned off/on under "Audio Preferences > Use legacy audio effect playback pre-roll".
-
TaranVH
commented
@Thomas Berglund
I have not tried that. Interesting idea... will do.. -
Thomas Berglund
commented
@TaranVH Have you tried to turn off "Play audio while scrubbing" (shift+s, or Preferences > Audio)? This dramatically improves scrubbing performance overall. I have reported this to Adobe for many years, including screen recordings. It is especially noticeable with longer videos, and even more so with long highly compressed Long-GOP videos.
-
Matyáš Levíček
commented
I usually generate DNx proxies and that solves the issue
-
colin cameron
commented
yes just did an edit tonight where i had to find the start of a minute and a half segment in four and a half hours of footage and this feature would have been incredibly useful.
-
harry rogers
commented
I'm having the same problem