Optimize layered videos.
just, watch the attached video. There's no reason for a moving plate to be hundreds of times easier to compute than a still one. How many optimizations are so niche that they actually hurt performance overall? I get it, an optimized program will no doubt not always have the same efficiency on one frame as another, but this is just silly.
This is the reason my old compositions were all lagging. They all... had un-transformed video files in them. in what world is it less efficient to not calculate movement?
They're the same video, and when they're on top of one another, perfectly matched, why does it get less efficient? At the very worst, wouldn't it just stay the same? I'd imagine doing math for transparency, edge pixels, translation, and so on would be more taxing than just doing transparency.
perhaps a more modular approach to an effects api would be better than... whatever this is. A proper API shouldn't differentiate between transparency calculations or algorithms here, or there, unless absolutely necessary.

-
Jess McMillan (AE Dev @ Adobe) commented
Hello Nick,
There is a lot of nuances to the information you see in the render time output.
Anything listed in the twirl open of a layer gets combined to be the value for the overall layer time. Just from looking at this - the extra time being taken on your first layer is due to Luma Key.
As for the frames where you are seeing 0ms* - the * represents that this render was able to pull the results from cache where the part or all of the image for a layer was completely stored. It is likely that the frame with the heaviest hit is because that was the first frame rendered and the others were rendered after it. If you were to clear your cache and have the CTI resting on a different frame, I imagine that would be where you would see the render hit instead.
The following page has a lot of additional details that might help you interpret what you are seeing represented in those values. https://helpx.adobe.com/si/after-effects/using/composition-profiler.html
All the best,
Jess