set matte, set channels, calculations,etc to treat & take in an account layer's own transforms.
Set Matte ignores if the layer had been transformed in some way when it is used as a track matte for ex, it treats its transformrs as well, but any matte options in any effect would ignore original transformations if there is any...
Roei Tzoref commented
it does treat the layer transformations if the layer is continously rasterized or collapsed. see if this helps: https://vimeo.com/274047695