I've found a bug in the current build of After Effects CC2018 (Version 15.1.1, Build 12). There's a scaling glitch that manifests such that Ae Render Queue setup treats the scaled preview resolution as the composition maximum resolution (SD for a UHD comp), and treats a UHD output as an upscale of the SD preview size. It also appears to affect MOGRT files exported to use in Premiere Pro.

I've been troubleshooting this all day today and made little headway, but after finding this thread the proposed solution (to set the "Resolution/Down Sample Factor Popup" to "Full") allows the comp to render properly from the Ae Render Queue, as a Dynamic Linked comp in Premiere Pro, and as a MOGRT.

Originally discussed on the forum here: https://forums.adobe.com/message/10496314#10496314

iMac (Retina 5K, 27-inch, 2017)
3.4 GHz Intel Core i5
16 GB 2400 MHz DDR4
Radeon Pro 570 4096 MB
macOS High Sierra 10.13.6 (17G65)