RenderQueue.render interacts poorly with undo.
I understand that it doesn't make a lot of sense to undo rendering but I maintain a suite of scripts (written by others) which do a lot of complicated project manipulation along with rendering here and there.
The scripts stopped working in CC2017; app.project.renderQueue.render() wouldn't do anything. No errors, it would just immediately return without running the render. app.onError would catch a couple of lines suggesting a render was started but only for the first item in the RQ.
During the course of ripping everything apart and putting it back together again I started getting these "undo group mismatch, attempting to fix" messages from After Effects. Not wanting to deal with that problem just yet, I commented out all of the app.beginUndoGroup("message") and app.endUndoGroup() calls. Suddenly, the scripts started working properly.
Wanting to know more, I just added two lines around my call to render:
app.beginUndoGroup("test");
app.project.renderQueue.render();
app.endUndoGroup();
I began to get those mismatch messages. There's obviously no mismatch. In the simple tests the rendering does seem to actually work but it's exposed the fact that rendering interacts poorly with undo groups. This is simply a bug; if it's an error to be in an undo group when rendering then make it an error! If rendering gets aborted, render() should return an error, not just silently fail.

-
Saar Oz commented
Any updates on this?
-
Teague Chrystie commented
I'm having similar issues with Undo Groups, but not just with render stuff — with pretty much anything.
Suppose I have a UG called "Example UG," in which a project item is moved. The syntax is correct, as with Tom's examples above — and yet, when I run the script and then pull up the Edit menu to 'undo' it, the NAME of the operation AE wants to 'undo' is... "Move Item." Not "Example UG."
(Windows 10, AE 15.0.0, Build 180)