I understand that it doesn't make a lot of sense to undo rendering but I maintain a suite of scripts (written by others) which do a lot of complicated project manipulation along with rendering here and there.

The scripts stopped working in CC2017; app.project.renderQueue.render() wouldn't do anything. No errors, it would just immediately return without running the render. app.onError would catch a couple of lines suggesting a render was started but only for the first item in the RQ.

During the course of ripping everything apart and putting it back together again I started getting these "undo group mismatch, attempting to fix" messages from After Effects. Not wanting to deal with that problem just yet, I commented out all of the app.beginUndoGroup("message") and app.endUndoGroup() calls. Suddenly, the scripts started working properly.

Wanting to know more, I just added two lines around my call to render:

app.beginUndoGroup("test");
app.project.renderQueue.render();
app.endUndoGroup();

I began to get those mismatch messages. There's obviously no mismatch. In the simple tests the rendering does seem to actually work but it's exposed the fact that rendering interacts poorly with undo groups. This is simply a bug; if it's an error to be in an undo group when rendering then make it an error! If rendering gets aborted, render() should return an error, not just silently fail.