Well, I had that set to Full HD from the start.
It's multiple scenes, some of them had the camera behave strangely.
I've managed to circumvent the issue though: I realized that the lower edge of the background always lined up neatly with the top of my inventory bar.
After setting that interface to "Absolute" instead of "Bottom", I can walk my Character straight out of the picture and the camera stays where it should.