Graphics seems to be basically working now, both on QEMU and rpi4 (plus/minus a
few quirks).  This patch is now causing more problems than it is solving, so
let's revert it for now.

We need to take care that slow rendering is handled properly.  We might have
more test cases failing due to this after dropping this patch.

Signed-off-by: Esben Haabendal <esben@geanix.com>