Pointermove event latency in web browsers

I&#x27;ve done a lot of work testing this type of latency in web browsers: <a href="https:&#x2F;&#x2F;google.github.io&#x2F;latency-benchmark&#x2F;" rel="nofollow">https:&#x2F;&#x2F;google.github.io&#x2F;latency-benchmark&#x2F;</a><p>On Windows, DWM&#x27;s display compositing adds one frame of latency to every window on screen. It&#x27;s not possible to render a dragged object in any window that sticks to the mouse cursor without at least one frame of latency.<p>But when you drag whole windows around they <i>do</i> stick to the mouse cursor with apparently zero frames of latency; how does DWM do it? Easy, they cheat by disabling the hardware mouse overlay during window dragging so that the mouse cursor gets that extra frame of latency too. You can prove this by enabling &quot;Night light&quot; in settings; watch the mouse cursor change colors as it transitions from hardware overlay to software rendering when you start dragging a window.

&gt;This happens in a buffer and is normally one display update behind in time.<p>This assumes compositors perform their work right after each display refresh. Compositors can decide to perform their work later, some amount of time before the next display refresh (e.g. a few milliseconds). This allows to reduce latency because the new buffers submitted by clients (such as web browsers) can be displayed with less than 1 refresh period worth of latency. For instance the browser can update its buffer at last display refresh + 8ms, then the compositor can composite at last display refresh + 13ms, and the new frame can be displayed at last display refresh + 16ms.<p>Here&#x27;s for instance how Weston does it: [1]. Sway has a similar feature.<p>&gt;However since pointing with a cursor is such a core experience in these OS&#x27;es, the &quot;screen compositor&quot; usually have special code to draw the cursor on screen as late as possible—as close in time to an actual display refresh as possible—to be able to use the most recent position data from the input device driver.<p>That&#x27;s not entirely true. Nowadays all GPUs have a feature called &quot;cursor plane&quot;. This allows the compositor to configure the cursor directly in the hardware and to avoid drawing it. So when the user just moves the mouse around the compositor doesn&#x27;t need to redraw anything, all it needs to do is update the cursor position in the hardware registers.<p>Compositors don&#x27;t have code to draw the cursor as late as possible. Instead, they program the cursor position when drawing a new frame. (On some hardware this allows the compositor to &quot;fixup&quot; the cursor position in case some input events happen after drawing and before the display refresh.)<p>But in the end, all of this doesn&#x27;t really matter. What matters is that the app draws before the compositor draws, thus the compositor will have a more up-to-date cursor position.<p>[1]: <a href="https:&#x2F;&#x2F;ppaalanen.blogspot.com&#x2F;2015&#x2F;02&#x2F;weston-repaint-scheduling.html" rel="nofollow">https:&#x2F;&#x2F;ppaalanen.blogspot.com&#x2F;2015&#x2F;02&#x2F;weston-repaint-schedu...</a>

The neglect for latency in current popular systems such as Linux sickens me.<p>I suggest experimenting with cyclictest from rt-tests. On all hardware I&#x27;ve tried, I get 30ms+ peaks after running it on the background for not even very long. I can&#x27;t comprehend how anybody could find this acceptable.<p>I do run linux-rt for this reason. Then again, while linux-rt provides the tools to make latency reasonable, the rest of the system hardly does use them.<p>As we move from the likes of Linux to better architected systems, potentially based on seL4, I do hope the responsiveness will return to sanity. Until then, I&#x27;ll have to keep going back to my Amiga hardware as cope mechanism.

If you try to &quot;predict the present&quot; based on the past (and when you use previous points to calculate velocity and acceleration, that&#x27;s what you&#x27;re doing) it will overshoot when there&#x27;s a change in direction, and how much depends on how aggressively you try to extrapolate. For the one-dimensional case in signal processing, doing this with a quickly-changing signal like a square wave will result in ringing.<p>It can smooth things a bit but it&#x27;s not that good a substitute for actually improving latency.<p>(There are probably consequences for coronavirus charts as well, since they&#x27;re based on lagging data.)

I&#x27;m seeing &lt;2ms in Edge Chromium and ~10ms in Firefox on a 144 Hz display. I&#x27;m curious how that compares to what other people are seeing.<p>I&#x27;ve been doing some WebGL work recently and I&#x27;ve noticed that while it reaches ~144 fps using requestAnimationFrame() in Firefox, there&#x27;s a lot of stuttering. It&#x27;s very smooth at 144 fps in Edge Chromium, while Edge Legacy is below 80 fps. As far as I can tell it&#x27;s not CPU bound, and it&#x27;s definitely not GPU bound. It would be nice if I could get it running smoothly in Firefox but I don&#x27;t know what to investigate.

&gt; If you move your pointer left and right (or up and down) in sweeping motions and follow it with your eyes, you&#x27;ll notice that the rectangle is trailing behind the pointer by quite a long distance<p>that&#x27;s definitely not what I am observing (<a href="https:&#x2F;&#x2F;streamable.com&#x2F;9u4cpx" rel="nofollow">https:&#x2F;&#x2F;streamable.com&#x2F;9u4cpx</a>). Enabling the predictive tracking, however, is quite nauseating especially in circular motions. Please don&#x27;t play with your users&#x27; cursors !

I didn&#x27;t read the article, but I did try the checkboxes. What I saw surprised me and I will go read the article to see if it addresses my experience, but in case it isn&#x27;t:<p>1. The predictive checkbox improved tracking my cursor.<p>2. Disabling `requestAnimationFrame` improved it more.<p>This is not what I&#x27;d have expected, so I&#x27;ll include details about my environment:<p>- macOS 10.15.4<p>- Safari 13.1<p>- 2019 16&quot; MBP with maxed RAM and ~25GB swap<p>I have no idea whether the browser or the memory pressure made same-thread tracking more accurate, but something did.

Also great bench for browsers and perf tester <a href="https:&#x2F;&#x2F;www.vsynctester.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.vsynctester.com&#x2F;</a>

I recently experimented with implementing certain pointer-controlled effects on a &lt;canvas&gt;, and was discouraged by the jerky feeling caused by latency.<p>But I noticed that if I rendered the effect with motion blur, it suddenly started to feel <i>much</i> smoother, and the perception of jerkiness was mostly gone. I felt that it completely restored my sense of control of the motion.<p>It’s surprising considering that motion blur actually adds one half frame of extra latency.<p>Since trying this, I’m bothered by how jerky fast mouse movement always feels in MacOS. 60 fps leaves these enormous, ugly gaps between the pointer at each frame, and makes it hard to perceive the motion correctly. I can’t unsee these gaps now! I’m convinced that system-wide motion blur just for the pointer would be a simple way to make the whole OS feel much smoother and more responsive.

The dead-reckoning algorithm seems to do well when moving in a straight line but my impression is that it does worse if there are curves because it veers ourside the path that mouse actually traced. For example, when moving the mouse in a circle the predicted squaretrace appears to move in a circle with a larger radius.<p>What kind of algorithm could be used to improve the accuracy for curves?

I got a bit excited thinking this may go into latency of dereferencing pointers in C.