Algorithms that simulate eye tracking are very tempting because of how fast the data is produced. The effort required to analyze a stimulus based on image processing is significantly less than running a user study with real testers.
While some areas of high attention are indeed correctly detected, the limitations of the simulation quickly become apparent in our comparison with real eye tracking.
On the one hand, human perception is driven by visual stimuli (aka bottom-up processing). Attention is generated by colors, contrasts, or movements. These processes can be simulated by analyzing the stimulus directly.
On the other hand, human perception is also driven by what we already know to interpret new information (aka top-down processes). These mental processes and experiences influence the way our attention is directed. The gaze data on specific a task cannot be replicated by a simulation. For example, the home page of a website is scanned completely differently, depending on which product is being searched for. Also, strong effects like the habit of ignoring advertisements on a page as a learned way of interacting with websites cannot be simulated.
We examined three providers of simulated eye-tracking and quickly found weaknesses in the approach. For example, a top menu was not recognized by the algorithms as attention-generating, since the design and color scheme were rather inconspicuous. However, the menu was immediately noticed and used by the test subjects in a real study setting. Other areas received a disproportionate amount of attention from the users because a product they were looking for was displayed there.
Conclusion: The implementation of AI eye-tracking studies is significantly easier than that of eye-tracking studies with real subjects. However, our small comparison showed the limitations of the algorithms. Some areas of high attention are correctly detected. Others, at the same time, are falsely estimated, since cognitive processes such as concentrating on a menu for product search or ignoring the logo of an online store cannot be reproduced.