Thanks to new AI technologies, expanded possibilities for analyzing user sessions are now available. By integrating AI with eye-tracking data and speech recordings, it is now possible to capture information about the user’s language and emotions. These valuable data can then be analyzed using further specialized AIs to gain additional insights into user behavior and needs.
The platform huggingface.co offers publicly accessible AIs for local use or as a cloud service. The individual AIs can be easily merged into a pipeline to solve complex tasks.
Using multiple AIs available there, we analyzed the web sessions of an eye-tracking study. With the first AI, the spoken words in the screencast were transcribed. A second AI was then able to identify and separate the interviewer’s spoken words and test user using speaker diarization. Finally, another AI determined the user’s emotions on this transcript. The sentiments were then combined with the gaze data and assigned to the viewed areas of the website. This extends the classic eye-tracking heatmap with a new sentiment color coding. For example, red for negative and green for positive expressions.
During a user study with eye-tracking, this subject said “I already chose L but here I have to click on L again,” which was recognized as a negative sentiment by the AI. Finally, another AI was used to determine the time point of the gaze data when which word was spoken.
What’s particularly exciting is how quickly these technologies are advancing. In the past year, we’ve seen huge improvements in speech recognition, with tools like OpenAI’s “whisper” achieving near-perfect transcription in both English and German. And with a low resource requirement, these models can be easily used on a standard notebook computer.
Overall, these developments open up new possibilities for businesses looking to better understand their users’ needs and behaviors. Whether you’re working on a website, an app, or something else entirely, it’s worth exploring the many AI tools now available to help you make the most of user sessions!