Relationship between facial expressions and human emotions

The article describes our approach to assessing emotions from facial expressions in the context of watching a video. The approach involves using a proprietary algorithm to compute the intensity of a smile, as a proxy for amusement; an eyebrow as a proxy for surprise and a frown, as a proxy for confusion. The algorithm uses a facemesh, which is a representation of 478 key points on the face, to analyze the movement and contraction of the facial muscles. The output is a score between 0.0 and 1.0 that reflects the intensity of the smile or frown. The approach is designed to be real-time and private, leveraging local processing power to analyze the key points of a facemesh.

• Smiling is often associated with happiness and positive emotions
• Studies have shown a relationship between smiling and emotions
• Smiling can be used as a proxy for amusement
• Frowning is often associated with negative emotions such as sadness, anger, or frustration
• Frowning can also indicate concentration or contemplation
• Frowning can be used as a proxy for confusion
• Asking questions is necessary to disambiguate confusion and surprise proxies
• The approach involves using a face detection model and a facemesh to estimate the intensity of a smile or frown
• The output is a score between 0.0 and 1.0 that reflects the intensity of the expression.

Smiling & amusement

Relationship between smiling and human emotion

Smiling is often associated with happiness and pleasure. 

When a person smiles, it typically indicates that they are feeling positive emotions such as joy, amusement, or contentment. Smiling can also be a way of expressing friendliness or goodwill towards others. 

However, it's important to note that smiling is a complex behavior that can have many different meanings and can be influenced by a variety of factors, including cultural and individual differences.

Studies pointing to a relationship between smiling and emotions

There is a foundational study from Paul Ekman that has investigated the relationship between smiling and emotions, in a paper from 1990 called: The Duchenne Smile Emotional Expression And Brain Physiology  

A Duchenne smile is a genuine smile characterized by the activation of the muscles around the eyes, resulting in the creation of crow's feet wrinkles. 

Some key points to know about a Duchenne smile are:

• It is named after the French neurologist Guillaume Duchenne, who studied the physiology of facial expressions in the mid-19th century.
• It is also known as a "real" or "true" smile, as opposed to a fake or forced smile.
• It is typically associated with feelings of happiness and warmth.
• It is typically considered to be more intense and sincere than a non-Duchenne smile.
• It is often considered to be more attractive than a non-Duchenne smile.

Estimation of smile as a proxy to amusement

Amusement is a state of experiencing humorous and entertaining events or situations.

Our approach is to compute an evaluation of amusement based on smile detection involving several steps.

First, an image is captured using a traditional RGB or IR camera. This image is then passed through a face detection model that runs locally on the device. If a face is detected, a facemesh is estimated using the local processing power of the device. This facemesh is composed of 478 key points that represent the various landmarks on the face, such as the corners of the mouth, the eyebrows, and the nose.

Next, the stream of facemesh data is given as input to a proprietary algorithm that is specifically designed to estimate the intensity of a Duchenne smile. This algorithm uses the key points of the facemesh to analyze the movement and contraction of the facial muscles around the eyes and mouth, which are characteristic of a Duchenne smile. Based on this analysis, the algorithm computes a score between 0.0 and 1.0 that reflects the intensity of the Duchenne smile.

Overall, this approach allows for real-time and private by design evaluation of amusement based on smile detection, by leveraging local processing power and a specialized algorithm to analyze the key points of a facemesh.

Frowning & confusion

Relationship between frowning and human emotion

Yes, frowning is often associated with negative emotions such as sadness, anger, or frustration. 

When a person frowns, it typically indicates that they are feeling unhappy or displeased. Frowning can also be a way of expressing disapproval or dissatisfaction with something.

Like smiling, however, frowning is a complex behavior that can have many different meanings and can be influenced by a variety of factors.

Frowning can have several different meanings, depending on the context in which it is used. Some common meanings of frowning include:

• Disapproval or dissatisfaction: Frowning can be a way of expressing that you are unhappy with something or someone. For example, if someone tells a joke that you don't find funny, you might frown to show that you don't approve of the joke.
• Concentration or contemplation: Frowning can also be a sign of deep thought or concentration. For example, if you are trying to solve a difficult problem, you might frown in order to focus your attention on the task at hand.
• Frustration or annoyance: Frowning can also be a way of expressing frustration or annoyance. For example, if you are stuck in traffic and are running late for an appointment, you might frown in frustration.
• Pain or discomfort: Frowning can also be a sign of physical discomfort or pain. For example, if you have a headache, you might frown in response to the pain.
• Sadness or disappointment: Frowning can also be a way of expressing sadness or disappointment. For example, if you receive bad news, you might frown in response.

Overall, the meaning of frowning can vary depending on the situation and the individual who is frowning. In general, however, frowning is often associated with negative emotions or feelings.

What does it mean when someone frowns while watching a video?

If someone frowns while watching a video, it could mean that they are not enjoying the video or that they are dissatisfied with it in some way. It is possible that the person is frowning because they find the content of the video to be boring or uninteresting, or because they are confused or frustrated by something that is happening in the video.

Alternatively, the person might be frowning because they are concentrating or thinking deeply about something that is happening in the video. For example, if the video is presenting information that is difficult to understand or that requires careful attention, the person might frown in order to focus their attention on the video.

Overall, the meaning of someone frowning while watching a video can vary depending on the context and the individual who is frowning. It is important to ask the person explicitly about their feelings in the context of the video, in order to determine the reason for their frown.

To disambiguate the meaning of a detected frown while watching a video, some potential survey questions could include:

1. What were you feeling at the time when you frowned?
2. Were you feeling confused, frustrated, or unhappy at that moment?
3. What specifically was happening in the video that caused you to frown?
4. Was the frown a result of your concentration or contemplation on something in the video, or was it a reaction to something negative or unpleasant in the video?
5. Was the frown a genuine expression of your emotions, or was it a deliberate, intentional expression for some reason?

Are there some studies pointing to a relationship between frowning and emotions?

Yes, there have been many studies that have investigated the relationship between frowning and emotions.

For example the study of Vaish et al. (2013), pointed out that:

• The study used electrophysiology to explore the mechanisms responsible for the impact of frowning on emotional reasoning.
• The study found that frowning influenced sustained attention to emotional stimuli, as measured by LPP.
• Frowning also showed a trend for a negative effect on the inference process measured by the N400.
• The study provides insight into the link between reasoning and emotions in the body.
• The findings suggest that frowning impacts sustained attention and possibly type 2 processes (ie. automatic or non-conscious processes.

Overall, the available evidence suggests that there is a strong connection between frowning and emotions.

Estimation of frowning as a proxy to confusion

Our approach to compute an evaluation of confusion based on frown detection involves several steps. 

First, an image is captured using a traditional RGB or IR camera. This image is then passed through a face detection model that runs locally on the device. If a face is detected, a facemesh is estimated using the local processing power of the device. This facemesh is composed of 478 key points that represent the various landmarks on the face, such as the eyebrows, the forehead, and the corners of the mouth.

Next, the stream of facemesh data is given as input to a proprietary algorithm that is specifically designed to estimate the intensity of a frown. This algorithm uses the key points of the facemesh to analyze the movement and contraction of the facial muscles around the eyebrows and forehead, which are characteristic of a frown. Based on this analysis, the algorithm computes a score between 0.0 and 1.0 that reflects the intensity of the frown.

Overall, this approach allows for real-time and private by design evaluation of confusion based on frown detection, by leveraging local processing power and a specialized algorithm to analyze the key points of a facemesh.

Raising eyebrows & surprise

Relationship between raising eyebrows and human emotions

Raising eyebrows is often associated with surprise or confusion. 

When a person raises their eyebrows, it typically indicates that they are surprised or unsure about something. Raising eyebrows can also be a way of expressing skepticism or disbelief. 

Like smiling and frowning, however, raising eyebrows is a complex behavior that can have many different meanings and can be influenced by a variety of factors.

Raising eyebrows can have a variety of meanings depending on the context and the individual who is raising their eyebrows. Some possible meanings of raising eyebrows include:

• Surprise or disbelief: Raising eyebrows can be a way of expressing that you are surprised or skeptical about something. For example, if someone tells you an unexpected piece of news, you might raise your eyebrows to show that you are surprised.
• Interest or curiosity: Raising eyebrows can also be a sign of interest or curiosity. For example, if you are listening to someone talk about a topic that you find interesting, you might raise your eyebrows to show that you are engaged and want to hear more.
• Skepticism or doubt: Raising eyebrows can also be a way of expressing skepticism or doubt. For example, if someone makes a claim that you find difficult to believe, you might raise your eyebrows to show that you are skeptical.
• Confusion or uncertainty: Raising eyebrows can also be a sign of confusion or uncertainty. For example, if you are trying to understand something that is complex or difficult, you might raise your eyebrows in order to show that you are unsure.

In their study Namba et al. (2021), conduct extensive research on distinct temporal features of genuine and deliberate facial expressions of surprise. These are the key findings:

• The study investigated the physical differences between genuine and deliberate expressions of surprise.
• The study used video recordings of participants experiencing genuine surprise and producing deliberate surprise through different methods.
• The study found that genuine expressions of surprise included horizontal movement in the left part of the face and weak movement coupling of all facial parts.
• In contrast, deliberate expressions of surprise showed faster eyebrow and eyelid movement and stronger coupling of movements.
• The findings suggest that dynamic information is important in distinguishing genuine and deliberate expressions of surprise.

Overall, the meaning of raising eyebrows can vary depending on the situation and the individual who is raising their eyebrows. In general, however, raising eyebrows is often associated with a feeling of surprise or uncertainty.

What does it mean when someone raises eyebrows while watching a video?

It is difficult to determine the exact meaning of someone raising their eyebrows while watching a video without further context or information. Raising eyebrows can indicate a variety of emotions or reactions, such as surprise, skepticism, curiosity, or disbelief. The meaning of the raised eyebrows would depend on the person's body language and other facial expressions, as well as the content of the video they are watching.

To disambiguate the meaning of an eyebrow raising while watching a video, we would need to consider the context in which the eyebrow raising occurred. This could include factors such as the person's facial expression and body language, the content of the video, and the person's verbal response to the video. By taking all of these factors into account, we may be able to determine the specific meaning of the eyebrow raising. For example, it may indicate surprise, skepticism, interest, or disbelief, depending on the situation.

Here are some possible survey questions that could help disambiguate the meaning of an eyebrow raising while watching a video:

1. What was the context of the eyebrow raising? (e.g. was it in response to a particular moment in the video, or was it a general response to the video as a whole?)
2. What was your emotional state while watching the video? (e.g. were you happy, sad, surprised, confused, etc.?)
3. What do you think caused the eyebrow raising? (e.g. was it a reaction to something in the video, or was it a spontaneous expression of your own emotional state?)
4. Can you describe the facial expression you were making when you raised your eyebrows? (e.g. were you frowning, smiling, neutral, etc.?)
5. How did the eyebrow raising affect your overall experience of watching the video? (e.g. did it make the experience more enjoyable, less enjoyable, or had no impact?)
6. If you could go back and watch the video again, do you think you would still raise your eyebrows in the same way? (e.g. would your response be the same, or would it be different?)

Estimation of eyebrow raising as a proxy to surprise

Our approach to compute an evaluation of surprise, related to raising eyebrow detection, involves the use of a traditional RGB or IR camera to capture an image of the face. 

This image is then applied to a face detection model, which is run locally on the device. If a face is detected in the image, a facemesh is estimated locally using keypoints on the face. 

This facemesh is composed of 478 keypoints, which represent the various facial features, such as the eyes, nose, and mouth. 

The stream of the facemesh is then given as input to our proprietary algorithm, which computes the facemesh input stream to estimate the intensity of an eyebrow raising. 

The output of this algorithm is a value between 0.0 and 1.0, which indicates the intensity of the eyebrow raising. This value can be used to evaluate the level of surprise exhibited by the person in the image.

Final word

We have developed an approach for assessing emotions from facial expressions in the context of watching a video.

Our approach involves using a proprietary algorithm to compute the intensity of a smile, which we use as a proxy for amusement; a frown, which we use as a proxy for confusion and an eyebrow raising which we use as a proxy for surprise. The algorithm uses a facemesh, which is a representation of 478 key points on the face, to analyze the movement and contraction of the facial muscles. The output is a score between 0.0 and 1.0 that reflects the intensity of the smile, eyebrow raise or frown.

We note that it is important to ask questions in order to disambiguate confusion and surprise proxies. This approach allows for real-time and private emotion assessment.