[BUG]: I rotated the image by 90 degrees clockwise just to test if Deepface properly aligns that

[jogendra-india]

Before You Report a Bug, Please Confirm You Have Done The Following...

• I have updated to the latest version of the packages.
• I have searched for both existing issues and closed issues and found none that matched my issue.

DeepFace's version

v0.0.94 (directly cloned the master branch)

Python version

3.9

Operating System

Ubuntu

Dependencies

absl-py==2.1.0
amqp==5.3.1
asgiref==3.8.1
asttokens==3.0.0
astunparse==1.6.3
async-timeout==5.0.1
backcall==0.2.0
beautifulsoup4==4.12.3
billiard==4.2.1
black==24.10.0
blinker==1.9.0
celery==5.4.0
certifi==2024.12.14
cfgv==3.4.0
charset-normalizer==3.4.1
click==8.1.8
click-didyoumean==0.3.1
click-plugins==1.1.1
click-repl==0.3.0
contourpy==1.3.0
cycler==0.12.1
decorator==5.1.1
deepface @ git+https://github.com/serengil/deepface.git@ca6afd98efb87dc61fb022ddd2892da23dc75f68
distlib==0.3.9
Django==4.2
django-celery-results==2.5.1
django-cors-headers==4.6.0
django-extensions==3.2.3
django-filter==24.3
djangorestframework==3.15.2
executing==2.1.0
filelock==3.16.1
fire==0.7.0
Flask==3.1.0
Flask-Cors==5.0.0
flatbuffers==24.12.23
fonttools==4.55.3
gast==0.6.0
gdown==5.2.0
google-pasta==0.2.0
greenlet==3.1.1
grpcio==1.68.1
gunicorn==23.0.0
h5py==3.12.1
identify==2.6.3
idna==3.10
importlib_metadata==8.5.0
importlib_resources==6.4.5
ipython==8.12.3
itsdangerous==2.2.0
jedi==0.19.2
Jinja2==3.1.5
keras==3.7.0
kiwisolver==1.4.7
kombu==5.4.2
libclang==18.1.1
Markdown==3.7
markdown-it-py==3.0.0
MarkupSafe==3.0.2
matplotlib==3.9.4
matplotlib-inline==0.1.7
mdurl==0.1.2
ml-dtypes==0.4.1
mtcnn==0.1.1
mypy-extensions==1.0.0
namex==0.0.8
nodeenv==1.9.1
numpy==2.0.2
opencv-python==4.10.0.84
opt_einsum==3.4.0
optree==0.13.1
packaging==24.2
pandas==2.2.3
parso==0.8.4
pathspec==0.12.1
pexpect==4.9.0
pgvector==0.3.6
pickleshare==0.7.5
pillow==11.0.0
platformdirs==4.3.6
pre_commit==4.0.1
prompt_toolkit==3.0.48
protobuf==5.29.2
psycopg2==2.9.10
ptyprocess==0.7.0
pure_eval==0.2.3
Pygments==2.18.0
pyparsing==3.2.0
PySocks==1.7.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
pytz==2024.2
PyYAML==6.0.2
redis==5.2.1
requests==2.32.3
retina-face==0.0.17
rich==13.9.4
six==1.17.0
soupsieve==2.6
SQLAlchemy==2.0.36
sqlparse==0.5.3
stack-data==0.6.3
tensorboard==2.18.0
tensorboard-data-server==0.7.2
tensorflow==2.18.0
tensorflow-io-gcs-filesystem==0.37.1
termcolor==2.5.0
tf_keras==2.18.0
tomli==2.2.1
tqdm==4.67.1
traitlets==5.14.3
typing_extensions==4.12.2
tzdata==2024.2
urllib3==2.3.0
vine==5.1.0
virtualenv==20.28.0
wcwidth==0.2.13
Werkzeug==3.1.3
wrapt==1.17.0
zipp==3.21.0

Reproducible example

I have provided a rotated image image to .represent() method with retinaFace face detection model and FaceNet512 as recognition model.

Debugged the below method in modules/detection.py


def align_img_wrt_eyes(
    img: np.ndarray,
    left_eye: Optional[Union[list, tuple]],
    right_eye: Optional[Union[list, tuple]],
) -> Tuple[np.ndarray, float]:
    """
    Align a given image horizantally with respect to their left and right eye locations
    Args:
        img (np.ndarray): pre-loaded image with detected face
        left_eye (list or tuple): coordinates of left eye with respect to the person itself
        right_eye(list or tuple): coordinates of right eye with respect to the person itself
    Returns:
        img (np.ndarray): aligned facial image
    """
    # if eye could not be detected for the given image, return image itself
    if left_eye is None or right_eye is None:
        return img, 0

    # sometimes unexpectedly detected images come with nil dimensions
    if img.shape[0] == 0 or img.shape[1] == 0:
        return img, 0

    angle = float(np.degrees(np.arctan2(left_eye[1] - right_eye[1], left_eye[0] - right_eye[0])))

    (h, w) = img.shape[:2]
    center = (w // 2, h // 2)
    M = cv2.getRotationMatrix2D(center, angle, 1.0)
    img = cv2.warpAffine(
        img, M, (w, h), flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_CONSTANT, borderValue=(0, 0, 0)
    )

    return img, angle

### Relevant Log Output

I have provided this image:
![shahrukh_rotated](https://github.com/user-attachments/assets/e7fa4433-569c-4b2a-b9bc-7724b96cf2f0)

Just before getting into align_img_wrt_eyes, the img looked like this

![img_from_align_img_wrt_eyes_rotated](https://github.com/user-attachments/assets/d1a5412f-6553-4762-9845-045403e99e78)

align_img_wrt_eyes() method returned this:
![final_img](https://github.com/user-attachments/assets/70b0149f-9c56-43fe-8c92-b29a34494b42)



### Expected Result

I expected two things:

1. The bounding box should have rotated along with the image, rather than remaining fixed. The current behavior causes parts of the face to be cropped.
2. The image should have been properly aligned horizontally, but it is still misaligned.

### What happened instead?

1. The bounding box remained fixed while only the image rotated, causing parts of the face to be cropped.
2. The image did not align properly and is still tilted instead of being horizontally straightened.

### Additional Info

_No response_

[serengil]

I tested the given image with retinaface. You didn't mention the detector backend you are using. Also, I assume you didn't share the original image because the one in the link is low quality and opencv cannot find a face in it.

As you can see, it finds the coordinates of eyes with highlighted points.

TLDR: deepface can align faces without any bug, the problem here is that face detectors fail to detect eye coordinates properly when images aligned 90 degrees. So, unfortunately, nothing can be done for this.

[jogendra-india]

I tested the given image with retinaface. You didn't mention the detector backend you are using. Also, I assume you didn't share the original image because the one in the link is low quality and opencv cannot find a face in it.

As you can see, it finds the coordinates of eyes with highlighted points.

TLDR: deepface can align faces without any bug, the problem here is that face detectors fail to detect eye coordinates properly when images aligned 90 degrees. So, unfortunately, nothing can be done for this.

Thanks for clarifying this.
I used the RetinaFace detector (as mentioned earlier in the bug report), and it appears that the issue lies with the face detector.

Regarding my other point, the bounding box doesn’t seem to rotate along with the image during alignment, causing parts of the face to fall outside the box (as shown in these examples). Should we address this?

Unaligned image:

aligned image:

[serengil]

oh i just understand what you mean

here, we should replace w and h beucase angle is too much

[jogendra-india]

oh i just understand what you mean

here, we should replace w and h beucase angle is too much

I tried rotating the bounding box along with the image, but it seems this approach also requires additional adjustments as the accuracy was decreasing. I'll continue exploring to see if I can come up with a more reliable solution. Thanks for taking the time to look into this!

[serengil]

feel free to create a PR if you can find a way, I will be able to work on this to night