lookAt() function on camera not behaving as expected in WebXR after VR headset moves

[bencf]

Description

We’re having an issue with using the lookAt() function to try to direct the camera position in WebXR. We are not sure if this is a bug in ThreeJS code or that we are not doing this properly or misunderstanding what the expected behavior should be, but it seems like a bug so we figured we’d file it here and see what the response is.

The goal is to position the viewing direction for the user when they are in XR mode. Our understanding of how to do this now is to create a parent object of the camera (which we are calling cameraRig) as a THREE.Group and then add() the camera to it. Then when we want to move the XR camera in the scene, we change the position of the cameraRig, which essentially moves the camera as well without conflicting with the user’s own movement of the headset. This has mostly been working well for us to this point.

There have been some complications with the fact that the scene seems to be oriented by default where the origin is the current location of the user’s headset. That is, if the user enters the scene, then moves the headset 1 meter in the x direction, and we want to move the cameraRig back to the original spot they started, we cannot move it to (0, 0, 0) (the absolute coordinates of the spot we want them to be at), as that will keep the positioning the same. Instead we have to move it to (-1, 0, 0) since relative to the new position of the headset, that is now how to get it back to the origin of the scene. We have accounted for this by adding offsets into our code for the positional movements of the headset when this occurs, which fixes the issue, though it does make for more complicated code.

However, we have encountered a problem when doing this and then using the lookAt() function on the cameraRig. If a user enters XR mode, moves the headset, and then we try to lookAt() a specific point, the user’s view seems to come from a different position. We were under the impression that lookAt() should only change the orientation of the camera, not the positioning of it, but it seems to in this case. We have even tried to set the quaternion directly and avoid the lookAt() function to achieve similar functionality, but this also has the same effect (assuming we are doing this correctly, as that code/math is more complicated and so we could easily have errors in how we’re doing it).

Reproduction steps

We have produced a minimal reproduction of this (linked below). If you run it with a WebXR emulator extension in a browser, you should see the following behavior:

• Click Enter VR. You are placed inside a green cube.
• Type the letter B. It should position you in such a way that you can see red and blue boxes.
• Type the letter L. It should then have you look toward the pink box.
• Reload the page
• Click Enter VR
• Use the emulator to move the headset a significant amount in any direction.
• Type the letter B. It should once again position you in such a way that you can see red and blue boxes as before, since we are accounting for the headset movement as an offset here when we reposition the cameraRig.
• Type the letter L. It is running the exact same code as before, but this time the view is entirely different from before, not just the direction but also the positioning of where the camera is viewing things from.

Again, in our experimentation with the full code we’re working on, this happens any way we try to set the orientation of the cameraRig after the headset has moved. It is entirely possible we are misunderstanding something, but think it might be an issue with how the orientation transform is being applied when the coordinates are relative to the headset and not relative to a stable world origin. But after a fair amount of experimentation, we cannot figure out another way to orient the camera in the direction we want (toward a specific set of coordinates) that doesn’t also change the location of where the camera seems to be looking from.

Code

The relevant portion of the code that is performing this change:

  const targetPoint = new THREE.Vector3(5, 1.6, -1);
  cameraRig.lookAt(targetPoint);
  cameraRig.updateMatrixWorld(true);

Live example

https://codepen.io/bencf/pen/ZYzbxQe

Screenshots

No response

Version

r171

Device

Desktop

Browser

Chrome

OS

MacOS

[mrxz]

The described behaviour does not sound like a bug, although it might appear counter-intuitive at first. During a WebXR session the camera is positioned and orientated based on the headset pose coming from the WebXR API. This is also the reason you can't directly set the position of the camera, but instead need to use a camera rig that holds the camera.

The (0, 0, 0) coordinate inside this camera rig corresponds with the origin of the reference space used by the WebXRManager (see getReferenceSpace). This allows the physically movement of the user to be combined with any virtual movement (e.g. locomotion, teleporting, etc...). The camera effectively represents the head of the user, while the camera rig represents the reference space the user is in.

When moving or rotating cameraRig you aren't directly moving or rotating the user, but the reference space they are in. As you've already noticed, this means you have to take into account where the user is inside this reference space.

But after a fair amount of experimentation, we cannot figure out another way to orient the camera in the direction we want (toward a specific set of coordinates) that doesn’t also change the location of where the camera seems to be looking from.

Unless the user is precisely at the origin of the reference space, rotating the reference space will also "move" the user in world space. Instead you'll have to both rotate and translate the camera rig such that the camera ends up in the desired location and with the desired orientation.

Assuming the camera is a direct child of cameraRig and that cameraRig is a direct child of the scene, it would look something like this:

const targetMatrix = /* Compute the world matrix you'd want the camera to have */;
const cameraMatrixInverse = camera.matrix.clone().invert();
cameraRig.matrix.multiplyMatrices(targetMatrix, cameraMatrixInverse);
cameraRig.matrix.decompose(cameraRig.position, cameraRig.quaternion, cameraRig.scale);

Also, I'd recommend avoiding any rotations that case the +Y axis of the camera to no longer align with the world up. It can be really disorientating and nauseating for users if the horizon is slanted/tilted.

[bencf]

This makes a lot of sense and is an incredibly helpful explanation that we didn't find elsewhere. I really appreciate you taking the time to write it out in such detail and provide the code for us to try to build off of. As I said at the start, I thought there was a significant chance that it was simply a misunderstanding on our part, and I'm glad that is what it ended up being!