React Three Fibre follow mouse respect controls

Question

I'm trying to get something to follow the mouse, after controls have changed the camera position

Here is an example https://codesandbox.io/s/r3f-mouse-forked-to146p?file=/src/index.js

If you drag the screen, the object is no longer next to the mouse

I need to update this position in relation to the camera position I assume.

const ref = useRef()
  useFrame(({ mouse }) => {
    const x = (mouse.x * viewport.width) / 2
    const y = (mouse.y * viewport.height) / 2
    ref.current.position.set(x, y, 0)
    ref.current.rotation.set(-y, x, 0)
  })

Answer 1

Gave up and used the RayCaster instead

  const updateMousePosition = mouse => {
    raycaster.setFromCamera(mouse, camera);
    const intersects = raycaster.intersectObjects(scene.children, true);
    if (intersects.length == 1) {
      vec3.copy(intersects[0].point);
      elRef.current.worldToLocal(vec3);
      setMousePos([vec3.x, vec3.y, vec3.z]);
    }
  };

  useFrame(state => {
    if (selected) updateMousePosition(state.mouse);
  });