Raycast in Three.js with only a projection matrix : <p>I'm rendering some custom layers using Three.js in a Mapbox GL JS page following <a href="https://docs.mapbox.com/mapbox-gl-js/example/add-3d-model/" rel="noreferrer">this example</a>. I'd like to add raycasting to determine which object a user has clicked on.</p>

<p>The issue is that I only get a projection matrix from Mapbox, which I use to render the scene:</p>

<pre class="lang-js prettyprint-override"><code>class CustomLayer {
  type = 'custom';
  renderingMode = '3d';

  onAdd(map, gl) {
    this.map = map;
    this.camera = new THREE.Camera();
    this.renderer = new THREE.WebGLRenderer({
      canvas: map.getCanvas(),
      context: gl,
      antialias: true,
    });
    this.scene = new THREE.Scene();
    // ...
  }

  render(gl, matrix) {
    this.camera.projectionMatrix = new THREE.Matrix4()
      .fromArray(matrix)
      .multiply(this.cameraTransform);
    this.renderer.state.reset();
    this.renderer.render(this.scene, this.camera);
  }
}
</code></pre>

<p>This renders just great, and tracks changes in view when I pan/rotate/zoom the map.</p>

<p><a href="https://i.stack.imgur.com/iQXGc.png" rel="noreferrer"><img src="https://i.stack.imgur.com/iQXGc.png" alt="A cube on Liberty Island"></a></p>

<p>Unfortunately, when I try to add raycasting I get an error:</p>

<pre class="lang-js prettyprint-override"><code>  raycast(point) {
    var mouse = new THREE.Vector2();
    mouse.x = ( point.x / this.map.transform.width ) * 2 - 1;
    mouse.y = 1 - ( point.y / this.map.transform.height ) * 2;
    const raycaster = new THREE.Raycaster();
    raycaster.setFromCamera(mouse, this.camera);
    console.log(raycaster.intersectObjects(this.scene.children, true));
  }
</code></pre>

<p>This gives me an exception:</p>

<pre><code>THREE.Raycaster: Unsupported camera type.
</code></pre>

<p>I can change from a generic <code>THREE.Camera</code> to a <code>THREE.PerspectiveCamera</code> without affecting the rendering of the scene:</p>

<pre class="lang-js prettyprint-override"><code>this.camera = new THREE.PerspectiveCamera(28, window.innerWidth / window.innerHeight, 0.1, 1e6);
</code></pre>

<p>This fixes the exception but also doesn't result in any objects being logged. Digging a bit reveals that the camera's <code>projectionMatrixInverse</code> is all <code>NaN</code>s, which we can fix by calculating it:</p>

<pre class="lang-js prettyprint-override"><code>  raycast(point) {
    var mouse = new THREE.Vector2();
    mouse.x = ( point.x / this.map.transform.width ) * 2 - 1;
    mouse.y = 1 - ( point.y / this.map.transform.height ) * 2;
    this.camera.projectionMatrixInverse.getInverse(this.camera.projectionMatrix);  // &lt;--
    const raycaster = new THREE.Raycaster();
    raycaster.setFromCamera(mouse, this.camera);
    console.log(raycaster.intersectObjects(this.scene.children, true));
  }
</code></pre>

<p>Now I get two intersections wherever I click, with two faces of the cube. Their distances are 0:</p>

<pre><code>[
  { distance: 0, faceIndex: 10, point: Vector3 { x: 0, y: 0, z: 0 }, uv: Vector2 {x: 0.5, y: 0.5}, ... },
  { distance: 0, faceIndex: 11, point: Vector3 { x: 0, y: 0, z: 0 }, uv: Vector2 {x: 0.5, y: 0.5}, ... },
]
</code></pre>

<p>So clearly something isn't working here. Looking at the <a href="https://github.com/mrdoob/three.js/blob/441fefadc017ebafd2d10089b6d48e1cd28795c2/src/core/Raycaster.js#L81-L83" rel="noreferrer">code for <code>setCamera</code></a>, it involves both <code>projectionMatrix</code> and <code>matrixWorld</code>. Is there a way I can set <code>matrixWorld</code>, or construct the raycaster's ray directly using only the projection matrix? It seems that I only need the projection matrix to render the scene, so I'd hope that it would also be all I need to cast a ray.</p>

<p>Full example <a href="https://codepen.io/danvk/pen/GRggjmg?editors=0010" rel="noreferrer">in this codepen</a>.</p>
