Let’s implement RTS-style mouse picking function

First, pass mouse position in screen-coordinate

We should register mouse button callback function to our window.

Then, convert it to clip coordinate.

Divided screen space x,y by window width,height, we don’t need z coordinate yet.

Next, Get inverse of projection matrix to send clip coordinate to view coordinate.

Finally, get view-coordinate to world-coordinate matrix, by multiplying with this matrix, we can get direction toward clicked position.

Then, check if the selected position is in herd’s area.

I used point vs AABB check.

If herd is selected, calculate mid point of herd and set direction toward position.

Done!