GPU based raycasting / depth testing

Kiminaze · 2023-02-16T10:42:25.268Z

Apologies for posting in the wrong category. Must have switched browser tabs unintentionally while creating the topic earlier today.

Problem

We are kind of limited in the way we can use raycasting (or ShapeTests as they are called in the natives; I’ll call them raycasts for this post).

These two are basically the only ones being used:

StartShapeTestLosProbe - FiveM Natives @ Cfx.re Docs
StartExpensiveSynchronousShapeTestLosProbe - FiveM Natives @ Cfx.re Docs

But they are kind of limited in their functionality.

Want to ignore more than one entity? Not possible (or using flags e.g. only all objects or all peds)
Cannot go through world hitboxes (unless flags specified which ignore all world objects) (e.g. a railing on a balcony that is not an object but has a rectangular hitbox although it is just 3 bars)

Example for the world hitbox issue:
This railing is a solid hitbox. If I use the collision flag to ignore world objects, it will not only go through the railing, but also the wall/floor behind it.

image1350×680 54.7 KB

Location:

image1309×653 91.4 KB

Suggestion

Wouldn’t it be really cool if we could have one raycast that acts as a depth test? Querying the depth buffer and returning the actual point in the world being rendered? Looking at a fence this would net the pixel position behind it without having to cast a ray and ignoring a giant manually created list of objects.

image888×545 72.9 KB

In general this should be possible (as I have already worked with DirectX and currently in Vulkan), however I don’t know if this is something that can be hooked into FiveM and it would probably take ages for me to get as far into this subject to be able to do it myself.

Use cases

… that I can think of right now:

Being able to select objects with disabled collision.
Selecting objects behind or inside other objects without ignoring those objects. (e.g. a crate on the backseat of a vehicle)
Ignoring two (or more) objects in a raycast without resorting to casting additional rays.

(“selecting” referring mostly to being able to use the ingame cursor to highlight and select objects like can be seen here [Release] ContextMenu v2.0 - Major Update!)

Another cool feature would be to be able to ignore certain models instead of a single object. E.g. a raycast that ignores a list of models (like all fences for example).

Linden · 2023-02-16T16:40:06.345Z

Kiminaze:

Cannot go through world hitboxes (unless flags specified which ignore all world objects) (e.g. a railing on a balcony that is not an object but has a rectangular hitbox although it is just 3 bars)

Example for the world hitbox issue:
This railing is a solid hitbox. If I use the collision flag to ignore world objects, it will not only go through the railing, but also the wall/floor behind it.

Setting p8 as 2 will ignore see through objects like the railing and chain link fence fwiw

Kiminaze · 2023-02-16T22:35:59.316Z

Linden:

Setting p8 as 2 will ignore see through objects like the railing and chain link fence fwiw

Thank you for that answer! I didn’t even consider changing that parameter. That’s definitely already a game changer.

I’ve done some additional testing and found the following things:

0 and 4 seem to do the exact same thing colliding with everything
1 works great with the metal fences, as the raycast goes through the chain link part, but not the metal pipes, sadly does not work on windows though
2 seems to go through everything that is (partially) seethrough? Like vehicle windows and glass in general which is extremely useful as well. Sadly it seems to go through the whole fences (and not just the chain link part).
3 doesn’t allow for combining both at the same time and just acts the same as 2 as far as I can tell.

Though I would still like to keep this thread open for the “ignore multiple objects/models raycast”. That would still be something useful in certain cases.

Linden · 2023-02-17T06:45:31.480Z

Kiminaze:

0 and 4 seem to do the exact same thing colliding with everything

1 works great with the metal fences, as the raycast goes through the chain link part, but not the metal pipes, sadly does not work on windows though

2 seems to go through everything that is (partially) seethrough? Like vehicle windows and glass in general which is extremely useful as well. Sadly it seems to go through the whole fences (and not just the chain link part).

3 doesn’t allow for combining both at the same time and just acts the same as 2 as far as I can tell.

1 ignores glass, though it doesn’t seem to work on car windows.

2 ignores see-through objects, yes including partially see-through. Worse case I’ve seen is the desk in the Paleto bank being one huge see-through entity. Vehicle windows fall under this as you mentioned.

4 is supposed to ignore “no_collision” but I’m not entirely sure what that includes (setting an entity to have no collision doesn’t work).

You can use these bits together, so passing 1|2 (or just 3) should ignore both glass and see-through objects, though I assume most glass is marked as see-through. The “default” value for this arg is 1|2|4. Mixing the flags also works (i.e. 2|4|8|16 to include vehicles, peds, ragdolls, and objects).

For the meantime I recommend flag switching and distance-checking to ensure you can target all objects and the world even when overlapping.

function RaycastFromCamera(flag)
    local coords, normal = GetWorldCoordFromScreenCoord(0.5, 0.5)
    local destination = coords + normal * 10
    local handle = StartShapeTestLosProbe(coords.x, coords.y, coords.z, destination.x, destination.y, destination.z,
        flag, PlayerPedId(), 4) -- you may want 2 or 6?

    while true do
        Wait(0)
        local retval, hit, endCoords, surfaceNormal, materialHash, entityHit = GetShapeTestResultIncludingMaterial(handle)

        if retval ~= 1 then
            return hit, entityHit, endCoords, surfaceNormal, materialHash
        end
    end
end

    local flag = 511

    while isActive do
        local playerCoords = GetEntityCoords(PlayerPedId())
        local hit, entityHit, endCoords = RaycastFromCamera(flag)
        local entityType = entityHit ~= 0 and GetEntityType(entityHit) or 0
        local distance = #(playerCoords - endCoords)

        if entityType == 0 then
            local _flag = flag == 511 and 26 or 511
            local _hit, _entityHit, _endCoords = RaycastFromCamera(_flag)
            local _distance = #(playerCoords - _endCoords)

            if _distance < distance then
                flag, hit, entityHit, endCoords, distance = _flag, _hit, _entityHit, _endCoords, _distance
                entityType = entityHit ~= 0 and GetEntityType(entityHit) or 0
            end
        end

        -- etc..
        if not next(options) then -- switch flag if the targeted entity had no registered options
            flag = flag == 511 and 26 or 511
        end
    end

Kiminaze · 2023-02-19T10:18:47.064Z

I think I can take this as a solution. Thank you for the detailed explanation!

Linden:

2 ignores see-through objects, yes including partially see-through. Worse case I’ve seen is the desk in the Paleto bank being one huge see-through entity. Vehicle windows fall under this as you mentioned.

Yea, this has some weird exceptions/objects but it is absolutely fine for the most part. I found a metal staircase that is still a big box although mostly hollow

Linden:

You can use these bits together, so passing 1|2 (or just 3) should ignore both glass and see-through objects, though I assume most glass is marked as see-through. The “default” value for this arg is 1|2|4.

Yea, 3 doesn’t really do a lot of a difference compared to 2.

Linden:

Mixing the flags also works (i.e. 2|4|8|16 to include vehicles, peds, ragdolls, and objects).

Already been doing that. No clue why I didn’t think about the collisions bits when trying different flags. For my purpose I am using 27 (world, vehicles, peds, objects).

Linden:

local coords, normal = GetWorldCoordFromScreenCoord(0.5, 0.5)

I’ll have to try that native. Is it already on production? I think I remember it being beta/unstable only though I could be wrong here.
Currently I’m doing something really stupid (although it works fine!) by creating a new cam with the gameplay cams settings, then getting the matrix from the new cam for the forward vector and then deleting the cam again