[DUI/NUI] Request for SEND_DUI_KEY_DOWN, SEND_DUI_KEY_UP

SQLabs · 2023-09-04T17:04:59.480Z

Why?

Currently, DUI is only able to Accept mouse input into the browser, which heavily limits functionality.
Pretty much all big sites (google, YouTube, disk cord, etc) block being loaded into iframes, which completely removes the possibility of a script adding their own top-level JS to handle it.

Allowing Keyboard Input allows so many more possibilities to what you can accomplish with DUI rendering

Usage/Examples

Browsing YouTube

(using voice chats and playing games, would be cool to be able to send actual messages)
Watch [Mycroft] DUI Discord Showcase | Streamable

Other Cool use cases
https://streamable.com/n789ho

*Notable usage: browsing https://google.com/

As you can see in the amazing previews above, I’m able to use mouse input to do anything I want, but, when it comes down to needing text input, I cannot do anything, from logging into sites, sending messages, searching google and YouTube, Hell it can even render native 4k video and images but not even accept a simple key press?

Implementation Suggestion

User Experience

My Suggestion for Implementation is to use the same methods as SEND_DUI_MOUSE_DOWN and SEND_DUI_MOUSE_UP, and they would be used in the exact same way:

if IsDisabledControlJustPressed(0, 38) then --Pressed E
            SendDuiKeyDown(duiObject,  "e")
end
if IsDisabledControlJustReleased(0, 38) then --Released E
            SendDuiKeyUp(duiObject,  "e")
end

they can also be combined by the user like so:

function KeyPressed(dui, key)
    SendDuiKeyDown(dui,  key)
    SendDuiKeyUp(dui,  key)
end

KeyPressed(duiObject, "e")

Backend

As for the backend, my suggestion is:

		.AddMethod("SEND_DUI_KEY_UP", &NUIWindowWrapper::InjectKeyUp)
		.AddMethod("SEND_DUI_KEY_DOWN", &NUIWindowWrapper::InjectKeyDown)

Adding these to the pre-existing hooks/injections would work great,
switching out the existing,

browser->GetHost()->SendMouseMoveEvent(ev, false);

with

browser->GetHost()->SendKeyEvent(keyEvent);

Similar to the existing KeyEvent System for normal NUI inputs, and developers should easily be able to add keyboard input into their DUI without having to use hacky and dangerous methods.

Ending

Thanks for Reading, and I hope we can massively improve how amazing DUI is, and allow more amazing scripts to be made.

DerDevHD · 2023-09-04T19:40:06.992Z

You can achieve this yourself using SendDuiMessage.

Just pass the wanted parameters, like a key in this case, to this method as a JSON string.

On the DUI part you wan’t to have a handler which intercepts the event (just like in NUI) and execute the keypress.

Simple JS keypress example:

var keyboardEvent = document.createEvent('KeyboardEvent');
var initMethod = typeof keyboardEvent.initKeyboardEvent !== 'undefined' ? 'initKeyboardEvent' : 'initKeyEvent';
    
keyboardEvent[initMethod](
  'keydown', // event type: keydown, keyup, keypress
  true, // bubbles
  true, // cancelable
  window, // view: should be window
  false, // ctrlKey
  false, // altKey
  false, // shiftKey
  false, // metaKey
  40, // keyCode: unsigned long - the virtual key code, else 0
  0, // charCode: unsigned long - the Unicode character associated with the depressed key, else 0
);
document.dispatchEvent(keyboardEvent);

Source: javascript - Is it possible to simulate key press events programmatically? - Stack Overflow

EDIT: may be a bit hacky to include custom js files inside webpages like youtube, etc. You may use a custom DUI site and embed your webpages there using iframes. (And some technolegies that bypass iframe-blocking websites).

P.S. I’ve done something similar some time ago.

SQLabs · 2023-09-04T19:46:47.027Z

DerDevHD:

On the DUI part you wan’t to have a handler which intercepts the event (just like in NUI) and execute the keypress.

that would require top Level JS, which is Not usable within this context.
since when you create a DUI, it sets the URL, like a browser tab. unless u set it to be your own html file, you have no control of key presses.

Iframes are not a solution here, as i explained in the post
that is just a really hacky and not really a solution to the issue.

Whereas, if u can just inject the KeyPress in, the same way the mouse events are, that is the proper way of achieving it

For example, if you did,

local duiObject =  CreateDui("https://youtube.com", 2048, 1024)

your Code would not work, because there is no where to put your code, and you cannot iframe that URL, so it remains impossible to go visit that URL in a DUI context, and search for videos, or sign in, within the DUI.

Why limit what you can do, when there is a path for more?

DerDevHD · 2023-09-05T20:00:00.360Z

I agree.

nta · 2023-09-10T12:00:13.689Z

SQLabs:

but not even accept a simple key press

Because it’s not ‘simple’. See the comment on your pull request: feat(nui/resources): SEND_DUI_KEY_DOWN & SEND_DUI_KEY_UP by Mycroft-Studios · Pull Request #2195 · citizenfx/fivem · GitHub.

Chang-nl · 2024-12-05T18:25:10.477Z

Hey just a quick question, how did you get the mouse working and send the correct mouse coordinates to the dui?