The doc page for the addon is here https://github.com/home-assistant/hassio-addons/blob/7a63283562aae8f3bc7e92b9bbec271fbae26e9d/dnsmasq/DOCS.md
I think for what you are trying to do you want to change the hosts section. That looks like it’s working since ui.42o4 is resolving to 192.168.0.100.
In some places you mention ui,home but it seems you have configured ui.42o4. Is the proxy server also configured for ui.4204?
If you type http://192.168.0.100:8443 in the browser does it bring up the expected page?
What does “Running this command in Ubuntu within windows 10 (not another VM)” mean? Are you running this under the windows subsystem for linux?