Connecting to Mosquitto in Home Assistant
The Mosquitto add-on listens for MQTT on 1883 and WebSocket on 1884, and with TLS on 8883 and 8884. From an HTTPS page the browser allows only wss://, so use 8884 with a certificate (the same /ssl/fullchain.pem and privkey.pem as Home Assistant) or an HTTPS proxy in front of 1884. Create a dedicated user in Settings → People (or logins in the add-on options) instead of using the admin account.
Refusal codes
The broker answers a connect with a code: 4 is a bad username or password, 5 is not authorised (the user exists but the ACL denies it, or anonymous access is off). If the WebSocket never opens, the problem is lower: a closed port, the wrong path (/mqtt on HiveMQ and EMQX, none on Mosquitto) or the certificate. Check a port from outside: port check. Certificate: SSL check.