Why the broker refuses
Since 2.0, Mosquitto with no config listens on localhost only and refuses anonymous clients. That is why you need listener 1883 and a password file. Code 5 (not authorised) means a bad login or ACL; a connection error means the port is closed or the broker listens elsewhere.
Test the broker from the browser (needs WebSocket, and wss on 8884 from an HTTPS page): MQTT client. Get a Let’s Encrypt certificate as for a website; restart Mosquitto after renewal.