If you chose to require an out-of-band password, you will need to somehow transmit that password to the receipient. Make sure to use another way than you used to transmit the retrieval link, or else there is no security benefit of using an additional password at all.
E.g.: If you send the retrieval link via email, transmit the password using Microsoft Teams, or even better, verbally over the phone.