Is there a way to disable public lookup when composing messages?

Yes there is an easy way to do this. Simply navigate to the Admin Dashboard -> Website Settings.

Then go the following option “Enable public user search and message composition”. Make sure it is disabled as shown in the screenshot below and save settings. This will disable the user lookup completely from both Web UI and REST API.