“So I sent messengers to them, saying, ‘I am doing a great work and I cannot come down. Why should the work stop while I leave it and come down to you?'” Nehemiah 6:3
You can not allow any negative voices inside of your head while on your way to the completion of your task.
Much like those who attempted to discourage Nehemiah by negative words, innuendo and even the threat of bodily harm he would not discouraged.
A rule of success: Do not go down to those who will not come up to you.