A difficult person can indirectly affect the overall success of a whole department or company. When there is dissention in your ranks or among your peers, the losses in energy, enthusiasm, and productivity diminish results in ways that are difficult to quantify.
Team efficiency suffers if you and your subordinates simply avoid the person causing the problems. Information gets lost when people don't communicate. The weak spots in your departmental efforts will create points of dysfunction and disconnect with customers and other departments.
As problems mount, your reputation as an effective leader and team player will suffer, and your whole department will feel the consequences. The mud slide of problems sucks the vitality out of your workplace and saps the energy that you and your best people owe to your jobs.