A spirit alone might not be a direct threat to them, but perhaps operating with the minds, bodies, and extensive abilities of the villagers opens up some dangerous doors.

[This is just a guess, of course.]