Leaving town simply wouldn't work, see the rise of Julius Caesar. He went into Rome untouched and his armies easily beat Pompey's after Caesar gained the favor of Rome's denizens.
I'd start public executions of well known conspirators to silence the potential upheaval. I'd enact a curfew to no one out of their house after work, having my 500 soldiers patrol the streets. If conspiracies persisted, I'd just line up any person that has any esteem in the community: teachers, priests, lawyers, scientists, etc. etc. and have them brutally murdered. Fear is the best option in this case.
If by some odd chance that didn't work, and a coup was still planned, I'd just leave for a neutral country and lead a good life there.