So far it hasn't been a problem (fingers crossed). I have a couple ways into the server in case I get myself banned. Also, it's my personal server, so the only user that would complain is me :)

For other sites, I'm honestly not sure how they go about choosing a limit. 20 does seem to be more reasonable, while still being perfectly safe.