You can keep the git folder in your machine, then use sshfs to mount the remote into your directory where: project-root -.git/ -code/ <= remote/code

This way you don't have to give git access to the potentially unsafe server. Git hook attacks are still possible so disable those by defaultç

This is an unusual folder structure but works fine, let me know if there is anything iffy