I am using a ssh tunnel and screen to connect to the server.
ssh $USER@$IP "screen -S $USER$PORT -t remoter$PORT -d -m Rscript -e 'remoter::server(port=$PORT, showmsg=TRUE, userpng=FALSE)'"
Server R process dies when I call 'load("my.RData")' and client halts. Executing in an interactive R session on the server works without any problem.