The old parable about the man with one clock knowing what time it is but the man with two never being sure...
First off: I use arch now more than a year on a raspberry and I love it!
NTP needs more than one server to compare against, vote and pick the 'best' time source from.
If this is a virtual server then there are other things you need to do to keep ntp in check on linux.
Two servers is the "worst possible configuration" according to the NTP authors since it will never know which one is "more" correct.
Once you get to three servers NTP can begin to mark clocks as "falsetickers" based on some internal metrics - those will show up with a status of 'x' when you run 5 is not a special number.