I never noticed that the entrance to the Crystal Desert is in the Field of Ruin. Apparently, only by order of Queen Jennah will the guards open it! Not to be deterred, I did some poking around until I found myself a way past those pesky guards!