Hm... So I made my character a little bigger than I needed, then changed the scale factor to 50 in the character properties so it is the size I want. I guess that's what messed it up. But, if I don't set any scale in the waypoints, it still messes up. So, I set the first point to 50 then it was fine from then on.
I guess I just misunderstood the scaling system to begin with.