Vision-Language-Action (VLA) models are gaining attention in autonomous driving, inheriting commonsense priors from large language models and widening the navigation interface from a fixed command set to free-form natural-language instructions. This open-ended interface, however, exposes a new vulnerability surface: an inattentive driver may issue instructions that are unsafe given the current scene, and a malicious driver may issue dangerous ones by design. We first construct a suite of benchmarks that evaluate VLA models along three axes: (i) safe navigation following, (ii) navigation-scene conflict awareness, and (iii) robustness to adversarial instructions, revealing that existing policies are deficient on both safe following and conflict awareness. Guided by these observations, we propose SafeDriveVLA, which decouples conflict reasoning from action generation through an interpretable conflict prediction before the action generation, together with navigation-conditioned world model dreaming that exposes whether the instructed maneuver is feasible before the policy commits. SafeDriveVLA reaches state-of-the-art 83.64 DS and 59.82% SR on Bench2Drive, lifts closed-loop language following by 48.7%, and reduces collisions under unsafe instructions from 66 to 45 compared to prior state-of-the-art.