Many rail level crossings (RLXs) have only passive protection, such as static signs instructing road users to stop, yield, or look for trains. Stop signs have been suggested as a low-cost option to improve safety at passive RLXs, as requiring drivers to stop should encourage safe behaviour. However, field observations have noted high rates of non-compliance at stop-controlled RLXs. To explore this further, we conducted an on-road study to identify factors that influence compliance at stop-controlled RLXs. Twenty-two drivers drove a 30.5km route in rural Australia, encompassing three stop-controlled RLXs. In over half of all cases (59%) drivers stopped completely at the RLX; on 27% of crossings drivers executed a rolling stop, and on 14% of crossings drivers violated the stop controls. Rolling stops were defined as a continuous deceleration to <10km/h, but remaining above 0km/h, before accelerating to >10km/h. Behavioural patterns, including visual checks and decision-making, were similar when comparing drivers who made complete versus rolling stops. Non-compliant drivers did not differ from compliant drivers in approach speeds, but spent less time visually checking for trains. Post-drive interviews revealed some drivers wilfully disregarded the stop sign, whereas others did not notice the stop sign. Those who intentionally violated noted trains were infrequent and suggested sight distance was good enough (even though all crossings had been formally assessed as having inadequate sight distance). Overall the results suggest most drivers exhibit safe behaviour at passive RLXs, but a notable minority disregard or fail to notice signs. Potential avenues for redesigning passive RLXs to improve safety are discussed.