Trying to fit "Person who doesn't drive" into a privileged/not privileged framework ultimately doesn't make much sense outside of further context.

A homeless person living out of a tent and a celebrity with a valet are both non-drivers.

An impoverished person taking the bus in Brooklyn is not "outsourcing driving to the poor" when the MTA driver is making an order of magnitude more than they are.