@uwunix@mastodon.technology they'd need both the private key and the passphrase to do anything with it

If you're keeping your key secure in all the usual suggested ways then the passphrase doesn't matter much, it's like putting a luggage lock on something going in a safe