My suggestion: Clean out the Speed potentiometer with some Deoxit spray (or similar cleaner). What I think is happening is that you’re getting intermittent, spurious values from the Speed pot. While it *could* be a circuit issue, cleaning out the pot is a cheap fix.
You say that it’s always the Speed pot, so it may be jittering between values. The ModFactor “thinks” that you’re adjusting the Speed, so it switches to fine-tune encoder mode. Just a theory, but I’d try that first. You may be able to dislodge dust using compressed air. DON’T use WD-40 (it leaves a sticky residue). Turn the pot several times after applying the cleaner; fully clockwise and counter-clockwise.