The problem you are experiencing is inherent with the state
variable filter which is the topology on which the modfilter is
based. The problem is that when the cutoff frequency gets to about
2/5ths of the sample rate, you will get that noise and distortion.
There is no solution at the moment. Avoiding that 10KHz threshold is the only possible way to use it right now.
Hopefully a new module or a newer version of the current one will be released.
The bug is in the database.
all the best