The delay in Shimmer isn't in any direct dry path like a conventional delay. The Shimmer algorithm outputs come directly from the stereo reverb. The delays sit in a feedback path that goes from this output back to the input of the reverb (the pitchshifters are also in this path), so really you can think of the delay as "growth" control for the pitch swells. Lower delays will cause pitchshifting to occur faster and vice versa.
We felt that bounding the delay was the best choice for the sound, as it allowed the really large pad-like verb to breathe just a little before we routed a pitch shifted version of itself back into itself. The decision really had nothing to do with it being too close to anything in Pitchfactor, because fundamentally Shimmer is a very different algorithm stucture. So, no, it is not currently possible to have no delay.
Hope this helps, and thanks for digging in and asking good questions!