I really like your ideas.  There was some talk at one point about global controllers, assignable to parameters, such as an LFO or two.  I say a step sequencer with smoothing & one-shot (or X repetition) capability covers almost everything.  ADSRs, wavetable shapes, etc.

No idea the difficulties involved to implement that.  Your concept seems more practical, contained inside an algorithm package.  That excellent PolySynth envelope follower needs more targets.  Maintain the EF, use its detection trigger, split & reshape that envelope before branching off to amplitude, cutoff, resonance (and dare I add pitch shifting).

The same basic concepts are there in HotSawz, in a limited sense.  As for existing general control over envelope, there are a few options you can try.  With a few threads on this forum about the upsides & downsides in each one.

DynaVerb / Omnipressor.  UltraTap / Chop Machine, using the Swell & Trigger selections.  TremoloPan, using an Envelope Shape.  (The Modulation ADSRs have a short, fixed profile).  Ducked Delay at zero delay times.  Might have missed some.  Not complete solutions, but situationally useful.  They can be tricky to adjust, and find your way around the core limitations.