One simple thing I'm missing is a way to point at a word, and let the LLM know that that's where it went wrong (so it can generate an alternative output).
LM studio allows you to edit the context of an existing message like this, and you can even insert whatever writing to continue a thought, then have it continue generating.