RAG with Pinecone + GPT-5 for generating new math problems: incoherent outputs, mixed chunks, and lack of originality

I think this is just a constraint with the current models, but would be happy to be proven otherwise. As far as LLMs go, they cannot be creative, they will, as for now atleast, always only transform already known content. Maybe you can try extracting certain topics and concepts and then try to let the model combine them to create more demanding and creative problems?

single pass RAG performs poorly for these kind of tasks...
to address this issue we have to go for multi-agent... need to introduce generate-and-verify loop.

use rag only for retrieving high-level patterns, then use these in generation step... now give this new problem to an LLM (solver) which only task is to solve the problem.

this self-verification will help you generate better problems.