When using a graph database you can build a knowledge Graph out of the long term memory. Storing it only in a vector database means that you'll only find things that are similar to the user question and miss a lot of information that is an aggregation of different memories.
It's a very similar reason on why using Graph for RAG can help you get much more accurate responses than with Vector RAG. See a blog I wrote about it https://www.falkordb.com/blog/knowledge-graph-vs-vector-data...