And if you click the "Ellen Ternan" link there you will see that they lived together for 10 years and are even suspected of having a child (who died early).
Yeah, but he wrote this before he and his wife separated. Living with Ellen Ternan for ten years after that isn't good evidence that they had an affair before it.