If a CSRF attack is not coming from a browser, how are you going to get the victim's session cookies? If you do get them, it's not called CSRF anymore.
What's stopping an attacker from harvesting session cookies, and then using them in a separate attack? I'm not that familiar with it, but possibly phantom.js hooked up to an exploit tool could work as well?
I'm not saying referrer checking is useless, but once again, you never trust user input. Ever. If you follow that rule you're decreasing attack vectors to essentially social engineering, brute force, and complex technological exploits using buffer overflows and the like (which I guess is still kind of a side effect of trusting user input).
Well, a CSRF attack doesn't involve reading cookies. I believe you'd need an XSS vulnerability in order to do that. (Or, if the site isn't using SSL, just sniff the network traffic.) CSRF just transmits the legitimate cookies to the server - but with a request the victim probably didn't want to actually send. At no point are the victim's cookies able to be harvested.
You could indeed possibly forge a request after harvesting session cookies - but that isn't CSRF. It's either XSS or traffic snooping.
One way to cut down the (time) window of opportunity for an attack is to have the server generate new session tokens, and store them in the cookie, on each request, but require a valid token in order for a new token to be generated. If you have an XSS vulnerability, this is not an adequate solution, as it requires the victim to submit another request between the time the attacker steals the cookie and the time the attacker submit the cookie to the server.
So basically, use SSL and be really careful about XSS. You can trust the REFERER header and session cookies together if and only if they are both present. "Never trust user input that you have to escape or that will be rendered to the browser" is very reasonable though.