> why don't you just host your papers on your website then - or, if you do, why don't you think it's enough?
Most publishers prevented you from doing this as of a few years ago (I left academic research around that time). You were not allowed to post any content that contains any work by the publisher (even formatting/editing changes after first round of reviews). Thus you could only publish a "preprint" which no-one can rely on to cite because they don't know what's in the final peer-reviewed version. Some publishers are more lenient than others but there's definitely friction induced.
The norm at least in parts of physics is to update the arxiv version once the paper is accepted, with a note saying "v3: matches version to appear in XXX", with a DOI link. Where "matches" means the content, but not the journal's formatting, spelling & comma preferences, etc.
(And ideally this note would also mention any substantial changes, like "new section 4 explaining..." or "derivation in 3.2 re-written, but no change to conclusions", for the benefit of those who already read v2.)
I do the same (over in comp sci.) - usually the journals will let you put pre-print articles up on arxiv, researchgate, and web pages, and then the final article - sometimes with added spelling mistakes - many of the journals have off shored the editorial process - is on their site.
I'm curious what would happen if a new paper tried to reference previous papers published in ArXiv instead of the traditional publishers. From a strictly academic perspective, I can't think of any reason why this would be a problem, apart from the existential problem it causes for traditional publishers.
Citing papers which aren't yet published by their arxiv numbers is routine. (And, before 1991, people did the same using the author's institute's preprint numbers.) If they have appeared in a journal by the time yours is accepted by a journal, then you should add the journal info too. (Although many journals now allow you to leave the arxiv numbers as an additional part of each reference.)
I (not OP) do, https://jan.hermann.name/publications/ (note the copyright notices, required by the journals, which pop up when you hover of the pdf link [didn't figure out a good alternative on mobile]). In rare cases, the journals don't allow this at all, in some cases they allow you to self-publish only the submitted manuscript, before any modifications based on peer review.
As for why it's not enough—because hiring committees and funding agencies almost never take unpublished (that is, in a proper journal) manuscripts into account when evaluating you and your funding proposals. This is what needs to change in the first place to break the loop.
Google Scholar still gets its PDFs from preprint servers and author's websites. I suppose you meant to say that Google Scholar's web indexing is how many people got their PDFs pre-scihub?
Im not sure if it's legally flawless to upload the published journal article to your Website and open it up to the public.
As far as i know, you're only allowed to pass it on a personal, per-request basis. This explicitly refers to the final peer reviewed article and not to the manuscript, which can make a great difference.
Many publishers explicitly allow you to post your own articles on your personal/professional website. Some only allow "preprint" form and not the published version.
Most publishers prevented you from doing this as of a few years ago (I left academic research around that time). You were not allowed to post any content that contains any work by the publisher (even formatting/editing changes after first round of reviews). Thus you could only publish a "preprint" which no-one can rely on to cite because they don't know what's in the final peer-reviewed version. Some publishers are more lenient than others but there's definitely friction induced.