[patio11]

This is a deep topic. Here's an answer to it, which might not be everyone's go-to answer:

Consider the case where you know either the software or the system which is the target. Consider also the case where your goal is "get a reliably exploitable vulnerability while coming in under budget" as opposed to e.g. "enumerate a sufficiently broad swathe of vulnerabilities such that a stakeholder is pleased with your diligence." You will probably prioritize vulnerability classes which, in your experience and/or that of the industry, are numerous and high-severity. You will probably also prioritize "the joints" of the system, because (if you've done software development professionally) you know that handoffs between scripts, teams, servers, processes, etc etc are never as well-implemented or well-tested as something which is deeply within a particular set of borders.

Thomas has posted a list of features which are high-probability candidates for game-over vulnerabilities several times on HN; see second half of this comment: https://news.ycombinator.com/item?id=7936921

Research prioritization in a wide open space is research prioritization in a wide open space, and is broadly a hard problem. Broadly speaking you look for leverage: how does the set of things you are capable of causing get to a disproportionate (potential) impact? If I were hypothetically in Google Zero, I'd be looking primarily for widely deployed software which sits in poorly-understood places, operates at least part of the time on user-supplied data, and is colocated with terrifyingly sensitive systems. Bonus points if that software is boring and so hasn't had anyone take a serious look at it in a while, like e.g. png rendering libraries or request parsing libraries.