[isotypic]

You might like the book "A History of Abstract Algebra" by Israel Kleiner - it goes over specifically the developments leading to the invention of the abstract group. The answer to your questions is that nobody really sat down and invented the group from the ether - its more accurate to say someone sat down and said "Hey, all these things we've been studying for the past 50 years are all the same thing if we think of it this way", and then the mathematical community eventually gets around to realizing its a useful abstraction (if it is one) as people build on it, or work more without it and eventually realize the abstraction would be helpful. For groups, this played out in how Cayley defined the abstract group in the 1850s, but it only started to gain more widespread usage in the 1870s. As for what they were doing, the main areas ways appeared around this time were through permutation groups (roots of polynomials), abelian groups (various number theoretic constructions/statements), and geometry (study geometry by studying groups of transformations, like isometries for Euclidean geometry).