Technically I'm taking a large liberty saying you're "activating layers", all the layers are affecting the output and you don't pick and choose them
But you can imagine the model like a plinko board: just because the ball passes every peg, doesn't mean every peg changed it's trajectory.
When you fine tune a model, you're trying to change how the pegs are arranged so the ball falls through the board differently.
When you prompt tune you're changing how the ball will fall too. You don't get to change the board, but you can change where the ball starts or have the ball go through the board several more times than normal before the user sees it, etc.
You can't see the ball falling (which layers are doing what), only where it falls, but when you spend long enough building on these models, you do get an intuition for which prompts have an outsized effect on where the ball will land.
But you can imagine the model like a plinko board: just because the ball passes every peg, doesn't mean every peg changed it's trajectory.
When you fine tune a model, you're trying to change how the pegs are arranged so the ball falls through the board differently.
When you prompt tune you're changing how the ball will fall too. You don't get to change the board, but you can change where the ball starts or have the ball go through the board several more times than normal before the user sees it, etc.
You can't see the ball falling (which layers are doing what), only where it falls, but when you spend long enough building on these models, you do get an intuition for which prompts have an outsized effect on where the ball will land.