This is not the case at least on most major networks.
On UTxO (Unspent Transaction Output) based networks the transaction is signed by the sender and generates new UTxO for each recipient. The UTxO aren't signed by the recipient but they can be redeemed by the recipient using their private keys (as the UTxO is either signed by or derived from the recipient's public key/address). If you wanted to "return" the funds you'd just create a transaction containing the received UTxO as an input and deduct whatever fees that are used in the Tx from said UTxO. This "returns" the Tx without you spending any funds or having "kept" any amount of their funds.
On the Account/Balance model I believe the way it works is similar. The sender signs the Tx with their private key and the receiver signs the Tx with their public key. In this case the "return" Tx is just going to involve sending the fund amount back to the sender since there isn't any separation of Tx outputs in the account balance.
On UTxO (Unspent Transaction Output) based networks the transaction is signed by the sender and generates new UTxO for each recipient. The UTxO aren't signed by the recipient but they can be redeemed by the recipient using their private keys (as the UTxO is either signed by or derived from the recipient's public key/address). If you wanted to "return" the funds you'd just create a transaction containing the received UTxO as an input and deduct whatever fees that are used in the Tx from said UTxO. This "returns" the Tx without you spending any funds or having "kept" any amount of their funds.
On the Account/Balance model I believe the way it works is similar. The sender signs the Tx with their private key and the receiver signs the Tx with their public key. In this case the "return" Tx is just going to involve sending the fund amount back to the sender since there isn't any separation of Tx outputs in the account balance.