Short answer. The same rules that govern application of payments. Article 1289 of the Civil Code says that when a person has several debts susceptible of compensation, the rules on application of payments apply to determine the order in which compensation cancels them, rather than a separate set of rules for compensation itself.
What the law says
If a person should have against him several debts which are susceptible of compensation, the rules on the application of payments shall apply to the order of the compensation.
Civil Code, Article 1289 — Order of Compensation. Read the full provision →
Why compensation borrows rules meant for payment
Compensation is a way of extinguishing debts when two parties are creditors and debtors of each other, effectively cancelling out mutual obligations instead of requiring an actual exchange of money. Article 1289 addresses what happens when several of your debts could be extinguished this way: if a person should have against him several debts which are susceptible of compensation, the rules on the application of payments shall apply to the order of the compensation. Rather than creating a new set of rules just for this scenario, the law reuses the existing framework for application of payments.
What this means in practice
Because the rules on application of payments govern here, the same considerations that would decide which debt gets credited first when you make an actual payment without specifying which debt it covers also decide which of your debts gets cancelled first by compensation. This includes looking first at whether a designation can be made or inferred, and if not, defaulting to the debt that is most onerous to you, or dividing proportionately among debts of equal burden, following that same framework.
Why this consistency matters
Applying the same rules to both actual payment and compensation keeps the treatment of your debts consistent regardless of how they are actually extinguished. Whether a debt is wiped out because you handed over money or because it was set off against what the other party owed you, Article 1289 ensures the order in which your various debts are treated as satisfied follows the same logic, rather than compensation working by an entirely separate and potentially inconsistent set of priorities.
What this means for you with multiple debts
If you have several debts to the same person who also owes you money, and compensation would extinguish some of them, working out which specific debt is cancelled first means applying the application-of-payments framework to your situation, looking at whether any debt was designated, whether one is more onerous than the others, and whether several debts share the same nature and burden. Article 1289 is what tells you to use that existing framework, rather than treating compensation as governed by some different, unstated rule of its own.