Narratively, affinity is the level of trust and co-operation felt between the characters and somewhat defined in their relationships.
This varies between the characters, with some having a naturally higher base affinity with specific party members - i.e because of their closeness, Yuri and Karin start with a fairly high affinity toward each other - and some having a naturally lower base affinity - Lucia and the whole party, for example, for attempting to feed them to her giant pet cat, Andre.
Affinity is gained by the members in the active party successfully pulling off combos, double combos, and combo magic. The stat only ever increases; never decreasing even if a character is removed from the active party, or for story reasons.
Higher affinity between characters will cause the combo ring to vanish slower, and allowing more time to correctly input the /// button prompt, as well as increasing the overall damage dealt from combos.
A hidden benefit is that there is also a chance that if a character is in critical health, the party member with whom they share the highest affinity may block an otherwise fatal attack from an enemy - behaviour otherwise unseen, unless Gepetto is equipped with the Silver Chair.