Webhooks fire twice, so plan for it

Last week one of our integrations ran the same job twice. A payment provider sent its "payment succeeded" notice, our server was slow to reply, the provider tried again, and two members ended up with two upgraded memberships each. Nobody lost money, but I spent an evening cleaning it up and writing apologetic messages.

The lesson is old and well known, and I still had to learn it in person: anything that arrives over the network can arrive more than once. Plan for it.

Three habits we adopted

• Store the event ID before acting. Every incoming event has an ID. We write it to a small table first. If the ID is already there, we reply "OK" and do nothing else.
• Reply fast, work later. The handler now only records the event and answers. The actual work runs from a queue a few seconds later, so the sender never times out and tries again.
• Make actions safe to repeat. "Set membership to Supporter until 1 April" is safe to run twice. "Add 30 days to membership" is not. We rewrote our handlers to set states instead of adding to them.

Test it on purpose

Most payment and form tools let you resend a test event. Send the same one two or three times in a row to a staging copy of your site and check that nothing doubles. It takes five minutes and would have saved me an evening.

If you've built integrations on UNA and have your own patterns for this, I'd love to hear them in UNA Builders, especially from anyone who has wired up more than one payment provider at a time.