We're currently investigating a system-wide issue where events (listed in the Activity Stream of your Dashboard and individual subscriptions) and the associated Webhooks were not generated for some renewals.
The subscription renewals themselves occurred normally, and the subscriptions will hold the correct state now. Only the events and webhooks are affected.
We are not sure when we'll be able to replay these events, so in the meantime please use the API to fetch the current state of subscriptions if you rely on this information for your application.
You will now find a list of affected subscriptions when you log in. If you have any affected subscriptions, you will have a banner at the top of your screen that says "Click here to view a list of the effected subscriptions". Clicking the provided link will take you to your listing.
Q. What do you mean by "missed events"?
A. We keep a running list of your site's activity that is separate from the subscription records themselves. The subscriptions were not affected - they continued to renew, capture payment, update balances, and next billing dates. But, the activity records that record what happened and when were not generated in all cases. Since webhooks are based on this activity list, they were not sent when the event was not recorded.
Q. What types of events were missed?
A. The following types of events may have been affected
- Renewal Success/Failure
- Payment Success/Failure (only payments that were initiated as a part of a renewal or retry that was also missed)
- Subscription State Change (only those that resulted from a renwal or retry that was also missed)
We found the reason for this and events are no longer going missing. The dates of the outage have been changed to 12/5 to 12/10. We are working to construct a listing of the affected subscriptions so you can inspect them. These will be made available in the interface when you log in. We will likely try to reconstruct the events but it is a daunting task. We would like to hear from you about whether the ability to replay the associated webhooks is useful or not.
Q. Why did this happen?
A. We added additional capacity for processing subscription renewals, and some of the extra "workers" were run in an interactive manner. When we ran the workers like this, the code that controls our observers was not loaded. The observers are responsible for generating the webhooks and events.
Q. Will this happen again?
A. Now that we know why this happened, it shouldn't happen again. The observer code that has been handling the generation of these events has been in place and working without problem for over a year.
Q. What about subscription processing capacity?
A. We just launched a new backend queue and our capacity for renewing subscriptions has increased substantially. We should not be getting behind on subscription renewals again.