Howdy Everyone, Ben here:
Over the last few days we had one ping which we missed and another ping which we caught, but had an error sending to all members.
I wanted to take some time to go in-depth with each issue, how they happened, and how we resolved them.
Additionally, I wanted to apologize for the both of these incidences. We are working around the clock to continue to harden our system.
I appreciate y’all hanging in there with us as we work through these first month growing pains.
Natchez Missed Ping (Feb 15th):
On February 15th we missed a ping for Natchez dropping Federal 205’s.
This was very simply due to not monitoring the Federal 205 product.
This was a bummer, as we do monitor a number of Natchez SRP’s just not that specific one.
When doing inventory of what Natchez sells about a month ago, we did not see that they carried Federal 205’s.
The image below includes all of our existing monitors plus the new added federal 205 listing highlighted in green.
Incomplete Midsouth Ping (Feb 16th):
On February 16th we detected a drop of CCI 450’s from Midsouth.
Log from Scraper detecting the Ping.
After the scraper detected the product in-stock, it attempted to communicate this back to the main server responsible to sending text messages.
This communication was also successful.
Main server announcing in its logs that is sending out pings to the members.
Recapping: The scraper detected the product was in stock, and the successfully communicated that to the main server, which then decided to send out messages.
Now, where is my text?
Our next step will take us to the logs of our text message api. Where we find something interesting.
Logs from the Text Message system. Indicating that we did text members.
So, from the looks of the logs, we did text 10 members.
Unfortunately, we did not text everyone, this issue actually boils down to a very simple problem.
When our system spoke to the website, it only returned the most recent 10 members, which we did not realize as all of our tests were using less then 10 members.
This has now been fixed, and will not happen agian.
