PG-HQ Forums
Site Outage Details - Printable Version

+- PG-HQ Forums (https://www.pg-hq.com/comms)
+-- Forum: Panzer Grenadier (https://www.pg-hq.com/comms/forumdisplay.php?fid=3)
+--- Forum: PG-HQ Q&A (https://www.pg-hq.com/comms/forumdisplay.php?fid=8)
+--- Thread: Site Outage Details (/showthread.php?tid=2310)

Pages: 1 2

Site Outage Details - Shad - 07-13-2020

Site Outage Details

I apologize for the downtime. Here's the short version:
  • The outage was due to an automated tool used by our hosting provider
  • No data was lost or damaged
  • We're back
Longer version for the curious:
  • Microsoft's search engine Bing decided late Friday / early Saturday would be a good time to try and process the entire contents of PG-HQ's Library for its search archives
  • The sudden high volume of traffic from "BingBot" drove the server's CPU load beyond allowed limits
  • At about 0500 CT Saturday, an automated bit of software at our ISP shutdown the site and sent me a notification basically telling me that I had to figure out what was causing the load, fix it, tell them I fixed it, and pass their review of my changes before they'd re-enable the site
  • I was on a hiking trip and unable to get in front of a computer until Saturday evening
  • I quickly narrowed it down to BingBot and adjusted our code to block that particular webcrawler, then informed the ISP's tech center
  • Due I assume to it being the weekend, it took them ~16 hours to review and approve my changes and unlock the site
Conclusion
I again apologize for the outage. This is the only significant outage we've experienced in over 10 years of existence. Even so, the admin actions taken by our ISP are unacceptable. PG-HQ was shutdown without forewarning of any kind and then tech support was not ready to instantly reactivate the site. Had this been a business customer, there would have been a serious impact to revenue.

As such, starting this week I'm beginning the migration to an upgraded, more professional hosting provider. This will happen over the next few weeks and apart from some momentary downtime when we finally redirect pg-hq.com to the new server, you shouldn't notice anything different.

Thank you for your patience and understanding!

RE: Site Outage Details - Schoenwulf - 07-13-2020

Thanks! Good to see the site back up and appreciate all the work that you put in to make this such a great site.

RE: Site Outage Details - DAn_Huffman - 07-13-2020

Drew;

     Not your fault. You need some outdoor time, esp. in IL.

How did you narrow it down to the Bing bot? 

Curious form an IT web host POV.

Thanks

RE: Site Outage Details - Kithikor - 07-13-2020

Thank you for letting us know.
Gotta go hate on Microsoft some more now  Big Grin

RE: Site Outage Details - wargamer1939 - 07-13-2020

Thank you for aall you do Im new here and do the 5 a month but kudos to such a wonderful resource.

RE: Site Outage Details - Shad - 07-13-2020

(07-13-2020, 07:55 AM)DAn_Huffman Wrote: How did you narrow it down to the Bing bot? 

Curious form an IT web host POV.

Thanks

Library.php tripped the CPU limit, so I grabbed the raw server access logs for July and reviewed the 12 hours before we were nailed. BingBot declares itself via UserAgent when it makes requests and 95%+ of the requests to Library.php during the period in question were BingBot crawling from page to page and initiating scores of DB connections.

From there a simple update to Robots.txt took care of it. Googling around, I'm not the first person to be psuedo-DDoSd by BingBot...

RE: Site Outage Details - cjsiam - 07-13-2020

(07-13-2020, 08:24 AM)Shad Wrote:
(07-13-2020, 07:55 AM)DAn_Huffman Wrote: How did you narrow it down to the Bing bot? 

Curious form an IT web host POV.

Thanks

Library.php tripped the CPU limit, so I grabbed the raw server access logs for July and reviewed the 12 hours before we were nailed. BingBot declares itself via UserAgent when it makes requests and 95%+ of the requests to Library.php during the period in question were BingBot crawling from page to page and initiating scores of DB connections.

From there a simple update to Robots.txt took care of it. Googling around, I'm not the first person to be psuedo-DDoSd by BingBot...

Another fine feature brought to you by the world's new super-villian, Bill Gates...

RE: Site Outage Details - triangular_cube - 03-05-2021

Piggybacking on the old outage thread to talk about the new one. 

I tried to post a reply to the news post on the homepage but am still getting the HTTP 500 error (how the page was displaying on chrome during the outage) when doing so. Not sure if that functionality needs an update as well. Maybe its just a local cache issue for all I know. 

Everything else is working fine as far as I can tell, but I haven't tried to log any plays since it came online again.

Anyways, just wanted to thank you for all your hard work in getting us back up and running, but the site wont let me Smile

RE: Site Outage Details - JayTownsend - 03-05-2021

Thank you very much Andrew. The system still won't let me post an AAR however?

RE: Site Outage Details - cjsiam - 03-05-2021

Bless you for the time you spend keeping this going for us.

Do let us know if we can do anything to help....I'll go try and donate Smile
cjSmile