Author Topic: FREE READ: Server Resiliency  (Read 58209 times)

0 Members and 1 Guest are viewing this topic.

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
FREE READ: Server Resiliency
« on: March 26, 2014, 08:52:11 AM »
Some of you have likely noticed an increase in the number and severity of server outages (downtime) of 49ersParadise.com.  I wanted to take a moment to clarify what is going on, what I am doing about it, and how you can help.

For quite a while now 49ersParadise.com has been running at or near the maximum RAM levels of our server.  When our RAM utilization stays this high for a prolonged period of time (like in peak periods of signing players, big rumors, games, the draft, etc) the server does one of two things: 1) It crashes or 2) The account is suspended.

When this happens I work with the hosting company to determine if we can go back online based on our load being reduced or to determine whether actions need to be taken to reduce our load.  So far, they have been very helpful in getting us back online and recommending ways to reduce our load, and prevent our server from going down.  Above all else though, they continue to suggest we add more resources to the server.

The past few outages have been caused by some abnormal activity:
1) some malware was found on the server running up RAM utilization.  This malware was removed
2 & 3) appear to be related to SPAMMERS using our server as a means to send out SPAM.  We have locked down our mail servers as best as we possibly can, and increased the SPAM protection with some additional software (which as you can imagine requires additional server resources)
3) The signing of Perish Cox, seemed to cause a flood of traffic to the site.  If Perish Cox can bring us down, imagine what the draft can do.

In the past people have suggested cutting back on what the site has to offer in order to save our resources.  Unfortunately, it is the things people use most often (the forum, the main page, the hot off the press, the extra point) that consume the most resources, and often utilize similar processes (cutting one aspect does not necessarily mean releasing the server resources.

As you know, we work on a very tight budget at 49ersParadise.com. We are not in this game to make a profit, and we are 9 subscribers away from me not having to support the site from my own pay check each month - as I have been doing for a very long time now.  My commitment has always to keep the site as open as possible, and as free from advertising as possible.  That is why I have taken the personal loss each month to keep us up and running.  Reality is, our traffic would not really justify much income in the way of banner advertising anyway.

What am I doing about it?
Rather than increase our RAM on our existing servers, which would run an extra $40 / month (yup that's it), I have chosen to add a second server at different hosting company for $10 / month.  As you can guess, that second server does not have "all the resources we need" but I am hoping it will accomplish two primary goals: 1) Provide enough relief to the main server that it does not go down as often and 2) host the entire forum on its own, thereby if one server crashes, hopefully the other does not and we should always have access to either or both of the main site and the forum.  In addition, I will be hosting "The Skinny on 49ersParadise.com" on the second server which will be a resource light version of 49ersParadise.com.  I feel it is very important to have this up and running prior to the draft.

In the coming days there will be a period of down time in the forum as I lock things down in order to make the move to the second server.  I am hoping this is just a few hours, but I really don't know.  I will do my best to communicate on facebook and twitter (and by proxy our main page) when the forum has been relocated.  At the same time as "The Skinny on 49ersParadise.com" becomes available I will make its address known to all.  My goal is to keep this site up and running as smoothly, and strongly as I possibly can for the long term.

What can you do about it?
It goes without saying that the biggest thing you can do to help this site is support it financially, and convince others to do so as well. 
Please consider:
1) Subscribing, increasing your subscription, or helping others to do so http://49ersparadise.com/subscribe.shtml
2) Donating using the PayPal button above
!! you do not need a paypal account to subscribe or donate !!
3) Purchasing NFL merchandise through 49ers Paradise (that big banner at the top of the main page of our site, or the text links at the top of the forum- anything bought after clicking those links send a small commission to our site)
4) Shop on Amazon.com through 49ersParadise.com (there's a search box for Amazon on the main page of the site at the bottom of the "49ers Shop" widget.  Purchasing anything on Amazon that way and our site earns a small commission).

Thank you for reading and your consideration of support!
Bryan

"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline Scoots

  • Super Bowl MVP
  • *****
  • Posts: 10766
Re: FREE READ: Server Resiliency
« Reply #1 on: March 26, 2014, 12:20:37 PM »
IIRC once the amazon.com cookie is set for 49ersparadise.com even if you go direct to amazon.com the 'dise get's a taste ... is that correct?

Are you running a greylist on the server?  I found that greylisting cut my spam server load a LOT.

I've had a few minor servers (largly ignored in closets in field offices) start struggling and in all 3 cases found bitcoin mining processes were running on the servers ... you'd think the people wanting to steam CPU clicks would write programs that would only use resources that were not being used so people wouldn't notice as quickly :)

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #2 on: March 26, 2014, 01:34:49 PM »
Yes, that's correct regarding Amazon.... I believe the cookie is by window session up to 30 days.

I am running a grey list but it's been finicky blocking some normal activity for a few non-North America users.  Do you have a reliable recommendation?

great point on the bitcoin mining, what did you notice in your logs to pick up on that?
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline Scoots

  • Super Bowl MVP
  • *****
  • Posts: 10766
Re: FREE READ: Server Resiliency
« Reply #3 on: March 26, 2014, 01:50:42 PM »
The mining was usually a perl or python thing running, the way I found it was looking at all of the running processes.  One of them started late and night and then stopped in the early morning.  :)

I'm using postgrey on my postfix mtas.  Unfortunately nothing is perfect.

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #4 on: March 31, 2014, 02:53:55 PM »
haven't been able to spot something like that on my end to this point.  SPAM is a problem though... working to try and at least get that under wraps.
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #5 on: April 22, 2014, 01:19:03 PM »
UPDATE:
1. Lots of work on the server has brought about way more stability as most of you have probably noticed

2. As planned I've moved the forum to a second server.  The hope is that it would be unlikely for both servers on different hosts to be down at the same time.  In doing this, I have also created a "Skinny" main page for the site too.  Depending on each server's status this page will offer a Skinny selection of most of 'Dise related content.

Please bookmark www.49Paradise.com

3. All links from the main site to the forum should be fully operational and automatically redirect back and forth.  If anything is not working for you, please let me know.

4. I am working on resolving the forum feed issue to the main page of the site. 
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #6 on: April 22, 2014, 01:49:43 PM »
#4 is solved.
Working on restoring username functionality to twitter posts
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline TIM

  • Super Bowl MVP
  • *****
  • Posts: 21404
  • I am ALWAYS right,(except when I'm wrong).
Re: FREE READ: Server Resiliency
« Reply #7 on: April 22, 2014, 02:07:06 PM »
The Forum is very slow for me this Morning Bry .
( JMHO )

JOE MONTANA IS
lord of The Rings.


GET#SIX...and seven and eight and nine and ten... !!!

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #8 on: April 22, 2014, 02:28:35 PM »
Really? I find with the exception of the shoutbox... it's flying... others?
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline TIM

  • Super Bowl MVP
  • *****
  • Posts: 21404
  • I am ALWAYS right,(except when I'm wrong).
Re: FREE READ: Server Resiliency
« Reply #9 on: April 22, 2014, 02:32:25 PM »
It's all slow as molasses for me Bry. I am waiting forever for any page I click on to load. First thing I did was check to see if it was a problem from my side but the Dise is the only site I am having this problem with .
( JMHO )

JOE MONTANA IS
lord of The Rings.


GET#SIX...and seven and eight and nine and ten... !!!

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #10 on: April 22, 2014, 02:38:05 PM »
ok... will keep an eye on it.  the new server may need load balancing.  I submitted a support request to the host to see if they can double check this
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline TIM

  • Super Bowl MVP
  • *****
  • Posts: 21404
  • I am ALWAYS right,(except when I'm wrong).
Re: FREE READ: Server Resiliency
« Reply #11 on: April 22, 2014, 02:39:34 PM »
I am using IE but I tried Google Chrome and I could not even get logged in to the site with Chrome (also had trouble logging in with IE but finally got in).
( JMHO )

JOE MONTANA IS
lord of The Rings.


GET#SIX...and seven and eight and nine and ten... !!!

Offline Scoots

  • Super Bowl MVP
  • *****
  • Posts: 10766
Re: FREE READ: Server Resiliency
« Reply #12 on: April 22, 2014, 05:41:23 PM »
Seems considerably slower than normal to me as well.

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #13 on: April 22, 2014, 07:43:50 PM »
I'm still in discussions with them.  CPU and RAM on this server are about 1.5 and 2x what the rest of the site was running on... doesn't make sense to me... geography is about the same too...
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline cervant

  • Super Bowl MVP
  • *****
  • Posts: 8019
Re: FREE READ: Server Resiliency
« Reply #14 on: April 22, 2014, 08:58:34 PM »
Maybe you've lost your database indexes? (Or is that "indices"?)
"Show me a man with a sense of pride, and I'll show you a guy with limited options."

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #15 on: April 22, 2014, 10:02:32 PM »
... how would one check that?
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline cervant

  • Super Bowl MVP
  • *****
  • Posts: 8019
Re: FREE READ: Server Resiliency
« Reply #16 on: April 22, 2014, 10:17:06 PM »
Your mileage may vary, but kinda like this:

1) Open a shell window
2) Connect to the database
3) I don't recall the exact syntax, but something on the order of SHOW INDEXES ON <table_name>

You might also get it from

DESCRIBE <table_name> and see if columns are marked as keys.

Syntax will vary between oracle, mysql & (hairball) Microsoft databases, whichever your server has.
"Show me a man with a sense of pride, and I'll show you a guy with limited options."

Offline Scoots

  • Super Bowl MVP
  • *****
  • Posts: 10766
Re: FREE READ: Server Resiliency
« Reply #17 on: April 23, 2014, 01:36:39 AM »
running faster for me now

Offline Scoots

  • Super Bowl MVP
  • *****
  • Posts: 10766
Re: FREE READ: Server Resiliency
« Reply #18 on: April 23, 2014, 01:38:59 AM »
But still slow ... not as fast as it was on old server.

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #19 on: April 23, 2014, 07:12:20 AM »
Flying for me right now...?
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline Scoots

  • Super Bowl MVP
  • *****
  • Posts: 10766
Re: FREE READ: Server Resiliency
« Reply #20 on: April 23, 2014, 11:29:17 AM »
Front page of the forum was quick, bringing up this thread took 27 seconds.

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #21 on: April 23, 2014, 11:48:32 AM »
Working with the hosting company to change servers and see if that helps.. it will require some down time again...
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #22 on: April 23, 2014, 05:21:03 PM »
Flying now? Server upgrade..
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline Scoots

  • Super Bowl MVP
  • *****
  • Posts: 10766
Re: FREE READ: Server Resiliency
« Reply #23 on: April 23, 2014, 06:02:44 PM »
Seems as fast or faster than the old server on a good day!

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #24 on: April 23, 2014, 06:09:49 PM »
Good news... Expensive.. But good news....  Others?
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline hojer49

  • Super Bowl MVP
  • *****
  • Posts: 6743
Re: FREE READ: Server Resiliency
« Reply #25 on: April 23, 2014, 06:35:53 PM »
Using my Samsung tablet at home, the Forum seems MUCH faster than normal. So well done B, from this side of the Atlantic :)


Success is temporary, loyalty is forever.

Offline cervant

  • Super Bowl MVP
  • *****
  • Posts: 8019
Re: FREE READ: Server Resiliency
« Reply #26 on: April 23, 2014, 07:43:15 PM »
Seems fast here. Much of the delays were in loading Huddle, but that seems fine now.
"Show me a man with a sense of pride, and I'll show you a guy with limited options."

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #27 on: April 23, 2014, 09:35:16 PM »
Good news
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline TIM

  • Super Bowl MVP
  • *****
  • Posts: 21404
  • I am ALWAYS right,(except when I'm wrong).
Re: FREE READ: Server Resiliency
« Reply #28 on: April 23, 2014, 09:56:17 PM »
No more problems Bry ! The site is loading fast and the shoutbox and all the threads are also loading very quickly !
Thank You !
( JMHO )

JOE MONTANA IS
lord of The Rings.


GET#SIX...and seven and eight and nine and ten... !!!

Offline TIM

  • Super Bowl MVP
  • *****
  • Posts: 21404
  • I am ALWAYS right,(except when I'm wrong).
Re: FREE READ: Server Resiliency
« Reply #29 on: April 23, 2014, 10:06:20 PM »
...Except I don't see any spell check .(didn't use it much anyway, as anyone who reads my posts can attest).
( JMHO )

JOE MONTANA IS
lord of The Rings.


GET#SIX...and seven and eight and nine and ten... !!!

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #30 on: April 23, 2014, 10:54:13 PM »
No idea what happened to spell check. Nothing to do with server to server. My best guess is that it was retired by the forum software I use in one of the ongoing updates... Most web browsers do it automatically now...
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #31 on: May 01, 2014, 08:29:22 PM »
Spoke to soon... the world went boom... some domains are crossing path with 49ersParadise.com we are working on a resolution ASAP.  Some turbulence is expected for the next 24-48 hours
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #32 on: May 12, 2014, 09:32:36 AM »
I'm rather relieved at how the servers held up over the weekend.  This was the cleanest experience I have had on draft day in eons
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi

Offline Scoots

  • Super Bowl MVP
  • *****
  • Posts: 10766
Re: FREE READ: Server Resiliency
« Reply #33 on: May 12, 2014, 11:02:15 AM »
Congrats.  I had one connect failure and a couple slow loads, but nothing that I could be sure was the dise at all.  Good job you did there!

Offline Bryan_Admin

  • Administrator
  • Super Bowl MVP
  • *****
  • Posts: I am a geek!!
    • 49ers Paradise
Re: FREE READ: Server Resiliency
« Reply #34 on: May 12, 2014, 05:45:55 PM »
Thanks Scoots!  It was refreshing... I can't imagine having had to deal with a server outage during that crazy trade sequence.
"Winning is not a sometime thing; it’s an all time thing. You don’t win once in a while, you don’t do things right once in a while, you do them right all the time. Winning is habit. Unfortunately, so is losing" - Lombardi