49ers Paradise > Administrative Messages

FREE READ: Server Resiliency

(1/8) > >>

Bryan_Admin:
Some of you have likely noticed an increase in the number and severity of server outages (downtime) of 49ersParadise.com.  I wanted to take a moment to clarify what is going on, what I am doing about it, and how you can help.

For quite a while now 49ersParadise.com has been running at or near the maximum RAM levels of our server.  When our RAM utilization stays this high for a prolonged period of time (like in peak periods of signing players, big rumors, games, the draft, etc) the server does one of two things: 1) It crashes or 2) The account is suspended.

When this happens I work with the hosting company to determine if we can go back online based on our load being reduced or to determine whether actions need to be taken to reduce our load.  So far, they have been very helpful in getting us back online and recommending ways to reduce our load, and prevent our server from going down.  Above all else though, they continue to suggest we add more resources to the server.

The past few outages have been caused by some abnormal activity:
1) some malware was found on the server running up RAM utilization.  This malware was removed
2 & 3) appear to be related to SPAMMERS using our server as a means to send out SPAM.  We have locked down our mail servers as best as we possibly can, and increased the SPAM protection with some additional software (which as you can imagine requires additional server resources)
3) The signing of Perish Cox, seemed to cause a flood of traffic to the site.  If Perish Cox can bring us down, imagine what the draft can do.

In the past people have suggested cutting back on what the site has to offer in order to save our resources.  Unfortunately, it is the things people use most often (the forum, the main page, the hot off the press, the extra point) that consume the most resources, and often utilize similar processes (cutting one aspect does not necessarily mean releasing the server resources.

As you know, we work on a very tight budget at 49ersParadise.com. We are not in this game to make a profit, and we are 9 subscribers away from me not having to support the site from my own pay check each month - as I have been doing for a very long time now.  My commitment has always to keep the site as open as possible, and as free from advertising as possible.  That is why I have taken the personal loss each month to keep us up and running.  Reality is, our traffic would not really justify much income in the way of banner advertising anyway.

What am I doing about it?
Rather than increase our RAM on our existing servers, which would run an extra $40 / month (yup that's it), I have chosen to add a second server at different hosting company for $10 / month.  As you can guess, that second server does not have "all the resources we need" but I am hoping it will accomplish two primary goals: 1) Provide enough relief to the main server that it does not go down as often and 2) host the entire forum on its own, thereby if one server crashes, hopefully the other does not and we should always have access to either or both of the main site and the forum.  In addition, I will be hosting "The Skinny on 49ersParadise.com" on the second server which will be a resource light version of 49ersParadise.com.  I feel it is very important to have this up and running prior to the draft.

In the coming days there will be a period of down time in the forum as I lock things down in order to make the move to the second server.  I am hoping this is just a few hours, but I really don't know.  I will do my best to communicate on facebook and twitter (and by proxy our main page) when the forum has been relocated.  At the same time as "The Skinny on 49ersParadise.com" becomes available I will make its address known to all.  My goal is to keep this site up and running as smoothly, and strongly as I possibly can for the long term.

What can you do about it?
It goes without saying that the biggest thing you can do to help this site is support it financially, and convince others to do so as well. 
Please consider:
1) Subscribing, increasing your subscription, or helping others to do so http://49ersparadise.com/subscribe.shtml
2) Donating using the PayPal button above
!! you do not need a paypal account to subscribe or donate !!
3) Purchasing NFL merchandise through 49ers Paradise (that big banner at the top of the main page of our site, or the text links at the top of the forum- anything bought after clicking those links send a small commission to our site)
4) Shop on Amazon.com through 49ersParadise.com (there's a search box for Amazon on the main page of the site at the bottom of the "49ers Shop" widget.  Purchasing anything on Amazon that way and our site earns a small commission).

Thank you for reading and your consideration of support!
Bryan

Scoots:
IIRC once the amazon.com cookie is set for 49ersparadise.com even if you go direct to amazon.com the 'dise get's a taste ... is that correct?

Are you running a greylist on the server?  I found that greylisting cut my spam server load a LOT.

I've had a few minor servers (largly ignored in closets in field offices) start struggling and in all 3 cases found bitcoin mining processes were running on the servers ... you'd think the people wanting to steam CPU clicks would write programs that would only use resources that were not being used so people wouldn't notice as quickly :)

Bryan_Admin:
Yes, that's correct regarding Amazon.... I believe the cookie is by window session up to 30 days.

I am running a grey list but it's been finicky blocking some normal activity for a few non-North America users.  Do you have a reliable recommendation?

great point on the bitcoin mining, what did you notice in your logs to pick up on that?

Scoots:
The mining was usually a perl or python thing running, the way I found it was looking at all of the running processes.  One of them started late and night and then stopped in the early morning.  :)

I'm using postgrey on my postfix mtas.  Unfortunately nothing is perfect.

Bryan_Admin:
haven't been able to spot something like that on my end to this point.  SPAM is a problem though... working to try and at least get that under wraps.

Navigation

[0] Message Index

[#] Next page