I will have a few thousand customers going to be on my website purchasing all at once (about 4K in a few minutes). Last 3 times I had this we had issues with bugs or server. I believe we've handled the server issues, we use Rackspace and scaled up more servers last 2 times & didn't see any issues with servers. But we had a few bugs each time that caused major issues. I've done testing myself and had 3 others review the site and I've corrected those issues. But really the problem is that we don't discover these bugs until we are hit with this much traffic all at once. I've done load testing using a third party site but that doesn't really show us bugs as it's not real users utilizing the site & preforming functions they would normally do. We also don't have much of any money to throw at this. FYI, with our web app/site our customers send their customers to purchase goods from our site, which charges them a fee based off the number in line they are to come to the site. This "number in line" model is what causes most of the complexity and problem. But we can't change that as it's what our customer's want.
I have been in this situation as well as almost every other web application developer and I learned my lesson along the way.
You said you and 3 others tested, but I feel that 4 people may not be enough. I think at least 50 to 75 people is a much more sufficient number and here is how I do it with my own web apps:
I think that is what alpha and beta versions are for. If an app is just released without it being in beta version, there will be problems, but putting something in beta tells the public that there will be bugs and that those bugs can be tested, so releasing a Beta version prior to your actual release date is one way to go.
Also, when testing for bugs before releasing to massive traffic, all scenarios need to be tested and it is possible that 4 people will not test all of those scenarios or even contemplate them.
I suggest to companies who release software and websites to test in all platforms including, but not limited to:
1. different web browsers including IE, Firefox, chrome, safari
2. different mobile devices including smart phones and tablets
3. operating systems
4. various times of the day where internet traffic may vary and even different days of the week
5. different screen sizes and resolutions
Since every has a different version of all of these factors, testing is needed for more than 4 people.
I said that 50 to 75 people is efficient testing and you may think that is overkill, but it really shows you bugs that you never would have thought of. For that many people, give away something for free as an incentive like a free product or ebook. You will get many people willing to help out for just the incentive.Works for me.