« previous next »
Pages: [1]

Author Topic: oops, the case for "LIMIT 1"  (Read 3088 times)

micah

  • A real person, on the Internet.
  • Ass Wipe
  • Posts: 6915
  • Karma: +58/-55
  • Truth cannot contradict truth.
    • micahj.com
oops, the case for "LIMIT 1"
« on: January 02, 2008, 01:16:47 PM »
so I made a change to an application and forgot the WHERE clause in an update query....

and sort of overwrote 175,498 user records by mistake...

I don't think my boss is happy.
« Last Edit: January 02, 2008, 01:56:25 PM by micah »
Logged
"I possess a device, in my pocket, that is capable of accessing the entirety of information known to man.  I use it to look at pictures of cats and get in arguments with strangers."

charlie

  • Jackass In Charge
  • Posts: 7903
  • Karma: +84/-53
Re: oops, the case for "LIMIT 1"
« Reply #1 on: January 02, 2008, 02:03:38 PM »
Eeek...

That sucks. Can the original data be restored?
Logged

hans

  • Guitar Addict
  • Jackass In Charge
  • Posts: 3523
  • Karma: +46/-18
Re: oops, the case for "LIMIT 1"
« Reply #2 on: January 02, 2008, 02:18:13 PM »
Don't you have a test system to try things out on first?

If not, it's your boss's fault for not having the required environments available. (or his fault for letting you do it by hand) ;)

Dev - Stage - Prod (at least)


I wonder when they'll put ctrl-z into the database.
Logged
This signature intentionally left blank.

micah

  • A real person, on the Internet.
  • Ass Wipe
  • Posts: 6915
  • Karma: +58/-55
  • Truth cannot contradict truth.
    • micahj.com
Re: oops, the case for "LIMIT 1"
« Reply #3 on: January 02, 2008, 02:22:14 PM »
yeah, everything is cool now (I think).

We had to take our 30,000+ page site down for about a half hour and my boss rebuilt the missing data from yesterday's back up

Now its just a matter of responding to the zillion people who sent support questions in the last hour or so.

What happened was that each time an existing user would renew their membership (which happens on average about once every 5 minutes during the day) their e-mail address would overwrite the e-mail address of every other user in the system.

a number of users noticed that their address was wrong so they changed it back to their old address.  The compounded problem is that when you change your e-mail address it sends a verification e-mail to your old address - which in this case was the address of whoever most recently just renewed their membership.

So users were getting e-mail confirmation change notices for seemingly random people. 

Also, other users were using applications that send e-mails to their address... but since we had the wrong address, the confirmations were being sent to the wrong people.

Luckily, a mass mailing was in cue that got stopped at the last second before it sent 175 thousand copies of itself to one person.

Quote
Don't you have a test system to try things out on first?

yeah I do - and I tested it and it did what I thought it would do -- i just didn't bother to look at what ELSE it did before I copied it over to the live server.

The thing is, it was just such a small change - like those that I make all the time - that I just didn't think much of it at the time and didn't give it as much testing as needed.
Logged
"I possess a device, in my pocket, that is capable of accessing the entirety of information known to man.  I use it to look at pictures of cats and get in arguments with strangers."

Phil

  • Best friends with Mr Donkey
  • Jackass III
  • Posts: 188
  • Karma: +13/-0
Re: oops, the case for "LIMIT 1"
« Reply #4 on: January 03, 2008, 11:16:53 AM »
Quote
The thing is, it was just such a small change - like those that I make all the time - that I just didn't think much of it at the time and didn't give it as much testing as needed.

In my experience this is the probably the source of so many major problems. I work in a completely different field but this sounds so familiar. The big changes get lots of time allocated for checking and testing, re-checking and re-testing etc..etc.

It's the apparently small and innocuous 'tweaks' that end up wreaking havoc because the developer forgets about a knock-on effect on code that was written yonks ago.

 :)
Logged

hans

  • Guitar Addict
  • Jackass In Charge
  • Posts: 3523
  • Karma: +46/-18
Re: oops, the case for "LIMIT 1"
« Reply #5 on: January 03, 2008, 01:06:00 PM »
Looks like you've just discovered a new test case to add to the system.

It's all a learning process. That's why good systems have backups and logging. Makes the impact of mistakes less than they could be. Now just imagine if you didn't have a backup....

Good luck with all the confused customers.
Logged
This signature intentionally left blank.

charlie

  • Jackass In Charge
  • Posts: 7903
  • Karma: +84/-53
Re: oops, the case for "LIMIT 1"
« Reply #6 on: January 03, 2008, 01:37:18 PM »
Generally you should have a QA staff that tests changes for you. Obviously on a small team that might not be possible, but if it is then it helps (assuming competent QA) because it is their job to go through all of the possible test cases rather than the specific one you know should be tested.

Of course, if your QA staff is managed by developers who don't know good QA practices, then you can have a situation like we had on a product elsewhere in our company. The development managers created QA teams for each individual section of the main application, and those QA teams only tested the section they were assigned to. There was no QA assigned to test the product as a whole. Oops! There was actually a bug that caused data corruption in the customers' databases that got released to actual users. Yeah, that probably sucked for that product's team.
Logged
Pages: [1]
« previous next »