Narrative
IncyWincyWebApp stores some of its persistent data on MongoDB. On a not so
great day, Oliver, the Ops guy, issues an accidental query that updated the
profile collection in users database making status of private profiles
public and also changing the related visible database in the process.
In his attempt to restore those documents to original state, he contacts
Adam, the Admin and describes what happened.
Adam: "Oliver, Do you know the time when this query was issued?
Oliver: "Around 1 o'clock today afternoon"
Adam: "Would it help if I did not restore the portion of log pertaining to that accidental update?"
Oliver: "Yes but I also have important documents belonging to transaction database in that time duration"
Adam: "Ok, I will exclude only the documents belonging to profile collection and visible database within that time window and retain transaction database"
Oliver: "Yes, that would work!"
Given source replica set and target node are running |
with configuration |
source {unsecureSrcNode} on port {unsecureSrcPort} , target {unsecureTgtNode} on port {unsecureTgtPort} |
1. Some CUD Operations were done and a Wrong update was made
run mongo command on |
source |
Comments |
use database |
users |
|
run |
db.profile.insert({name:"One",type:"private"}) |
|
run |
db.profile.save({name:"Two",type:"private"}) |
|
run |
db.profile.insert({name:"Three",type:"public"}) |
|
run |
db.profile.save({name:"Four",type:"public"}) |
|
use database |
transactions |
|
run |
db.orders.insert({orderNo:456}) |
|
run |
db.orders.save({orderNo:457}) |
|
use database |
users |
|
run |
db.profile.update({name:"One"},{$set:{type:"public"}}) |
«-- Wrong update |
use database |
visible |
|
run |
db.profile.list.insert({userName:"One"}) |
«-- Cascaded wrong change |
use database |
transactions |
|
run |
db.orders.insert({orderNo:458}) |
|
use database |
visible |
|
run |
db.details.save({viewedLinks:3}) |
«-- Cascaded wrong change |
use database |
users |
|
run |
db.profile.update({name:"Two"},{$set:{type:"public"}}) |
«-- Wrong update |
use database |
visible |
|
run |
db.profile.list.insert({userName:"Two"}) |
«-- Cascaded wrong change |
use database |
transactions |
|
run |
db.orders.insert({orderNo:459}) |
|
run |
db.orders.save({orderNo:460}) |
|
use database |
users |
|
run |
db.profile.insert({name:"Five",type:"private"}) |
|
2. Run Backup Utility
Open terminal |
and run |
backup -s {unsecureSrcNode} --port={unsecureSrcPort} -f {file} |
and show |
stdout |
and ensure |
stdout |
contains |
Process started |
3. Run Restore Utility With Dry run to analyze the corrupt data
Open terminal |
and run |
restore -d {unsecureTgtNode} --port={unsecureTgtPort} -f {file} --dry-run |
and show |
stdout |
4. Run Restore Utility to exclude corrupt data after analysis is done
- Dry-run indicates that wrong update query was fired at 6th document from last.
So we adopt a strategy to first restore all documents before it.
-
Further ignore the two documents containing wrong update, and then restore
excluding them, i.e. restore last four documents.
4a. 1st Restore all correct documents until error.
Open oplog for node |
source |
and travel |
9 |
documents back in time and save timestamp in |
{sSince} |
Open oplog for node |
source |
and travel |
4 |
documents back in time and save timestamp in |
{sUntil} |
Open terminal |
and run |
restore -d {unsecureTgtNode} --port={unsecureTgtPort} -f {file} --sExclude --sUntil={sUntil} --sSince={sSince} --sNs=users.profile,visible |
and show |
stdout |
and show |
stderr |
and ensure |
stdout |
contains |
Process started |
5. Ensure Target does not contains corrupt data
Ensure source and target has
|
find documents for collection |
profile |
in database |
users |
where |
null |
and cleanup databases |
true |
source value | destination value |
{ "name" : "One" , "type" : "public"} | { "name" : "One" , "type" : "private"} |
{ "name" : "Two" , "type" : "public"} | { "name" : "Two" , "type" : "private"} |
{ "name" : "Three" , "type" : "public"} | { "name" : "Three" , "type" : "public"} |
{ "name" : "Four" , "type" : "public"} | { "name" : "Four" , "type" : "public"} |
{ "name" : "Five" , "type" : "private"} | { "name" : "Five" , "type" : "private"} |
Checking transactions database is intact
Ensuring target is consistent with source
|
find documents for collection |
orders |
in database |
transactions |
where |
null |
and cleanup databases |
true |
source value | destination value |
{ "orderNo" : 456.0} | { "orderNo" : 456.0} |
{ "orderNo" : 457.0} | { "orderNo" : 457.0} |
{ "orderNo" : 458.0} | { "orderNo" : 458.0} |
{ "orderNo" : 459.0} | { "orderNo" : 459.0} |
{ "orderNo" : 460.0} | { "orderNo" : 460.0} |
Checking documents of database 'visible' in source is not restored to target
Ensure source and target has
|
run in database |
visible |
query |
db.profile.list.count() |
and cleanup databases |
false |
source value | destination value |
2 | 0 |
Ensure source and target has
|
run in database |
visible |
query |
db.details.count() |
and cleanup databases |
true |
source value | destination value |
1 | 0 |