Tayra can help with analysis of documents for restore

Narrative

IncyWincyWebApp stores some of its persistent data on MongoDB. On a not so great day, Oliver, the Ops guy, issues an accidental query that updated the profile collection in users database making status of private profiles public and also changing the related visible database in the process. In his attempt to restore those documents to original state, he contacts Adam, the Admin and describes what happened.

Adam: "Oliver, Do you know the time when this query was issued?

Oliver: "Around 1 o'clock today afternoon"

Adam: "Would it help if I did not restore the portion of log pertaining to that accidental update?"

Oliver: "Yes but I also have important documents belonging to transaction database in that time duration"

Adam: "Ok, I will exclude only the documents belonging to profile collection and visible database within that time window and retain transaction database"

Oliver: "Yes, that would work!"


Given source replica set and target node are running
with configuration source {unsecureSrcNode} on port {unsecureSrcPort} , target {unsecureTgtNode} on port {unsecureTgtPort}

1. Some CUD Operations were done and a Wrong update was made

run mongo command on
source
Comments
use database users
run
db.profile.insert({name:"One",type:"private"})
run
db.profile.save({name:"Two",type:"private"})
run
db.profile.insert({name:"Three",type:"public"})
run
db.profile.save({name:"Four",type:"public"})
use database transactions
run
db.orders.insert({orderNo:456})
run
db.orders.save({orderNo:457})
use database users
run
db.profile.update({name:"One"},{$set:{type:"public"}})
«-- Wrong update
use database visible
run
db.profile.list.insert({userName:"One"})
«-- Cascaded wrong change
use database transactions
run
db.orders.insert({orderNo:458})
use database visible
run
db.details.save({viewedLinks:3})
«-- Cascaded wrong change
use database users
run
db.profile.update({name:"Two"},{$set:{type:"public"}})
«-- Wrong update
use database visible
run
db.profile.list.insert({userName:"Two"})
«-- Cascaded wrong change
use database transactions
run
db.orders.insert({orderNo:459})
run
db.orders.save({orderNo:460})
use database users
run
db.profile.insert({name:"Five",type:"private"})

2. Run Backup Utility

Open terminal
and run
backup -s {unsecureSrcNode} --port={unsecureSrcPort} -f {file}
and show
stdout
and ensure
stdout
contains Process started

3. Run Restore Utility With Dry run to analyze the corrupt data

Open terminal
and run
restore -d {unsecureTgtNode} --port={unsecureTgtPort} -f {file} --dry-run
and show
stdout


4. Run Restore Utility to exclude corrupt data after analysis is done

  • Dry-run indicates that wrong update query was fired at 6th document from last. So we adopt a strategy to first restore all documents before it.
  • Further ignore the two documents containing wrong update, and then restore excluding them, i.e. restore last four documents.


4a. 1st Restore all correct documents until error.

Open oplog for node
source
and travel 9 documents back in time and save timestamp in
{sSince}

Open oplog for node
source
and travel 4 documents back in time and save timestamp in
{sUntil}

Open terminal
and run
restore -d {unsecureTgtNode} --port={unsecureTgtPort} -f {file} --sExclude --sUntil={sUntil} --sSince={sSince} --sNs=users.profile,visible
and show
stdout
and show
stderr
and ensure
stdout
contains Process started

5. Ensure Target does not contains corrupt data

Ensure source and target has
find documents for collection
profile
in database
users
where
null
and cleanup databases true
source valuedestination value
{ "name" : "One" , "type" : "public"} 
{ "name" : "One" , "type" : "private"}
{ "name" : "Two" , "type" : "public"}
{ "name" : "Two" , "type" : "private"}
{ "name" : "Three" , "type" : "public"}
{ "name" : "Three" , "type" : "public"}
{ "name" : "Four" , "type" : "public"}
{ "name" : "Four" , "type" : "public"}
{ "name" : "Five" , "type" : "private"} 
{ "name" : "Five" , "type" : "private"}

Checking transactions database is intact
Ensuring target is consistent with source
find documents for collection
orders
in database
transactions
where
null
and cleanup databases true
source valuedestination value
{ "orderNo" : 456.0} 
{ "orderNo" : 456.0}
{ "orderNo" : 457.0} 
{ "orderNo" : 457.0}
{ "orderNo" : 458.0} 
{ "orderNo" : 458.0}
{ "orderNo" : 459.0} 
{ "orderNo" : 459.0}
{ "orderNo" : 460.0} 
{ "orderNo" : 460.0}

Checking documents of database 'visible' in source is not restored to target
Ensure source and target has
run in database
visible
query
db.profile.list.count()
and cleanup databases false
source valuedestination value
20

Ensure source and target has
run in database
visible
query
db.details.count()
and cleanup databases true
source valuedestination value
10