Status: Fixed
The implementation is a mix of the the three mentioned methods (issue #247).
There is a commandline option called --open-file-policy, which can be set to
"ignore", "snapshot" or "copy".
The "ignore" setting does the same as previous version, simply exclude the file from the
backup set.
If the setting is either "snapshot" or "copy", and the file it locked, it is opened in non-excusive read mode.
Once open, a in-memory signature file is generated. This is compared to the previous signature to detect changes,
just as the normal operation. If the file has not changed, the file is skipped as it would normally be.
If the file has changed and the policy is "snapshot", the file will be processed as normal,
but during the file read, a new signature is generated. This new signature is stored
in the signature archives. When writing the new signature, it is compared to the original signature.
If the file has changed during the backup operation, a warning is written to the output log.
If the policy was set to "copy", the modified file is copied to a temporary location,
while a new signature is generated. If the copy fails due to a file change, the file
is omitted from the backup set. If the copy succeeds, the file is known to not have changed
during the first access and the copy, which gives some assurance that the file is not
actively being written, at the expense of a copy operation.
The default setting is "snapshot", because this is likely to work in most scenarios,
and potentially produces a broken file rather than guarantee a missing file (by "ignore").
Note that when reading an open file, there is always a chance that the file is being written,
so you may end up with a broken file in the backup. This is also true for the VSS and LVM snapshots,
except for VSS-aware applications.
The "copy" method is meant to minimize this risk, but there is no guarantee that the file is in an
acceptable state, just because it has not been written for any period.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@588 59da171f-624f-0410-aa54-27559c288bec
Status: Fixed
After a lot of work, I have now re-written the entire SharpRSync library.
My tests indicate that generating a diff file for an 8gb file took 10 minutes before, and now takes 8 minutes.
A delta for the same (unchanged) 8GB file took 58 minutes, and now takes 7 minutes.
The implementation seems to run faster under 64 bit, but I'm unsure if it is
related to the actual instructions issued, or the address simulation/conversion done by the OS.
I have also greatly optimized the amount of memory required by the program.
On windows, a delta for an 8GB file can be now be created with a total of 125 mb required memory.
The base app takes 65mb, so the used memory is around 60mb, and the raw diff file is approximately 44mb.
I compared it a bit to the rdiff program on linux.
For a 150mb file, rdiff takes about 1 second to generate a diff, and the same for a delta.
SharpRSync runs the same in about 7 seconds, although some of that is the mono startup and library loading.
For a 8gb file rdiff can generate a diff in about 500 seconds, where SharpRSync takes about 590 seconds.
When generating a delta, rdiff uses 530 seconds, and SharpRSync uses 770 seconds.
The last measurement is inaccurate as the memory requirement under mono is greater, and the test machine
had low memory, so there was a lot of swapping going on.
An interesting side effect is that the rdiff generated delta file is about 380kb, where the SharpRSync delta is 15 bytes.
This is because rdiff does not do the sequence matching optimization mentioned on the librsync page (last paragraph):
http://rsync.samba.org/tech_report/node4.html
Further optimization should probably focus on getting the MD4 algorithm to run faster.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@480 59da171f-624f-0410-aa54-27559c288bec
Updated some sub-projects to VS 2008, fixed .Net 3.5 dependencies to 2.0.
Added a language selection option to the settings dialog.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@222 59da171f-624f-0410-aa54-27559c288bec
Added WixProjBuilder to build the Wix projects without MSBuild support.
Simplified the build/release process.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@215 59da171f-624f-0410-aa54-27559c288bec
Made some lists in the wizard double-click enabled.
Added a warning if password settings are changed for an existing backup.
Filters are now first-touch, as that is more efficient.
Added a filter interface, because regular expressions are very inefficient if they contain more than a thousand subgroups.
Restore now works as expected.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@86 59da171f-624f-0410-aa54-27559c288bec
Updated to the newest version of LightDataModel.
Switched from duplicity to duplicati, this runs a bit faster.
Now supports authentication on network shares (windows only).
git-svn-id: https://duplicati.googlecode.com/svn/trunk@66 59da171f-624f-0410-aa54-27559c288bec
kenneth@hexad.dk