Status: Fixed
Updated default buffer size to 64kB for copying, gives drastically improved speed for FTP, and possibly all backends.
git-svn-id: https://duplicati.googlecode.com/svn/branches/1.3.1@1140 59da171f-624f-0410-aa54-27559c288bec
Status: Fixed
The implementation is a mix of the the three mentioned methods (issue #247).
There is a commandline option called --open-file-policy, which can be set to
"ignore", "snapshot" or "copy".
The "ignore" setting does the same as previous version, simply exclude the file from the
backup set.
If the setting is either "snapshot" or "copy", and the file it locked, it is opened in non-excusive read mode.
Once open, a in-memory signature file is generated. This is compared to the previous signature to detect changes,
just as the normal operation. If the file has not changed, the file is skipped as it would normally be.
If the file has changed and the policy is "snapshot", the file will be processed as normal,
but during the file read, a new signature is generated. This new signature is stored
in the signature archives. When writing the new signature, it is compared to the original signature.
If the file has changed during the backup operation, a warning is written to the output log.
If the policy was set to "copy", the modified file is copied to a temporary location,
while a new signature is generated. If the copy fails due to a file change, the file
is omitted from the backup set. If the copy succeeds, the file is known to not have changed
during the first access and the copy, which gives some assurance that the file is not
actively being written, at the expense of a copy operation.
The default setting is "snapshot", because this is likely to work in most scenarios,
and potentially produces a broken file rather than guarantee a missing file (by "ignore").
Note that when reading an open file, there is always a chance that the file is being written,
so you may end up with a broken file in the backup. This is also true for the VSS and LVM snapshots,
except for VSS-aware applications.
The "copy" method is meant to minimize this risk, but there is no guarantee that the file is in an
acceptable state, just because it has not been written for any period.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@588 59da171f-624f-0410-aa54-27559c288bec
I have now fixed the issue, and tested it on the file provided.
It seems to compress better now, although the file contains sporadic changes,
even if only a single line was added. The restore also works as expected,
and the update works on my unittest.
Could you please try the updated version and see if it works as expected for you as well?
With regards to the "no else statement" (comment 7), I think that you discovered another
bug, which I have also fixed. The else statement is there to ensure that the buffer is completely filled,
so long chains of unmatched data gets written with as few commands as possible. So, on the first entry,
the copy command is flushed and the force_buffer_refill is set, causing the buffer to refill.
The bug was that IF the buffer was already full, the stream would read 0 bytes, and return a count of 0.
This was then interpreted as if the stream was empty (it returns zero when empty). I have now fixed that
with a check for this special case.
I have also included an update to the ChecksumFileReader that splits the previous m_weakTable into two
separate arrays, as that gives a faster lookup, than using a custom comparer. I also added support for
very large files (files larger than blocksize * int.MaxValue, appx. 4TB).
git-svn-id: https://duplicati.googlecode.com/svn/trunk@548 59da171f-624f-0410-aa54-27559c288bec
The purpose of this change is to prevent loading a defunct MD4 implementation,
which exists in the Mono library in all versions prior to this path (july 2010):
http://github.com/mono/mono/commit/6be3c20a61d4c9937d4b9f1e95783a6203f9f9c7
The drawback is that IF the system has a native MD4 library, it is not used.
That is of course a negative thing, but so far I have not seen any stock
windows/linux system with native MD4 support.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@481 59da171f-624f-0410-aa54-27559c288bec
Status: Fixed
After a lot of work, I have now re-written the entire SharpRSync library.
My tests indicate that generating a diff file for an 8gb file took 10 minutes before, and now takes 8 minutes.
A delta for the same (unchanged) 8GB file took 58 minutes, and now takes 7 minutes.
The implementation seems to run faster under 64 bit, but I'm unsure if it is
related to the actual instructions issued, or the address simulation/conversion done by the OS.
I have also greatly optimized the amount of memory required by the program.
On windows, a delta for an 8GB file can be now be created with a total of 125 mb required memory.
The base app takes 65mb, so the used memory is around 60mb, and the raw diff file is approximately 44mb.
I compared it a bit to the rdiff program on linux.
For a 150mb file, rdiff takes about 1 second to generate a diff, and the same for a delta.
SharpRSync runs the same in about 7 seconds, although some of that is the mono startup and library loading.
For a 8gb file rdiff can generate a diff in about 500 seconds, where SharpRSync takes about 590 seconds.
When generating a delta, rdiff uses 530 seconds, and SharpRSync uses 770 seconds.
The last measurement is inaccurate as the memory requirement under mono is greater, and the test machine
had low memory, so there was a lot of swapping going on.
An interesting side effect is that the rdiff generated delta file is about 380kb, where the SharpRSync delta is 15 bytes.
This is because rdiff does not do the sequence matching optimization mentioned on the librsync page (last paragraph):
http://rsync.samba.org/tech_report/node4.html
Further optimization should probably focus on getting the MD4 algorithm to run faster.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@480 59da171f-624f-0410-aa54-27559c288bec
Fixed a timing issue in the SSH backend, which hopefully solves issue #59.
An error message indicating missing executable filenames was switched between linux and windows.
Added a number of app.config files.
Updated version numbers.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@229 59da171f-624f-0410-aa54-27559c288bec
Updated some sub-projects to VS 2008, fixed .Net 3.5 dependencies to 2.0.
Added a language selection option to the settings dialog.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@222 59da171f-624f-0410-aa54-27559c288bec
Added WixProjBuilder to build the Wix projects without MSBuild support.
Simplified the build/release process.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@215 59da171f-624f-0410-aa54-27559c288bec
Made some lists in the wizard double-click enabled.
Added a warning if password settings are changed for an existing backup.
Filters are now first-touch, as that is more efficient.
Added a filter interface, because regular expressions are very inefficient if they contain more than a thousand subgroups.
Restore now works as expected.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@86 59da171f-624f-0410-aa54-27559c288bec
Updated to the newest version of LightDataModel.
Switched from duplicity to duplicati, this runs a bit faster.
Now supports authentication on network shares (windows only).
git-svn-id: https://duplicati.googlecode.com/svn/trunk@66 59da171f-624f-0410-aa54-27559c288bec
