I have now fixed the issue, and tested it on the file provided.
It seems to compress better now, although the file contains sporadic changes,
even if only a single line was added. The restore also works as expected,
and the update works on my unittest.
Could you please try the updated version and see if it works as expected for you as well?
With regards to the "no else statement" (comment 7), I think that you discovered another
bug, which I have also fixed. The else statement is there to ensure that the buffer is completely filled,
so long chains of unmatched data gets written with as few commands as possible. So, on the first entry,
the copy command is flushed and the force_buffer_refill is set, causing the buffer to refill.
The bug was that IF the buffer was already full, the stream would read 0 bytes, and return a count of 0.
This was then interpreted as if the stream was empty (it returns zero when empty). I have now fixed that
with a check for this special case.
I have also included an update to the ChecksumFileReader that splits the previous m_weakTable into two
separate arrays, as that gives a faster lookup, than using a custom comparer. I also added support for
very large files (files larger than blocksize * int.MaxValue, appx. 4TB).
git-svn-id: https://duplicati.googlecode.com/svn/trunk@548 59da171f-624f-0410-aa54-27559c288bec
Status: Fixed
After a lot of work, I have now re-written the entire SharpRSync library.
My tests indicate that generating a diff file for an 8gb file took 10 minutes before, and now takes 8 minutes.
A delta for the same (unchanged) 8GB file took 58 minutes, and now takes 7 minutes.
The implementation seems to run faster under 64 bit, but I'm unsure if it is
related to the actual instructions issued, or the address simulation/conversion done by the OS.
I have also greatly optimized the amount of memory required by the program.
On windows, a delta for an 8GB file can be now be created with a total of 125 mb required memory.
The base app takes 65mb, so the used memory is around 60mb, and the raw diff file is approximately 44mb.
I compared it a bit to the rdiff program on linux.
For a 150mb file, rdiff takes about 1 second to generate a diff, and the same for a delta.
SharpRSync runs the same in about 7 seconds, although some of that is the mono startup and library loading.
For a 8gb file rdiff can generate a diff in about 500 seconds, where SharpRSync takes about 590 seconds.
When generating a delta, rdiff uses 530 seconds, and SharpRSync uses 770 seconds.
The last measurement is inaccurate as the memory requirement under mono is greater, and the test machine
had low memory, so there was a lot of swapping going on.
An interesting side effect is that the rdiff generated delta file is about 380kb, where the SharpRSync delta is 15 bytes.
This is because rdiff does not do the sequence matching optimization mentioned on the librsync page (last paragraph):
http://rsync.samba.org/tech_report/node4.html
Further optimization should probably focus on getting the MD4 algorithm to run faster.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@480 59da171f-624f-0410-aa54-27559c288bec
kenneth.skovhede@gmail.com