Gusted and GitHub
356c5055ad
lib/sha256: Remove it ( #9643 )
...
### Purpose
Remove the `lib/sha256` package, because it's no longer necessary. Go's
standard library now has the same performance and is on par with
`sha256-simd` since [Since Go
1.21](https://github.com/golang/go/commit/1a64574f42b95594cf9c8a12e9ca13d75585429c ).
Therefore using `sha256-simd` has no benefits anymore.
ARM already has optimized sha256 assembly code since
https://github.com/golang/go/commit/7b8a7f8272fd1941a199af1adb334bd9996e8909 ,
`sha256-simd` published their results before that optimized assembly was
implemented,
https://github.com/minio/sha256-simd/commit/f941fedda826b68a196de2e0a9183e273ec0cb91 .
The assembly looks very similar and the benchmarks in the Go commit
match that of `sha256-simd`.
This patch removes all of the related code of `lib/sha256` and makes
`crypto/sha256` the 'default'.
Benchmark of `sha256-simd` and `crypto/sha256`:
<details>
```
cpu: AMD Ryzen 5 3600X 6-Core Processor
│ simd.txt │ go.txt │
│ sec/op │ sec/op vs base │
Hash/8Bytes-12 63.25n ± 1% 73.38n ± 1% +16.02% (p=0.002 n=6)
Hash/64Bytes-12 98.73n ± 1% 105.30n ± 1% +6.65% (p=0.002 n=6)
Hash/1K-12 567.2n ± 1% 572.8n ± 1% +0.99% (p=0.002 n=6)
Hash/8K-12 4.062µ ± 1% 4.062µ ± 1% ~ (p=0.396 n=6)
Hash/1M-12 512.1µ ± 0% 510.6µ ± 1% ~ (p=0.485 n=6)
Hash/5M-12 2.556m ± 1% 2.564m ± 0% ~ (p=0.093 n=6)
Hash/10M-12 5.112m ± 0% 5.127m ± 0% ~ (p=0.093 n=6)
geomean 13.82µ 14.27µ +3.28%
│ simd.txt │ go.txt │
│ B/s │ B/s vs base │
Hash/8Bytes-12 120.6Mi ± 1% 104.0Mi ± 1% -13.81% (p=0.002 n=6)
Hash/64Bytes-12 618.2Mi ± 1% 579.8Mi ± 1% -6.22% (p=0.002 n=6)
Hash/1K-12 1.682Gi ± 1% 1.665Gi ± 1% -0.98% (p=0.002 n=6)
Hash/8K-12 1.878Gi ± 1% 1.878Gi ± 1% ~ (p=0.310 n=6)
Hash/1M-12 1.907Gi ± 0% 1.913Gi ± 1% ~ (p=0.485 n=6)
Hash/5M-12 1.911Gi ± 1% 1.904Gi ± 0% ~ (p=0.093 n=6)
Hash/10M-12 1.910Gi ± 0% 1.905Gi ± 0% ~ (p=0.093 n=6)
geomean 1.066Gi 1.032Gi -3.18%
```
</details>
### Testing
Compiled and tested on Linux.
### Documentation
https://github.com/syncthing/docs/pull/874
2024-08-10 12:58:20 +01:00
Jakob Borg and GitHub
61b94b9ea5
lib/db: Drop indexes for outgoing data to force refresh (ref #9496 ) ( #9502 )
...
### Purpose
Resend our indexes since we fixed that index-sending issue.
I made a new thing to only drop the non-local-device index IDs, i.e.,
those for other devices. This means we will see a mismatch and resend
all indexes, but they will not. This is somewhat cleaner as it avoids
resending everything twice when two devices are upgraded, and in any
case, we have no reason to force a resend of incoming indexes here.
### Testing
It happens on my computer...
2024-04-08 11:14:27 +02:00
Jakob Borg and GitHub
acd767b30b
all: Remove lib/util package ( #9049 )
...
Grab-bag packages are nasty, this cleans it up a little by splitting it
into topical packages sempahore, netutil, stringutil, structutil.
2023-08-21 19:44:33 +02:00
Simon Frei and GitHub
591e4d8af1
gui, lib: Fix tracking deleted locally-changed on encrypted ( fixes #7715 ) ( #7726 )
2021-11-10 09:46:21 +01:00
greatroar and GitHub
f96c211198
lib/db: Replace SipHash with hash/maphash ( #7962 )
2021-09-24 21:26:07 +02:00
greatroar and GitHub
7fa141ea39
all: Unused args, retvals, assignments ( #7926 )
2021-09-08 00:11:16 +02:00
Simon Frei and GitHub
aeca1fb575
lib/db: Check if sequences change when repairing metadata ( #7770 )
2021-06-17 13:53:39 +02:00
Jakob Borg and GitHub
ce65aea0ab
lib/db: Use a more concurrent GC ( fixes #7722 ) ( #7750 )
...
This changes the GC mechanism so that the first pass (which reads all
FileInfos to populate bloom filters with block & version hashes) can
happen concurrently with normal database operations.
The big gcMut still exists, and we grab it temporarily to block all
other modifications while we set up the bloom filters. We then release
the lock and let other things happen, with those other things also
updating the bloom filters as required. Once the first phase is done we
again grab the gcMut, knowing that we are the sole modifier of the
database, and do the cleanup.
I also removed the final compaction step.
2021-06-07 10:52:06 +02:00
greatroar and GitHub
95c9561e97
lib/db: Clean up Timer and wait for logging before return in GC ( #7720 )
2021-05-31 09:50:21 +02:00
Simon Frei and GitHub
58592e3ef1
lib/db: Add logging for GC ( #7707 )
2021-05-22 21:36:43 +02:00
Simon Frei and GitHub
0d054f9b64
lib/model: Don't use empty folder cfg for index sender ( fixes #7649 ) ( #7671 )
2021-05-15 11:13:39 +02:00
Simon Frei and GitHub
f30f9c50f8
lib/db: Handle indirection error repairing sequences ( fixes #7026 ) ( #7525 )
2021-04-05 10:24:16 +02:00
Jakob Borg and GitHub
6e5514419d
lib/db: Fix some omitted error checks, unused variable ( #7489 )
2021-03-17 21:41:07 +01:00
Simon Frei and GitHub
273ee09925
lib/db, lib/model: Allow needing invalid files ( fixes #7474 ) ( #7476 )
2021-03-15 07:58:01 +01:00
Simon Frei and GitHub
27a34609a1
all: Failure reporting fixes ( #7331 )
2021-02-05 11:21:14 +01:00
Simon Frei and GitHub
070bf3b776
lib/db: Report number of repaired items from checkGlobal ( #7329 )
2021-02-04 14:42:46 +01:00
Simon Frei and GitHub
a20a5f61f0
lib/ur: Send unreported failures on shutdown ( #7164 )
2020-12-22 20:17:14 +01:00
Simon Frei and GitHub
78bd0341a8
all: Handle errors opening db/creating file-set (ref #5907 ) ( #7150 )
2020-12-21 12:59:22 +01:00
Simon Frei and GitHub
4a787986cd
lib/db: Prevent IndexID creation race ( #7211 )
2020-12-21 11:32:59 +01:00
Simon Frei and GitHub
bd0c9913cf
lib/db: Remove index ids when dropping folder ( #7200 )
2020-12-21 11:10:59 +01:00
Simon Frei and GitHub
fa40ccece1
lib: Consistently set suture logging ( #7202 )
2020-12-18 19:44:00 +01:00
Simon Frei and GitHub
9524b51708
all: Implement suture v4-api ( #6947 )
2020-11-17 13:19:04 +01:00
Simon Frei and GitHub
01a7ef3b0f
lib/db: Undo adding user info to panic msgs (ref #7029 ) ( #7040 )
2020-10-19 08:40:37 +02:00
Simon Frei and GitHub
23c935b05a
lib/db: Ignore not found on delete in recalcGlobal (ref #7026 ) ( #7041 )
2020-10-19 08:28:53 +02:00
Simon Frei and GitHub
d828adb648
cmd/stcrashreceiver, lib/db: Improve panic message handling ( #7029 )
2020-10-08 17:37:45 +02:00
Simon Frei and GitHub
14ae330eff
lib/db: Improve error handling checking db (ref #7026 ) ( #7027 )
2020-10-06 20:14:09 +02:00
Simon Frei and GitHub
08bebbe59b
lib/db, lib/syncthing: Don't repair DB on upgrade, but on error ( fixes #6917 ) ( #6971 )
2020-09-10 10:54:41 +02:00
Simon Frei and GitHub
7b821d2550
lib/db: Add local need check&repair ( #6950 )
2020-09-04 14:01:46 +02:00
Simon Frei and GitHub
cc1f6e4d4a
lib/db, lib/model: Cover exec-paths with debug logging ( #6918 )
2020-08-20 16:11:20 +02:00
Simon Frei and GitHub
8f5215878b
lib/db: Don't put truncated files (ref #6855 , ref #6501 ) ( #6888 )
2020-08-18 09:20:12 +02:00
Simon Frei and GitHub
424d1b1608
lib/db: Commit meta when dropping device ( #6862 )
2020-07-28 16:46:42 +02:00
Simon Frei and GitHub
1b9e5c0937
lib/db: Include blocks in db check (ref #6855 ) ( #6861 )
2020-07-28 16:25:07 +02:00
Audrius Butkevicius and GitHub
55147f5901
lib/db: Rework flush hooks ( #6838 )
2020-07-19 08:55:27 +02:00
greatroar and GitHub
9f92f8c609
lib/db: Use SipHash to deal with hash collision in GC ( #6826 )
...
If the GC finds a key k that it wants to keep, it records that in a
Bloom filter. If a key k' can be removed but its hash collides with k,
it will be kept. Since the old Bloom filter code was completely
deterministic, the next run would encounter the same collision, assuming
k must still be kept.
A randomized hash function that uses all the SHA-256 bits solves this
problem: the second run has a non-zero probability of removing k', as
long as the Bloom filter is not completely full.
2020-07-11 09:36:09 +02:00
Simon Frei and GitHub
8cf9d91ed4
lib: Print nicely rounded durations ( #6756 )
2020-06-18 10:55:41 +02:00
Simon Frei and GitHub
cbe0d2fffc
lib/db: Improve error message on meta inconsistency ( #6751 )
2020-06-17 10:03:39 +02:00
André Colomb and GitHub
46536509d7
lib/protocol: Avoid panic in DeviceIDFromBytes ( #6714 )
2020-06-07 10:31:12 +02:00
1f8e6c55f6
lib/db: Refactor to use global list by version ( fixes #6372 ) ( #6638 )
...
Group the global list of files by version, instead of having one flat list for all devices. This removes lots of duplicate protocol.Vectors.
Co-authored-by: Jakob Borg <jakob@kastelo.net >
2020-05-30 09:50:23 +02:00
Jakob Borg and GitHub
94beed5c10
lib/db: Add Badger backend ( fixes #5910 ) ( #6250 )
2020-05-29 13:43:02 +02:00
Audrius Butkevicius and GitHub
a1c5b44c74
lib/model: Fix rename handling (ref #6650 ) ( #6652 )
2020-05-16 14:39:27 +02:00
Simon Frei and GitHub
974551375e
lib/db: Dont add symlinks to blocks map ( fixes #6637 ) ( #6639 )
2020-05-13 20:38:21 +02:00
Jakob Borg and GitHub
531ceb2b0f
Add indirection for large version vectors. ( #6376 )
...
This adds indirection of large version vectors in the same manner as we
already to block lists. The effect is the same: less duplicated data in
some situations.
To mitigate the impact for when this indirection
wouldn't be needed I've added an indirection cutoff for both blocks and
the new version vector stuff: we don't do the indirection at all for
small block lists or small version vectors, instead storing it directly
like we used to do. This is faster for small files and small setups.
2020-05-13 14:28:42 +02:00
Audrius Butkevicius and GitHub
decb967969
all: Reorder sequences for better rename detection ( #6574 )
2020-05-11 20:15:11 +02:00
Simon Frei and GitHub
a94951becd
lib/db, lib/model: Keep need stats in metadata (ref #5899 ) ( #6413 )
2020-05-11 15:07:06 +02:00
Jakob Borg and GitHub
c0c18a568c
lib/db: Hold the bloom filter the right way ( fixes #6614 ) ( #6617 )
2020-05-08 14:18:00 +02:00
greatroar and GitHub
0e5ba3ca05
lib/db: Upgrade to Blobloom v0.1.1 ( #6553 )
...
Now faster and Apache-licensed.
2020-04-20 14:23:36 +02:00
greatroar and GitHub
44b0f0b456
lib/db: Switch to faster blobloom Bloom filter pkg ( #6537 )
2020-04-20 09:02:33 +02:00
Jakob Borg and GitHub
0e67c036bb
lib/db: Make database GC a service, stop on Stop() ( #6518 )
...
This makes the GC runner a service that will stop fairly quickly when
told to.
As a bonus, STTRACE=app will print the service tree on the way out,
including any errors they've flagged.
2020-04-12 10:26:57 +02:00
Simon Frei and GitHub
df318ed370
lib/db: Don't get blocklists on drop and missing continue (ref #6457 ) ( #6502 )
2020-04-07 13:13:18 +02:00
Simon Frei and GitHub
7f23de4f03
all: Pass db intervals as args not env vars ( #6448 )
2020-03-24 13:53:20 +01:00
