cgra: i have two more test runs to add to the earlier ones:
snsabot: Logged on 2020-08-30 10:54:24 cgra: some numbers follow, but could be garbage due to some of my mistake. suitable anyway for estimating if even remotely realistic:
cgra: unmodified trb (incl. malleus_mikehearnificarum.vpatch): ~17h runtime, start height ~289k, 24k blocks, 16k bastards, 191MB debug.log
cgra: low-footprint variation of item 2), tx off, sending blocks off, malleus_m enabled: runtime ~21h, start height ~315k, 26k blocks, 2200 bastards, 45MB debug.log
snsabot: Logged on 2020-08-30 10:44:19 cgra: based on this, i've got currently two distinct avenues on my table, both under the same name, "initial sync mode": 1) stop all the tx activity until done with the catch-up, and 2) sync from one peer at a time. after one set of "inv", cycle to the next peer
cgra: while there's still that nat unknown, the numbers suggest to me that:
snsabot: Logged on 2020-08-30 10:49:10 cgra: b) i'm behind nat, so no "inbound" peers
cgra: 1) the highest impact of this tweak is merely in increased prb tolerance, and 2) some bandwidth efficiency is gained from lower amount of bastard blocks.
cgra: OTOH, with low-footprint variation run, i also measured ProcessBlock time vs other activity, and there was ~38% of that other activity. it seemed to me that most of the time not in ProcessBlock was waiting for occasional, slow-uplink peers delivering blocks.
cgra: first impression i get is that prioritizing faster peers could help, but don't immediately see an obvious, compact implementation for such a thing. experimenting next with the 'cement' approach feels more appealing.
cgra: in case there's any reason to go further: the "low-footprint" here would meant: some ~33 lines of code (braces incl), 2 new primitive variables
cgra: http://logs.nosuchlabs.com/log/therealbitcoin/2020-08-30#1001110 << in theory, i suppose there could be an operator-defined stop condition, like "turbo on until block hash xxx", or "until height x", or "until minimum work x"
snsabot: Logged on 2020-08-30 16:54:16 asciilifeform: actually doesn't have any objection to 'sync mode' where node doesn't even attempt to relay tx, etc -- aside from the fact that one would have to manually switch the thing to standard mode by hand when decide that 'synced'
asciilifeform: http://logs.nosuchlabs.com/log/therealbitcoin/2020-09-01#1001121 << my orig. test of that patch ran for year+, fwiw.
snsabot: Logged on 2020-09-01 04:14:56 cgra: low-footprint variation of item 2), tx off, sending blocks off, malleus_m enabled: runtime ~21h, start height ~315k, 26k blocks, 2200 bastards, 45MB debug.log
asciilifeform: ( malleus, that is. )
asciilifeform: http://logs.nosuchlabs.com/log/therealbitcoin/2020-09-01#1001126 << this aint surprising, considering that the thing can only service 1 peer at a time.
snsabot: Logged on 2020-09-01 04:15:36 cgra: OTOH, with low-footprint variation run, i also measured ProcessBlock time vs other activity, and there was ~38% of that other activity. it seemed to me that most of the time not in ProcessBlock was waiting for occasional, slow-uplink peers delivering blocks.
asciilifeform: ( and often enuff the peer aint even delivering block of any kind, but simply hogging the pipe )
asciilifeform: http://logs.nosuchlabs.com/log/therealbitcoin/2020-09-01#1001125 << (1) aint worth it even if somehow gave 2x faster sync (and it does not.) prb often enuff sends bogus blox, bogus tx, and other crapola. observe that the current trb fleet functions just fine w/out 'prb tolerance.'
snsabot: Logged on 2020-09-01 04:15:26 cgra: 1) the highest impact of this tweak is merely in increased prb tolerance, and 2) some bandwidth efficiency is gained from lower amount of bastard blocks.
asciilifeform: !w poll
watchglass: Polling 12 nodes...
watchglass: 205.134.172.4:8333 : (172-4.core.ai.net) Alive: (0.083s) V=70001 (/therealbitcoin.org:0.7.0.1/) Jumpers=0x1 (TRB-Compat.) Blocks=646294
watchglass: 205.134.172.6:8333 : (172-6.core.ai.net) Alive: (0.081s) V=99999 (/therealbitcoin.org:0.9.99.99/) Jumpers=0x1 (TRB-Compat.) Blocks=646294
watchglass: 205.134.172.26:8333 : Alive: (0.084s) V=99999 (/therealbitcoin.org:0.9.99.99/) Jumpers=0x1 (TRB-Compat.) Blocks=646294
watchglass: 108.31.170.3:8333 : (pool-108-31-170-3.washdc.fios.verizon.net) Alive: (0.098s) V=99999 (/therealbitcoin.org:0.9.99.99/) Jumpers=0x1 (TRB-Compat.) Blocks=646294 (Operator: asciilifeform)
watchglass: 205.134.172.27:8333 : Alive: (0.145s) V=99999 (/therealbitcoin.org:0.9.99.99/) Jumpers=0x1 (TRB-Compat.) Blocks=646294 (Operator: asciilifeform)
watchglass: 143.202.160.10:8333 : Alive: (0.310s) V=70001 (/therealbitcoin.org:0.7.0.1/) Jumpers=0x1 (TRB-Compat.) Blocks=646294
watchglass: 192.151.158.26:8333 : Alive: (0.218s) V=70001 (/therealbitcoin.org:0.7.0.1/) Jumpers=0x1 (TRB-Compat.) Blocks=646294
watchglass: 208.94.240.42:8333 : Alive: (0.227s) V=99999 (/therealbitcoin.org:0.9.99.99/) Jumpers=0x1 (TRB-Compat.) Blocks=646294
watchglass: 213.109.238.156:8333 : Alive: (0.328s) V=99999 (/therealbitcoin.org:0.9.99.99/) Jumpers=0x1 (TRB-Compat.) Blocks=646294
watchglass: 188.121.168.69:8333 : (rev-188-121-168-69.radiolan.sk) Alive: (0.391s) V=99999 (/therealbitcoin.org:0.9.99.99/) Jumpers=0x1 (TRB-Compat.) Blocks=646294
watchglass: 103.36.92.112:8333 : (terebe.ns01.net) Alive: (0.624s) V=99999 (/therealbitcoin.org:0.9.99.99/) Jumpers=0x1 (TRB-Compat.) Blocks=646294
watchglass: 176.9.59.199:8333 : Busy? (No answer in 20 sec.) (Operator: jurov)
asciilifeform: http://logs.nosuchlabs.com/log/therealbitcoin/2020-09-01#1001127 << 'cement' aint implemented yet, note.
snsabot: Logged on 2020-09-01 04:15:43 cgra: first impression i get is that prioritizing faster peers could help, but don't immediately see an obvious, compact implementation for such a thing. experimenting next with the 'cement' approach feels more appealing.
cgra: asciilifeform: re "sync mode" related points, yeah, i was in my own mind more or less concluding there wasn't much to improve from to begin with, even though it might've seemd so to an untrained eye (mine)
cgra: i mean, to improve in where i was looking the improvements from
asciilifeform: cgra: i don't mean to suggest that it's 100% impossible to improve the sync mechanism. but did want to illustrate that doing so safely and compactly aint trivial.
asciilifeform: and that most of the historically proposed 'improvements' are anyffin but.
asciilifeform: ( and, importantly, that many of the headaches in the current sync are rooted elsewhere, e.g. snail-slow db )
cgra: http://logs.nosuchlabs.com/log/therealbitcoin/2020-09-01#1001153 << yes. i intend to conjure up something on my own, that resembles the idea (taking into account the starting point of yours you mentioned)
snsabot: Logged on 2020-09-01 12:12:47 asciilifeform: http://logs.nosuchlabs.com/log/therealbitcoin/2020-09-01#1001127 << 'cement' aint implemented yet, note.
asciilifeform: cgra: you'll want to reg yer pubkey w/ deedbot if you want to propose patches for consideration in trb.
cgra: asciilifeform: yes, thanks.
asciilifeform: cgra: i wrote in '18 a patch to enable ~generating~ 'cement' . but not the receiving end.
cgra: funny thing, thought i had found a trb DoS issue (while studying trb's p2p), tried it out and all, but in the last minute, discovered it was already found (obey_sendbuffersize)
asciilifeform: cgra: indeed was found. ( empirically, at first. )
asciilifeform: cgra: see also 'wedger.py'.
cgra: will take a look there too
asciilifeform: cgra: ^ is simply proof of concept for that exploit.
asciilifeform: i.e. reliably wedges any trb noad that wasn't built w/ 'obey_sendbuffersize' .
asciilifeform: ( after which it'll stay hung until manually cycled by operator )
cgra: asciilifeform: i was going to read up ProcessBlock() on my own next, but might as well ask the q i have: how much lighter could the cement-based verification be?
asciilifeform: cgra: not sure i understand -- what means 'lighter verification' here ?
cgra: 'faster' in the most practical sense
asciilifeform: cgra: not substantially. you still gotta let it hash the contents of the incoming block to determine that it matches the cemented hash.
cgra bbl
asciilifeform: what one'd get outta 'cement' is strictly a solution to the 'node waits to have next block dropped in its gaping mouth' problem. for 1st N blox.
asciilifeform: ( i.e. when you ask for ~concrete~ hash, and peer has the block, generally it ~will~ send )