cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079525 << to give a some estimate, if a block verifies in 5 sec on average, 720000*5/60/60/24 = about 42 days to verify the whole chain. the 5 secs may be a gross underestimate for the apu1 box, is merely a hunch based on my sync tests with an 9th gen usg-intel cpu
dulapbot: Logged on 2022-02-16 09:26:39 asciilifeform: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079519 << indeed will be useful for standing up nodes 'quickly' (will still be limited by $machine's bw, and by the time to verify blox, imho it'd be suicidally foolish to skip verification even for 'cemented' blox)
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079517 << possibly 'half' cuz just this, but i'll try anyway...: did you already come up with a clean algo how to parallelize verification of next and download of next+n blocks?
dulapbot: Logged on 2022-02-16 00:11:56 asciilifeform: has a half-done cement-mode vpatch, not rdy for primetime justyet
dulapbot: Logged on 2022-01-28 13:27:25 cgra: this is again starting to take time beyond imagination, so i thought i'd make an attempt to loosely describe meanwhile, perhaps asciilifeform grasps what i mean:
cgra: i recalled i had some actual measurements recorded, which add up to these figures: in height range 228k-288k, my avg verification time was ~760 ms, in range 289k-317k it was ~1810 ms, and in range 427k-649k it was ~5050 ms.
cgra: the slowest verification i could find off-hand from own (somewhat messy) records was 9.2 seconds, block 0000000000000000000a34fb7816747e8d0e09b535e5d07f70f60c690a64c17b.
cgra: whaack: re podcasts in general, personally i find listening inefficient also because for a difficult/unfamiliar subject to properly sink in, i would need multiple re-reads over time. but i ack your typing pains issue
asciilifeform: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079578 << prolly won't surprise cgra that asciilifeform concluded that doing ~anything~ in parallel in trb is ~impossible w/out removing locks and potentially opening gates of hell. asciilifeform's draft 'usecement' is 100% serial, asks for 1 block at a time (and wastefully, 'getdata' to each connected peer, thinking atm how to reduce the bw guzzling)
dulapbot: Logged on 2022-02-17 05:37:32 cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079517 << possibly 'half' cuz just this, but i'll try anyway...: did you already come up with a clean algo how to parallelize verification of next and download of next+n blocks?
asciilifeform: cgra: i suspect that the only safe way to dl them in parallel would be a ~separate proggy~ which does so and feeds to noad via eatblock
asciilifeform: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079582 << >> http://logs.nosuchlabs.com/log/trilema/2018-07-13#1834130
dulapbot: Logged on 2022-02-17 07:35:19 cgra: i recalled i had some actual measurements recorded, which add up to these figures: in height range 228k-288k, my avg verification time was ~760 ms, in range 289k-317k it was ~1810 ms, and in range 427k-649k it was ~5050 ms.
dulapbot: (trilema) 2018-07-13 asciilifeform: http://therealbitcoin.org/ml/btc-dev/2017-February/000256.html << for reference; the block timers.
whaack: !e view-height
trbexplorer: block_height: 723775
trbexplorer: mins_since_last_block: 2
whaack: Ran another query last night, 8.83 mn coins are in p2pkh address UTXOs
whaack: the query to doso was:
whaack: sqlite3 db "SELECT SUM(value) FROM output_input JOIN address ON output_input.address_id = address.address_id WHERE spending_tx_id IS NULL AND hex(address.address) like '76a914%88ac'"
shinohai: So p2pkh still outpacing p2sh?
shinohai: (iirc was ~3mn)
whaack: shinohai: i believe there are 5.2mn in p2sh https://txstats.com/dashboard/db/p2sh-statistics?orgId=1
whaack: i haven't ran the query myself, that's the next one i'm going to construct
whaack: the ~3mn and change i posted yesterday are in addresses that start with 0x00, which are not p2sh, they are the easiest form of 'anyonecanspend'
whaack: so in the trb-addy vs. segwit war i see it as p2pkh vs. (0x00 addresses + p2sh addresses)
whaack: which looks something like 8.8mn vs. 8.5mn, if trends continue it's about to cross over into gavincoin's favor
whaack: cgra: i concur that podcast is a pretty shitty medium, recording is also a pita, when i say something i want to word differently i'm stuck with it unless i rerecord or start doing a collage of audio clips
shinohai: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079569 <<< tbh I had almost forgotten the 'low-S' DER thing, but shouldn't be too difficult to detect if first byte is > 0x7f.
dulapbot: Logged on 2022-02-16 21:17:31 asciilifeform: shinohai: you can do it but 1) the openssl utils hash the thing, you want to sign raw turd 2) they dun enforce the 'low-S' thing
whaack: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079587 <-- My intuition firmly believes that this is the right idea. trb should come with a shit-filterer, another proggy it connects to. This other proggy exists solely to greedily grab blocks from all sources, heathen etc. It then feeds trb the info via the rpc command
dulapbot: Logged on 2022-02-17 09:53:10 asciilifeform: cgra: i suspect that the only safe way to dl them in parallel would be a ~separate proggy~ which does so and feeds to noad via eatblock
whaack: i don't know why you would feed via eatblock, because IMO the goal is to replicate a 'traditional sync' as much as possible
whaack: also in this case the cement logic belongs in the shit-filterer, not in trb itself
asciilifeform: whaack: i'ma release the draft 'usecement' patch, when i'm satisfied that it worx. aint a 'final' solution, but 1) quite lightweight 2) make for sumthing close to fastest possible sync w/out major surgeries
dulapbot: Logged on 2022-02-17 09:52:45 asciilifeform: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079578 << prolly won't surprise cgra that asciilifeform concluded that doing ~anything~ in parallel in trb is ~impossible w/out removing locks and potentially opening gates of hell. asciilifeform's draft 'usecement' is 100% serial, asks for 1 block at a time (and wastefully, 'getdata' to each connected peer, thinking atm how to reduce the bw guzzling)
asciilifeform: note, btw, that in the earlier linked ml experiment, asciilifeform demonstrated that most of block verification cost is in the idjit slowmo db
dulapbot: Logged on 2021-11-12 12:48:05 asciilifeform: replacing the db with proper o(1) nqb+cryostat would fix this, bdb is monstrously slow
asciilifeform: even the monstrous bdb thing, observe, is 'fast enuff' if noad is 'up to date'. for the most part only becomes an issue when revving up a new one. hence 'cement'.
asciilifeform: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079607 << perhaps obv., but multiple problems w/ such approach ( in what write the 'filter' ? foist python on n00bs on top of the rest of the horror? ... or anuther cpp turd ? or try in ada, where we dun even have tcp glue yet? )
dulapbot: Logged on 2022-02-17 10:55:47 whaack: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079587 <-- My intuition firmly believes that this is the right idea. trb should come with a shit-filterer, another proggy it connects to. This other proggy exists solely to greedily grab blocks from all sources, heathen etc. It then feeds trb the info via the rpc command
asciilifeform: the other thing is, the functionality of trb aint as cleanly separable as one would wish -- if 'filter' actually 100% knew what is a valid tx or block, you would scarcely need trb itself.
asciilifeform: ( recall e.g. the 'gales wallet' people and how mp lost his shit when was explained to him that it still needs to plug into a live noad to have any notion of what inputs to use for a new tx )
asciilifeform: the problem with accepting ~anything~ other than the immediately next block is fundamental, rather than incidental
asciilifeform: i.e. the only kinds of block you can mechanically determine validity of are 1) one that connects immediately after one that you have 2) a cemented one
asciilifeform: any other kind, if you so much as agree to keep it for 5 seconds, opens you up to drowning in liquishit.
asciilifeform did quick napkin calculation re bw : it'd take <1h to move all blox to date through a gb/s pipe. i.e. bw aint the major concern if yer on a decent pipe.
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079616 << does your cement-mode somehow bypass completely, or in part, the bdb bog?
dulapbot: Logged on 2022-02-17 11:36:39 asciilifeform: even the monstrous bdb thing, observe, is 'fast enuff' if noad is 'up to date'. for the most part only becomes an issue when revving up a new one. hence 'cement'.
asciilifeform: cgra: not at all
asciilifeform: cgra: only bypasses the 'ask over9000 times for next block and get nuffin pertinent for n hours / d days' bog.
cgra: right, as i thought
cgra: on account of the 'fast enuff' + 'cement soon' points, the continous sync portion on my table (that 'getheaders-sync') prolly best without any kind of parallelization tricks, cuz less code
cgra: asciilifeform: re cement getdata algo, wanna point out that in theory there's also a risk of sendbuffer flooding some trb peer, especially with a lower-than-default sendbuffersize, if peer's upload is slow, and/or peer stuck at the same time doing smth else
cgra: a fast node responds fast to your getdata chain and allows it going forward, while slow peers' sendbuffers build up at that pace
asciilifeform: cgra: this is a headache even w/out cement neh
asciilifeform: (i.e. currently)
cgra: asciilifeform: normal getdata's are limited by the getblocks response size of the peer, or some other case in mind?
cgra: i mean, a peer won't advertise blocks (in response to getblocks) more at a time than a half of it's sendbuffer can hold
asciilifeform: cgra: in asciilifeform's current draft, asks for 1 block at a time
cgra: asciilifeform: but then another round of getdata when receives the block from the fastest node?
cgra: asciilifeform: this is when the slowest node may still be wrestling with smth else, but has your first getdata queued
asciilifeform: cgra: trying to think of better algo, which doesn't result in n peers attempting to send in same $block
cgra not convinced yet, after all, that shouldn't try formulating some kinda minimalistic pipeline... cement seems to want similar too
dulapbot: Logged on 2022-02-17 12:37:29 cgra: on account of the 'fast enuff' + 'cement soon' points, the continous sync portion on my table (that 'getheaders-sync') prolly best without any kind of parallelization tricks, cuz less code
asciilifeform: cgra: not impossible, but would be very tricky to limit memory footprint.
cgra: can't get rid of the idea that downloading a bounded buffer of N next blocks could somehow work, and without horror
asciilifeform: for cemented blox certainly could work
asciilifeform: tho because of the asinine pseudo-multithreadedness, only 1 peer can send/recv at a time
cgra: asciilifeform: if can find it for cement, would be enough, cuz 'fast enuff' for normal operation
asciilifeform: possibly even w/out this, 'fast enuff'
asciilifeform wrote errything but the part which actually bakes the getdata's. possibly tonight will finish & testfire
cgra keeps thinking of smth like "spread getdatas of the block range N to peers, follow peer response times and avoid overloading slow peers. buffer populates, verify block every time the lowest block in range N available, adjust buffer afterwards"
asciilifeform: gnarly. 'slow' peer at a given time is usually simply 1 who is busy. 30sec later the same may be 'fastest' peer.
asciilifeform: other problem is that any peer who is sending anyffin at all, ties up the whole noad.
asciilifeform: whether he's sending the requested block, a duplicate, or 'latest' crapola that aint wanted at all
asciilifeform: the practical pill is likely to be to use -connect and pick a reliable peer when syncing w/ cement
asciilifeform: the sheer halfbakedness of the whole design is mindboggling -- e.g. wai did shitoshi do the merkle hash thing, and then decided to send blocks as monolithic turd ?
asciilifeform: would've been entirely possib. to send the header, then the 1st level of merkelisms, then n+1st, etc.
asciilifeform: (for n00bz: block hash is a 'merkle tree', i.e. permits reordering tx and get same hash)
asciilifeform must bbl
signpost tuned in, but sick. probably resurface next week or so.
signpost: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-04#1078280 << this is done.
signpost: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-12#1079042 << good old sneakernet
dulapbot: Logged on 2022-02-12 21:44:32 whaack: lol maybe this is the right warez protocol (@ signpost), a mail-in your hard drive service
signpost: was burned CDs "back in my day"
signpost: probably best density to weight ratio for mailed items is going to be tape backup.
asciilifeform: signpost: afaik presently the winner of 'MB per kg for parcel' is mech. hdd. (with caveat that it takes 2nd place if both parties own a -- rather costly -- current-day tape drive)
asciilifeform: a pretty tight 2nd place tho
asciilifeform: this afaik is on acct of the fact that human-priced tape drives have virtually disappeared, and what remains is strictly in the 'tbtf' golden toilet market, where ancient laws concretely demand the use of tape
crtdaydreams: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079662 << Thanking you kindly.
dulapbot: Logged on 2022-02-17 14:43:06 signpost: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-04#1078280 << this is done.
asciilifeform has buncha 8 and 4mm scsi tape decks, hasn't powered'em up in some yrs
crtdaydreams: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079661 << Get well soon :)
dulapbot: Logged on 2022-02-17 14:40:36 signpost: tuned in, but sick. probably resurface next week or so.
signpost: asciilifeform: ah, that makes sense.
signpost looked into these recently and yep, was taken aback at the prices for drives.
signpost: ty crtdaydreams
asciilifeform: e.g.: 1 vendor offers lto-9 tape, i.e. 18tb, for ~200 $; a randomly-picked shop offers 18tb hdd for 325 $.
asciilifeform: the latter gives random access, can be bought in over9000 places, and doesn't need a ~5k deck to r/w it.
asciilifeform: but if you need over9000 x 18tb in-house -- the tape wins.
crtdaydreams: I spent some ~160 $ USD for 8TB D:
asciilifeform: crtdaydreams: when?
crtdaydreams: sec, Im going to get actual price
crtdaydreams: ST8000DM004 @ $195 AUD apiece
crtdaydreams: which is some $120 USD
crtdaydreams: mebbe less
crtdaydreams: the ex. rate is like .75 so $140 USD / 8TB
asciilifeform: sounds moar or less ordinary
crtdaydreams: Yeah, It's not like WD Blacks or anything.
asciilifeform: the mega-drives always cost moar 'per mb' than the 'garden variety'
asciilifeform: folx who actually need to pack 'as much as can' into $cabinet, pay the premium
crtdaydreams: hehe data hoarders
asciilifeform: for folx who are buying handful and not over9000, normally makes sense to simply get n moar of the thinner ones
crtdaydreams requires morning sustenance
crtdaydreams: Speaking of economy of scaled, you'd expect moar drives == moar power. There's a balance to be struck with premium disk
asciilifeform: moar power, moar space, moar cooling cost
crtdaydreams: well, less (physical space) per drive? The whole point is to strike up a balance between individual drive draw while increasing capacity in the same sized box, no?
shinohai likes how USSA always appoints "Czars" but we never get any with those sweet Brezhnev eyebrows .....
verisimilitude: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079576 That's impressive in its awful nature. I recall asciilifeform calling Bitcoin Microsoft-style shitware, but that's impressive all the same.
dulapbot: Logged on 2022-02-17 05:24:00 cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079525 << to give a some estimate, if a block verifies in 5 sec on average, 720000*5/60/60/24 = about 42 days to verify the whole chain. the 5 secs may be a gross underestimate for the apu1 box, is merely a hunch based on my sync tests with an 9th gen usg-intel cpu
verisimilitude: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-18#1074264 I forgot to explicitly note this, jonsykkel, but this is how I write an ``ancient Greek program'', or more appropriately an ``ancient Roman program''.
dulapbot: Logged on 2022-01-18 17:02:10 jonsykkel: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-18#1074159 << looking 4wd to reading
verisimilitude: I'd be elated to know what was thought about it. I'll be working on a successor article and the Ada version of the program described in this article, today, but I've been mulling over details for weeks anyway.
asciilifeform: cgra, whaack , et al ^
asciilifeform: ^ currently testing w/ the cement from previous experiment and sync from local noad. in fact spends ~0 time 'sitting with mouth open', reliably grabs next block errytime thus far. currently in the 50k's (goin' for ~20m nao)
asciilifeform: cement activity is piped to debug log.
asciilifeform: e.g.: 'Proposed block 00000000059f962b47efef20b5038f79cecbf406a1ab6ff9accaefdbe72b815f (57259) ACCEPTED by cement.'
asciilifeform: notably, disabled 'inv' processing when 'in cement', so receives ~0 extraneous blox.
asciilifeform: (also ~no tx, but these aint usefully edible when yer syncing anyway)
asciilifeform: gives pretty close to the fastest possible (w/out parallelisms) sync afaik.
asciilifeform: ... in 70k's, 0 pauses thus far...
vex: I must say, cement is an insightful dectriptor asciilifeform
vex: any allcaps hardware on the horizon?
asciilifeform: vex: nope
asciilifeform: ... 90+k, no pauses. (these are the lightweight blox, tho)
vex: !e height
trbexplorer: vex: my valid commands are: src, uptime, version, help, view-height, view-address, view-utxos, view-balance, view-block, view-raw-block, view-tx, view-raw-tx, push
vex: !e view-height
trbexplorer: block_height: 723831
trbexplorer: mins_since_last_block: 19
vex: what's that in 14 fours alfie?
vex: in 1.44 inch floppies. big stack
asciilifeform: whaack: comment in yer spam queue
jonsykkel: verisimilitude: red now, good shit
asciilifeform: wb jonsykkel
vex: cunning linguists in full effect.
asciilifeform: ... 100k+, 0 pauses
vex: get well soon signpost
jonsykkel: cementing as we speak from blok235000
asciilifeform: jonsykkel: a++. no pauses ?
asciilifeform: ... 125k+
asciilifeform: interestingly, seems to 'automagically' sync from 'fastest' atm peer
asciilifeform: (rather than one-from-1, one-from-anuther)
jonsykkel: depends wat constitues a pause
jonsykkel: (not sure wat trbsync normally looks like)
dulapbot: Logged on 2022-02-17 19:35:15 asciilifeform: e.g.: 'Proposed block 00000000059f962b47efef20b5038f79cecbf406a1ab6ff9accaefdbe72b815f (57259) ACCEPTED by cement.'
asciilifeform: jonsykkel: normally, w/out cement, the thing will stall, sometimes for hours, 'waiting with open mouth' for sumbody to feed it
asciilifeform: (when 'behind', that is, or freshly syncing)
jonsykkel: dont have no timestamps in debuglog so hard to say
asciilifeform: jonsykkel: i left mine w/ tail -f debug.log, so visually apparent
jonsykkel: a nice, puted now tail -f debug.log | xargs -IL date +"%Y/%m/%d %H:%M:%S L" >> debug2.log
jonsykkel: will chek bak tomorow
asciilifeform: jonsykkel et al : can try 'unloadcement' to see the difference.
asciilifeform: ( then can load it again w/ 'loadcement' )
asciilifeform notes that it is possible w/ this patch to 'catch up' any 'fallen behind' noad via cement, generate start .. end on a synced one, loadcement on the sad noad
dulapbot: Logged on 2022-02-17 22:10:03 jonsykkel: cementing as we speak from blok235000
jonsykkel: indeed, my sadnode was set up couple days ago
asciilifeform: jonsykkel: what height nao?
jonsykkel: asciilifeform: 235941
jonsykkel: that "235000" was some 235xxx
asciilifeform: jonsykkel: paste coupla tho. ln. of tail ?
asciilifeform curious wai his seems to be going ~6x faster than jonsykkel's
jonsykkel: its a 15yo laptop with mech hdd
asciilifeform: prolly explains
asciilifeform: my testbed is a 'ryzen' on 1g pipe, lol
jonsykkel: this node a bit more sad yep
asciilifeform: w/ nvme
asciilifeform: jonsykkel's test is prolly moar typical tho
asciilifeform: so a++
asciilifeform: at any rate log loox visually correct.
asciilifeform: i.e. dun see any stalls
asciilifeform: ty jonsykkel
jonsykkel: well see if can get syncd before newyear
asciilifeform must bbl