Show Idle (>14 d.) Chans
← 2022-02-16 | 2022-02-18 → ↓
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079525 << to give a some estimate, if a block verifies in 5 sec on average, 720000*5/60/60/24 = about 42 days to verify the whole chain. the 5 secs may be a gross underestimate for the apu1 box, is merely a hunch based on my sync tests with an 9th gen usg-intel cpu
dulapbot: Logged on 2022-02-16 09:26:39 asciilifeform: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079519 << indeed will be useful for standing up nodes 'quickly' (will still be limited by $machine's bw, and by the time to verify blox, imho it'd be suicidally foolish to skip verification even for 'cemented' blox)
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079517 << possibly 'half' cuz just this, but i'll try anyway...: did you already come up with a clean algo how to parallelize verification of next and download of next+n blocks?
dulapbot: Logged on 2022-02-16 00:11:56 asciilifeform: has a half-done cement-mode vpatch, not rdy for primetime justyet
cgra: facing same (or a variant of) question in my attempt to formulate a 'getheaders-sync' patch
dulapbot: Logged on 2022-01-28 13:27:25 cgra: this is again starting to take time beyond imagination, so i thought i'd make an attempt to loosely describe meanwhile, perhaps asciilifeform grasps what i mean:
cgra: i recalled i had some actual measurements recorded, which add up to these figures: in height range 228k-288k, my avg verification time was ~760 ms, in range 289k-317k it was ~1810 ms, and in range 427k-649k it was ~5050 ms.
cgra: the slowest verification i could find off-hand from own (somewhat messy) records was 9.2 seconds, block 0000000000000000000a34fb7816747e8d0e09b535e5d07f70f60c690a64c17b.
cgra: whaack: re podcasts in general, personally i find listening inefficient also because for a difficult/unfamiliar subject to properly sink in, i would need multiple re-reads over time. but i ack your typing pains issue
asciilifeform: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079578 << prolly won't surprise cgra that asciilifeform concluded that doing ~anything~ in parallel in trb is ~impossible w/out removing locks and potentially opening gates of hell. asciilifeform's draft 'usecement' is 100% serial, asks for 1 block at a time (and wastefully, 'getdata' to each connected peer, thinking atm how to reduce the bw guzzling)
dulapbot: Logged on 2022-02-17 05:37:32 cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079517 << possibly 'half' cuz just this, but i'll try anyway...: did you already come up with a clean algo how to parallelize verification of next and download of next+n blocks?
asciilifeform: cgra: i suspect that the only safe way to dl them in parallel would be a ~separate proggy~ which does so and feeds to noad via eatblock
dulapbot: Logged on 2022-02-17 07:35:19 cgra: i recalled i had some actual measurements recorded, which add up to these figures: in height range 228k-288k, my avg verification time was ~760 ms, in range 289k-317k it was ~1810 ms, and in range 427k-649k it was ~5050 ms.
dulapbot: (trilema) 2018-07-13 asciilifeform: http://therealbitcoin.org/ml/btc-dev/2017-February/000256.html << for reference; the block timers.
whaack: !e view-height
trbexplorer: block_height: 723775
trbexplorer: mins_since_last_block: 2
whaack: Ran another query last night, 8.83 mn coins are in p2pkh address UTXOs
whaack: the query to doso was:
whaack: sqlite3 db "SELECT SUM(value) FROM output_input JOIN address ON output_input.address_id = address.address_id WHERE spending_tx_id IS NULL AND hex(address.address) like '76a914%88ac'"
shinohai: So p2pkh still outpacing p2sh?
shinohai: (iirc was ~3mn)
whaack: shinohai: i believe there are 5.2mn in p2sh https://txstats.com/dashboard/db/p2sh-statistics?orgId=1
whaack: i haven't ran the query myself, that's the next one i'm going to construct
whaack: the ~3mn and change i posted yesterday are in addresses that start with 0x00, which are not p2sh, they are the easiest form of 'anyonecanspend'
whaack: so in the trb-addy vs. segwit war i see it as p2pkh vs. (0x00 addresses + p2sh addresses)
whaack: which looks something like 8.8mn vs. 8.5mn, if trends continue it's about to cross over into gavincoin's favor
whaack: cgra: i concur that podcast is a pretty shitty medium, recording is also a pita, when i say something i want to word differently i'm stuck with it unless i rerecord or start doing a collage of audio clips
shinohai: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079569 <<< tbh I had almost forgotten the 'low-S' DER thing, but shouldn't be too difficult to detect if first byte is > 0x7f.
dulapbot: Logged on 2022-02-16 21:17:31 asciilifeform: shinohai: you can do it but 1) the openssl utils hash the thing, you want to sign raw turd 2) they dun enforce the 'low-S' thing
whaack: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079587 <-- My intuition firmly believes that this is the right idea. trb should come with a shit-filterer, another proggy it connects to. This other proggy exists solely to greedily grab blocks from all sources, heathen etc. It then feeds trb the info via the rpc command
dulapbot: Logged on 2022-02-17 09:53:10 asciilifeform: cgra: i suspect that the only safe way to dl them in parallel would be a ~separate proggy~ which does so and feeds to noad via eatblock
whaack: i don't know why you would feed via eatblock, because IMO the goal is to replicate a 'traditional sync' as much as possible
whaack: also in this case the cement logic belongs in the shit-filterer, not in trb itself
asciilifeform: whaack: i'ma release the draft 'usecement' patch, when i'm satisfied that it worx. aint a 'final' solution, but 1) quite lightweight 2) make for sumthing close to fastest possible sync w/out major surgeries
dulapbot: Logged on 2022-02-17 09:52:45 asciilifeform: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079578 << prolly won't surprise cgra that asciilifeform concluded that doing ~anything~ in parallel in trb is ~impossible w/out removing locks and potentially opening gates of hell. asciilifeform's draft 'usecement' is 100% serial, asks for 1 block at a time (and wastefully, 'getdata' to each connected peer, thinking atm how to reduce the bw guzzling)
asciilifeform: note, btw, that in the earlier linked ml experiment, asciilifeform demonstrated that most of block verification cost is in the idjit slowmo db
asciilifeform: plumbing in a sane single-purpose db would shave perhaps 90% of it
dulapbot: Logged on 2021-11-12 12:48:05 asciilifeform: replacing the db with proper o(1) nqb+cryostat would fix this, bdb is monstrously slow
asciilifeform: even the monstrous bdb thing, observe, is 'fast enuff' if noad is 'up to date'. for the most part only becomes an issue when revving up a new one. hence 'cement'.
asciilifeform: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079607 << perhaps obv., but multiple problems w/ such approach ( in what write the 'filter' ? foist python on n00bs on top of the rest of the horror? ... or anuther cpp turd ? or try in ada, where we dun even have tcp glue yet? )
dulapbot: Logged on 2022-02-17 10:55:47 whaack: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079587 <-- My intuition firmly believes that this is the right idea. trb should come with a shit-filterer, another proggy it connects to. This other proggy exists solely to greedily grab blocks from all sources, heathen etc. It then feeds trb the info via the rpc command
asciilifeform: the other thing is, the functionality of trb aint as cleanly separable as one would wish -- if 'filter' actually 100% knew what is a valid tx or block, you would scarcely need trb itself.
asciilifeform: ( recall e.g. the 'gales wallet' people and how mp lost his shit when was explained to him that it still needs to plug into a live noad to have any notion of what inputs to use for a new tx )
asciilifeform: the problem with accepting ~anything~ other than the immediately next block is fundamental, rather than incidental
asciilifeform: i.e. the only kinds of block you can mechanically determine validity of are 1) one that connects immediately after one that you have 2) a cemented one
asciilifeform: any other kind, if you so much as agree to keep it for 5 seconds, opens you up to drowning in liquishit.
asciilifeform did quick napkin calculation re bw : it'd take <1h to move all blox to date through a gb/s pipe. i.e. bw aint the major concern if yer on a decent pipe.
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079616 << does your cement-mode somehow bypass completely, or in part, the bdb bog?
dulapbot: Logged on 2022-02-17 11:36:39 asciilifeform: even the monstrous bdb thing, observe, is 'fast enuff' if noad is 'up to date'. for the most part only becomes an issue when revving up a new one. hence 'cement'.
asciilifeform: cgra: not at all
asciilifeform: cgra: only bypasses the 'ask over9000 times for next block and get nuffin pertinent for n hours / d days' bog.
cgra: right, as i thought
cgra: on account of the 'fast enuff' + 'cement soon' points, the continous sync portion on my table (that 'getheaders-sync') prolly best without any kind of parallelization tricks, cuz less code
cgra: asciilifeform: re cement getdata algo, wanna point out that in theory there's also a risk of sendbuffer flooding some trb peer, especially with a lower-than-default sendbuffersize, if peer's upload is slow, and/or peer stuck at the same time doing smth else
cgra: a fast node responds fast to your getdata chain and allows it going forward, while slow peers' sendbuffers build up at that pace
asciilifeform: cgra: this is a headache even w/out cement neh
asciilifeform: (i.e. currently)
cgra: asciilifeform: normal getdata's are limited by the getblocks response size of the peer, or some other case in mind?
cgra: i mean, a peer won't advertise blocks (in response to getblocks) more at a time than a half of it's sendbuffer can hold
asciilifeform: cgra: in asciilifeform's current draft, asks for 1 block at a time
cgra: asciilifeform: but then another round of getdata when receives the block from the fastest node?
asciilifeform: correct
cgra: asciilifeform: this is when the slowest node may still be wrestling with smth else, but has your first getdata queued
asciilifeform: cgra: trying to think of better algo, which doesn't result in n peers attempting to send in same $block
cgra not convinced yet, after all, that shouldn't try formulating some kinda minimalistic pipeline... cement seems to want similar too
dulapbot: Logged on 2022-02-17 12:37:29 cgra: on account of the 'fast enuff' + 'cement soon' points, the continous sync portion on my table (that 'getheaders-sync') prolly best without any kind of parallelization tricks, cuz less code
asciilifeform: cgra: not impossible, but would be very tricky to limit memory footprint.
cgra: can't get rid of the idea that downloading a bounded buffer of N next blocks could somehow work, and without horror
asciilifeform: for cemented blox certainly could work
asciilifeform: tho because of the asinine pseudo-multithreadedness, only 1 peer can send/recv at a time
cgra: asciilifeform: if can find it for cement, would be enough, cuz 'fast enuff' for normal operation
asciilifeform: possibly even w/out this, 'fast enuff'
asciilifeform wrote errything but the part which actually bakes the getdata's. possibly tonight will finish & testfire
cgra keeps thinking of smth like "spread getdatas of the block range N to peers, follow peer response times and avoid overloading slow peers. buffer populates, verify block every time the lowest block in range N available, adjust buffer afterwards"
asciilifeform: gnarly. 'slow' peer at a given time is usually simply 1 who is busy. 30sec later the same may be 'fastest' peer.
asciilifeform: other problem is that any peer who is sending anyffin at all, ties up the whole noad.
asciilifeform: whether he's sending the requested block, a duplicate, or 'latest' crapola that aint wanted at all
asciilifeform: the practical pill is likely to be to use -connect and pick a reliable peer when syncing w/ cement
asciilifeform: the sheer halfbakedness of the whole design is mindboggling -- e.g. wai did shitoshi do the merkle hash thing, and then decided to send blocks as monolithic turd ?
asciilifeform: would've been entirely possib. to send the header, then the 1st level of merkelisms, then n+1st, etc.
asciilifeform: (for n00bz: block hash is a 'merkle tree', i.e. permits reordering tx and get same hash)
asciilifeform must bbl
signpost tuned in, but sick. probably resurface next week or so.
dulapbot: Logged on 2022-02-04 03:48:53 crtdaydreams: signpost: http://paste.deedbot.org/?id=h-Sc
dulapbot: Logged on 2022-02-12 21:44:32 whaack: lol maybe this is the right warez protocol (@ signpost), a mail-in your hard drive service
signpost: was burned CDs "back in my day"
signpost: probably best density to weight ratio for mailed items is going to be tape backup.
asciilifeform: signpost: afaik presently the winner of 'MB per kg for parcel' is mech. hdd. (with caveat that it takes 2nd place if both parties own a -- rather costly -- current-day tape drive)
asciilifeform: a pretty tight 2nd place tho
asciilifeform: this afaik is on acct of the fact that human-priced tape drives have virtually disappeared, and what remains is strictly in the 'tbtf' golden toilet market, where ancient laws concretely demand the use of tape
dulapbot: Logged on 2022-02-17 14:43:06 signpost: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-04#1078280 << this is done.
asciilifeform has buncha 8 and 4mm scsi tape decks, hasn't powered'em up in some yrs
dulapbot: Logged on 2022-02-17 14:40:36 signpost: tuned in, but sick. probably resurface next week or so.
signpost: asciilifeform: ah, that makes sense.
signpost looked into these recently and yep, was taken aback at the prices for drives.
signpost: ty crtdaydreams
asciilifeform: e.g.: 1 vendor offers lto-9 tape, i.e. 18tb, for ~200 $; a randomly-picked shop offers 18tb hdd for 325 $.
asciilifeform: the latter gives random access, can be bought in over9000 places, and doesn't need a ~5k deck to r/w it.
asciilifeform: but if you need over9000 x 18tb in-house -- the tape wins.
crtdaydreams: I spent some ~160 $ USD for 8TB D:
asciilifeform: crtdaydreams: when?
crtdaydreams: sec, Im going to get actual price
crtdaydreams: ST8000DM004 @ $195 AUD apiece
crtdaydreams: which is some $120 USD
crtdaydreams: mebbe less
crtdaydreams: the ex. rate is like .75 so $140 USD / 8TB
asciilifeform: sounds moar or less ordinary
crtdaydreams: Yeah, It's not like WD Blacks or anything.
asciilifeform: the mega-drives always cost moar 'per mb' than the 'garden variety'
asciilifeform: folx who actually need to pack 'as much as can' into $cabinet, pay the premium
crtdaydreams: hehe data hoarders
asciilifeform: for folx who are buying handful and not over9000, normally makes sense to simply get n moar of the thinner ones
crtdaydreams requires morning sustenance
crtdaydreams: Speaking of economy of scaled, you'd expect moar drives == moar power. There's a balance to be struck with premium disk
asciilifeform: moar power, moar space, moar cooling cost
crtdaydreams: well, less (physical space) per drive? The whole point is to strike up a balance between individual drive draw while increasing capacity in the same sized box, no?
shinohai: https://archive.is/LVgKK <<< lol DOJ has appointed a Director of Buterin's Waterfall.
shinohai likes how USSA always appoints "Czars" but we never get any with those sweet Brezhnev eyebrows .....
verisimilitude: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079576 That's impressive in its awful nature. I recall asciilifeform calling Bitcoin Microsoft-style shitware, but that's impressive all the same.
dulapbot: Logged on 2022-02-17 05:24:00 cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079525 << to give a some estimate, if a block verifies in 5 sec on average, 720000*5/60/60/24 = about 42 days to verify the whole chain. the 5 secs may be a gross underestimate for the apu1 box, is merely a hunch based on my sync tests with an 9th gen usg-intel cpu
verisimilitude: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-18#1074264 I forgot to explicitly note this, jonsykkel, but this is how I write an ``ancient Greek program'', or more appropriately an ``ancient Roman program''.
dulapbot: Logged on 2022-01-18 17:02:10 jonsykkel: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-18#1074159 << looking 4wd to reading
verisimilitude: I'd be elated to know what was thought about it. I'll be working on a successor article and the Ada version of the program described in this article, today, but I've been mulling over details for weeks anyway.
asciilifeform: cgra, whaack , et al ^
asciilifeform: ^ currently testing w/ the cement from previous experiment and sync from local noad. in fact spends ~0 time 'sitting with mouth open', reliably grabs next block errytime thus far. currently in the 50k's (goin' for ~20m nao)
asciilifeform: cement activity is piped to debug log.
asciilifeform: e.g.: 'Proposed block 00000000059f962b47efef20b5038f79cecbf406a1ab6ff9accaefdbe72b815f (57259) ACCEPTED by cement.'
asciilifeform: notably, disabled 'inv' processing when 'in cement', so receives ~0 extraneous blox.
asciilifeform: (also ~no tx, but these aint usefully edible when yer syncing anyway)
asciilifeform: gives pretty close to the fastest possible (w/out parallelisms) sync afaik.
asciilifeform: ... in 70k's, 0 pauses thus far...
vex: I must say, cement is an insightful dectriptor asciilifeform
vex: any allcaps hardware on the horizon?
asciilifeform: vex: nope
asciilifeform: ... 90+k, no pauses. (these are the lightweight blox, tho)
vex: !e height
trbexplorer: vex: my valid commands are: src, uptime, version, help, view-height, view-address, view-utxos, view-balance, view-block, view-raw-block, view-tx, view-raw-tx, push
vex: !e view-height
trbexplorer: block_height: 723831
trbexplorer: mins_since_last_block: 19
vex: what's that in 14 fours alfie?
vex: in 1.44 inch floppies. big stack
vex: inches?
vex: ah
asciilifeform: whaack: comment in yer spam queue
jonsykkel: verisimilitude: red now, good shit
jonsykkel: tho i fail to see why u didnt just do smth greek like http://zzz.st/up/NeHlDMls/
asciilifeform: wb jonsykkel
vex: cunning linguists in full effect.
asciilifeform: ... 100k+, 0 pauses
vex: get well soon signpost
jonsykkel: cementing as we speak from blok235000
asciilifeform: jonsykkel: a++. no pauses ?
asciilifeform: ... 125k+
asciilifeform: interestingly, seems to 'automagically' sync from 'fastest' atm peer
asciilifeform: (rather than one-from-1, one-from-anuther)
jonsykkel: depends wat constitues a pause
jonsykkel: (not sure wat trbsync normally looks like)
asciilifeform: jonsykkel: let's say a full minute w/out an eat
dulapbot: Logged on 2022-02-17 19:35:15 asciilifeform: e.g.: 'Proposed block 00000000059f962b47efef20b5038f79cecbf406a1ab6ff9accaefdbe72b815f (57259) ACCEPTED by cement.'
asciilifeform: jonsykkel: normally, w/out cement, the thing will stall, sometimes for hours, 'waiting with open mouth' for sumbody to feed it
asciilifeform: (when 'behind', that is, or freshly syncing)
jonsykkel: dont have no timestamps in debuglog so hard to say
asciilifeform: jonsykkel: i left mine w/ tail -f debug.log, so visually apparent
jonsykkel: a nice, puted now tail -f debug.log | xargs -IL date +"%Y/%m/%d %H:%M:%S L" >> debug2.log
jonsykkel: will chek bak tomorow
asciilifeform: jonsykkel et al : can try 'unloadcement' to see the difference.
asciilifeform: ( then can load it again w/ 'loadcement' )
asciilifeform notes that it is possible w/ this patch to 'catch up' any 'fallen behind' noad via cement, generate start .. end on a synced one, loadcement on the sad noad
asciilifeform: as i understand this is in fact what jonsykkel was doing
dulapbot: Logged on 2022-02-17 22:10:03 jonsykkel: cementing as we speak from blok235000
jonsykkel: indeed, my sadnode was set up couple days ago
asciilifeform: jonsykkel: what height nao?
jonsykkel: asciilifeform: 235941
jonsykkel: that "235000" was some 235xxx
asciilifeform: jonsykkel: paste coupla tho. ln. of tail ?
asciilifeform curious wai his seems to be going ~6x faster than jonsykkel's
jonsykkel: its a 15yo laptop with mech hdd
asciilifeform: prolly explains
asciilifeform: my testbed is a 'ryzen' on 1g pipe, lol
jonsykkel: this node a bit more sad yep
asciilifeform: w/ nvme
asciilifeform: jonsykkel's test is prolly moar typical tho
asciilifeform: at any rate log loox visually correct.
asciilifeform: i.e. dun see any stalls
asciilifeform: ty jonsykkel
jonsykkel: well see if can get syncd before newyear
asciilifeform must bbl
jonsykkel: cool
← 2022-02-16 | 2022-02-18 → ↑
Random(asciilifeform) | Download hourly DB snapshot | Get Source Code