Show Idle (> d.) Chans
Results 1 ... 250 found in asciilifeform for 'from:cgra' | →
cgra: asciilifeform: saw, looking forward to it
cgra: also tried "tk-tagging" 300k rows of text. filtering text window with tags is instantaneous
cgra: perhaps interestingly, found out that tcl (my version at least) won't eat unicode points above 0xFFFF, causing tkinter to crash and complain
cgra also fiddled with tkinter. duct-taped a simple gui into blatta and hoping to show up eventually on the #pest side.
cgra: jonsykkel: ty!
cgra: signpost, lobbes: thanks!
cgra willing to peer up in pest, if any of billymg, jonsykkel, thimbronion, mod6, unpx want. ditto for any other pest-regular not currently on this medium
cgra: asciilifeform: ty! i'll try and show up in a few days
cgra: oh, and forgot to mention that my ip address changes ~daily (partly because of the power cycling)
cgra: i'm currently usually powering gear off overnight, and often have completely offline days. would the current pestnet handle that? or would it perhaps make an interesting experiment?
cgra: but ^ i accidentally forgot the replay attack angle...
cgra: irc-variant could simply show console warnings, with given msg and station-local timestamp comparisons
cgra: (at least for me personally) hard-to-picture scenarios like above, seem to exist how the stale check is bypassed, and hence seems cleaner to not try to force a peer to behave. instead let him spam or let his clock drift, and let the problem show up as is, with minimal indirection and gotchas. then you can decide whether some action required
cgra: asciilifeform: peer B sent funny timestamps first, then sent a good one to a common peer C of A and B. C now getdatas the funny items and sends the latest, good one to A. A concludes a broken chain, and getdatas also the funny items from C
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-07-20#1111878 << i kinda assumed constant 'prodding' going on, but don't seem to actually be the case per current spec draft. is it going to be? if regularly prodding, now the advertised, badly timestamped segment becomes subject to getdata, and bypasses the stale check
cgra: asciilifeform: does this affect your intent to implement the 'chain locking', or still want it the same? if still the same, how do you view 'deterministic S-turd' approaches, like: "S-turd := HASH(S0 + chain_msg_seq_no)"?
cgra: fwiw, in my possibly uneducated opinion, i find the 'my name is' msg type meaningful. cuz a selfchain is a more fitting item to label, than a single msg. while msgs in a selfchain have most of the time the same handle.
cgra: asciilifeform: since the pest getdata does, and more or less has to, bypass the staleness check, the whole checking of stales may be useless. even 'future' messages will eventually get through (even if a now omitted getdata future check was specified). OTOH, if you don't bother with checking stales, you now get more accurate reception timestamps of these badly timestamped messages, and more accurate picture of what's (been) going on
cgra: or otherwise got only a vague impression upon read
cgra: asciilifeform: coupla explicit mentions of packet rejection missing i think
cgra: ok. another, simpler point: prolly want to reject any packet with any invalid AString or UString field, including non-null(s) after null char?
cgra: yeah, was next gonna highlight the continuity issue
cgra: asciilifeform: when usenet-style threads come to pest, is a null netchain BM value becoming commonplace? ie. 'a new thread'
cgra: sorry, mis-enter earlier
cgra: asciilifeform, checking that we're on the same page: i'm trying to understand whether those tricksters are an actual issue. cuz selfchains, assuming ~smooth operation re forks/breaks, seem to give clear identity for every bob
cgra: asciilifeform: checking that we'
cgra: ah ok
cgra: is the 'squaring the circle' part here that there's two conflicting (given two pestnets joining) povs, both requiring "my bob must stay bob."?
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-07-13#1111437 << do you mean here that one solution would be that whoever talked in pest (whichever net) first deserves the handle? and could check from genesis message timestamp. or do you mean that "my bob must stay bob"?
cgra: or, i'm not sure what the bob q was meant to highlight exactly. an irc specific issue?
cgra: asciilifeform: aren't you asking who's bob while they got their own barcode (selfchain) in their back of the head, and could check there instead?
cgra: good maybe being realistic
cgra: i suppose i could use a good example, maybe i have no imagination
cgra: i mean, by asking the participants what's going on
cgra: what i mean by questioning the locking mechanism, that is there a break/fork case that wouldn't resolve just by ~asking out?
cgra: yeah, trying to take one step at a time. first step being not constraining to irc specifics
cgra: (and the hash icon feature you said earlier)
cgra: asciilifeform: i suppose a gui pestron feature that reads text 'bob' on screen, but links to a distinct selfchain, worked until agreed upon, new name(s)
cgra: asciilifeform: but now that the circle is larger, don't the both bobs need commonly agreed names?
cgra: asciilifeform: and temporary two bobs are ok if there's at least some distinguishing factor, like the intact selfchain down to the day 1?
cgra: (bot l2 case hardly warrants locking mechanism though)
cgra: right, that's a clear l2 case
cgra: can you imagine a case for l3?
cgra: so, it's still down to trusting the choke-point, as with without the locking mechanism
cgra: because received the just-revealed S-turd before others
cgra: the choke-point may still plant his own chain extending msg in place of the original sender
cgra: asciilifeform: i grasped the algo, not necessarily all implications though
cgra: or maybe not automatically, but thought that since it's down to trust, wouldn't need anything else but simple mechanics
cgra: asciilifeform: you pointed out the thread, i could try re-reading if you think there's an obvious explanation
cgra: i was wondering whether there's any demand to automatically resolve either case
cgra: does 'forked' mean specifically branched?
cgra: i'm suggesting the leaky part is plugged by everybody sending a msg nobody else wouldn't and couldn't
cgra: asciilifeform: i mean, in (leaky) theory, two selfchains are usually distinguishable, even just because the subjects in q wrote different things over time. right?
cgra: otoh, the locking idea would make selfchains reliably distinguishable (assuming proper 'S-turd' entropy). instead, would somekinda chain claiming convention, where you send a standard message with a random string in it, similarly guarantee distinguishable selfchains?
cgra: asciilifeform: selfchain having the locking thing makes me think how much does it make sense if you still gotta trust the choke-point peer(s) between you and some semi-rando (l2). and how big biz is there with a semi-rando anyway
cgra: and branch is reliably detectable
cgra: i'm trying to isolate a branching selfchain as a condemnable situation vs >1 parallel chains
cgra: asciilifeform: no funny business
cgra: asciilifeform: but (1) only good explanation for selfchain fork?
cgra: asciilifeform: you mean (1)?
cgra: (and not counting multiple genesis messages as fork)
cgra: is there any good reason for selfchain fork?
cgra: only now noticed you said 'netchain' there, was still in 'selfchain' thinking mode
cgra: asciilifeform: yea was just about to point out ^
cgra: right
cgra: obviously a rough draft still, for N next revisions, and understandably
cgra: similarly all the two-letter symbols. perhaps helped with added explanation, or symbol length, or increased consistency. for consistency example, the "T" in "Tw" means point in time, while "T" in "Te" means duration. maybe "Dt" or "It" instead (I as in 'interval')?
cgra: side note: upon a handful of first spec reads, i found 'long buffer' and 'short buffer' didn't stick as well as perhaps smth like 'dedup buffer' or 'embargo buffer' would've
cgra: currently afaik only said way out of order buffer is the timeout
cgra: ah, ^ mention prolly missing
cgra: asciilifeform: if you go by the current spec draft letter, doesn't the order buffer necessarily always release the newly synced chain segment backwards, from newest to oldest? obv though, can't help out-of-orderism if sync takes longer than 5min, the order buffer timeout
cgra: right
cgra: asciilifeform: aiming for cool diagrams and all?
cgra: asciilifeform: how's your 'hevea' rewrite getting along?
cgra: imo the long buffer desc could use similar explanation
cgra: ok, makes sense
cgra: asciilifeform: any comment re 'long buffer portion being cached anyway'?
cgra: asciilifeform: in other words, is this correct: 1) netchain forks are desirable, but selfchain forks not. 2) multiple selfchain genesises desirable only if for a good reason (like joint nets). 3) while selfchain forks and excess genesises not desirable, difficult to prevent, so perhaps better to just tolerate
cgra: and not just genesis messages, any retroactive forks like ^
cgra: asciilifeform: if multiple genesis messages are allowed, then prolly otherwise retroactively injected selfchains must be allowed too?
cgra: asciilifeform: any comment re prod and loose chain ends?
cgra: now, if selfchain forks are allowed, prod seems to have a problem: how to sync the possible loose ends? should apply to the joint nets case too
cgra: another, related consideration: doesn't getdata as now specced, allow retroactive selfchain fork? feed new branch backwards, starting from the last of the chain, which meets the 'not-stale' requirement
cgra: asciilifeform: if two pest nets join, they're going to cross-sync their histories and end up having two separate branches. if somebody was part of both nets, but separately, he's now going to have two 'genesis messages', one from each net. correct?
cgra: asciilifeform: in replay scenario, doesn't the 15 min timestamp margin limit the disk access, so that the queried entries are ~always in cache, and no actual disk access caused?
cgra: asciilifeform: another q, what was the 'long buffer' for again? why not just always check the 'very long buffer', (tm)(r) jonsykkel?
cgra: asciilifeform: did you still have this idea that separate pest nets ought to be able to join?
cgra: asciilifeform: by 'plugged shut' i mean, even if all the known trb oom disease was cured, there's still a middle-man who can fake incoming messages to whatever he feels like, such as invalid, ban-inducing blocks, and cause trb to ban each and every traditional peer
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-04-11#1094984 << was thinking of trb with a pest interface in addition to the traditional one. so if you're plugged shut on the traditional front, can still communicate with your friends, assuming have enough friends not all of them being similarly plugged shut at the same time
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-04-09#1094664 << may also find interesting (like i do) a band performing using heavy eastern finnish dialect (savo) - https://www.youtube.com/watch?v=ocHbfpzjDE4 :D
cgra: jonsykkel: fwiw, i was impressed by your welcome =)
cgra: while instead, gain a predictable bound or two
cgra: and perhaps this means that you could just have a relatively small list of recorded peer addresses and/or banned addresses, and not lose much, compared to current mechanisms
cgra: asciilifeform: also been in the process of grasping what it means for trb being completely at the mercy of the middle-man as it currently is. only apparent cure being the wot integration. currently thinking that banning some peers, keeping a list of anothers, is therefore not an extremely important aspect, if a middle-man could fuck you up anytime anyway: like distort all messages from your peers so that they end up all being banned.
cgra: been lately trying to revive a youth grappling hobby, for physical exercise (to balance office-like activity), and personal enjoyment
cgra: anyway, hojo hojo mangol
cgra still tuned in and breathing, while not sure what was meant here. possibly a confusion of someone/somewhere?
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-27#1081208 << close enough to ponder whether finland's next, or what was going on again
cgra: billymg: gives you more choice in picking your favourite sewing needle. and even if pages are stored in the db, prolly still applies. if wanted multi-line code highlight, then you'd likely best off with a minimal mpwp tweak on top of that
cgra: ah and if blog owner puts it there anyway, not his l1, what's the difference if the html generator runs on his desktop?
cgra: i'm not sure if vpatch volume is high enough that it needed server-side automation, while phf's chart certainly is neat
cgra: ah, but i'm already adding links as a last, separate step, so may not apply
cgra: if i had to guess though, long link texts might still throw your concentration annoyingly off from actual writing
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-26#1081000 << atm the hack i have, works reasonably well, so it's now sunk somewhat low in priority to think of a better replacement
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-26#1080994 << i had the same feeling from the earlier usage, but see for example my latest, look for 'storycontent', and it's pretty alright wysiwyg output
cgra: billymg: oh, and one minor detail: personally i would find yearly archive grouping more suitable myself than monthly. and anyone else with 0-2 articles/month :)
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-26#1080978 << as these are static pages, they don't need a server-side generator. perhaps only something for highlight-linking, say, multi-line highlights at minimum.
cgra: an another look sometime
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-26#1080979 << can't say i'm too familiar with the php stack so thought i'd use that name :) but overall, i fell back to the tinymce approach (knitted together a localhost "wysiwyg" editor hack), because couldn't stand hand-cranking that many trb links and code tags as i did. another thing is not sure yet why couldn't i access the admin page through ssh-forwarded port, intending to give it
cgra: asciilifeform: yeah, understandably.
cgra: the item you sampled, was added by hand to the presented set of files, the others are bulk-generated from a vtree dir
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-25#1080721 << it's python 'pygments' for colors, spiced up with my own dog-food to add per-line links. in the url, "p" stands for "patch", and "r" for revision; most of the trb items are there, only if you give the correct url. i'm planning to add index pages, too, so can eventually also browse it.
cgra: ack
cgra: ah ok
cgra: asciilifeform: for sniffing around later, could you mention some specific, slow blocks?
cgra: varies, 200kB/sec seen too
cgra: asciilifeform: re slow peers: 205.134.172.6 is slow for me, some 20-120kB/sec transfer rate. also inv's only one block at a time, when blocks 800kB+
cgra: asciilifeform: ^
cgra: asciilifeform: perhaps cement ought to replace it more explicitly
cgra: asciilifeform: if cement stays, prolly removed?
cgra: yea, other alternative, and clearly worse
cgra: right
cgra: right... though is it any different from noob having to pick a client software?
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-19#1080068 << it's useful/necessary to prevent early-chain spam-alts innit? or why hesitated?
cgra: was kinda interested to see that effect
cgra: asciilifeform: with these first, miniscule blocks, that 50% diff of ours could be due to cement doing getdata-block exchange for every block, where the old sync runs in multi-MB bulks per each ask-inv-getdata-blocks exchange
cgra: asciilifeform: iirc you said ryzen, so perhaps close enuff to my i7-9900k. was only looking for whether any apparent difference between cement-block-by-block and old-sync-in-block-bursts. somewhat related to the q whether any use to have overlap between verification and download of next
cgra: asciilifeform, just gave this a try: first 125232 blocks took 53 minutes to sync with the old non-cement sync (before entirely stopped syncing, stuck now for 4h). how fast were your first 125k blocks?
cgra: i mean, random shots could've all hit on fast nodes, unless otherwise clearly didn't
cgra: asciilifeform: did you hit a slow node, or not sure?
cgra: hmm, i remember as if some trb peer used to be a slow-link one. could try making cement-mode start off with such a peer, and see if makes operator sad or not
cgra: asciilifeform: prolly just gotta take time to see experimentally/think through staring at the code, whether significantly still could lock-in to a slomo-peer
cgra: no dupes prolly cuz always download and verification from first giver succeeded under 2min
cgra: asks, more or less, schedule with 2min intervals
cgra: right
cgra: asciilifeform: what signs of gravitation you've seen? not just the first node it started with, and still going with the same?
cgra: asciilifeform: yeah, true. depends on whether can lock-in to a 100kb/sec node as well a 1gb/sec node, depending on the phase of the moon
cgra: asciilifeform: the only part i was thinking the parallelization for, was that now it alternates between downloading and verifying, these don't overlap other than perhaps worth a 64k tcp buffer
cgra: ("sätös", btw, in finnish, means "a haphazard, poorly designed or built creation")
cgra: asciilifeform: trb's locks are recursive, yeah. sätöshi's been doing nested same-locks too
cgra: asciilifeform: cement segment makes sense for catch-up yeah, not yet come up with a case where gapped beneficial
cgra refrains from further lock comments until has pressed item to look at
cgra: asciilifeform: right (re lock likely unnecessary as cement now is)
cgra: a minor point: for consistent only-public-methods-with-lock, would move cs_cement from currentEntry() to getExpected()
cgra: asciilifeform: why are gapped cements supported in your draft patch?
cgra: can only see branch division
cgra: asciilifeform: how do txes divide into levels?
cgra: example: lv 3: tx hashes A B C D E; lv 2: merkle hashes AB CD EE; lv 1: merkle hashes ABCD EEEE; lv 0, merkle root: ABCDEEEE.
cgra: see CBlock::BuildMerkleTree() here, which returns the merkle root hash
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079659 << asciilifeform prolly agrees that, strictly speaking, block hash is derived from the 80-byte block header, while the 'merkle root' hash is ~embedded~ in the same header. but i don't grasp how would merkleism allow tx reordering without the merkle root changing
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079658 << does 'level-by-level' here also mean 'branch-by-branch'? ie. can validate transactions one by one, as they stream from peer, if provided with enough related higher-level merkle hashes
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079773 << mech hdd with the bdbmay prove unfeasible, while still an interesting experiment
cgra keeps thinking of smth like "spread getdatas of the block range N to peers, follow peer response times and avoid overloading slow peers. buffer populates, verify block every time the lowest block in range N available, adjust buffer afterwards"
cgra: asciilifeform: if can find it for cement, would be enough, cuz 'fast enuff' for normal operation
cgra: can't get rid of the idea that downloading a bounded buffer of N next blocks could somehow work, and without horror
cgra not convinced yet, after all, that shouldn't try formulating some kinda minimalistic pipeline... cement seems to want similar too
cgra: asciilifeform: this is when the slowest node may still be wrestling with smth else, but has your first getdata queued
cgra: asciilifeform: but then another round of getdata when receives the block from the fastest node?
cgra: i mean, a peer won't advertise blocks (in response to getblocks) more at a time than a half of it's sendbuffer can hold
cgra: asciilifeform: normal getdata's are limited by the getblocks response size of the peer, or some other case in mind?
cgra: a fast node responds fast to your getdata chain and allows it going forward, while slow peers' sendbuffers build up at that pace
cgra: asciilifeform: re cement getdata algo, wanna point out that in theory there's also a risk of sendbuffer flooding some trb peer, especially with a lower-than-default sendbuffersize, if peer's upload is slow, and/or peer stuck at the same time doing smth else
cgra: on account of the 'fast enuff' + 'cement soon' points, the continous sync portion on my table (that 'getheaders-sync') prolly best without any kind of parallelization tricks, cuz less code
cgra: right, as i thought
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-17#1079616 << does your cement-mode somehow bypass completely, or in part, the bdb bog?
cgra: whaack: re podcasts in general, personally i find listening inefficient also because for a difficult/unfamiliar subject to properly sink in, i would need multiple re-reads over time. but i ack your typing pains issue
cgra: the slowest verification i could find off-hand from own (somewhat messy) records was 9.2 seconds, block 0000000000000000000a34fb7816747e8d0e09b535e5d07f70f60c690a64c17b.
cgra: i recalled i had some actual measurements recorded, which add up to these figures: in height range 228k-288k, my avg verification time was ~760 ms, in range 289k-317k it was ~1810 ms, and in range 427k-649k it was ~5050 ms.
cgra: facing same (or a variant of) question in my attempt to formulate a 'getheaders-sync' patch
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079517 << possibly 'half' cuz just this, but i'll try anyway...: did you already come up with a clean algo how to parallelize verification of next and download of next+n blocks?
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-16#1079525 << to give a some estimate, if a block verifies in 5 sec on average, 720000*5/60/60/24 = about 42 days to verify the whole chain. the 5 secs may be a gross underestimate for the apu1 box, is merely a hunch based on my sync tests with an 9th gen usg-intel cpu
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-02-01#1077594 << per trb code seems that 'block work' is not per the hash value, but per the block target value. every valid block at the same height always have same weight in this sense. the deciding factor between competing blocks appears to only be the order a particular node received them. and consequently, will only relay the first-received block.
cgra: top bits equal meaning here for example in the extreme case, block 0000000000000000000e95d1cabfbb9dec4469bf0eb17645187acc848aeb1349, whose target value is 0000000000000000000e95de0000000000000000000000000000000000000000
cgra: target]
cgra: asciilifeform: i crunched through hashes and difficulty targets of block heights ~600k..650k and found that ~17% of them had one or more top bits equal. now i'm confused again how is the kingergarten logic peculiar. that many matching top bits suggests to me not every miner aims [http://logs.nosuchlabs.com/log/asciilifeform/2022-01-31#1077390][one top bit lower than the
cgra: i can see a theme here: 'i don't have my own yet, can i borrow yours?' explorer. paste bin.
cgra: signpost: would it work/make sense to you to make pastes linked in pest messages (seen by your station) permanent?
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-31#1077478 << prolly exactly the angle i was looking from (trb study). otoh, has same effect, and gets you two birds with one stone, when you get to publish the explorer sources >> http://logs.nosuchlabs.com/log/asciilifeform/2022-01-30#1077257
cgra were pointed out yet another snot blob hanging in the same, neverending strain :D
cgra: ah so while settle for "up to twice as difficult", gain elsewhere more than that
cgra: asciilifeform: re "quantized by upper bits", do you mean mining works more efficiently if in case of minimum difficulty, you aim at or under 000000007fffffffffffffffffffffffffffffffffffffffffffffffffffffff instead? ie. guaranteed less top bits than the threshold
cgra: though any idea how to connect? i still don't see any peculiarity in my kindergarten approach
cgra: hehe, and i was heading to bed when saw your first comments re this overwork math
cgra: asciilifeform: i think i said exactly so, here at least
cgra 's clutch is slipping. difficulty grasping asciilifeform's thinking
cgra: asciilifeform: re bottom segment not helping, do you mean 00000000ffff0000000000000000000000000000000000000000000000000001 would've been a valid hash under minimum difficulty (note the LSB)?
cgra: asciilifeform: this comparison is the pov i'm looking at this from. you just try a random number and it's either <= the threshold, or above it. if above, keep trying.
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-30#1077292 << for difficulty=1, the "target" (ie. the maximum allowed hash value) = 00000000ffff0000000000000000000000000000000000000000000000000000; so: target / genesis_hash ~= 2536 (dunno why i first rounded up, to 2600)
cgra: for example, the last dig i did was "what's the last block mined with the hard-coded, minimum difficulty the genesis block started off with?"
cgra: whaack: i'm possibly not aware of all implications, but i keep contemplating a 'pesttronic block explorer' accepting read-only sql queries from wot
cgra: asciilifeform: btw, did you ever comment on this one? valid block species now here and here (obv the eating node must've chain height below the builtin checkpoints)
cgra: sometime earlier, i made two experiments to verify some upper limits of blocks. at the time they weren't valid blocks, but i rectified that in the mean time: it took perhaps <1 hour per block to mine, with an 8-core CPU, and a python(!) miner
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-29#1077198 << note that, peculiarly, the genesis hash was nominally ~2600x harder to mine than the genesis difficulty required! for easy check, see next block hash: ~1300x larger integer than the genesis hash.
cgra: signpost: right
cgra: asciilifeform: hehe, perhaps best forgotten
cgra: somekinda eclipse ada thingy supposedly existed, which i so far didn't get around to dig up -- already deprecated, in favor of GPS i think. the thingy might also be a torso as well
cgra: * 'fits this scenario better' --> 'fits this scenario well'
cgra: then there's that, say, the node operator had a knob to decide how long a lead declares the losers/the winner. then the operator found out he had underestimated realistic reorg length and wants to adjust: now a previously declared 'loser' and the old 'winner' are swapped. i kinda feel the term 'resurrected' fits this scenario better, but prolly someone better in english can find a fitting 'sports term' here, too
cgra: dying/dead, 'dying' until declared 'dead'/'extinct', then a 'recoverer'/'recovering chain'?
cgra: coupla examples: losing/loser, 'losing' until declared 'loser'. how about the chain that recovers after initial losing position, a 'runner-up'?
cgra: re the 'orphan' misnomer, drawing the line between 'losing' and 'loser', instead of 'extinct' and 'resurrected', perhaps worth considering.
cgra: signpost or whaack have any idea whether the jetbrains/intellij family also have a decent ada variant? (although now that i checked again, the GPS may be fine after all. forgot why i thought previously otherwise)
cgra: asciilifeform: genuinely a work of '(f)art'
cgra: i believe the spam guard can cause a stuck sync, for example, when all the peers send their new best block inv's to us too early, and now register that block as 'known by us', and won't respond with hash of that block anymore to our later getblocks requests
cgra: asciilifeform: i mean, not ~only~ as an initial sync, does help both
cgra: asciilifeform: i'm not proposing this as an initial sync, but for continuous operation, perhaps primarily because of the 'spam guard'
cgra: asciilifeform: right
cgra: (but had plenty of peers to play with)
cgra: asciilifeform: my test node was always behind a nat...
cgra: (otoh, my head being solid bone not ruled out as a possibility either)
cgra: asciilifeform: i suspect you've forgotten something about the 'inv' mechanism
cgra: afaik not many concrete blocks get thrown around, while plenty inv adverts flying
cgra: asciilifeform: to me an 'unsolicited block' is a concrete block being thrown at me, not an inv advert of one. what's 'unsolicited block' to you?
cgra: isn't any more headers-firstism than the getblocks/inv is -- first asking for hashes, then receiving a bunch of them in response
cgra: asciilifeform: imo this isn't headers-firstism, why do you think so?
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-28#1077040 << they send an "inv" of the latest block (ie. not the block itself), which you needn't "AskFor()" in response, send "getheaders" instead and proceed from there.
cgra: while admittedly headers-first-esque, i suppose some 3-5MB block buffer wouldn't hurt, if a reasonably clean implementation exists, to allow parallel processing and download of blocks
cgra: this "one block at a time" approach could afaik mean ~zero bastards and ~zero blocks out-of-order. the downside is that blocks are processed ~and~ downloaded synchronously. next block is requested only after the previous one is fully processed, not simultaneously
cgra: the unbounded "headers" message is ~160kB in size, but the node asking for blocks could use an occasionally updated stop hash, seems like some 63 blocks ahead of the first in the unbounded headers list would give an optimal, average "headers bandwidth". every time the stop hash is reached, another unbounded headers is requested, and new stop hash is decided.
cgra: while the "headers" message's item count isn't regulated by the originating node, like an "inv" in reply to "getblocks" is (to allow the receiving node to request the whole advertised list of blocks at once), the "headers" recipient could simply ask for one block at a time and repeat "getheaders" on new block reception
cgra: the proposed approach is to, instead of sending "getblocks", send "getheaders". no change in how to respond to other nodes' "getblocks", only change how own node requests new blocks. then, "AskFor()" blocks according to "headers" responses, not "inv" messages. on "inv" messages, can just reply with "getheaders" to get a more reliable "block advert": a "headers" message, and now further "AskFor()"
cgra: getblocks/inv exchange sucks because the inv response may end up distorted at least for: 1) new best block(s) mixing up with it, 2) the inv spam guard, by preventing repeats, will cause gaps, and in the worst case, stuck sync, 3) unlike the "headers" message, it's not an explicitly described hash segment of the block chain, just a list of block hashes, so any mixups and gaps are not obvious
cgra: the message exchanges "getblocks"/"inv" and "getheaders"/"headers" are similar, but the former ends up being pretty clumsy in practice. also, ofc, the latter otoh isn't currently fully implemented
cgra: this is again starting to take time beyond imagination, so i thought i'd make an attempt to loosely describe meanwhile, perhaps asciilifeform grasps what i mean:
cgra: (this re the pile of trb notes looking to partially publish)
cgra: asciilifeform: currently evaluating whether sticking with the reStructuredText pile of mine or try to find a (semi-)custom approach. tables seem like potentially getting annoying with RST
cgra: right
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2021-09-13#1058076 << did asciilifeform already have a clear idea how to implement tables here? imitate html equivalents?
cgra considering publishing similar text re trb, to support further discussion of sync (have had such a draft around getting shape since the first months of study)
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-16#1073393 << document type starting to really look like what the supposed trb documentation also could look like...
cgra: and nicely collides with the opposite meaning
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-11#1072474 << whaack, my "100GB or so" was way off, though might match better some of the other burst defect variants
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-11#1072541 << the hole obey_sendbuffersize plugs, still remains a mystery to me why causes slow death. at one point i thought the above "getheaders" processing slowing down every 4GB was it, but i didn't know about the 4GB-ism of it then. obv not the key point, cuz apu1 would be oom way before accumulating 4GB of extra garbage.
cgra: for interested readers: the above defects require the attacker himself to upload ~1:1 of the garbage being parked into victim's memory, while, for example, a carefully arranged 10e6 byte-burst of "getheaders" messages blows up into ~530GB (10e6 / 1021 * 718305 * 81) of peak memory load (also node locks up until done with the burst, and the processing slows down every 4GB). other "burst defects" are less severe, but similar.
cgra: http://logs.nosuchlabs.com/log/asciilifeform/2022-01-11#1072528 << you're right, i forgot the collection had a handful of these as well. the only, strictly permanent leaking item is "mapAlreadyAskedFor", while "setInventoryKnown" leaks until peer disconnects, "mapAddresses" accumulates for ~two weeks until gc starts to follow, and some others have 2min-24h accumulation times.
cgra: whaack: meant a restart loop that would restart only on crash (asciilifeform suggested sometime earlier)
cgra: whaack: that restart loop at least mitigates the issue
cgra: whaack: yeah possibly. unless nodes have >100GB ram/swap or so
cgra: whaack: i found new steam to chew on trb, so chose that for now
cgra: pete_rizzo_: and the ada gnat mentioned is a separate src package, don't use whatever is found from distros' own repositories, might cause headaches. did asciilifeform have a mirror?
cgra: pete_rizzo_: on a quick glance at the trb build instructions to refresh my memory -- if you find a playground linux with the required versions of gcc, g++ and gnupg as default, it helps. other people may have noob distro suggestions, while i happened to work on ubuntu 16.04 (not endorsing), which defaulted to wrong gcc/g++ (version 5).
cgra: right
cgra: maybe should try myself one time, for kicks
cgra: i vaguely remember talks about cash deposits in finland that some new questionnaire were implemented in recent years. granpa deposits 50€ and now cashier responds with a request to explain origin
cgra slowly recalled using cash for buying a car some 10y ago
| →
Random(asciilifeform) | Download hourly DB snapshot | Get Source Code