Hide Idle (>14 d.) Chans
← 2023-02-12 | 2023-02-14 → ↓
jonsykkel[asciilifeform]: unpx: crtdaydreams: AAAAAAAAAAAAACHHHHHHHHHHTUNG!!!!!! CRITICAL update, fixing unset netchain: http://zzz.st/progz/
jonsykkel[asciilifeform]: i will now, as the only reasonable course of action, commit sudoku in town square as atonement for this mistake
cgra: jonsykkel, will you provide outdoor town square photos of 1) clean sudoku page, 2) same sudoku completed, as proof?
jonsykkel[cgra|asciilifeform]: cgra: ofc, with timestamps and ID visible
cgra: responsibility appreciated
unpx[cgra|asciilifeform]: hm, my station missed a message after upgrade. Will try to debug this.
jonsykkel[cgra|asciilifeform]: unpx: which message in particular?
jonsykkel[cgra]: pls to paste tail of smalpest log
unpx[asciilifeform]: cgra's before yours, but looks oke
cgra: awt, peering info for you if want. i might have some blatta details to ask about, but perhaps not generally interesting points
cgra: http://logs.bitdash.io/pest/2023-02-12#1023114 << address cast somewhat useful here, because my included AT entry above is already outdated
bitbot[cgra]: Logged on 2023-02-12 14:41:07 awt: I spent a lot of time getting it to a functional state and I would really like to see it get used.
cgra: if, as specced in current 0xfa draft (4.2.1.3, 4.2.1.2), timestamps in a chain were never decreasing, would it guarantee a reasonable order of messages if ordered primarily by timestamp and secodarily by chain dependency? or were there known issues with this appro
cgra: ach, too?
cgra: there's that waiting of peer whose clock is running fast, which i'm aware of, but asking specifically about ordering problems
cgra: re peer with fast-running clock: footnote 21 states that you have to wait the time difference before sending your next-in-chain message. any obv downside to just add the difference to your new message's timestamp and send immediately instead?
asciilifeform: cgra: that part of the spec defo needs over9000 moar thought. (adding to your timestamp, for instance -- then it's same as if yer own clock runs fast, with the obv consequences!)
asciilifeform: is also likely to result in msgs being mysteriously discarded , when sumbody else's reply 'outruns' yours (client would need to notice this and reissue ?) -- gnarly
asciilifeform: ( worse yet, discarded but ~not by all stations~ , depending on who outruns whom )
asciilifeform: the 'monotonic timestamp' thing in 0xfa will prolly have to go.
awt[cgra]: cgra
awt[cgra]: cgra: will peer
cgra: asciilifeform, what do you mean by 'outrun'?
cgra: awt, ty!
asciilifeform: cgra: 'outrun' i.e. 2 stations issue replies with same netchain (to same prev msg) and some net participants receive 1 prior to the other
asciilifeform: suppose both have same timestamp (or the 'winner', at particular receiver, has greater one than the 'loser'). nao the 'loser', from that station's pov, is malformed
asciilifeform: this is a catastrophic lul because potentially splits the net, in the sense where there now exists a msg that is valid on some stations but not others
asciilifeform: (and yes, current thought is that malformed msgs get stored anyway, and displayed with special markings, as they may otherwise end up getdata'd infinitely, if they were discarded. but still ugh)
asciilifeform: validity of a msg should not depend on the moar or less random order in which packets may be received.
asciilifeform: 1 possible pill would be to keep the monotonicity requirement, but 'loser' notices that he 'lost' and auto-reissues $msg
asciilifeform: gnarly tho
cgra: asciilifeform, you mean no two netchains may refer to same message? aren't they racing like that independent of fast clocks?
asciilifeform: cgra: per current 0xfa predraft, 'NetChain of a Message may not refer to one with a Timestamp greater than its own'
asciilifeform may be mistaken re the implication, will have to think
asciilifeform: come to think of it, pretty much errything above is fulla shit, so nm
asciilifeform still waking up, lol
cgra: hehe, right
cgra: asciilifeform best caught off guard -- right off bed
cgra: asciilifeform, was thinking that the main consequence of your clock running fast is your messages propagate as getdata only, once too much time difference. if otoh, you wait to catch up the difference, might end up with a slower propagation. and faster propagation --> faster 'whose clock's off here?'
cgra: http://logs.bitdash.io/pest/2023-02-07#1022462 << btw, one consideration: you'd lose message reception times and break the ordering of old messages. old messages have two problems: occasional non-monotonic timestamp progression, and unreliable netchain path (contains zero netchain and netchain=selfchain messages). these to
cgra: gether leaves the reception time the best sorting criteria, especially for high-uptime stations
bitbot[cgra]: Logged on 2023-02-07 13:56:31 asciilifeform[5]: cgra: fwiw asciilifeform has no plans to migrate anyffin other than keys from blatta db ( wainot let station sync from net the proper way ? )
phf: 􏿽http://logs.bitdash.io/pest/2023-02-13#1023172 << netchain/selfchain is an explicit claim by the sender about what he's implicitly replying to. the problem with `who cares about small drift` argument, is that it is primarily semantically an issue in a near-realtime communication, w
phf: 􏿽here small drift will be counter acted by packet delay, but the semantics rely heavily on correct ordering. in other words in cases of `and turns out she was a fish!` `lol!`, that lol really should go after the `joke`
bitbot[cgra]: Logged on 2023-02-13 06:36:22 cgra[jonsykkel]: if, as specced in current 0xfa draft (4.2.1.3, 4.2.1.2), timestamps in a chain were never decreasing, would it guarantee a reasonable order of messages if ordered primarily by timestamp and secodarily by chain depende
cgra: phf, if 'monotonic timestamp', 'lol' sender must've seen the joke already, so will have a >= timestamp. if equal timestamps, netchain decides order
phf: ah, i got the idea, but i don't think it's in spec. the way i read it monotonic timestamp is per-machine, but there's no requirement that timestamp must be >= lastmessage, or am i missing it?
cgra: phf, in 0xfa predraft, i'm looking at sections 4.2.1.2 and 4.2.1.3
phf: ah, interesting, that seems like a new requirement, or maybe i just punted on it
cgra: fun fact, looking at this today, tried to search "Timestamp greater than" from every spec version and didn't find. turned out firefox pdf search tripped on link + italic, or some detail in it, and didn't match
bitbot[cgra|asciilifeform]: Logged on 2022-11-15 10:54:31 jonsykkel: pg.22 "Note that the NetChain of a Message may not refer to one with a Timestamp greater than its own" << this efectively requires cloks to be synced within seconds, doesnt it. how to know if someone else is fast or you is slow
cgra: phf, first time introduced in that doc afaik
phf: well, considering that i've re-read entire knuth to implement the graph solution, and this requirement basically eleminates sort order as a problem...
cgra: right :P
phf: but on other hand i think ascii is smoking crack :> per jonsykkel a packet from a station with (possibly temporarily) broken clock results in entire network doing ???
cgra: phf, was this a pest comment, got log link handy?
asciilifeform: phf: confirmed re crack, it doesn't of course permanently push ~erryone's~ time fwd
asciilifeform: will only 'push' until there's a natural pause
asciilifeform: still potentially headache, if e.g. asciilifeform's clock is 14min 59s 'fast' vs erryone else's, nao any immed. replies have a strong chance of expiring before being received
asciilifeform: http://logs.bitdash.io/pest/2023-02-13#1023208 << not new, iirc that was in the 1st copy of 0xfa 'pre-draft' asciilifeform posted
bitbot[asciilifeform|cgra]: Logged on 2023-02-13 11:20:13 phf[awt|deedbot]: ah, interesting, that seems like a new requirement, or maybe i just punted on it
asciilifeform: (which was why asciilifeform was puzzled re phf's dig into graph walkers)
asciilifeform: may yet turn out to need the graph walker tho, not clear atm to asciilifeform that we don;t in fact
bitbot[cgra]: Logged on 2023-02-13 12:14:46 cgra[jonsykkel|signpost]: phf, was this a pest comment, got log link handy?
bitbot[asciilifeform]: Logged on 2023-02-13 11:20:33 cgra[jonsykkel|signpost]: fun fact, looking at this today, tried to search "Timestamp greater than" from every spec version and didn't find. turned out firefox pdf search tripped on link + italic, or some detail in it, and didn't match
phf: http://logs.bitdash.io/pest/2023-02-13#1023221 << graph walker idea predates the current spec, and i just thought it was "cool" :) which is also how e.g. btcbase has a very very fast grep over logs
bitbot[cgra]: Logged on 2023-02-13 12:22:08 asciilifeform[4]: (which was why asciilifeform was puzzled re phf's dig into graph walkers)
phf: also imho a graph walker solution is appropriate for a lisp implementation, on account of a kind of retro-grandeur that's inherent to all the approaches that those guys took. bit twiddling fighting for each byte c virgin vs my chat program requires a 3 hour GC every night genera giga-chad
phf: i already learned something as a result, in khan if you replace the set of all nodes S with a priority queue, you get the secondary constraint for when graph doesn't enforce one "for free"
phf: which in our case is netchain/selfchain for graph, and timestamp for priority queue
phf: i can theoretically ignore the timestamp entirely, and use arrival order for secondary constraint
awt[cgra]: cgra: address cast seems to have worked using the key you gave me
cgra: apparently!
asciilifeform: graph walker will surely come in handy sumwhere.
asciilifeform: (maybe replace the rubbish one in vtron... )
asciilifeform: re timestamps -- asciilifeform still thinking about this possible algo
bitbot[cgra|asciilifeform]: Logged on 2022-12-25 15:59:26 asciilifeform[jonsykkel|awt|deedbot]: for thread-completeness, anuther solution could be to actually track clock diff. b/w you and $peer, and treat binary msgs as expiring in e.g. 1min from timestamp (adjusted for the diff) rather than 15.
asciilifeform: ( i.e. actually track the clock drifts, rather than always sitting on ~erry~ msg for up to 30min-epsilon )
asciilifeform: ... if a peer is 'warm', yer clock can be thought of as effectively synced
phf: some kind of running average on clock_time vs packet_time_stamp across all peers
asciilifeform: imho averaging wouldn't getcha anyffin but a slow collective drift
asciilifeform: instead track delta per peer. if $peer is 'warm', you now have effectively synced clocks, within bounds of oscillator noise
phf: well, we're already agreeing that packets reporting 15 minute difference from their actual time ain't a big deal. what's a little drift among friends
asciilifeform: lol 14min 59s drift 'no big deal', 15 -- goes to /dev/null and ends up having to be getdata'd (which in turn requires at least 1 to get through, at least a prod, to trigger)
asciilifeform: but what you ~actually~ want is simply to be immune to replays. which can get (for 'warm' peer) by tracking the actual delta, and nao your expir time with respect to that peer can be e.g. 10sec, or whatever multiple of the ping time with that peer yer comfortable with, rather than +/-15min
asciilifeform: ( for msg from previously 'cold' peer, still need sumthing like the old logic , tho, afaik )
asciilifeform: the point of this, if not obv, is simply to unclutter the antireplay buffer.
asciilifeform: ( 'filter', in 0xfa predraft, sec. 5.3.1. )
bitbot[asciilifeform|cgra]: Logged on 2022-12-25 12:39:07 asciilifeform[5]: jonsykkel et al : runnable example of asciilifeform's filter . and somewhat braindead demo coad for same.
phf[asciilifeform|cgra]: ah yeah i just ignore this kind of implementation policy details :}
asciilifeform also 'i'ma ignore the boring parts', is how we ended up with the lulnurseries etc. headaches
asciilifeform: at some pt gotta unignore'em, to get usable proggy
phf: well, no lulnurseries are a a result of somebody explicitly not ignoring your prescriptions!
asciilifeform: other side of same medal, neh
asciilifeform: i.e. asciilifeform not properly worked out implications of proposed algo, result is a barfalicious rube goldbergism
phf: well, it's more like i have a live system that's being constructed as it's being used, so i don't have to do it athena style. i can e.g. observe real effects and adjust things as needed
phf: right now the entire history of pest, pushed into a hash-table, returns a result in under ms
asciilifeform: in general imho this is sound, but in actual practice potentially painful, so makes sense to at least a little think-before-doing
phf: in fact doing a vector store, so one array for hash, one array for message, where indexes are the same, will give a naive hash query of walking the hash array until you find the right one in under ms even for the whole history of republican logs
phf: and if you run out of performance space there you do what them industrial routers do and just bloom filter the hell out of it
asciilifeform: no doubt thing small enuff that almost fits in l1 cache, for nao
asciilifeform: orig. thrd was re wat-do when gb's of warez moving b/w peers, tho
asciilifeform: then -- no longer '1ms'
asciilifeform: tracking clock deltas would letcha keep the antireplay guarantee while escaping sudden need for buying shoebox fulla ram stix
phf: but we don't have implementation of gb's of warez
asciilifeform: (recall that filter , per current predraft spec, stores hashes of ~all~, not only chained, msgs )
phf: right now the most we know about it, is that it's probably going to be luby and there's going to be a special inline message type to announce it
asciilifeform: phf: indeed we haven't yet. but will, and the moving parts gotta work correctly under warez conditions
asciilifeform: key there is that these msgs will still end up hashed and hashes kept in filter (otherwise yer open to replay)
asciilifeform: unless (as someone -- phf? -- proposed?) they can be discarded quickly via a different heuristic in the case of replay
asciilifeform: fwiw asciilifeform expects that once warez starts moving regularly, most replay attemps featuring randomly stolen traffic will logically involve bin msgs
asciilifeform: (given as they'll constitute the bulk of pest traffic)
asciilifeform: atm, noshit, 'uncatchable joe', and we can't reply on 'naturally occurring' jokers to test the antireplay logic, will need to do it deliberately
asciilifeform: *rely on
asciilifeform: ( and for that matter, most of the possible 'misbehaviour space' of the protocol remains unexplored )
phf: but wait if you want timestamp to be anti-replay, than it has at be > rather than >=
asciilifeform: ( and see also. this is 1 place where arguably gotta get it right ab initio, rather than iteratively )
bitbot[asciilifeform|cgra]: Logged on 2023-01-29 12:41:45 asciilifeform[4]: whereas if, hypothetically, next yr there is a pestnet with 400 people, and someone discovers 'magick packet' that floods it to the point of unusability, even after 'fix', 395 of'em will go back to 'telegram' or whatever and that'll be it.
asciilifeform: phf: atm we have in 4.2.1.1 'Any Message bearing a Timestamp more than 15 minutes away, in either direction, from the Time at a receiver’s station at the time of its receipt, is deemed stale...'
asciilifeform: so already > . asciilifeform's examples upstack were malformed.
phf: http://logs.bitdash.io/pest/2023-02-13#1023247 << i guess the thinking is that further restricting time buffer will let you store less data in filter. so instead of it being a +-15m it's essentially +15m
bitbot[cgra]: Logged on 2023-02-13 13:31:20 asciilifeform[jonsykkel|awt|deedbot]: the point of this, if not obv, is simply to unclutter the antireplay buffer.
asciilifeform: +/- 1min, potentially, for most msgs
asciilifeform: or even tighter
phf: but that's at the expensive of either broken timestamp, or strong awareness of the counterparty's clock
asciilifeform: would only need to track the delta vs. each warm peer
phf: well, no you have to track delta which is a probabilistic value, that's split between time drift and packet delay
asciilifeform: phf: idea is to reduce the effective window to span any plausible packet delay
asciilifeform: (factoring out the long-term drift when possible)
phf: i got the idea, i'm just clarifying that it's not "just timedelta" it's a time delta + packet delay
asciilifeform: for a msg coming from 'cold' peer, naturally will need the full window
phf: how does hearsay factor into this?
phf: l1 station will have to update hearsay timestamp, or else you're also tracking hypothetical drifts with hypothetical l>1
asciilifeform: in asciilifeform's notion, would retain the full window for hearsays. which aint much of a problem , as bin msgs cannot be hearsayed
asciilifeform: the reduced window contemplated for bin msgs strictly
asciilifeform: (they're what threatens to bloat filter to multi-GB)
phf: i think i'm going back to considering luby packets as a totally special thing, that will be explored once it's out
phf: the crack fumes from pg county might be drifting too close to your house!
asciilifeform: ~all we know about'em so far is that they'll be in the bin msg class, which suffices to discuss q of timestamps
asciilifeform: but no particular reason to hurry
phf: there are other potentially unpleasant replay packets, like addresscast
phf: or prod
asciilifeform: fwiw asciilifeform not convinced that this, rather than phf's earlier proposed algo ('find fast way to detect replayed luby that not relies on timestamp') is the correct pill
asciilifeform: remains to be seen.
asciilifeform: phf: they'd get the full window (and not eat over9000 ram)
asciilifeform: but in general asciilifeform would prefer the algo with fewest moving parts, ideally would treat all msg timestamps rather than exempting some types from staleness entirely
phf: see i want a mechanism that doesn't involved message timestamps at all :)
asciilifeform currently thinking 'narrow window where obviously possible'
asciilifeform: phf: if you devised one, plox to post
phf: in fact it's unlikely that i'll ever implement the 15 minute window, i'll just make my hash lookup really really really fast
asciilifeform: if you've fast enuff ssd, could even work at line rate, conceivably
asciilifeform: (until it burns a hole, lol)
asciilifeform: thing is, only chained msgs, per spec, actually get to live on disk
asciilifeform: errything else only lives in filter (until expires)
asciilifeform: what asciilifeform thought , is that phf devised a way to enforce 'no message is valid if seen a second time' dictum ~without timestamps~ entirely
asciilifeform: that'd be over9000 nifty.
asciilifeform: (timestamp is in the validity criteria strictly to limit the mass of hashes that have to be stuffed into filter, if not obv)
asciilifeform: if erryone had 'infinite' ram (and somehow with o(1) -- parallel? -- lookup) -- would not need any such thing..
asciilifeform: lol wtf is happening to the window in that screenshit ??
phf: special effects!
asciilifeform not seen the pictured spam , just yet
asciilifeform: phf: bloom is potentially exactly what's needed. ( asciilifeform not up to date on subj, will refresh )
phf: i think bloom is inside out, can say "possibly not in set" and "definitely not in set"
asciilifeform associates bloom filter with a braindamaged implementation in ye olde prb, but this aint bloom's fault, seems to be precisely what oughta have in a fast pestron
phf: but yeah, bloom is what they put in routers and such for when need to guard bajilion something with "should i even check"
signpost[cgra]: god, this twee-ass shit. bunny planet.
phf: signpost, it's really really bad
asciilifeform: watsa 'twee' ?
asciilifeform can guess
signpost[cgra]: "I'm just a widdle harmless code monkey; isn't Engineering Fun?"
phf: asciilifeform, i recon originally reference to a stereotypical disney bird, mocked in "who framed roger rabbit"
phf: it's a tiny adorable over the top cartoon bird that goes "twee twee twee". and it has all the most adorable manerisms, and it's so sacharine that you want to shoot the damn creature
asciilifeform: a, yes, that one
signpost[cgra] has a brief flashback to hatespeech training in same tone
asciilifeform recalls 'terrorism training'. 'a grenade falls through yer office window. what position should you take' etc
asciilifeform: errybody in salt mine luvved these, they were a break from the monotony of the actual salt mining
asciilifeform: prolly is half of what drives the various idjit 'training quizes' etc
signpost[cgra]: sounds way more fun. anyway I wont derail you gents further.
asciilifeform: ( answer was 'lie down when machine gun firing through window, but crouch if grenade', theoretically frags form a 'hourglass' pattern. but lulzy if q is posed re 4 m^2 'office', 'fish in barrel' lol )
asciilifeform prefers the ww1 answer : 'what do you do if a shell lands in yer trench?' 'jump up 20m and scatter yourself around'
phf: signpost, yeah i but asciilifeform's mines are a lot more grownup than ours, he can't fully appreciate just how disgusting this bullshit is
asciilifeform: this was possibly the most abjectly 'dilbertian' mine asciilifeform served in
asciilifeform: after that, escaped, to 'house arrest'
phf: well, i finished one task and now it wants me to log in into my official google account
asciilifeform: since then, the only lultraining encountered was, one time, 'prevention of money launderers'. was even moar hilarious than the 'terrorism'
asciilifeform: phf: you may already be a winner!111(tm)(r)(c)
asciilifeform: googlewagen dispatched.
phf: apparently it's a wellknown thing, i left feedback along the lines of "i ain't loging in and you suck". now i can tell people that i almost was a google developer.
asciilifeform: phf: asciilifeform recalls similarly lulzy 'employment olympiads' in 2010s. iirc there was even 1 conducted by nsa.
asciilifeform: some of'em are somewhere in the old #t log.
asciilifeform at one pt found himself participating in one, 'reverse this 13337 fresh chinese trojan', while doing it realized that the perp co was likely farming out its workload to applicants, phree labour, lol
asciilifeform: in theory is possible to run entire biz that way, and someone, somewhere is doing it
asciilifeform: 'they insist on working for phree, wainot let'em'
asciilifeform: ( the 'prize', ftr, is invariably an abject 'entry' jerb, the kind suitable for 19yo )
← 2023-02-12 | 2023-02-14 → ↑
Random(pest) | Download hourly DB snapshot | Get Source Code