cgra: asciilifeform, why isn't version and command included in the pest message hash? why would: less room for funnybiz
cgra: there's at least one instance: somebody could send a direct "hello" first to victim, then broadcast same text with same timestamp. now victim's broadcast reception is stuck at uniqueness constraint fault. i suppose you, alternatively, could handle the case by separating direct and broadcast logs
cgra: "hello" as in first ever message, and could even in theory happen accidentally if done rapidly on same second
asciilifeform: cgra: the random padding in message text makes this scenario ~impossible
asciilifeform: (in message text, in unused bytes of speaker field, and, obviously, the chain hashes)
asciilifeform: ... a valid direct msg can't have same chain hashes as any broadcast msg, thinkabout it
asciilifeform: hash covers from timestamp to the end of the packet.
asciilifeform: (fwiw current pestrons will accept a msg with arbitrary chain hashes, however)
cgra: the first direct has zero chain hashes, like the first broadcast
cgra: and i suppose a malefactor can pad with zeroes
cgra: ahh, broadcast doesn't have zero net chain, if not fiddled. only first msg of the net...
asciilifeform: what would that do, if victim doesnt pad with zeroes
cgra: asciilifeform is it possible that broadcast zero net-chain will be(come) spec-acceptable in the future? besides for a couple of first-of-net messages
asciilifeform: current spec is silent re subj. but imho oughta reject a broadcast msg with zero netchain unless yer db is empty. a new speaker on net oughta receive at least 1 msg before speaking. (and prod makes it simple to getdata for 1 even if no one has spoken in a while)
cgra: right, so broadcast and direct cannot collide then
cgra: another case: not sure how much funny to be had, but a peer could respond to a getdata with false commands. extra code checks if nothing else
cgra: re version, maybe some trick between incompatible message versions becomes possible, if version fiddled. cannot picture a concrete example though.
cgra: while at it (compatiblity-breaking (borderline-?)nitpicks): why specify 'speaker' field in messages, if not going to use it: ignore, getdata, prod, keyoffer, keyslice?
cgra: and another: why follow text messages' format in some other message types even if not going to use chain hash and speaker fields?
asciilifeform: http://logs.bitdash.io/pest/2023-01-16#1020487 << imho obv. but perhaps oughta be included in spec -- a getdata reply oughta go 'where was expected' and never elsewhere
bitbot[asciilifeform]: Logged on 2023-01-16 08:29:50 cgra[deedbot]: another case: not sure how much funny to be had, but a peer could respond to a getdata with false commands. extra code checks if nothing else
asciilifeform: (regardless of cmd code)
asciilifeform: http://logs.bitdash.io/pest/2023-01-16#1020488 << in principle, an old pestron (e.g. 1 that not knows about multipart) can getdata for an incompat. msg and choke on it. there's little to be done about this imho
bitbot[asciilifeform]: Logged on 2023-01-16 08:32:37 cgra[jonsykkel|signpost]: re version, maybe some trick between incompatible message versions becomes possible, if version fiddled. cannot picture a concrete example though.
asciilifeform: (does say 'prototype' on the box, neh)
asciilifeform: http://logs.bitdash.io/pest/2023-01-16#1020489 + http://logs.bitdash.io/pest/2023-01-16#1020490 << compatibility artifact. (originally was gonna have'em in all msgs, to simplify parsing. in new spec, are absent only in bin msgs)
bitbot[asciilifeform]: Logged on 2023-01-16 08:35:41 cgra[jonsykkel]: while at it (compatiblity-breaking (borderline-?)nitpicks): why specify 'speaker' field in messages, if not going to use it: ignore, getdata, prod, keyoffer, keyslice?
bitbot[asciilifeform]: Logged on 2023-01-16 08:36:57 cgra[jonsykkel|signpost]: and another: why follow text messages' format in some other message types even if not going to use chain hash and speaker fields?
asciilifeform in general proponent of mandatorily-present, fixed-length fields -- eliminates broad classes of exploitable bugs
asciilifeform: http://logs.bitdash.io/pest/2023-01-16#1020494 << apropos, anuther item that oughta be included in spec -- 'broken heart' to represent a getdata'd msg that for 1 reason or anuther invalid and not displayed. (iirc discussed previously)
bitbot[asciilifeform]: Logged on 2023-01-16 09:44:07 asciilifeform[5]: http://logs.bitdash.io/pest/2023-01-16#1020488 << in principle, an old pestron (e.g. 1 that not knows about multipart) can getdata for an incompat. msg and choke on it. there's little to be done about this imho
asciilifeform: (i.e. avoid infinite getdata attempt)
phf: i think your thinking about getdata is linear. there's technically no guarantee that anybody is going to respond to your getdata request, at any guaranteed time
asciilifeform: ^ entirely correct
phf: so the process should be upfront assumed to be fully desynchronized: you're going to have "holes" in your log, potentially indefinitely, and you're going to make getdata requests, until you personaly decide to give up, and they might come back at any random time in the future
asciilifeform: ( among other reasons, a msg could even contain garbage where the chain hashes oughta be, and so no one will ever supply a msg with that hash , no matter what )
asciilifeform: ^ for this reason, getdata attempts gotta be finite (or will clutter the list of hashes exempt from timestamp check 4evah )
asciilifeform: (so no, not 'any' time in future)
asciilifeform: otherwise yer open to memory exhaustion attack
phf: well, any time in the future, even after you already gave up, or before you did
asciilifeform: if a getdata response shows up after you've already removed the hash from getdata list (i.e. exempt from timestamp) -- tough cookies
asciilifeform: it gets thrown
phf: well, it's tough cookies ~for you~ so obv gotta weigh your options
asciilifeform: ( how long to keep'em around -- up to operator )
asciilifeform: aaha
asciilifeform: atm spec not mentions broken hearts, or the need to represent holes in log (permanent or otherwise) -- but oughta
asciilifeform recalls early pest thrd with a briefly returned oldtimer (ben_vulpes?) where the latter argued that spec is a bloat warcrime because mentions not only protocol, as in classical rfcs, but tries to be 'complete bakeable recipe'
asciilifeform: ^ imho not avoidable -- a p2p protocol includes good measure of 'social expectation' of station behaviour, on top of packet format per se
phf: i am frankly of similar opinion, ought to separate "recommendations for author" from "spec". e.g. details of various nursery buffers, or even things like "mandatory" timeouts
phf: because the irc nursery behavior turned out to be entirely broken as soon as we started test running it, even though the base protocol can be made to work
phf: you like to defend even your bad decisions like they are strokes of genuis, but if original pest wasn't written perfect like athena from the brow of her father, then awt wouldn't be saddled with a lot of "legacy" code, that clearly didn't pan out
asciilifeform: phf: there's no doubt bad decisions, and would rather find out what they are nao, than in hypothetical 'later' where >dozen folx tuned in
asciilifeform: fwiw in current draft spec no 'mandatory' constants, but only 'recommended'.
phf: like e.g. i'd still like to see a poc packet management code, that can put or reject any arbitrary packet into the backlog
phf: which to me appears like in the worst case it's actually a topological sort on a graph, that can arbitrarily reorder log order
asciilifeform: phf: plox to expand re above
asciilifeform entirely on board re 'nursery buffers were idjit idea', threw'em entirely out in 0xfa, were forced mistake on acct of ircism
phf: well, the order of packets in selfchain are fixed, we know as much. but the order of packets in netchain are subjective to each speaker
asciilifeform: entirely correct (and, on top of this, notionally in gui client can set arbitrary netchain, to make 'reply')
asciilifeform in his attempt at pestron, aims to display msgs in timestamp order, but with clickable arrows to self/net chain predecessors
phf: and the construction of netchain is the artifact of "what's the last packet i got"
asciilifeform: by default -- exactly that
asciilifeform: ( and on a net even a little larger than the current, there's not even any notion that erryone will see the same broadcasts; so inescapable )
phf: so the points at which multiple parallel subjective netchains link up are fully driven by random packet loss behavior
phf: loss/delay
asciilifeform: that, and bounce decay (on larger net)
phf: now under normal circumstances, this will be near realtime
phf: but lets say one station is on a unreliable connection. it's lost subchain (by either fully arriving, or having to be reconstructed with getdata) will have to be inserted into past log, by essentially random injection. either as a large, expanding chunk "there's 10 messages missing five screen pages ago"
phf: or smaller subchunks "there was 2 messages missing here" "there was 5 messages missing here", depending on how many messages got through to the unreliable station in the process
asciilifeform: right; the station, having plugged itself back into the wall, emits a msg with selfchain pointing to the latest msg sent 'into unplugged cable'; peers getdata for it; and so on
asciilifeform: aha, is possible that some got through, rather than 1 continuous interval of 'unplugged'
phf: because the unreliable station might be getting some packets, but not having any packets come out (was happening to me a couple of times, because internet)
asciilifeform: right
phf: but then if it happens to multiple stations, you can have all kinds of chaotic behaviors
phf: for example you can have two parallel chains A and B (two stations "talking to themselves"), but the chains actually got observed by a third station which was also talking to itself C. because there's nothing connecting A and B you will place them arbitrarily, depending on who you
phf: heard from first, but when C arrives it might say that your order of placement was actually incorrect
asciilifeform: fwiw asciilifeform suspects that 'erryone agrees on canonical order' is impossible.
phf: that is true, but the point is more that any moment anyone can chime in with "this there arbitrary order is actually fixed, because i saw it happen", at which point you have to rearrange your own log
asciilifeform: why wouldja 'have to' rearrange own log ?
phf: well, if you're trying to retain the guarantee that packets are "sorted" in the dependency of netchain
asciilifeform: likely the best that can be done is to stow 'obvious' (e.g. fills in existing brokenheart, with 1 packet or linear chain thereof) getdata replies in the obvious place, whereas all else is displayed in chrono order (with clickable arrows to nominal predecessors)
asciilifeform: but yes, stations with poor connectivity will often end up emitting chains of msgs which appear as clusters of 'replies' where was not intended
asciilifeform: nfi what a hypothetical pill against this would look like tho.
phf: see, i thought that some kind of heuristic insert is basically a requirement. probably you're correct and in a strongly connected pest network these issues won't manifest
phf: a public log on account of being batch can always just reorder the whole graph arbitrarily no matter how much it costs
phf: but in a realtime setting with a homegenous network, i think some amount of cleverness as to where you insert the packet on arrival will make the conversation smoother. but maybe that's not even a reasonable approach, because we want time "new" messages to always be at the bottom, unless they are getdata
asciilifeform: phf: ^ this was asciilifeform's notion -- 'always at bottom if new', and ui lets you walk the netchain; rather than 'threaded display' a la heathen chats (e.g 'slack') where indeed can sit & scratch yer head re 'in what order to display'
phf: http://logs.bitdash.io/pest/2023-01-16#1020554 << well, at the baseline, if you make a graph of netchain/selfchain dependencies and then flatten it with topological sort, whatever subset of packets you have it will give you a correct global order
bitbot[asciilifeform]: Logged on 2023-01-16 10:49:42 asciilifeform[jonsykkel|awt|deedbot]: nfi what a hypothetical pill against this would look like tho.
asciilifeform: the q of 'how to display recently filled holes' will need to be answered ('unread' marks on msgs?) but needs empirical massage to see what actually is most edible for people
phf: so in the most degenerate casey you could maintain a graph of your packets, new packets do a graph insert, and your render does a topological sort with a full rerender on each new packet :>
phf: that'll give you an absolutely correct "answer follows question" ordering
asciilifeform considered this, tho with a 'window' of e.g. 'last month' to keep cost in reasonable bound
asciilifeform not has any notion that he currently knows 'final solution to how to best display pest log', expects to try a # of approaches
phf: and on the other end of this design space you have "insert all packets to end as they arrive, getdata or not"
asciilifeform: for that matter, atm still fighting the gui lib rather than even touched subj
bitbot[asciilifeform]: Logged on 2023-01-09 23:04:17 asciilifeform[jonsykkel|deedbot|awt]: meanwhile, lulzily, found that the wx widget most suited to displaying log lines is atrociously slow on gtk/linux (blazing fast on crapple, for all the good that does asciilifeform) so will have
asciilifeform: phf: on disk, they go precisely 'as they arrive'
asciilifeform: anyffin else promises to be nightmare
asciilifeform threw in the towel for nao re custom db, and baking in, for the time being, sqlite, as awt did
asciilifeform: so long as you can pull out arbitrary msg by hash in o(~log n), can live
phf: right you're piggybacking on index b-tree
asciilifeform: at some pt, will likely end up with sumthing like this for msg store
asciilifeform: but is even possible that storing parsed (i.e. with indices of antecedents, rather than hashes) representation a la nqb is over9000 faster (and can still pull out original msg by index)
asciilifeform: ( i.e. if you know that given hash H points to msg # N on disk, store N instead of H, and store H 'next to' msg (or in brokenheart, if not have the msg) )
asciilifeform: ( to rephrase -- no doubt that faster, but may be fast enuff that no particular reason to use hand-sewn db rather than sqlite )
cgra wouldn't at this point consider other than easy-to-change db (ie. sqlite), given lack of his vision of how the db is really going to end up when 'done'
cgra: can asciilifeform give any other reason to not hash version and command with msg, other than cuz already ~set in stone?
cgra: i suppose, if non-zero net-chains were enforced (unlike now), you could distinguish direct and broadcast from each other by the net-chain value and wouldn't have to 100% trust the command value.
asciilifeform: cgra: can't think of any convincing reason , other than where it'd be a 1st class bitch to retrofit
cgra: does asciilifeform see possible somekinda 'reboot' where accumulated and identified, saddening wrinkles in spec get straightened? i keep thinking of this being still way experimental, and just a dozen or so d00ds 'just chatting', while seeing this command kind of nuisances somewhat pile up
cgra: in my limited imagination, how'd such a reboot possibly look like: new log era declared, newpest net forms, oldpest net is abandoned over time... or both nets go on, and eventually names are being called.
phf[asciilifeform]: but why?
cgra: just to get a cleaner spec
phf: why would it be cleaner?
cgra: phf, well for instance, that command field. could actually use it in full
cgra: and perhaps the field arrangement would be easier to remember in some cases. a group of binary messages now have a speaker field which isn't supposed to be used
cgra: for example, handful of other similar things
phf: yeah, but you have to make a case for why the change makes it are cleaner, and so far, you're arguing from the opposite
cgra could overestimate the eventual severity, though. not really saying 'need reboot now'
cgra: phf, for example getdata message has a speaker field, which is documented as 'do not use'. why isn't it cleaner if this field didn't exist in getdata message?
cgra: i'm looking at the pre-draft spec, above example in section 5.4.4.2
asciilifeform: cgra: thinking about it, really oughta revise that (and similar) subsects -- old pestrons expect the obv value for speaker (given as they dunhave the 'filter' and ~will~ think that a msg sent to $peer and somehow mirrored back is 'own' )
asciilifeform: ( peer->self and self->peer aint mechanically distinguishable w/out either speaker field or filter (where a mirrored msg will be zapped immed.) )
asciilifeform: http://logs.bitdash.io/pest/2023-01-16#1020582 << can picture changes that could justify the headache of 'reboot' but cosmetic fiddles imho aint in that category
bitbot[cgra]: Logged on 2023-01-16 16:28:35 cgra[jonsykkel|signpost]: does asciilifeform see possible somekinda 'reboot' where accumulated and identified, saddening wrinkles in spec get straightened? i keep thinking of this being still way experimental, and just a dozen or so d00ds 'just chatting', while seeing this command kind of nuis
bitbot[asciilifeform]: (asciilifeform) 2022-01-26 asciilifeform: thimbronion: current notion, summarized: we bite off 64byte from payload, and get 2 new fields, call'em 'unlock' and 'lock'. lock == h256(errything else in msg, incl. 'unlock', and unrevealed 32byte turd 'S'.); unlock = 'S' from yer previous msg.
asciilifeform: http://logs.bitdash.io/pest/2023-01-16#1020592 << from asciilifeform's pov, protocol is in fact cleaner with fixed-length, fixed-position errything (and in fact was reluctant to propose 'binary' msg types where the usual fields entirely missing)
bitbot[cgra]: Logged on 2023-01-16 16:46:54 cgra[jonsykkel|signpost]: phf, for example getdata message has a speaker field, which is documented as 'do not use'. why isn't it cleaner if this field didn't exist in getdata message?
bitbot[asciilifeform]: Logged on 2023-01-16 09:46:06 asciilifeform[jonsykkel|deedbot|awt]: in general proponent of mandatorily-present, fixed-length fields -- eliminates broad classes of exploitable bugs
asciilifeform: the protocol is in fact ~less~ complex with fixed fields (that may be meaningful or not in given case, but their presence prohibits 'frame shifting')
cgra: asciilifeform, right. and if binary messages didn't exist, no such dummy fields didn't exist either.
asciilifeform: ^ could be debated, but imho not justifies 'reboot'
asciilifeform: bad enuff that the multipart msg format breaks back-compat.
cgra: asciilifeform, frame shifting atm hebrew to me. how to grasp?
asciilifeform: term from geneticists
asciilifeform: i suppose the nearest commonly-used electronic equiv. is the 'nop sled'
asciilifeform: i.e. eater eats from unexpected place, with exploitable effect
asciilifeform must brb
cgra: does 'frame shifting' explain why getdata format "timestamp:8,noise:64,speaker:32,wanthash:32:noise:292" is better than "timestamp:8,wanthash:32,noise:388" would be?
cgra to be clear, not saying either that this alone would warrant a reboot. now just trying to grasp apparent fundamentals he wasn't aware of
cgra: http://logs.bitdash.io/pest/2023-01-16#1020590 << phf, in arguing that hashing here (pieces of preamble) and there (all message fields) was cleaner than just hashing there, i conflated "cleaner spec" and "cleaner impl". cleaner impl in the sense that can trust command of a msg being of same origin the msg contents are.
bitbot[cgra]: Logged on 2023-01-16 16:45:09 phf[deedbot|awt]: yeah, but you have to make a case for why the change makes it are cleaner, and so far, you're arguing from the opposite
phf: http://logs.bitdash.io/pest/2023-01-16#1020613 << something came up, so i had to run away, apologies; but i'd like to address this thread extensively, so i'll have to come back to it tomorrow
bitbot[asciilifeform]: Logged on 2023-01-16 17:49:05 cgra[jonsykkel]: http://logs.bitdash.io/pest/2023-01-16#1020590 << phf, in arguing that hashing here (pieces of preamble) and there (all message fields) was cleaner than just hashing there, i conflated "cleaner spec" and "cleaner impl". cleaner impl in the sense that can trust command of a msg
phf: (as in irl intervened so fast i didn't even have time to "afk", just as i was in the middle of writing things)
asciilifeform: cgra: there's anuther (entirely aside from above) 'seekrit' explanation -- asciilifeform was attempting to design pest proto for, among other things, clean fpgaization
asciilifeform: ... where a field that is or isn't used is simply set of 'wires', whereas if you allow a 'thing to be in different places' you gotta mux
asciilifeform: throwing speaker/nethash/chainhash outta binary packets was a concession to bw conservation (these, presumably, nao would need separate logic path on fpga...)
asciilifeform: call it megalomania if want, but asciilifeform would like to have at some pt nics, routers, radios, etc. that natively speak pest 'at line rate' with 0 cpu utilization.
asciilifeform: http://logs.bitdash.io/pest/2023-01-16#1020613 << recall that cmd and version are covered by the seal and thereby just as authenticable to $peer as the rest of packet
bitbot[asciilifeform]: Logged on 2023-01-16 17:49:05 cgra[jonsykkel]: http://logs.bitdash.io/pest/2023-01-16#1020590 << phf, in arguing that hashing here (pieces of preamble) and there (all message fields) was cleaner than just hashing there, i conflated "cleaner spec" and "cleaner impl". cleaner impl in the sense that can trust command of a msg
asciilifeform: the only thing they aint covered by is msg hash (and still not obvious to asciilifeform what'd be gained by doing so)
asciilifeform: ... is why left'em out of msg hash to begin with (... not because drunk)
asciilifeform: version in particular not belongs in msg hash any more than bounce does. ( is intented to be version of the pestron sending the msg in all cases. rather than descriptive of the particular msg per se )
asciilifeform: i.e. it will be of current pestron, even when relaying an old msg originally received and logged to disk by an older one
asciilifeform: and this is proper, because it is there to tell counterparty what kind of msgs the sender knows how to eat.
asciilifeform: ( and ditto when relaying a broadcast received from peer running older version of protocol )
asciilifeform: hence, gotta be mutateable.