Show Idle (>14 d.) Chans
← 2022-12-28 | 2022-12-30 → ↓
jonsykkel[asciilifeform]: correct solution is obvious
jonsykkel[asciilifeform]: zero lolcats in chain, only hashes
jonsykkel[asciilifeform]: if none of ur peers wants to store lolcat on drive, u are fucked
jonsykkel[asciilifeform]: nice and simple
jonsykkel[asciilifeform]: then clients would have setting like "download all <3mb cats auto", and "store all <3mb cats forever" if can spare the disk
jonsykkel[asciilifeform]: puting them inline in chain seems like annoying way to force evryone to store 5000 lolcats or nothing. then when disk full u gota start deleteing evrything from old end rather than just the cats
jonsykkel[asciilifeform]: which seems sily given avg cat is 3000x the mass of avg text message
jonsykkel[asciilifeform]: if do the operator decides if inline or not, inevitably sombody will "i think this here 2000x2000 jpg reencoded as png is integral to disscusion" then u are stuck forever with ball of shit weighing same as 3years of text log
asciilifeform: jonsykkel: this was moar or less asciilifeform's argument. atm it may seem like 'wainot jpg in chain' because net is small, errybody has luxurious net pipes, disk/cpu to spare, etc.
asciilifeform: but all-to-all flooding of multi-MB pile (and demand to walk through it when standing up new station or resyncing a downed one) is a solid ugh imho.
jonsykkel[asciilifeform]: massive ugh indeed
asciilifeform: ideally, we'll have a reasonable direct-share mechanism for non-humanreadable payloads, and get ~same effect as 'in chain' for things people actually want to keep around.
bitbot[asciilifeform]: Logged on 2022-12-28 19:09:50 asciilifeform[jonsykkel|awt|deedbot]: per asciilifeform's lights, the way to get the tar itself oughta be to ask one's l1 for the hash, as broadcast; someone will have it, or at least have permitted his station to receive it from a peer who ... recurses
asciilifeform will introduce a direct packet type, 'getbinwithhash'
asciilifeform: once we have with-what.
jonsykkel[asciilifeform]: puts responsibility of deciding which blobs to store on the net rather than the guy pushing potentially large # of cats
jonsykkel[asciilifeform]: guess one could make arg that someone can push megabytes of text in same way, but will not happen by accident in that case
asciilifeform could even see having coupla-kB bins in chain, for thumbnail pics; these wouldn't bloat much moar than fat multipart texts. but not even convinced atm that this is needed
phf[asciilifeform]: that solution has already been proposed, at the beginning of original discussion. the exploration was whether or not the other ~additional~ behavior can work
asciilifeform: phf: aha
jonsykkel[asciilifeform]: sure, just giving 2c on problems of additional behavior
asciilifeform: 1 possible pill that hasn't yet been mentioned is a separate chain for broadcast bins (with separate db etc)
asciilifeform: ( still suffers from the 'if not errybody propagates'em, availability will suck' problem tho )
jonsykkel[asciilifeform]: wat would be advantage of that in comparsion to hash links in main chain?
asciilifeform: jonsykkel: notmuch afaik. mentioned for thread-completeness.
signpost[asciilifeform]: jonsykkel: at least worth considering why pest protocol can't communicate the whole pest spec (which includes illustrations) as something represented in the chain.
signpost[asciilifeform] finds this bandying of "obvious" lazy on the part of all.
signpost[asciilifeform]: "I am a narcissist and wish you to submit to my bare opinion" is the entire content.
signpost[asciilifeform]: good, we can all do that one again when we conquer earth
signpost[asciilifeform]: at any rate, seems like most (all?) agree that binwads of some size need to be skippable, but there's disagreement on whether that line is at size >0 or somewhere higher
jonsykkel[asciilifeform]: only cuz of practical considerations, cant have infinite sized chain, so line must be drawn in sand somewhere and seems to me simplest possible place to put it at "no blob in chain"
signpost[asciilifeform]: can't have infinite sized chain, so need to kill everyone in 2100
signpost[asciilifeform]: not being a dick here, just saying this is more an engineering tradeoff subj than cut and dry.
jonsykkel[asciilifeform]: better to kill them in 2100 than in 2025
signpost[asciilifeform] cannot disagree, lol
jonsykkel[asciilifeform]: to some extent exagerating cut and dryness of conclusion for comedic efect
jonsykkel[asciilifeform]: also, final spec already poised to be reasonably complex, makes sense imo to value simplicity where posible if nothing else
signpost[asciilifeform]: yep. fwiw I don't see that image inlining actually forces anyone to retain images. I can hack my client to silently discard all but a before-after marker around the images.
signpost[asciilifeform]: this is perhaps made more cumbersome but nothing prevents it any more than "headers-only" mode in prb
jonsykkel[asciilifeform]: true, would still have to walk them during backwards sync though
jonsykkel[asciilifeform]: making full sync take watever number of times longer than otherwise would
signpost[asciilifeform]: not a proposal. using the example to call the representation of the image arbitrary when talking about whether to retain.
phf[asciilifeform]: that's a technical detail though, we're having an ontological conversation about what the log is, whether or not some kind of binary blobs are its essential properties. is it a pdf or txt
signpost[asciilifeform]: yup, agreed. was about to switch to the social question.
signpost[asciilifeform]: what kind of artifact a pest station accumulates (and leaves for others to find, perhaps) is the design question.
signpost[asciilifeform] would go full zealot and call a pest station's job to record History.
signpost[asciilifeform]: or at least be the primary index into same.
phf[asciilifeform]: from technical perspective you can devise a fairly simple schemes for storing binary blobs separately and just enough meta to recover their packets. so you have A -> B -> <meta for a pdf> -> G on drive, which becomes A->B->C->D->E->F->G in flight
phf[asciilifeform]: and if you deleted a pdf, and somebody asks for getpacket C,D,E,F you fail to respond
phf[asciilifeform]: but that of course means that /somebody/ has to keep the entire log at all times, which is when you run into the whole 10GB of log with all kinds of lolcats
jonsykkel[asciilifeform]: if expect that some will store inline blobs and some wont, doesnt this defeat purpose of puting blobs in the chain in first place
jonsykkel[asciilifeform]: u end up with same availiability problem if enough ppls decide to not store cats
signpost[asciilifeform]: would it be useful to refocus on the question of push vs pull, rather than storage? that seems more important to me.
phf[asciilifeform]: i was going to get to that :)
signpost[asciilifeform]: let's say phf gets the pic of bill gates getting pissed on by epstein's girls
signpost[asciilifeform]: but he's about to be captured by the enemy
phf[asciilifeform]: deleting blobs becomes an equivalent of tearing pages out of a book, because it doesn't fit on your shelf, but what that signals is `your book is missing parts`.
phf[asciilifeform]: 􏿽i guess overall my argument hinges on authorial intent, and for their to be an intent. that is concsious construction of a shared document. the case of 2000x2000 jpg reencoded as png or even the general case of not understanding this complicated point that i'm trying to make. you
phf: 􏿽can't prevent somebody from attaching linux.iso into their word document, in other words, but i guess i don't really care about those people? what i'm trying to accomplish is if somebody is making some illustration heavy argument, i don't want to discover that all the pics got lost
phf[asciilifeform]: and to some extent the counterargument is `we can't have a rich document, because some idiot is going to attach linux.iso inline`
phf[asciilifeform]: 􏿽 because of technicality, because first-flight-1.png got equated to gzdoom-patched-nudes.tgz
jonsykkel[asciilifeform]: big difference is that if numskull attaches fedora27.iso to important word document it can be removed at any point in time without riping pages from book
signpost[asciilifeform]: to note where there isn't disagreement, sounds like nobody objects to an author being able to specify the intent that an image is inline in the log.
signpost[asciilifeform]: two questions remain. one's not super interesting to me, which is whether the byte array rides along with the inline item's hash, or lives elsewhere.
signpost[asciilifeform]: the other is whether this binwad's hash, byte-array, or both are pushed rather than my station having to ask to retrieve them.
signpost[asciilifeform]: the bill-gates-piss-pic example is the best argument I can give for why have push.
signpost[asciilifeform]: there's preserving for posterity sure, but also the question of whether a station will ever be able to get important contextualizing bits out to others before perishing.
phf[asciilifeform]: signpost, the two are tied together intimately, because the question becomes `we have a mechanism for delivering log that has properties X, do we want binary blobs to fully share those properties`
signpost[asciilifeform]: seems very useful for some binary blobs to have.
phf[asciilifeform]: right, even in trivial cases of a non-stationary station. post a lolcat, close the lid
phf[asciilifeform]: this end of the conversation is abstract though, because if binary blob messages don't use the same mechanism as broadcast messages we no longer have anything concrete to go by.
phf[asciilifeform]: the only mechanism discussed so far is having some kind of announcement message that is equivalent to broadcast, but as a payload it has id of file that you can then use to query some binary delivery mechanism. but we don't know anything about this binary delivery mechanism yet
asciilifeform: interesting thread
asciilifeform: http://logs.bitdash.io/pest/2022-12-29#1019867 << will observe, not errythng gets lost at same rate. e.g mp's s&m links mostly 404, but, otoh, 100% of fg schematics still up
bitbot[asciilifeform]: Logged on 2022-12-29 11:41:06 phf[jonsykkel|awt]: 􏿽can't prevent somebody from attaching linux.iso into their word document, in other words, but i guess i don't really care about those people? what i'm trying to accomplish is if somebody is making some illustration heavy argument, i don't want to discover that all the p
asciilifeform: if sumthing is genuinely interesting, even nao folx will keep copies around even when takes some elbow grease. not to mention when takes ~0 in a hypothetical sane p2p apparatus
asciilifeform: so imho there's a natural filter, which worx rather well when material is very easy to copy (not speaking here of '80s with their bolix tapes 'sank in atlantis' etc)
asciilifeform: http://logs.bitdash.io/pest/2022-12-29#1019874 << no reason why not have pushed-jpg in ~direct~ msgs, to l1. and a 'push this photo of gestapo behind my chair to erry peer!' cmd, if one likes
bitbot[asciilifeform]: Logged on 2022-12-29 11:49:02 signpost: the bill-gates-piss-pic example is the best argument I can give for why have push.
asciilifeform: ( from philosophical pov, pest supports both push ('here's a packet') and pull ('hey, i heard there was a $hash, nao plox getdata $hash') fwiw )
asciilifeform: http://logs.bitdash.io/pest/2022-12-29#1019880 << there was a napkin-level thread which cannot atm find, where 'luby has overhead, so only makes sense for $size, so what response to getbinwithhash oughta be is a tree of hashes until leaves represent frag with $size'
bitbot[asciilifeform]: Logged on 2022-12-29 12:00:05 phf[awt]: the only mechanism discussed so far is having some kind of announcement message that is equivalent to broadcast, but as a payload it has id of file that you can then use to query some binary delivery mechanism. but we don't know anything about this binary delivery mechanism yet
asciilifeform: no concretes tho, afaik these will require empirical #s
phf[asciilifeform]: is getbinwithhash going to follow the same 1-for-1 policy?
asciilifeform: phf: the notion iirc was that it won't, when comes down to level of 'leaf' (you get n luby packets, of which you need m, or until say 'enuff'. again need concretes to say what n/m oughta be)
asciilifeform: 1 of the reasons why the luby transfer absolutely gotta be direct-only (tho can be from >1 peer in parallel, sending diff frags)
asciilifeform: if you flood-route'em , no amt of net bw will ever suffice
asciilifeform: ( why luby to begin with? precisely so can be asynchronous and order-not-matters and occasional-loss-not-matters )
asciilifeform: the reason why 'need empiricals' is that there are unknowns ( what, for instance, happens if the sender and receiver have pipes of severely mismatched bw ? )
phf[asciilifeform]: here, catch!
asciilifeform: ideally oughta have some way to tell sender roughly what speed of ball yer able to catch; and conversely, for sender to be able to 'slow the balls'
phf[asciilifeform]: there's other unknowns, like for example i'm sending only one broadcast out(but i send ignore to three other stations, so they send me stuff back), and there are whole periods of dark, why? who knows. when my packets are silently lostt.
asciilifeform: phf: fwiw asciilifeform's original spec was braindamaged, in that it asked for getdata to be issued to the particular peer who had sent the msg that triggered it
asciilifeform: this will give perma-holes if specifically a<->b link is lossy (which is how a missed msg from b to start with)
asciilifeform: theoretically, tho, if erry station correctly handles getdata and prod -- you can 'talk into unplugged cable' for an hour and eventually folx will see what you had said, tho
asciilifeform: but afaik we aint there yet
phf[asciilifeform]: that's vaguely how it works at the moment
asciilifeform: rright, but evidently not 100% (tho asciilifeform not knows whether phf's station currently implements 100% getdata & prod)
phf[asciilifeform]: i send out getdata for whatever packet to all comers
phf[asciilifeform]: and i've never gotten prod request from anyone, but i handle prod responses
asciilifeform: hmm not even jonsykkel ? iirc his sends prods
phf[asciilifeform]: i'm not peered with him yet, because i still can't remember my gpg password :D
asciilifeform: could try setting up a smalpest locally to test prods, if you've the cycles
phf[asciilifeform]: i'm kind of getting concerned, but i was also sick for a week, so hopefull once my head clears out a bit…
asciilifeform: phf: worst case, 1 of us takes a drive to where the other is, and 'key party', asciilifeform remembers what phf looks like
asciilifeform: (and hopefully vice versa)
shinohai[busybot]: phf: "You need a man that knows how to handle your prod responses ...."
phf[asciilifeform]: if i permanently forgotten the password that would be Very Bad™. i have backups keys encrypted with it, etc.
asciilifeform: hopefulyl it's still there sumwhere in phf's head, and will float
bitbot[asciilifeform]: Logged on 2022-12-29 12:46:27 phf[awt]: i send out getdata for whatever packet to all comers
phf[asciilifeform]: i mean, it's not a very complicated logic!
asciilifeform: phf: does your pestron have provision to retry a getdata if not answered in $interval ?
asciilifeform trying to implement his earlier 'broken heart' idea, where when one finds a hash pointing 'nowhere', a db entry is made to represent the fact, and then asynchronously getdata's are issued to try & replace the placeholder
asciilifeform: ( rather than synchronously shooting one out immediately when see the hash, and then 'fughet about it' )
asciilifeform: phf's logit ftr is correct per current spec tho
asciilifeform: also phf is that a bolix screenshit ?
phf[asciilifeform]: http://logs.nosuchlabs.com/log/pest/2022-12-29#1019993 << that part is totally not to spec. i don't ever send getdata immediately, just plop the packet into the backlog. and in (periodic) handler there's logic to spool getdata for everything that's missing.
dulapbot: Logged on 2022-12-29 13:05:22 asciilifeform: phf: does your pestron have provision to retry a getdata if not answered in $interval ?
asciilifeform: wasn't obv from the photo
phf[asciilifeform]: 􏿽http://logs.bitdash.io/pest/2022-12-29#1019935 << i have two parallel tracks in code, very simple operator driven conditional stuff. so e.g. i call (prod) -> there's prod response -> it triggers getdata -> resulting packet goes into packet log. if that loop fails i don't care much
phf[asciilifeform]: 􏿽because i'll just rerun it manually. the second track is automatic stuff. the packet log maintains holes, there's a code in periodic handler to try and fill those holes with getdata requests. it's basically all-in on the broken heart idea.
bitbot[asciilifeform]: Logged on 2022-12-29 13:15:26 asciilifeform[6]: wasn't obv from the photo
bitbot[asciilifeform]: Logged on 2022-12-29 13:09:10 asciilifeform[jonsykkel]: also phf is that a bolix screenshit ?
← 2022-12-28 | 2022-12-30 → ↑
Random(pest) | Download hourly DB snapshot | Get Source Code