Show Idle (>14 d.) Chans
← 2023-02-02 ↓
cgra: signpost, in ocpy Decoder.solve(), wouldn't "destination_block_buffer ^= block.buffer" be better as "destination_block_buffer[:] = block.buffer" ?
cgra: first operand in a series of xor ops not needed to xor with initial zero
cgra: signpost, did you ever consider if aux blocks could be generated on-the-fly, instead of pre-generated set? (i thought this option was mentioned in the paper(s), but can't find it anymore, or i was just imagining it)
cgra: appears to me you cannot get exact same aux block probability distribution if you want to map from an aux block to 0..n message blocks, instead of mapping each message block to q aux blocks, but perhaps don't need exact same distribution?
signpost[cgra]: will need moar coffee to answer this question, but you're not supposed to use the same prng for aux blocks and check blocks, so when you generate aux blocks shouldn't affect that, I'd think.
signpost[cgra]: check blocks all get seeded with their block ID
signpost[cgra]: *production of check-block edges gets seeded with ^
signpost[cgra]: (also thanks for reading the thing! glad my thick skull isn't the only one aimed at it!)
signpost[cgra]: ty phf also
cgra: signpost, yeah i didn't mean to mix aux and check block prng's, but was pondering whether by slightly(?) changing the aux rng usage could you avoid mandatory aux pre-gen and generate aux blocks (still deterministically) as they are required
signpost[cgra]: aux blocks are a lot fatter than check blocks, and there's only say 3% of them compared to check blocks. I'd think doing lazy aux block generation would introduce chug in encoding each time one's hit as a check block edge
signpost[cgra]: that said there's no reason not to do the experiment.
signpost[cgra]: you're supposed to be able to run the second-phase of encoding indefinitely, while first-phase that produces the aux blocks is finite and small-ish, so I just ran it up front.
signpost[cgra]: one thing you don't necessarily want to leak to an outside adversary is which blocks rely upon aux blocks, because that'd give targeting info to the adversary. and if all aux-block-yielding check blocks are nuked, you're not getting the payload on the other side
signpost[cgra]: encoding's guarantees rely upon the enemy not being able to discern which blocks to zap
cgra: signpost, don't pre-genned aux blocks already give such info? because don't need any genning
signpost[cgra]: I don't think so, because to the check block pass they're just more message blocks
signpost[cgra]: there's not more or less work in the check-block phase because something was an aux block vs message block. it's just another edge that gets xor'd into check block payload
cgra: ah right
signpost[cgra]: vs if every time a check block edge ended up pointing at an aux block we then generated the aux block (which has many more edges than most check blocks) there'd be discernible choog
cgra: yes, true
cgra: could you say that on-the-fly aux-blocks only giving operator a ~choice~ whether to pre-gen/cache or not, makes the timing leak a minor issue?
cgra: signpost, np re reading ocpy, and double the ty for creating it!
signpost[cgra]: as I think about it, the timing leak is both an issue there and in one other case. in the case we're discussing, I'd rather not introduce configurability that persuades folks to do something which makes it easier for an attacker to undermine the reason for the encoding in the first place.
signpost[cgra]: and it occurs to me that this timing issue applies to all block generation.
signpost[cgra]: if the sender *ever* exposes their rate of block generation by letting blocks go out faster than they can be generated, an attacker can infer by timing the degree of all check blocks.
signpost[cgra]: at which point it's as easy as zapping degree-one check blocks, which ensure nothing ever gets solved.
signpost[cgra]: degree-1 blocks having one edge, and being identical to the underlying message or aux block.
cgra: signpost, i suppose on-the-fly auxes beneficial also to pre-gen by not requiring to process all aux blocks in one unit
signpost[cgra]: solving a degree-n block is impossible when total solved blocks is not at least the number of that block's degree, right?
signpost[cgra]: so if you don't get any degree-1 blocks... "we ain't found shit!"
bitbot[cgra|asciilifeform]: Logged on 2023-02-03 09:06:37 signpost: as I think about it, the timing leak is both an issue there and in one other case. in the case we're discussing, I'd rather not introduce configurability that persuades folks to do something which makes it easier for an attacker to undermine the reason for the encoding in the first p
cgra: http://logs.bitdash.io/pest/2023-02-03#1022100 << is that off-by-one? otherwise agreed
bitbot[cgra|asciilifeform]: Logged on 2023-02-03 09:08:48 signpost: solving a degree-n block is impossible when total solved blocks is not at least the number of that block's degree, right?
signpost[cgra]: herp, yep
signpost[cgra]: at least degree-minus-one blocks have to be solved for it to be possible (but not guaranteed) that a check block can be solved
signpost[cgra] sips moar coffee
signpost[cgra]: hm. I'm having a hard time seeing how this can be solved without reducing send rate to the worst encoding case, throwing random delays or random dummy blocks in, etc.
signpost[cgra]: curious what folks think.
signpost[cgra]: via radio, would be harder to zap particular blocks, but via internet, appears not.
jonsykkel[cgra|asciilifeform]: if enemy can zap at all, they can just zap all ur bloks no
signpost[cgra]: depends whether we're modeling DDoS or packet-analysis/silent-dropping
signpost[cgra]: but yeah, that's true
cgra: signpost, say you randomize it. wouldn't some slip through eventually?
signpost[cgra]: if DDoS, the encoding can be tuned to make use of whatever loss rate, can be 99.999%. if there's any signal getting through, reassembly of the message will be possible over some amount of time.
signpost[cgra]: cgra: randomizing, keeping a check-block hopper full and limiting transmission to a rate which ensures hopper doesn't run dry, all those kinds of mitigations.
cgra: right
jonsykkel[cgra]: yep, i mean specificaly re the picky zapping. guess they can make it 3% less obvious that zapping is going on
signpost[cgra]: just worthwhile to evaluate whether this encoding does anything, or whether it'd be better to just piss file-chunks as fast as possible, and maaaaybe have this thing available as the tool it's designed for, to handle squeeking out information when under heavy loss.
signpost[cgra] happy to say the thing doesn't do what's needed if it doesn't.
signpost[cgra]: another thing that oughta be modeled out is the swarm download aspect. the fact that separate nodes can create compatible streams of a given file makes for easy, uncoordinated collaboration to get another node a file.
signpost[cgra] will think moar and return in a bit
asciilifeform not grasps how the 'enemy knows which packets to drop' problem could be relevant outside of a compromised key
signpost[cgra]: yeah, I guess you'd have to know some luby-farts are going on at all, before deciding to drop the easiest to generate
signpost[cgra]: but I'd think the lubystreams would look different from the outside. constant packet spray, variable time between packets
asciilifeform: snoop does not know which blacket packets contain what, tho
asciilifeform: even if can make good guess somehow that packet p is part of a luby stream -- has nfi what part
signpost[cgra]: maybe not a problem, just figured it oughta be pushed on from this side, see if stands.
signpost[cgra]: the "what part" would be inferrable via timing if packets leave the station as soon as they're encoded, right?
signpost[cgra]: the encoded packets have a linear variability in difficulty to produce. another xor-encoding edge from encoded block to underlying message block would add same amount of encoding time.
asciilifeform: aint the ~whole point of lubyism to make order (as well as loss of some arbitrary %) not matter ?
signpost[cgra]: *if and only if the enemy cannot discern packets.
signpost[cgra]: and I'm pushing on whether the timing leak makes sufficiently discernibl
asciilifeform: so buffer'em and send at fixed rate (which you gotta do anyway if width of receiving and transmitting pipes differ)
signpost[cgra]: yup, was mentioned as a solution
asciilifeform still catching up on log
signpost[cgra]: if that's satisfactory to you too, sounds good.
asciilifeform: aha. buffer in such way that a burst containing mix of 'easy' and 'hard' chunks is pre-genned and then sent
asciilifeform: hypothetically not even need buffer, if can generate 'easy' and 'hard' chunks in parallel, avoiding 'interesting' idle intervals on the line
bitbot[cgra|asciilifeform]: Logged on 2022-12-29 12:34:01 phf[awt]: here, catch!
signpost[cgra]: ah yeah, the more parallelism you have going in encoding the better off you are on this front, as well as the obvious one of having more throughput.
signpost[cgra]: ok, pretty satisfied this isn't a problem since we have to control the spray rate in any case.
signpost[cgra] meanwhile seeing about chopping a simpler xorshift into the py version. will leave both in the code since phf has implemented the current one. not planning on making this configurable, but rather want to confirm that behavior on my end doens't change, or if so why.
cgra: signpost, if feel like it, there's also the 'un-floating' to consider simultaneously, plus the on-the-fly aux is related
signpost[cgra]: could you restate the benefit of on the fly aux generation?
cgra: 1) smaller memory requirement: processing one aux block at a time vs all aux blocks at once, 2) no mandatory extra state, 'just files' and the encoder
signpost[cgra]: yeah but would have to reprocess these aux blocks every time referenced, or cache
cgra: yes
signpost[cgra]: oughta be measured maybe, and see the hit to different payload sizes
cgra: also uncertain to my feeble brain if can 'invert' the probability math for it
signpost[cgra]: another solution to "what if the payload's large, and we're talking about 30G of aux blocks" is to chunk streams into finite encoding scopes.
signpost[cgra]: doing that might give what's needed to estimate how fast the other end can eat.
signpost[cgra]: and establishes points where adjustments can be made to encoding parameters.
cgra: would poor man's version of ^ be splitting bluray images into .part1,.part2 etc?
signpost[cgra]: yeah, encoder would chunk the payload like that.
signpost[cgra]: say receiver stops receiving blocks, and hasn't solved a chunk. can signal back "hey, I'm at 300/305 blocks solved over here on chunk 5"
signpost[cgra]: then sender can compare that to his previous estimate and adjust because loss was 5 blocks.
signpost[cgra]: receiver can also politely signal back that they received more blocks than they needed, and sender can adjust down.
signpost[cgra]: or sender can periodically decay and see what happens.
signpost[cgra]: were I to generate the aux blocks on the fly, next decision would be whether to lazily advance the prng which produces the message-aux edges, or precompute that.
signpost[cgra]: feels like fine tuning, not a core issue.
phf: http://logs.bitdash.io/pest/2023-02-03#1022148 << xorshift1024 fails LinearComp on "Big Crush tests" per https://arxiv.org/pdf/1810.05313.pdf, i don't know what that means for us
bitbot[cgra|asciilifeform]: Logged on 2023-02-03 11:03:37 signpost: meanwhile seeing about chopping a simpler xorshift into the py version. will leave both in the code since phf has implemented the current one. not planning on making this configurable, but rather want to confirm that behavior on my end doens't change, or if so why.
phf: but then there's another prng quality analysis page, where it's not listed as failing linearcomp, https://prng.di.unimi.it
signpost[cgra]: not sure. fwiw I've got a test in the pywad that drops packets and confirms that the message can still be solved.
phf: yah, i don't understand oc at all yet, so i'm just leaving this here
phf: http://logs.bitdash.io/pest/2023-02-02#1022068 << i figured out why it generates garbage on the last multiply. (unsigned-byte 64) is not immediate on a 64-bit lisp, on account of there not being enough space to put a tag. so to not make it generate garbage it needs to be inlined
bitbot[cgra|asciilifeform]: Logged on 2023-02-02 17:19:57 phf[deedbot|awt]: e.g. xorsfhit1024 becomes http://paste.deedbot.org/?id=UZma, and it still generates garbage on that last multiply (it bignums and then truncates, i can't figure out how to make it use a int64bit version of *)
phf: otherwise the value is boxed on return
phf: niether here nor there problem. to make things be easily fast on a lisp ought to keep bit size under most-positive-fixnum, which is short two bits under sbcl from whatever system word size
cgra: signpost, pre-gen/on-the-fly prolly otherwise a minor point, but affects aux randomization algo, so must choose which one to use
phf: cgra, i don't understand, why would on-the-fly require a different aux randomization algo?
cgra: phf, well, current is from a message block at a time, to aux blocks, but on-the-fly is 'this one here aux, which message blocks?'
cgra: i mean, the algo part around prng
phf: that didn't clarify, but i'll keep the point in mind while reading the code again
cgra: i'll try rephrasing sometime soon, atm bit busy off-desk
asciilifeform: http://logs.bitdash.io/pest/2023-02-03#1022167 << afaik non-ideal prng, in context of lubyism, simply gives less-than-ideal 'mileage' (excluding the case of a catastrophically-broken prng, e.g. with laughably short period; there, could fail entirely to send $block)
bitbot[cgra|asciilifeform]: Logged on 2023-02-03 11:40:23 phf[awt|deedbot]: http://logs.bitdash.io/pest/2023-02-03#1022148 << xorshift1024 fails LinearComp on "Big Crush tests" per https://arxiv.org/pdf/1810.05313.pdf, i don't know what that means for us
signpost[cgra]: yeah, the channel would seem lossier the worse the rng.
signpost[cgra]: this property made for hilarious pitfalls throughout the implementation, where brain-dead bugs would still eventually work
asciilifeform: biased prng will simply send subset of chunks moar often than they oughta be sent, at the expense of some other subset
phf: yeah, but that's not what linear complexity is, hence the question of how lincomp affects us
phf: as in, we can just go and choose an algo that doesn't fail linearcomp, but there might be no reason to.
asciilifeform: phf: as asciilifeform understands, from pov of lubyism (rather than cryptographic context) all statistical failures of prng 'are same thing'
phf: yeah, well, that's wrong
asciilifeform: phf: possibly; but then plox to elaborate
phf: e.g. https://www.turingfinance.com/hacking-the-random-walk-hypothesis/#linear-complexity-test suggests that linear complexity test is a test by insinuation. it attempts to automatically construct a lsfr out of bit stream
phf: *for the bit stream rather
phf: so it approximates one from a bit stream, and then compares it's length against "what would be expected from a true uniform random sequence"
asciilifeform: so, for instance, does 'prng fails linear complexity test' imply that one can construct a luby payload that will never properly send (i.e. some blox systematically avoided) ? or 1 that takes 10x the expected # of chunks to send..?
phf: no, it doesn't even live in the same realm
phf: it's explicitly a xorshift detector (xorshift IS an lsfr), and if it detects that the sequence could've been produced with a short xorshift it says "that's awfully short, it's probably not random"
signpost[cgra]: there's something head-cracking in there, at least for a dummy like me.
signpost[cgra]: first, why xorshift is so good at this, for as few steps as it contains.
signpost[cgra]: second, that apparently a lot of prng behavior is reducible into something like xorshift
signpost[cgra] would like to understand the math better.
phf: asciilifeform, it's a kind of walking in circles :) my understanding so far is that lct is a "short" lsfr detector, but the main appeal of xorshift family is that it's a short lsfr. i don't understand (nor have i found laymen documentation) on what other implications this have
phf: i'll just leave this here as is,
phf: 􏿽”These scrambled Xorshift generators output 64-bit integers. Ideally, a 64-bit PRNG should be tested with a 64-bit test suite; using a test suite designed for 32-bit PRNGs (or actually 31-bit PRNGs in the case of Big Crush) necessarily involves compomises. Vigna’s approach was
phf: 􏿽to interleave the least-significant 32 bits and most-significant 32 bits of each 64-bit output, thus each 64-bit output becomes two 32-bit outputs. This interleaving strategy may detect flaws in some 64-bit PRNGs, but it may also hide statistical weaknesses. Such an approach is use
phf: 􏿽ful but is insufficient. A more comprehensive strategy is to also test the most-significant and least-significant 32 bits separately. When we focus on the least-significant 32 bits, and, specifically, in the case where their bits are reversed, we find that the scrambled Xorshift ge
phf: 􏿽nerators systematically fail Big Crush.”
phf: and then later
phf: “The generators under consid- eration require relatively long seeds (at least 16 bytes). To generate a sufficiently long seed (128 or 1024 bits), we produce a random integer with the Bash shell’s internal RANDOM function as an initial seed for to the 64-bit SplitMix gen- erator [8].”
asciilifeform went to read phf's 'random walks' link but head not came back yet, may take a while
phf: http://logs.bitdash.io/pest/2023-02-03#1022196 << from reading luby ideal selection is not random, but follows some kind of fancy distribution. i imagine oc is similar
bitbot[asciilifeform]: Logged on 2023-02-03 12:23:22 asciilifeform[4]: so, for instance, does 'prng fails linear complexity test' imply that one can construct a luby payload that will never properly send (i.e. some blox systematically avoided) ? or 1 that takes 10x the expected # of chunks to send..?
phf: not ideal, but best case
bitbot[cgra]: Logged on 2023-02-03 12:21:31 phf[deedbot|awt]: e.g. https://www.turingfinance.com/hacking-the-random-walk-hypothesis/#linear-complexity-test suggests that linear complexity test is a test by insinuation. it attempts to automatically construct a lsfr out of bit stream
asciilifeform: phf: aha. was trying to think of 'what happens to it under variously sad prng'
asciilifeform: ( given that transmitted chunks are reorderable, potentially, using an actual trng -- 'enemy may infer order' failure mode can be ruled out )
asciilifeform: so afaik an attack would have to consist of a constructed payload that eats surprising bw, or even not completes
asciilifeform: ( e.g. a hypothetical lolcat.jpg that loox unremarkable and may end up distributed on pest nets but somehow takes 1tb of lubyistic chunks to actually move )
asciilifeform: rather like a de bruijn sequence worx against von neumann's fortune teller
phf: well, there's all kinds of questions as far as "malevolent counterparty"
asciilifeform: well on pestnet, properly malevolent counterparty can spam at will until unpeered. moar interesting imho q is re subtly boobytrapped payloads from third parties
asciilifeform: would seem possible if prng has a 'usable' flaw
phf: oc can be done with an explicit switch on and an explicit switched off, so e.g. can ask counterparty to send you lolcat then never ask to stop
phf: because only reciever know if fully recieved "oh, keep sending, keep sending, i still don't quite have the whole thing" can be an entirely legitimate state
asciilifeform: for instance. which is wai may need at least 'soft' peer bw quota, e.g. 'hey asciilifeform , why do you eat 99% of my bw asking for lolcat_part1_100.jpg 24/7 ?'
asciilifeform: need to at least indicate to operator that it is happening, imho
asciilifeform: given peer oughn't be able to soak up station bw 24/7 silently
phf: right, or some probabilistic checks "ok, dude, you should have had the thing by now unless your channel have 0.9999 error rate"
asciilifeform: even nao a broken or mischevious peer can throw getdata at you repeatedly
asciilifeform: (if you dun ration resources, it'll thrash disk and slow yer station to a crawl)
phf: yeah, but that's to your point of "one for one", luby explicitly fails the one for one
asciilifeform: correct, tho with getdata, 'one for one' is only tru re bw, given as sending getdata is considerably cheaper than answering it
asciilifeform: (if results in disk seek)
phf: rright
phf: also you can definitely do ddos with luby. as of right no handshake in protocol, so you can e.g. forge a valid "ignore" packet, but with bogus origin ip, so that will update AT, and then you follow it up by "turn on luby please for lolcat.jpg"
asciilifeform: ( hence wai per spec, getdata is only serviced if 'cpu cycles available' )
phf: *naive luby that is
phf: or better yet "turn on luby for all the files that i know you have"
asciilifeform: if you've the key, can indeed trigger lubyism to arbitrary ip, if can route forged packet to receiver
asciilifeform: fwiw this needs not only peer key, but also to know of a valid file hash to make luby req for
asciilifeform: i.e. joker can try to use his l1 as a 'ddos cannon'
phf: right
asciilifeform: imho the Right Thing to do is to make such lulz visible to operators, rather than try to rule out mechanically
asciilifeform: 'hey asciilifeform , what do you think yer doing?'
phf: there can be a legitimate reason for doing something like that, for example that would be direct effect of initial log sync
asciilifeform: the other obv pill would be to require a handshake before firing off luby sequence
asciilifeform: at least 1 packet each way
phf: yah
asciilifeform: sorta like the handshake used in 'rekey'
phf: http://logs.bitdash.io/pest/2023-02-03#1022247 << your initial sync is a bandwidth heavy cornucopia. long stream of getdata requests, and depending on how you implement, a getdata for a message with luby payload results in "oh yeah and by the way also give me stream for this luby"
bitbot[cgra]: Logged on 2023-02-03 14:15:00 phf[awt|deedbot]: there can be a legitimate reason for doing something like that, for example that would be direct effect of initial log sync
phf: in fact i can see how a naive log sync might result in a self-ddos :D
asciilifeform: well per current spec, only text msgs are getdata'ble
asciilifeform: lubyism in asciilifeform's conception is entirely separate protocol, with transport via pest directmsgs
phf: yes, but yes, but log will eventually contain hashes for payload, as per discussion
asciilifeform: doesn't mean one necessarily has to fetch erry single one during log sync tho
phf: but it would be a legitimate use case
asciilifeform: sure. you'll end up with over9000 linux isos etc if you do it, tho
asciilifeform: typically not what you want
phf: that's probably not true, on account of y'all punting on the "essential" payload question
phf: if you're catching up with log, it's sensible then to have some kind of heuristic for automatic luby payload request, "give me all png/jpg/gif/pdf files that are less than 20mb" or whatever
asciilifeform not expects to find a satisfying mechanical answer to that q
asciilifeform: in practice, will prolly end up as e.g. 'gimme erry bin signpost offered up'
phf: well, that too
phf: but for example whatever my client can display inline (images, video, audio), i'd want to fetch as soon as it's available
asciilifeform: how to not turn 'hey l1, gimme luby of lolcat.jpg' into a guaranteed own-goal is genuinely open q (suppose yer l1 is 20 stations and they all have it, say) atm
asciilifeform: sorta wai asciilifeform not convinced that it oughta default to 'hey l1...' but rather start with 1, and request others as-needed
asciilifeform: elementarily, if you're on e.g. 1g/s pipe, you dun win anyffin from having 20 1g/s pipes spewing at you at line rate simult.
asciilifeform: if yer isp is typical konsoomer sack of sad, may even trigger a ddos sensor somewhere and getcha dropped a la pizarro and you'll be picking up phone to complain 'to central laundry'
phf: but isn't that the mechanical beauty of luby. you get 20 streams, you get the result 20x faster :>
phf: n stations sending same luby stream is part of the same probability space
phf: right, but the requirement that you should be able to negotiate pipe rate was heavily discussed because there are other reasons
phf: don't want to get a single line rate stream from a 1 20g/s pipe either
phf: kek, internet went down so a handful of messages didn't go out. on reconnect i asked it to bump http://glyf.org/screenshots/pest-bump.png
asciilifeform[cgra]: lol until saturated line (supposing you even want the thing badly enuff that you dun mind saturating)
phf: asciilifeform, yeah but you ask n peers for maxrate/n luby streams
asciilifeform: phf's station seems to be pioneering testbed for 'can operate on poor-quality net pipe' design goal
asciilifeform: phf: right, can do this if you know for fact that the folx yer asking in fact all have lolcat.jpg
phf: it's entirely roaming too. like i just close the laptop and open it at cafe, it self recovers (by spewing ignore packets to everyone, once it can)
asciilifeform: (and that all have cycles to spare, and will send, etc)
phf: asciilifeform, so you have to have a provision to renegotiate. in fact "send lolcat at 10" followed by "send lolcat at 100" should produce correct rate adjustment result
asciilifeform: phf: is precisely how is supposed to work (still need the 'hammer' thing, for nat, tho)
asciilifeform: asciilifeform's notion is that one would request subhashes, e.g. 1MB at a time
phf: why then luby at all?
asciilifeform: (if payload smaller than MB, there are no subs)
asciilifeform: phf: balance between synchronicity and 'don't throw 1TB at me asynchronously plox'
asciilifeform: i.e. avoid the fairy tale magick pot that endlessly spews porridge and drowns the world
phf: my understanding is the whole point of fountain code is that you can ask bunch of random places to start sending you stuff, and with a high probability you're going to full your block holes in a reasonable amount of time
asciilifeform: limiting subchunk to e.g. MB lets ya say 'pot, stop'
cgra: what's wrong with 1MB of check blocks per 'luby getdata'?
asciilifeform: right, but imho not unreasonable to require that the process stops after certain # of blox, unless requested again
asciilifeform: otherwise yer in a position where e.g. asciilifeform asks for lolcat.jpg, then loses net connection and never says 'enuff', and nao yer sending 4evah (or until some rather large timeout)
phf: that should not be timeout based
cgra: say, 'luby getdata' = '1MB more plz'
asciilifeform not knows ideal cutoff, but does not think it oughta be much larger than 1MB
asciilifeform: the point of making exception from 'want a packet? send 1 packet' is to economize outgoing bw. but ought not to be at the cost of potentially drowning yerself errytime
asciilifeform: (as well as wasting over9000 of errybody's outgoing bw by having'em piss at you over9000 moar packets than actually need)
asciilifeform: another angle to think of it from, is that the total avail. bw to all the participants in a net is a finite resource, and ought not be burned carelessly
phf: i think some kind of hard limit on luby stream makes sense, so like getme lolcat 10, will send you a 1mb of lolcat blocks at 10b/s or whatever. if that was not enough you can ask again and again etc.
phf: but the idea of "gimme a specific chunk but with luby" ought to be tested against the inherent properties of luby
asciilifeform still suspects that best heuristic is '1 peer may not ever account for > 1/n bw (where n is # of peers) except by explicit permission of station operator'
asciilifeform: indeed oughta be tested.
dulapbot: Logged on 2023-01-29 13:43:03 asciilifeform: ... and many of these details, imho, can only be nailed down empirically.
cgra: why specific chunks again? to review partial data before full 60GB bluray image complete?
asciilifeform: cgra: you get a tree of subhashes, and then know when you've actually received given portions of the payload
asciilifeform: if e.g. some peer is sending you rubbish, you know about it sooner
asciilifeform: ( afaik none of the currently-known lubyistic encodings are at all resistant to rubbish )
cgra: asciilifeform, right
asciilifeform: if sumbody's 'poisoning' yer warez, you want to find out asap.
phf: asciilifeform, but you can do that using crc sum on a block, rather than a whole new subrange mechanism
cgra: asciilifeform, seems like most check block solving starts when like ~90% already received
cgra: (or similar high%)
asciilifeform: (how? let's say, if some chunk seems to never converge, you start excluding packets from various peers and see whether then converges)
asciilifeform: phf: making the requestable unit of transmission a e.g. 1MB block with a known hash, is reasonably simple way to do it
asciilifeform: you request H. H is either a hash of entire payload (if <1MB) or a hash of H1+H2, which a cooperating peer will give you; H1 and H2 may in turn be hashes of H1_1, H1_2, H2_1, H2_2, and so forth
asciilifeform: until get to the leaves
phf: you're trying to bolt a tradition chunk synchronization mechanism on top of a very different model, and there are ways to achieve what you're tyring to do, but in a more luby way
asciilifeform: possibly
jonsykkel[cgra|asciilifeform]: http://logs.bitdash.io/pest/2023-02-03#1022248 << if u can forge the ignore packet src ip to update AT, u can forge handshake ack in same way
bitbot[asciilifeform]: Logged on 2023-02-03 14:15:04 asciilifeform[5]: the other obv pill would be to require a handshake before firing off luby sequence
asciilifeform: jonsykkel: to do handshake, you gotta be able to both send ~and receive~ from $ip
asciilifeform: i.e. it's 'yours'
phf: jonsykkel, "handshake" implicitly involves a shared secret
jonsykkel[cgra]: ah ok i see
asciilifeform[cgra]: (err, both receive at and send from)
asciilifeform: and yes
asciilifeform: yet anuther reason wai you want to discretize subchunks -- say you switched cafes mid-download
asciilifeform: once yer peers run outta the given 1MB, they'll send the next one to yer new AT
asciilifeform: (given as that's where you'll be asking for the next MB from)
asciilifeform: (and you'll be re-requesting the interrupted subchunk, obv)
asciilifeform: the main reason you gotta, afaik, have the hash tree, tho, is to readily detect poisoning.
asciilifeform: otherwise you can sit there and, having received over9000 chunks, be left wondering wai never converges.
asciilifeform: whereas if yer expecting to converge on a 1MB chunk of known hash, and sumbody's already send you over9000x the nominally-req'd luby chunks, and still not converges -- you'll know that 1 or more of the suppliers is fucking you
asciilifeform: nao, the ~clever~ way to cut this knot would be if you could somehow nail down, via hash tree, the expected hashes of individual luby chunks
asciilifeform: rather than arbitrary cut into e.g. 1MB
asciilifeform: there'll still be a discrete subchunk size, tho, much larger than indiv. packet, tho
asciilifeform: (or tree itself would have to be impractically large)
asciilifeform: really oughta determine empirically.
asciilifeform must bbl
cgra: phf, trying this again:
bitbot[asciilifeform|cgra]: Logged on 2023-02-03 11:59:43 cgra[jonsykkel]: i'll try rephrasing sometime soon, atm bit busy off-desk
cgra: current randomization of edges between message blocks and aux blocks is by drawing exactly 3 aux blocks to each message block.
cgra: but if coming from on-the-fly angle, you'd have to first draw 'how many message blocks for this aux block', perhaps using the aux block itself as a seed of sorts, and after that draw as many message blocks
cgra: in theory, a single aux block can have 0..<message block count> edges while every message block has exactly 3 edges (with current ocpy params)
cgra: in above on-the-fly model, ocpy Header.message_block_edges and Header.aux_block_edges don't exist
cgra: (single aux block edges theoretical max 3 * <message block count> with current ocpy params, made mistake above)
cgra: a check block edge count ranges from 1 to 2114, averaging at ~8. if doing correct math here, aux block edge count averages at ~182
cgra: again ^ per ocpy current params
cgra: if you have a 100GB my-dumbuntu.zip, you have ~800 million message block/aux block edges
cgra: and ~4 million aux blocks
asciilifeform: http://logs.bitdash.io/pest/2023-02-03#1022328 << to elaborate: all you need is, before sending e.g. MB of luby, sent to the proposed addr 'here's 256 random bytes, plox to send'em back, then i'll start'
asciilifeform: *send to
bitbot[cgra|asciilifeform]: Logged on 2023-02-03 14:54:04 phf[awt|deedbot]: jonsykkel, "handshake" implicitly involves a shared secret
asciilifeform: the luby gets sent strictly to the addr which sent'em back, no AT renewal permitted between the 2 msgs
phf: should that be some kind of outside of luby mechanism?
phf: like you come online, start sending ignores, station recognizes change in at, and send you a challenge, that you need to respond to
phf: actually disregard, that seems like a bad idea, since it introduces all kinds of potential issues on the core messaging side
phf: one for one is such a nice property, once you violate it for reasons, suddenly need all kinds of extra mechanisms.
asciilifeform: phf: the 'send me back these rng bytes' thing would be part of the trigger for the luby (so 'outside', yes). imho would be enuff to avoid 'cannon' w/out adding much gnarl
bitbot[asciilifeform]: Logged on 2023-02-03 14:12:05 asciilifeform[jonsykkel]: i.e. joker can try to use his l1 as a 'ddos cannon'
asciilifeform: concretely, instead of a single binary direct msg 'gimme lubystream of hash H' being answered immed. by stream, the station will answer 'sure, but 1st send me back these rng bytes, stream will go to ip they get send back from and it has to be the same 1 from which you had asked just nao'
asciilifeform: no need to recognize, as such, change in AT; simply insist that the addr to which luby piss is to go is able to receive, as well as send, under the requester's key
asciilifeform: you can't forge, lol, ~receiving~ (or rather it aint called 'forging' anymoar if you can..)
asciilifeform: is very similar to the handshake spec already has for rekey, the purpose of which is to prevent a buggy, sad-rng'd, or saboteur peer from forcing the peering key to an apriori-known value
asciilifeform: ( there's naturally no way to prevent an idjit or traitor from divulging key to third party. but he can't force it to e.g. zero )
asciilifeform: ... this may be the 1 and only original idea, lol, in pest protocol. (tho for all asciilifeform knows, even it existed somewhere. but nfi where)
asciilifeform: contrast with e.g. ssl's 'null cipher'(tm) magick bit.
phf: right, i was over engineering things in my head. simply "guarding" luby with a handshake is good enough
phf: in totally unrelated can someone explain the china weather balloon story to me? i think ~that~ finally broke my clown world processing machinery
crtdaydreams[asciilifeform]: ^ second that, nfi where "spy balloon" nonsense origniated, but from a cursory glance anyone with a modicum of intelligence would observe that a) why worry about the bitch on the porch when there's a bear in the livingroom? and b) why not satellite imagery? lmao
crtdaydreams[asciilifeform] recalls a public satellite imagery website where could view .hk images and naviage around, could zoom in far enough to make out peoples faces quite easily
crtdaydreams[asciilifeform]: to the extent that faces etc. would be higher quality than found on e.g. passport
phf: that's not the angle i'm even interested in, more like "why is this such a big deal" and "why is there no sane grown up reaction"
phf: 􏿽1950s solution, announce that it's a weather balloon, immediately mount a televised recovery operation, have spooks look over it on a tight schedule, have a pr transfer of the balloon the chinese event, "our slanty eyed best partners seem to have lost their toy, we give it back, to
phf: 􏿽 bright future!" everyone drinks champagne and smiles
phf: 2023 zoomer solution "ugh awkward ugh we're monitoring *shuffles awkwardly around*"
phf: w t f
signpost[asciilifeform]: yeah, the lack of theatric message-sending is bizarre.
signpost[asciilifeform]: phf: I saw earlier that chinese state media of some kind was laughing at us being unable to control our airspace, but failed to find it a second time
asciilifeform: balloons aint obsolete, 'cheap & angry' if conspicuous snoop camera
asciilifeform: Official excuse for not popping it, apparently: 'could cause property damage on the ground' lol
signpost[phf]: curious if it's scanning radio
signpost[asciilifeform]: wifi, bluetooth, etc for exploitation
asciilifeform: hard to imagine wai else to bake such a large one if not to carry luxurious antennae
phf[asciilifeform]: the lack of any kind of sane grown up reaction is bizarre. i mean predictable, but that's like the ultimate failure of "hello yes hello" sanity check on there being present any kind of intelligence in the decision making aparatus
signpost[asciilifeform]: maybe they want to dump it in the ocean and recover
crtdaydreams[asciilifeform]: can't knock it down because "Das Racist!!1111!"
phf: signpost, in an obviously public event like that one ought to signal an intent, whatever the strategy is
asciilifeform pictures that there's likely a usg 'golden toilet' wunderwaffe designed to steal balloons, but not actually worx, and officials scheming to blame one anuther
signpost[asciilifeform]: sadder answer would be that biden's crackhead son relayed a threat from the chicoms.
signpost[asciilifeform]: and then he made up a reason to order no action.
signpost[asciilifeform]: mil could be internally wtf-ing at grandpa's order.
signpost[asciilifeform]: not claiming certainty, but any answer suggests woeful unpreparedness for war.
crtdaydreams[asciilifeform]: too busy waging war on own denizens
phf: so kommuniti consensus is wwiii?
crtdaydreams[asciilifeform]: phf: doubtful imo
phf: somehow in my world view chinese lost a balloon, "weather balloon" for diplomatic purposes, would've been mildly embarrassed but u.s. saved them face by being even more awkward and incompetent at diplomacy
signpost[asciilifeform]: chatter says the thing flew over a lot of air force bases.
signpost[asciilifeform]: and that it was maneuvering.
signpost[asciilifeform]: whiteman's a nuke base, b-2s etc
signpost[asciilifeform]: I think they scanned our MSFT-running trash.
phf: is it being piloted by one hide ping sam, the infamous ghost of beijing?
signpost[asciilifeform]: poor guy's instructed to jump out in the atlantic and start drinking.
phf: antenas are for show, hide ping sam remembers every detail of enemy installations by heart
← 2023-02-02 ↑
Random(pest) | Download hourly DB snapshot | Get Source Code