crtdaydreams[asciilifeform]: https://www.youtube.com/watch?v=bXJ5bZ8hq34 hits harder now
cgra: asciilifeform, in an attempt to enumerate sources that provide hints of specific pest messages' existence i came up with a following list: 1) an other message's self-chain or net-chain, 2) prod, 3) inv, 4) station operator (say, a manual getadata), and here's the question: 5) somebody else's getdata
cgra: any objections in using an unknown hash in somebody else's getdata as a trigger to send own getdata?
asciilifeform: cgra: from asciilifeform's pov -- massive potential for wasted bw ( sumbody gets 1 packet from a peer of his, and entire net is nao sending futile getdata's to one anuther ?! )
cgra: asciilifeform, right
asciilifeform: anyffin with potential for 'amplifications' aint even a 'measure 7 times, cut once' situation but 'measure over9000 times'
cgra: a reasonable rule of thumb
asciilifeform: the point of getdata is to 1) attempt to close gaps in yer chain 2) retrieve a possibly awol 'spearhead' (the prod case)
asciilifeform: both imho adequately covered by the current logic
asciilifeform: ( with the 1 possible exception of 'reply chains', fwiw )
asciilifeform: come to think of it, even the latter oughta be covered by selfchain and not need special case.
cgra: asciilifeform, perhaps another footnote to consider being added to spec, that 'why resist the urge of requesting an unknown want-hash'
asciilifeform: cgra: not that asciilifeform is opposed to making such a note, but if also went & made a similar note re: erry possible footshot he could think of , spec would be over9000 pgs and take anuther 20yrs to finish...
asciilifeform: maybe after spec is done (or at 'liquid helium' kelvin version, at any rate) could write a 'rationale' to go with it, like the ada people
asciilifeform: but atm this is a bridge too far imho
asciilifeform: for nao, the logs, with threads like this one, will have to suffice.
asciilifeform: there are prolly over9000 mis-optimizations one could make in pest that would ultimately make it unusable or impractical bordering on unusable (a la 'gnutella', say)
asciilifeform: asciilifeform's concern re protocol atm is to determine whether he ~already~ made such a footshot.
asciilifeform: see e.g. this thread.
bitbot[asciilifeform]: Logged on 2022-12-28 18:25:46 asciilifeform[6]: phf: i'm inclined to agree, i.e. 'let's have jpgs in chain, but Use Moderately' but plagued by suspicion that even very light use of it will be very painfully heavy for even a small pestnet with well-behaved inhabitants
asciilifeform: the ancients were not immune. early arpa, for instance, had to be manually rebooted on at least 1 occasion on acct of packet storm.
asciilifeform: this was survivable because there was only $smallint # of stations and the operators had working telephones and knew one another.
asciilifeform: and they had a strong interest in making the thing stand up.
asciilifeform: whereas if, hypothetically, next yr there is a pestnet with 400 people, and someone discovers 'magick packet' that floods it to the point of unusability, even after 'fix', 395 of'em will go back to 'telegram' or whatever and that'll be it.
signpost[cgra]: world's looking like fragmented internet with intermittent access may be in our near future, too.
signpost[cgra]: quite important to get pest right imho.
asciilifeform: signpost: in particular, the part where 'hypothetically work via radio' and 'not rely on ip carrier' imho.
signpost[cgra]: yep. not that anyone's owed, but the time for your item's coming, I think.
asciilifeform: ideally would at least demonstrate that it is ~possible~ before actually needed.
asciilifeform: as in the old proverb re inventing the parachute on one's way down from empire state building.
cgra: http://logs.bitdash.io/pest/2023-01-29#1021644 << yeah, is fine, just another nitpick of mine. and i prolly had a slightly wrong impression on how much of that 'rationale' you intended to do right away
bitbot[cgra]: Logged on 2023-01-29 12:28:14 asciilifeform[5]: cgra: not that asciilifeform is opposed to making such a note, but if also went & made a similar note re: erry possible footshot he could think of , spec would be over9000 pgs and take anuther 20yrs to finish...
asciilifeform: cgra: asciilifeform appreciates commentary re spec, and in fact key moving parts of the current protocol were on advice from other people (e.g. orig. spec not even had 'getdata')
asciilifeform: (instead had ludicrous kludge where packets were to be ack'd etc)
asciilifeform: ( & see also phf's point re designing p2pisms )
bitbot[asciilifeform]: Logged on 2022-12-25 12:57:51 phf[awt]: http://logs.nosuchlabs.com/log/pest/2022-12-25#1019348 << but on the other hand we don't want to discourage you writing experimental code, because unlike athena born as she was from the forehead of her father, distributed code has too much emergent behavior to attempt to reason it fr
dulapbot: Logged on 2022-12-25 12:47:42 asciilifeform: starts to think that in the process of posting this, reduced it to a trivial snoar. apologies to those who bored to tears.
asciilifeform: not even to mention the fact that awt independently conceived of ~80-90% of the thing and baked pre-pest proto , with only coupla ln of log to go from, back in '21
asciilifeform: ( in fact wrote & posted his 'alcuin' while asciilifeform was still sweating out 1st spec )
asciilifeform: in asciilifeform's lights, pest is an 'obvious, in retrospect' item, that 'could have existed in 1990s', but -- 'devil in details'.
asciilifeform: ... and many of these details, imho, can only be nailed down empirically.
asciilifeform: returning to this point, imho is major reason wai the luby thing aint optional luxury, but key moving part --
bitbot[asciilifeform]: Logged on 2023-01-29 12:42:39 signpost: world's looking like fragmented internet with intermittent access may be in our near future, too.
asciilifeform: conceivable that 'i have things with hashes h1, h2, ... h_n' 'i need h2' etc will at some pt be the primary, 'usenet-like' mechanism so to speak at all
cgra: http://logs.bitdash.io/pest/2023-01-29#1021672 << can easily agree. while though my brain-wiring is prolly in feather-weight class for this serious design biz
bitbot[asciilifeform]: Logged on 2023-01-29 13:44:18 asciilifeform[5]: ... and many of these details, imho, can only be nailed down empirically.
asciilifeform: fwiw the 'serious designers' gave us luldesigns like tcp.
signpost[cgra]: asciilifeform: I published the most recent revision to the python one. it's possible my dumb head is the reason why it's currently not able to do more than a coupla megabytes per second in transfer speed.
signpost[cgra]: that said, super-speed might not be priority number one. it does stand up to loss of packets as expected
asciilifeform: signpost: coupla mB/s as what fraction of the avail. bw ?
asciilifeform: (if bw was e.g. 100mb/s -- is pretty good)
signpost[cgra]: this is where encoder and decoder are sitting on the same box, so localhost traffic.
signpost[cgra]: bottleneck is the coder
asciilifeform: bottleneck is prolly the py proggy aha
signpost[cgra]: not saying I give up, just stating current status
cgra: signpost, =cpu-bound?
asciilifeform: imho that's actually pretty good, asciilifeform expected 100s of kB/s
signpost[cgra] running the profiler, sec
signpost[cgra]: yeah, it's not obscenely slow, but not saturating my pipe
cgra: signpost, was it the jetbrains thingy you're profiling with?
signpost[cgra]: yeah, but that doesn't give me a sense of memory churn, does it?
signpost[cgra]: open to suggestions on the proper way to instrument this thinger
signpost[cgra]: http://trinque.org/2023/01/18/ocpy_squirt_slurp/ << latest src
signpost[cgra]: one problem I have is that if I send too quickly, the receiving end misses some. I assume this is because I don't know how to write socket code properly.
signpost[cgra]: http://paste.deedbot.org/?id=zlJ- << e.g., sending with $cpuCount number of worker processes
cgra might try own hand on it tonight, to grasp better
signpost[cgra]: http://paste.deedbot.org/?id=qbNP << missed 3/4 on receving end
signpost[cgra]: (also measurements here are megabytes not megabit)
signpost[cgra]: http://paste.deedbot.org/?id=OZKZ << single-core send is received properly
signpost[cgra]: I set the recv buffer to larger than the file that's being sent, so I'm assuming there's pebkac going on with my socket code. would welcome anyone taking a look and seeing what I'm doing wrong.
signpost[cgra]: oh, I'm an idiot.
signpost[cgra]: /proc/sys/net/core/rmem_max limits what one can set on his socket.
signpost[cgra]: ok, found a few solvable problems. one's that I was a derp and didn't init a prng each post-fork. they're all getting the same state pre-fork, which ruins the distribution of check-block degree (number of edges from a check-block to its constituent message blocks)
signpost[cgra]: two is that the decoder doesn't eat udp packets fast enough, which was known. there's a barrier there in that python's not good at parallel code, and the decoder needs to share an mmap'd file among workers. advice welcome.
signpost[cgra]: I don't know that it's feasible to share access to the received file among processes rather than threads.
signpost[cgra]: I cranked net.core.netdev_max_backlog way up too, and now decoding works no matter how many workers are sending packets.
signpost[cgra]: seems like at this juncture it'd be better to jump over to C with this serving as experience, but if anyone has better advice to improve the python version that'd be mighty welcome, since blatta's also python
signpost[cgra] afk a bit
cgra: apparently no quick signpost ocpy try for me: i'm somewhat stuck with py3.5 and <=2019-12 scipy/numpy, and a couple of superficial code adjustments weren't enough. current show-stopper being scipy 1.4.1 (newest i can 'pip install') trying to use numpy.random.Generator.random_sample(), but not there. in numpy 1.16.3, there'
cgra: s not even numpy.random.Generator (versions tested chosen per various git history adventures and too numerous details)
signpost[cgra]: why are you trying to use older deps than are specified in requirements.txt ?
cgra: stuck with py3.5 atm etc older shit
cgra: thought i'd try my luck
signpost[cgra]: meaning no serious disrespect, this "I'm going to use imperial tech, but at a version I've sanctified by whatever ritual" thing is a wank.
signpost[cgra]: ocpy's an algo demonstrator, nothing more
cgra: imperial here meaning 'ubuntu', 'systemd' etc? old, new all the same
signpost[cgra]: yeah, meaning I don't see a lot of daylight between py27, py35, etc.
signpost[cgra]: python's foundational ideology is "yay Everyone can Program (TM)" since first version.
cgra: right
signpost[cgra]: would take patches that make the thing work on older versions I guess, but I think it's a waste of anyone's time.
signpost[cgra]: better to suss out if the algorithm's impl is sane-ish, then write the thing in an adultlang.
signpost[cgra]: probs time for the C version, which can be used by just about anything directly or by ffi
cgra: signpost, do you also find pointless sticking to older imperial softs in general? may as well upgrade OS when auto-update is pushed? or update only when have to? my old py is a product of the latter
signpost[cgra]: let's take the example of running blatta on py27
cgra: (to be clear, not suggesting signpost should go rewrite ocpy in py27)
signpost[cgra]: I don't recall what specific recv method it uses, but say it's this one
signpost[cgra]: y'all patching your py27 by hand?
signpost[cgra]: py3 versions are also susceptible, perhaps it happens not py3.5, but again, you're going to maintain this yourself?
signpost[cgra]: only way you actually get to where the affectation of "not going to upgrade" wants to go is to drastically cut away complexity which would include all of python.
signpost[cgra] wrote pentacle for this, and even there one can go "omg, there's a perl in it"
signpost[cgra]: at least in that case the perl isn't opening sockets
signpost[cgra]: "the master's ass graced this particular wordpress version. you can still see the crease and smell his cigars."
cgra: signpost, keeping thinking that letting the old, smaller pile steam, instead of increased & shuffled mass (~upgrade), is less bad. is this an obvious brain mis-wiring?
cgra got to hit the bed for a while. talk later
signpost[asciilifeform]: what's miswired is a bunch of d00ds all using different sanctified versions of these things.
signpost[asciilifeform]: barrier to rapid collaboration
signpost[asciilifeform] would sooner make sure ocpy works on dulap, for example, but you're on ???buntu
signpost[asciilifeform]: http://trinque.org/src/pentacle/build/ << or I would gladly target someone's python pbuild here.
signpost[asciilifeform]: but again, if it's going to be some old version, are you going to patch the thing's remote exploits, because in both blatta and ocpy we're talking about networking code, or what?
asciilifeform: signpost: old py had remote exploits in naked network stack ?!
asciilifeform fwiw has a working py2.7 errywhere, as that's what logotron & v.py were baked in
asciilifeform: on 1 box have py3.sumthing for ice40ism
asciilifeform: + whatever's on the crapple (nfi, prolly 3.x)
asciilifeform: http://logs.bitdash.io/pest/2023-01-29#1021739 << is part of wai asciilifeform set out to write a portable cpp pestron (deliberately constrained to cpp11, i.e. builds in gcc 4.9 & elsewhere)
bitbot[asciilifeform]: Logged on 2023-01-29 18:39:10 signpost: barrier to rapid collaboration
asciilifeform: (initially thought to glue the guism to jonsykkel's, but turned out to be over9000 gnarlier, interestingly)
asciilifeform: cpp11 aint ada, but did steal some of the civilized pheatures (e.g. for-each) and if written carefully, does help to keep a lid on rampant pointerism etc. and make for a readable proggy
asciilifeform has been testing each piece on crapple's compiler, sometimes uncovering & zapping implicit reliances on poorly-defined behaviours. will say more re subj later.
signpost[asciilifeform]: yeah, not criticizing cgra directly here, but making the broader point that relying on any of this shit is a tarpit.
signpost[asciilifeform]: and it's not less-tarpit by swimming in older tar.
signpost[asciilifeform]: http://logs.bitdash.io/pest/2023-01-29#1021747 << yep, seems sane, and I've got a gcc-4.9 builder cemented in place with minimum deps
bitbot[asciilifeform]: Logged on 2023-01-29 19:40:49 asciilifeform[4]: http://logs.bitdash.io/pest/2023-01-29#1021739 << is part of wai asciilifeform set out to write a portable cpp pestron (deliberately constrained to cpp11, i.e. builds in gcc 4.9 & elsewhere)
signpost[asciilifeform]: one's really signing up to own python if you intend to rely on an old version longterm
asciilifeform: signpost: asciilifeform's plan for python is (as discussed prev.) 'throw out the pythonisms', rather than 'care for it long-term' fwiw.
asciilifeform: ditto perl.
signpost[asciilifeform]: of exactly same opinion.
signpost[asciilifeform]: shitty thing about perl is all the autotools trash demands it. one doesn't want to use it for anything new, but didn't wanna pack gcc without the tools for all generated srcs
asciilifeform: fairnuff