asciilifeform: ACHTUNG trb operators! loox like all trb nodes known to me are presently wedged (at 618405)
shinohai: https://www.blockchain.com/btc/block/618405 <<< ~1.32 MB, likely lots of segshit in there.
asciilifeform: classical black hole condition, apparently, and both of mine ate 618406 when reset. still puzzling tho, not seen this sorta thing for past 2+ yrs
asciilifeform: shinohai: this has 0 i
asciilifeform: impact on trb normally
asciilifeform: shinohai: trb dun eat >1e6byte blox under any circumstance
shinohai: Being that ViaBTC mined it, I'm surprised block not empty
adlai: iirc, "prb" was poweranger, not publicrelations
asciilifeform: adlai: i seem to recall both. cant recall which earlier.
asciilifeform: anyways -- my 2 now running (but no peer seems to have 618412 atm) . all others in my 'addnode' table still bricks.
asciilifeform: last thought about this 2y ago, but afaik the only practical means to actually debug hypothetical 'magic mempool turd that wedges' would be to keep a ring buffer of all traffic (can reset when successful block eat, erry time) and dump to disk on sigkill
asciilifeform: mod6 ^
curiousd0g: no scaling problem?
asciilifeform: curiousd0g: nope. spam problem, since '15, periodically tho
curiousd0g: solution?
asciilifeform: curiousd0g: prolly variation on malleus theme
mod6: asciilifeform: yeah, my node seems to be 'blackholed' at the moment. investigating before a restart here.
asciilifeform: mod6: capture the log.
mod6: as to your thought on debugging, will think on it.
mod6: while I'm at it, did anyone try out those three new vpatches? I've been running with them just fine on test nodes.
shinohai likes being in the derpy alt-bitcoinwizards chan.
snsabot: (trilema) 2019-12-26 mp_en_viaje: in other housekeeping, hey diana_coman lobbes wouldja mind taking alf's derpy alt-bitcoinwizards off the loggers header list ?
asciilifeform: mod6: note that i've no evidence for 'magic packet'. could be easily race condition that pops up erry n yrs..
asciilifeform: shinohai: why ty
curiousd0g: "He who comes to us with a turd, by the turd shall perish."
curiousd0g: lol
mod6: asciilifeform: my log looks like your log. will provide if you really want to see tho.
shinohai: mod6: I hopefully can try 'em this weekend, it got put on back burner last weekend had a sick concubine to care for.
mod6: no worries shinohai
asciilifeform: mod6: initially had 4 hypotheses ( 1) magic tx in mempool 2) magic liquishit administered via plain tcp directly to noad 3) peculiarity in block 618406 4) failed reorg in vicinity of 618405 ). elim
mod6: but do let me know if/when they work for ya.
asciilifeform: eliminated 3 and 4
asciilifeform: ( add 5th -- race condition , for co
asciilifeform: completeness )
mod6: yeah, i mean, getting blackholed is nothing new. seems like what it is. that's what I would have thought if no one would have said anything.
asciilifeform grr will have to swap kbds
asciilifeform: mod6: i dun recall seeing hard wedge on my nodez since early '18
mod6: quick snippit of my one log looks like this: http://paste.deedbot.org/?id=qWkx
mod6: i guess generally speaking, if i let this just figure itself out, it usually does. just may take some time. hours usually.
asciilifeform: mod6: typical picture as seen during prolonged block verify. except, continues for hours.
mod6: ya.
asciilifeform: mod6: i saw last night, decided to hold off ringing the bell until wake up (if still wedged)
mod6: I think I'm going to restart my main node. And I'm gonna leave my test node in this state. See what happens.
asciilifeform: aite. ty mod6 . lemme know what you find.
mod6: Sure thing.
asciilifeform: i reset both of mine, they are presently operating ( tho sadly lacking 618412 atm )
mod6: LoadBlockIndex(): hashBestChain=0000000000000000000358b1f55d0fa15d185693974f0fa1dc3249109151f2b0 height=618405
mod6: (on restart)
mod6: So yah, had 618405, was waiting for, or trying to digest 618406
asciilifeform: mod6: both of mine ate through 618411 , currently operating normally (aside from no one offering 618412 to'em of yet)
asciilifeform: ( that is, after reset )
asciilifeform: wedge took place immed. prior to 618406 .
mod6: SetBestChain: new best=0000000000000000000686f8b07d684d9950f9b017ae1d9774de18c861ccb095 height=618413 work=3994228844468425383327109680
mod6: ploughing right back up.
asciilifeform: imho mempool ops really oughta have a timeout.
mod6: i saw the discussion before about txs that sit, often 0 fee. you mean like that?
asciilifeform: (in fact all ops oughta have a 10min timout. on adequate iron, no block verify ever exceeds the block interval. )
mod6: time out after 2 weeks.
asciilifeform: no
asciilifeform: timeout for externally-received command
mod6: oh. for msg ops.
asciilifeform: aha
mod6: maybe so. i'd like to get to the bottom of this weird issue. i kinda see it from time to time. maybe not as broadly impacting as this one tho.
asciilifeform: mod6: when was last time you saw >12h wedge ?
asciilifeform: i haven't since '18
asciilifeform: and can't recall a 'all known trbs hanging' since july of '15
mod6: heck, i see this kinda thing ... often. log looks just like it does (where I pasted above). thing doesn't respond to RPC commands.
asciilifeform: ( the latter was a reorg . whereas this, nao -- no reorg )
mod6: needs restart.
mod6: often tho, just seems to sort itself out after some time.
asciilifeform: mod6: often?! and never said?! tsk
mod6: i mean, more than since '18, sure. I typically think of these as 'blackholed'. I don't block ips quite like as encouraged on this one particular node. although, i should.
mod6: but i kinda like it to be beaten harder. per-se.
asciilifeform: mod6: the ips visible in the log aint in any way interesting, necessarily. you get same picture (folx disconnect) whenever verification aint working (because that thread wedged)
mod6: right
asciilifeform: yer seeing whatever peers you had at the time of wedge
asciilifeform: ( and not necessarily inclusive of the 1 who served you the hypothetical wedge cmd . that 1 could have long disconnected )
mod6: right. usually by the time I see it too, it's already been going on for a bit. hard to pinpoint.
mod6: never saw this thing since 18 with dulap?
asciilifeform: folx try to connect to wedged node, you get a 'accepted connection x.x.x.x:x'; then socket no message in first 60 seconds, 1 0
asciilifeform: mod6: nope
asciilifeform: mod6: and since '15 not once such that all known nodes were in this state simult.
mod6: hm. see I don't think it has anything to do with the block. which is why I try to stay away from 'wedged', unlike the other circumstances where we've said 'wedged'. i forget the block numbers off hand.
asciilifeform: not to do w/ 618406 at any rate
mod6: there've been a couple of cases where we've all been digging to figure out what the issue was.
mod6: right.
mod6: could be a heathen command tho. your dumping the cmds idea could be a helpful tool here.
mod6: (in debug moad)
superkuh: Fucking apple. Setting their browser to declare any TLS cert with a lifetime of greater than a year to be invalid. Corps gonna corp.
asciilifeform: mod6: well not 'heathen cmd' in the usual sense of 'named cmd unsupported by trb', these get malleus'd
asciilifeform: superkuh: whatcha doing on crapple box ?
superkuh: I'm not.
superkuh: But I do see apple users in my site logs.
superkuh: And they'll be gone now. Not a huge loss given apple users, but still.
asciilifeform: superkuh: take the next step and nix sslism.
mod6: asciilifeform: well 'magic turd' in there. what have you.
asciilifeform: mod6: supposing not simply race condition.
mod6: i guess i'd be surprised if that was the problem, a 'magic turd'.
mod6: perhaps, yeah, could be some sort of weird dead-lock.
asciilifeform: mod6: race cond. seems most likely imho. if 'magic turd' existed, would be served up to all trb daily, i suspect.
asciilifeform: trb really oughta have 'watchdog timer'
mod6: same. this seems to be something more rare. like I said, I don't see it all the time. but certainly not years between either.
asciilifeform: could be as simple as 'no new blox for 2h? graceful reset'
mod6: (could just be my noad too, fwiw, doing something weird)
mod6: now there's a helpful thing too. not sure I'd do it 2h's but, yeah, perhaps some interval.
asciilifeform: user knob
mod6: (im up to 618529 nao)
asciilifeform: a++
asciilifeform: mod6: i'ma leave this alone until nightfall, lessee
asciilifeform: mine also picking up blox atm
mod6: yeah, my test node is still derping. will write in when it recovers. won't wait forever, but may wait a few days to see.
asciilifeform: other folx appear to remain in wedge ( trinque et al )
asciilifeform: wonder how long until they notice.
asciilifeform: the main argument against race condition hypothesis, actually, is that all known (to asciilifeform) trb stations affected at afaik same moment
mod6: agreed.
asciilifeform: typically race cond. does not behave this way, but rather a 'planets align' thing
shinohai: http://logs.nosuchlabs.com/log/asciilifeform/2020-02-22#1007303 <<< will notice when they are given permission to look. xD
snsabot: Logged on 2020-02-22 11:15:24 asciilifeform: wonder how long until they notice.
asciilifeform: !q seen-anywhere trinque
snsabot: trinque last seen in #trilema on 2020-02-08 02:29:49: you're a cunt's dumpster, woof.
mod6: its possible that 406 just took a long time to verify for some reason, which causes a cascade of other issues that render it derped for a bit.
asciilifeform: shinohai: that 1 might notice , given how often logs in, some time in summer, lol
shinohai: kek
asciilifeform: mod6: took about 40s to verify on .27 (a slow, comparatively, box, 'apu1')
asciilifeform: when unwedged it, that is
mod6: yeah, fair enough, once restarted, verified very quickly.
asciilifeform: mod6: i considered timeout as early as in '15, but was loathe to bake it, given as it is 'symptomatic cure'
mod6: yeah, i get that too. this seems like it might be 1 problem or perhaps more likely a confluence of issues that build up at the right time. i'd rather fix that shit, than give it a pill.
mod6: only way to find out, really, is just to debug and wait for the issue to arise, then disect.
asciilifeform: mod6: the tricky bit is that in order to fix item like this in the proper sense, gotta reproduce on-demand
mod6: well, once one has an idea of wtf, then yes, can reproduce at will. otherwise, tilt at windmills.
asciilifeform: entirely possib. that it wont recur 'naturally' for another 2y...
mod6: to >1 trb node, this could be true.
mod6: in this case.
asciilifeform: BingoBoingo yer still wedged, btw
BingoBoingo has trb node 143.202.160.10:8333 sitting at blockheight 618530 at present, but sits on version string 70001
BingoBoingo: asciilifeform: Most of mine wedged, yes
asciilifeform: BingoBoingo: was speaking of what it was you had in the rack aha
asciilifeform: BingoBoingo: just ver string ? or is that actually yer pre-trb experimental thing
BingoBoingo: asciilifeform: Just version string. It's the rotor/stator recipe baked to give the nameservers something to do with their cycles.
asciilifeform: BingoBoingo: do you have dumpblock in there ?
asciilifeform: BingoBoingo: be sure to save the log before resetting any of your wedged ones, also
asciilifeform: ( grab the past 48hrs if you can )
asciilifeform: mod6: atm both of mine at 618417. not wedged, but also not yet seen next block from that.
BingoBoingo: I don't recall if I included dumpblock in the press, Will see if I can get anything interesting out of the logs.
asciilifeform: i'ma leave'em for a bit, see if climbs
mod6: hmm. ok. keep an eye on it. im on 618530
mod6: which now, looks like a few blocks behind. 618532 has been announced a few minutes ago.
BingoBoingo: mod6: Which version flag is your node flying and does it sit on port 8333?
mod6: setvernum=50400
mod6: I took it offline a minute ago tho.
adlai: http://logs.nosuchlabs.com/log/asciilifeform/2020-02-18#1006913 << i find your prose verbose; long paragraphs jar the eyes of seasoned lispers, so consider using short phrases and headings (similar to org-mode) to increase the chance that your prose actually gets read
mod6: Will restart it in a minute.
adlai intended to link a different line from that same conversation, although there is no need for a correction since the conversation should probably be read along with the article
asciilifeform: adlai: my objection there is a bit different -- just how many loc (and of what) in cl did that fella write, to then 'i'ma propose style guide' ?
asciilifeform: adlai: i wrote maybe 100kloc of cl and do not consider self qualified to tell others 'how to style' in it
adlai rarely has any disagreement with asciilifeform, and in this case, no objections to your objections !
adlai: especially, the bit about lispers not caring even one byte specifier about style
adlai: the word 'lispers' here refers to those who actually have to write and maintain the code; managers of teams of programmers are likely to care quite a bit about the optics.
asciilifeform: adlai: there's something to be said for consistent styling (for instance, can auto-conform via emacsism)
adlai: consistent styling also leads to fingerprinting.
asciilifeform: but it is 'cultural' problem, no one man can propose to solve it (unless he writes mega-opus that erryone in fact considers 'foundational' and reads, imitates, as the germans still imitate goethe )
asciilifeform: adlai: lolwat?
adlai: verisimilitude: thank you for the reminder about using loop keywords from the keyword package. this is a habit that i have yet to adopt, although i probably should.
adlai: fingerprinting, i.e. "this code smells like nakamoto"
asciilifeform: adlai: you may've noticed that asciilifeform and erryone he worked/works with, in fact signs his proggies
asciilifeform: i'm quite uninterested in whatever suffering anons face from their idiocy
adlai: you probably work very few net meters of cardboard/cement/etc from cubefarms were people scrub signatures before deploying code.
adlai: possibly even forging signatures and fingerprints to implicate their false flag of choice.
adlai: my guess is that it's much more fun, in the long run, to be able to claim your work as your own, and the suffering anons who can't, gotta console themselves with... dunno, salaries? conference chicken? etc
asciilifeform: adlai: lulzily, today's gen of virii artists often forgets even to remove debug symbols from built bin
verisimilitude: I appreciate your thoughts, adlai.
asciilifeform: ( yes, incl. usg's )
verisimilitude: I wrote my style document, because a fellow in my group wanted to read some, and I was lacking writing ideas.
asciilifeform: verisimilitude: there's nothing wrong w/ writing such thing. but i wanted to state why i would not .
verisimilitude: I neither expect to see nor care if anyone uses my style described therein.
asciilifeform: adlai: and at any rate src styling has 0 to do w/ 'make iran think nk wrote stuxnet' biz
adlai: eh, nobody cares about centrifuges anymore; these days the cool kids are all about that other kind of 'TMSR'
asciilifeform: adlai: is example.
verisimilitude: With no disrespect, adlai, I'm not particularly interested in changing my writing style so. I strive to avoid some circumlocution in my writing, but I'd rather not use headers and whatnot. I write for a very particular constraint, which would've been obvious if you'd seen the document before it was converted to HTML, and I'm uncertain if you'd.
adlai: lemme put it this way: your style guide seems written for recitation to students, rather than quick glances by a working programmer
verisimilitude: That's a good point; I suppose it's not written for quick glances, no.
adlai: (and there is no problem with this! provided that you can find any students who still want to learn lisp...)
adlai: )
curiousd0g: )
adlai: In other news, I do often wonder as to just how amazingly accurate asciilifeform was in once calling the people treating mp, dc, et al as their 21st century Elizae the 'patients'
asciilifeform: adlai: hm?
adlai: also, congratulations are in order for BingoBoingo, both for the birthday, and apparently for turning #ossasepia into another chapter of AA!
shinohai: lmfao
asciilifeform: i always assumed the 'aa' flavour there were deliberate thing
shinohai: Except step 3 isn't "God as you understand him" but "thou shalt have no other deity sans MP".
asciilifeform: see also : 'i was all fucked up on drugs. but then i found jesus. then i was all fucked up on jesus. but it's ok, i'm taking drugs for it'
adlai: http://logs.nosuchlabs.com/log/asciilifeform/2020-02-22#1007379 >> http://logs.nosuchlabs.com/log-search?q=from%3Aasciilifeform+patients&chan=trilema
snsabot: Logged on 2020-02-22 12:11:40 asciilifeform: adlai: hm?
asciilifeform: adlai: but what of it ?
adlai: shinohai: nonononono; "thou shalt have no other used car salesman but pete"
shinohai: pete_d has pretty nice cars, I'd deffo but his shared Lotus if it isn't too ragged out.
adlai: asciilifeform: well, mp claimed in one of the bits of legalese about "bitcoin as game currency, mpex as recursive game" that he's insane; yet look at that parade of 'normal' folks, just lining up to have him and his cohort set them straight about their troubles. it's almost as though the crazy are sane, and the sane -- crazy!
shinohai: s/but/buy
asciilifeform: adlai: understand, this is the biz of folx attending the church, not mine
asciilifeform: atm much moar interested in e.g. potential race wedge in trb, then in what happens in vatican
adlai goes back to eating oxford's ru-en+en-ru via retina
asciilifeform: adlai: enjoy
asciilifeform will bbl.
mod6: Node is back up, on block 618539
adlai: regrettably their idea of "dictionary" is almost entirely devoid of grammatical reference, so recommendations of particularly good summaries of the usage (either online or in print) would be much appreciated
adlai meanwhile starts crawling thru wiktionary
mod6: My node has fallen 4 blocks behind now, lot more of this in my debug.log: http://paste.deedbot.org/?id=-Rlj
adlai: ah cool, it has finally started to catch up entries from urbandictionary: https://en.wiktionary.org/wiki/Russian
mod6: So, ok, restarted my node yet again. This time in debug=true mode.
mod6: Was caught up, now fell behind. Here's something that ~might~ be of interest...
mod6 gathers up logs
mod6: http://paste.deedbot.org/?id=Oj7N << asciilifeform thoughts?
mod6: that address 138.197.207.236 is always getting disconnected directly after the flood control message -- a number of times in my 12Gb debug.log file.
mod6: anyone turn anything else up from that IP that looks "interesting"?
mod6: http://paste.deedbot.org/?id=l08E << Here's a grep of my log for that IP 4 lines before and after found. Might be just a version 70013 trying to sync. Timing seems odd to that flood control though.
mod6: Found basically ~same thing on test node with regard to behavior. Different IP's syncing. This time tho, 122 of the 'getdata' commands came in, then a flood control message, then ~32,161 more.
mod6: Then the weirdness/wedging seems to have started.
mod6: If anyone sees the same things in your debug.log, please write in.
mod6 been looking at this for some time now. Need food. bbs.
mod6: back around. will keep checkin' back.
shinohai: "Astounding work there asciilifeform, until I put you on shitlist then everything you ever made doesn't matter anymore"
shinohai: The cognitive dissonance of that lot amazes me "because mp says".
feedbot: http://qntra.net/2020/02/twitter-bans-bloomberg-campaign-deputy-field-organizers-for-platform-manipulation/ << Qntra -- Twitter Bans Bloomberg Campaign "Deputy Field Organizers" For "Platform Manipulation"
asciilifeform: mod6: has it occured to you to compare to usual debug output ? you aint wedged. not getting blocks as fast as you like cuz yer connected to a bunch of folx who ~are~. ( if yer node accepts e.g. getinfo cmd, you aint wedged )
asciilifeform: shinohai: lol 'ispless'
asciilifeform: shinohai: y'know, the trollage is for me, but for some reason it's always you who bites, lol
asciilifeform: BingoBoingo: 'great' job 'trb nodes have been wedged' w/out reporting who clued you in or linking to thrd. ( maybe next time will see how long it takes to notice, instead of giving what to crib? ) and et tu, mod6
snsabot: (ossasepia) 2020-02-22 BingoBoingo: In other news, a number of visible trb nodes have been wedged for some time.
asciilifeform wonders whether he's the only 1 who actually fucking watches the nodes
asciilifeform: adlai: ever considered doing something useful? even sumthing very simple. e.g. where's yer node ?
mod6: asciilifeform: I haven't gotten a new block since ~1pm on my last restart.
mod6: So, I'll clean my log, and restart again. It'll probably catch up a bit, but then we'll see.
mod6: Granted, could be just doing something wrong. I dunno. From what you're saying to me, sounds like your nodes are back to working 'fine'?
asciilifeform: 205.134.172.27 in my cage is at 618593
asciilifeform: the desk noad still behind, but receiving blox
mod6: *shrug* we'll see how it goes. just started my node. height = 618545 currently.
asciilifeform: mod6: if you recall, the ancient 'dun connect to >1 from same /24' logic is still in trb. so i rec to add .27 to yer addnode list. that way will get blox from mine even when peers end up on m
mod6: if it keeps up, great! if not, will publish debug.log.
asciilifeform: ... on mine instead
mod6: Ok.
asciilifeform: mod6: other thing , i rec to review what 'ordinary' debug log liquishit looks like
asciilifeform: we used to do this regularly, mod6 , you'd see a wedge and 'read clouds' , forgetting that the log ~always~ contains large amt of very similar liquishit
mod6: yeah, well, was looking for something that might point to why the fuck i wasn't getting any blocks.
mod6: well, it ~was~ getting blocks, upon immediate restart. then would promptly fall behind, and never get another, meanwhile not accepting any RPC commands.
asciilifeform: mod6: if you aint wedged -- then if aint getting blox, it is because other folx aint sending'em.
asciilifeform: ah so wasn't accepting rpc ?
mod6: no.
asciilifeform: next time you see this, mod6 , post the log plz.
mod6: just call 'getinfo' or 'stop' and just hang there in term.
asciilifeform: that, there, is wedge.
asciilifeform: and tcpdump -nnXSs 0 port 8333 plz .
mod6: i posted the log above, after the 'getdata' onslaught, RPC hang.
mod6: ok.
asciilifeform: mod6: this one ?
snsabot: Logged on 2020-02-22 15:00:46 mod6: http://paste.deedbot.org/?id=Oj7N << asciilifeform thoughts?
asciilifeform: i dun see any traces of attempted rpc in there
mod6: let me dig it up
mod6: need a few minutes
asciilifeform: mod6: i'ma bbl tomorrow.
asciilifeform: until then, rec to 'addnode' as described above, an' let it live
mod6: That doesn't show you much, but it does show that I issued the command after the wave of 'getdata'. It hung there for ~8 hours until I finally killed it off like about 10 minutes ago.
mod6: alright, have a good night.