Hide Idle (>14 d.) Chans
← 2022-01-11 | 2022-01-13 → ↓
jonsykkel[asciilifeform]: whaack: peer'd. didnt see until now
whaack[billymg|asciilifeform]: jonsykkel: damn, seems to not be working tho
whaack[asciilifeform|billymg]: last 3 people who tried to peer with me failed
whaack[asciilifeform|billymg]: jonsykkel: just checking, the port you have for me is 7780?
whaack[asciilifeform]: i can see from my handles table that 1-4 are peered but 5 is not peered, then 6-8 are jonsykkel, mod6, and peterl, the people who couldn't peer with me
whaack[asciilifeform]: when i run 'ss' i don't see the port 7780 in use, seems like there's a possibility that there's a thread that listens to new connections that has since died, but iirc there is no threading in blatta atm
whaack[asciilifeform|billymg]: hrm, nvm, netstat -ul shows i'm listening for udp packets on port 7780, and i must be, since i'm receiving msgs from my peers
whaack[asciilifeform|billymg]: mod6, jonsykkel, and Peterl: I am getting martian packets from 95.216.203.141:13337, 157.230.32.221:7778, and 162.247.151.243:55573 - do these belong to you?
whaack[billymg|asciilifeform]: as far as i can tell, there is something going wrong with the encryption/decryption
billymg: i had 55565 for PeterL, just updated
billymg is in a similar boat to whaack, peering mysteriously works for some but not others
billymg: i manually went through all my tables in the db yesterday and couldn't find anything that looked out of the ordinary, or any differences between the peers that work and the peers that don't
whaack[asciilifeform]: billymg: which peers are not working for you?
billymg: jonsykkel, PeterL, shinohai (and we've tried peering twice), signpost, mod6
jonsykkel[asciilifeform|billymg]: 95.216.203.141:13337 is me, yep
asciilifeform: billymg: pretty strange, asciilifeform is successfully peered w/ all of these (and he's behind nat, tho did need to set up an incoming fwd rule)
whaack[asciilifeform|billymg]: asciilifeform: per my investigation it seems like the problem is that the packets are not able to be decrypted
asciilifeform: billymg: iirc checked yer clock ?
asciilifeform: (imho debug log oughta indicate if rejects on stale. does it?)
billymg: yeah, not the clock
asciilifeform: for that matter billymg peered w/ asciilifeform , so clock ruled out
awt[asciilifeform|billymg]: if it were the clock there would be a corresponding error message
asciilifeform: billymg: yer on 9983 ?
billymg: yep
asciilifeform: billymg, whaack : how didja determine that packets from the given peers in fact reach billymg's station and then get tossed as martians?
billymg: asciilifeform: whack's talking about his own station, he's also having trouble peering with some people
awt[billymg|asciilifeform]: whaack: I will add the key you pasted the other day shortly
billymg: it's possible i have a different problem, yesterday searched logs for shinohai's IP, found only entries where i'm sending packets to him
billymg: no martian packets received *from* shinohai
asciilifeform: billymg: and you verified that you have valid AT entries for ea. other ?
billymg: yep
billymg: checked all tables in my db manually
bitbot: Logged on 2022-01-12 14:27:59 billymg: i manually went through all my tables in the db yesterday and couldn't find anything that looked out of the ordinary, or any differences between the peers that work and the peers that don't
billymg: and this is rather fresh db too, blew away and started new a few days ago
whaack[billymg]: asciilifeform: i'm only reporting my findings, not billymg's, i have log lines 'discarding x as martian' from certain peers, such as jonsykkel
asciilifeform: a so possib. 2 entirely diff bugs
asciilifeform: seems to asciilifeform that in both cases, blatta needs moar detailed max-verbosity debug output
whaack[asciilifeform|billymg]: asciilifeform: yes
whaack[billymg]: to be fair it does have lots of logging
awt[asciilifeform]: whaack I peered with your new key
whaack[asciilifeform|billymg]: awt: hm and it appears still no luck
asciilifeform: i.e. oughta be able to see readily whether packet received recently from $peer, and if rejected, why
awt[billymg|asciilifeform]: I'm still not getting immediate messages from you
asciilifeform: from $addr rather
billymg: ah, so for jonsykkel, am seeing martian packets
billymg: (yesterday only checked for shinohai since we recently rekeyed)
billymg: from PeterL - no martian packets, from signpost - no martian packets, from mod6 - no martian packets
asciilifeform: billymg: which peers do you have succesful peerings with atm ?
billymg: in all cases logs show that my station is sending packets to them
billymg: successfully peered with: awt, asciilifeform, bitbot, whaack
asciilifeform: is errybody currently on 9983 ?
asciilifeform: and billymg
asciilifeform: anyone isn't ?
awt[asciilifeform]: whaack: I'm not getting any rubbish messages from you
whaack[asciilifeform|billymg]: awt: that makes sense, you peered using my info, and i haven't received any msgs successfully from you, so how would my station even know where to send you msgs
asciilifeform: whaack: you can try adding him to your AT
whaack[billymg|asciilifeform]: asciilifeform: i will try shortly, but i want to first see if i can find out the bug
asciilifeform sadly not had time for a deep dig into blatta src w/ own hands just yet
asciilifeform: not since coupla versions ago at any rate
awt[asciilifeform|billymg]: whaack: did you comment out the if clause in update_at on 218?
asciilifeform ftr not yet applied the adhoc AT patch
awt[asciilifeform|billymg]: the logic there is busted and prevents at from being automatically updated
asciilifeform: hm i thought this only affected the timestamps
asciilifeform: (rather than the addrs per se)
awt[asciilifeform]: nope no fields are updated
whaack[billymg|asciilifeform]: awt: looks like i don't get any packets from you at all
whaack[billymg|asciilifeform] thinks he may have found the bug
whaack[asciilifeform|billymg]: awt: in station.py, you have a function is_duplicate which checks to see if a peer already exists in a current list of peers, the way you check to see if peers are the same is you check the address and port, but for peers that have not made 'first contact', the address and port are always set to 'None'
awt[asciilifeform]: whaack: yes signpost posted a patch for this
whaack[billymg]: so this likely why i had no trouble adding peers when i was doing the job one by one, but when i dispatched an invite to 4 peers, i started having issues
awt[asciilifeform]: Also I have logs of outgoing messages to you: DEBUG 2022-01-12 07:34:54,274: [whaacksip:6670] <- 3001237d76c66f2b
awt[asciilifeform|billymg]: will paste full ip if you don't mind
bitbot: Logged on 2022-01-08 00:13:26 signpost[billymg]: billymg: here's a draft patch http://paste.deedbot.org/?id=wQJn
whaack[asciilifeform|billymg]: awt: aye did you get my new msg? my port is 7780
billymg has that patch applied on his station already ^
awt[billymg|asciilifeform]: ok whaack you should be getting rubbish from me at 7780 now. have yet to receive an immediate message from you
whaack[asciilifeform]: awt: yes, and did you add the key i sent you in the new msg as well?
awt[billymg|asciilifeform]: whaack: key starts with: G3gcLMF
PeterL[asciilifeform]: awt: that post from signpost looks like it expired, do you have the patch available?
bitbot: Logged on 2022-01-09 18:11:38 whaack[asciilifeform]: awt: http://paste.deedbot.org/?id=YAU2
awt[billymg|asciilifeform]: whaack: ok will update
awt[billymg]: ok updated
PeterL[asciilifeform|billymg]: awt: that one didn't load for me either?
awt[asciilifeform|billymg]: PeterL: both work on my end
whaack[billymg|asciilifeform|asciilifeform]: howdy, just restarted my blatta,
whaack[billymg|asciilifeform]: oh hey awt: we're peered
awt[billymg]: ok whaack now I'm getting immediate messages from you
PeterL[asciilifeform]: awt: thanks, that one worked
whaack[asciilifeform|billymg]: peterl: we're peered now as well
whaack[billymg|asciilifeform]: i'm pretty certain that the bug that signpost posted a fix for was the problem for me
whaack[billymg|asciilifeform]: awt had the lowest pid and had the wrong key for me, and it was only the peers below him in the table that weren't working
awt[asciilifeform|billymg]: hopefully I can incorporate these various fixes into a release when I get back from CR later this month.
PeterL[asciilifeform|billymg]: hey, look at that!
billymg: jonsykkel: i just noticed i don't have a key for you, can you try adding this one? http://paste.deedbot.org/?id=XPiD
PeterL[asciilifeform|billymg]: billymg: I think you are the only one I have not managed to connect with yet that I have a key for
billymg: for the other users that i haven't yet successfully peered with i have: PeterL -- "G9qAwv"; signpost -- "OqlnKL"; mod6 -- "3j6a8x"; shinohai -- "KuU4cz"
billymg: PeterL: does that key look right for you?
billymg: (although i don't suspect is the key causing my issue for anyone but jonsykkel, since he's the only one i've received martian packets from)
billymg: it's*
billymg: for those i can't peer with, do your logs show that you're sending packets to me at least? (205.134.172.29:7778)
PeterL[asciilifeform|billymg]: billymg: key looks right
billymg: PeterL: and your station is sending packets to mine?
PeterL[asciilifeform|billymg]: hmm, I have a different port for you
billymg: aha
billymg: yeah, possibly in early days i was giving out my IRC port, not the UDP
billymg: PeterL: i got a packet from you now
billymg: looks like that was it
PeterL[billymg|asciilifeform]: ok, I updated my port, fixed!
PeterL[asciilifeform|billymg]: so to make blatta work with others (we decided that the other commands were causing it to crash, right?) would we just patch infosec.py line 219 to treat everything besides DIRECT and BROADCAST as having IGNORE command, or would we have to change somewhere else too?
jonsykkel[asciilifeform]: billymg: added
whaack[asciilifeform|billymg]: nice jonsykkel, looks like we're peered
whaack[asciilifeform]: great to see <-> implementations working
jonsykkel[billymg]: cool, restart is what did trick?
whaack[billymg]: err problem was from a bug i found that was first found by signpost
asciilifeform: soo, anyone still have a mysteriously failing peering? or does this cover errything?
whaack[asciilifeform|billymg] has successfully peered with erry1 he tried to.
asciilifeform: and billymg ?
signpost[asciilifeform]: looks like not yet peered with billymg, but we should retry from the top. billymg lmk when available to give it a go
signpost[asciilifeform]: tldr: yes, NIH et al made sars-cov-2, as we all knew.
signpost[asciilifeform|billymg]: https://archive.is/RZhXx << corporate press already in gear
signpost[asciilifeform]: and I'm sure the exhaustion mechanism I wrote about on my blog just days ago means precisely fuck-all will happen. the cows by now believe "of course we lied about it!1!!"
billymg: jonsykkel: i appear to be peered with you now, nice
billymg: that just leaves signpost, shinohai, and mod6
billymg: signpost: i have a key beginning in "OqlnKL" for you, and according to my logs i'm sending packets to you, but don't see any received from the ip:port that i have for you
billymg: in your logs do you see any packets being sent to my station (205.134.172.29:7778)?
signpost[asciilifeform|billymg]: billymg: believe I see me sending to that, but I don't see packets coming from it
billymg: signpost: your port is 7778?
signpost[asciilifeform]: yep, and ip 96.43.130.234
billymg: yeah, that's what i have
billymg: and `/wot signpost` yields that key from above
signpost[billymg|asciilifeform]: do you have martian packets from me?
billymg: nope
signpost[asciilifeform]: just tried to send you a DM
billymg: yeah, nothing in logs
billymg: only outgoing packets *to* that address:port
signpost[billymg|asciilifeform]: yeah I have the key you listed too
signpost[asciilifeform|billymg] mucks around in the sqlite deeb
signpost[asciilifeform]: nothing weird about your db entries compared to others
billymg: signpost: likewise, i checked yesterday every table in my db and found nothing different between the peers that work and those that don't
signpost[asciilifeform|billymg]: made sure I had the IP and port exactly right via search through blatta logs
billymg: signpost: wanna try a new key? http://paste.deedbot.org/?id=GZfd
billymg: just stabbing in the dark at this point, not sure what it coudl be
signpost[asciilifeform]: billymg: removed the old key and added that one
billymg: could*
signpost[asciilifeform]: still seeing ya via bounces atm
signpost[asciilifeform|billymg]: restarted my blatta too for shits and giggles
signpost[asciilifeform]: http://paste.deedbot.org/?id=riGb << looks like it's trying to send you that packet
billymg: lemme try that too, one sec
billymg: yeah, i've got entries like that as well
billymg: e.g.
billymg: INFO 2022-01-12 23:14:39,330: [96.43.130.234:7778 signpost] <- lemme try that too, one sec 0 2719a77678cf55bcc6f673f057a00d26a0cfe7ecf25aa2d2a3e9160d1354da32
signpost[asciilifeform|billymg]: yep might need to attach a debugger to this thing to figure out what's going on
billymg: ok, restarted
billymg: still nothing, but that's fine, can try another day
signpost[asciilifeform]: yep, probably good that there's a reproducible bug here to inspect
signpost[asciilifeform|billymg]: I'll get intellij's python debugger attached to this thing later and see what it's stepping through.
whaack[asciilifeform]: billymg: interesting, i got the msg 'ok, restarted' through asciilifeform, although all the other msgs from billymg i get directly
billymg: whaack: how about this one?
billymg: your message just now came through directly btw
whaack[asciilifeform]: yeah it was just 1 packet seemed to have been drop
bitbot: Logged on 2022-01-12 21:29:02 signpost[asciilifeform|billymg]: meanwhile lol @ us.mil finally leaking this https://assets.ctfassets.net/syq3snmxclc9/2mVob3c1aDd8CNvVnyei6n/95af7dbfd2958d4c2b8494048b4889b5/JAG_Docs_pt1_Og_WATERMARK_OVER_Redacted.pdf
asciilifeform: http://logs.bitdash.io/pest/2022-01-12#1002644 << we're gonna need some means of identifying keys for purposes like this w/out leaking bits. possibly e.g. crc32(sha512(key)). will amend spec after folx comment.
bitbot: Logged on 2022-01-12 22:38:51 billymg: signpost: i have a key beginning in "OqlnKL" for you, and according to my logs i'm sending packets to you, but don't see any received from the ip:port that i have for you
asciilifeform: ( prolly obv imho, but nobody but the 2 peers who are using a key have any biz knowing even leading N bits of it )
asciilifeform: oughn't to commit the idjicy of e.g. gpg (to use this string for anyffin internally) but for 'do you have key X?' oughta suffice imho.
signpost[asciilifeform|billymg]: http://logs.bitdash.io/pest/2022-01-13#1002686 << yeah, could easily interpret as cover for something even more inept/idiotic having happened.
bitbot: Logged on 2022-01-13 00:11:30 asciilifeform: http://logs.bitdash.io/pest/2022-01-12#1002638 << nfi whether authentic, but lulzy
signpost[asciilifeform|billymg]: http://logs.bitdash.io/pest/2022-01-13#1002688 << seems reasonable to me. and defo considering any keys with bits in logs trash that'll be rolled to new keys after debugging
bitbot: Logged on 2022-01-13 00:24:47 asciilifeform: http://logs.bitdash.io/pest/2022-01-12#1002644 << we're gonna need some means of identifying keys for purposes like this w/out leaking bits. possibly e.g. crc32(sha512(key)). will amend spec after folx comment.
asciilifeform: at any rate when we've rekeying this will matter rather less.
← 2022-01-11 | 2022-01-13 → ↑
Random(pest) | Download hourly DB snapshot | Get Source Code