billymg[asciilifeform]: $ticker btc usd
busybot[asciilifeform]: Current BTC price in USD: $20760.79
billymg[asciilifeform]: !c net-summary
crawlerbot[asciilifeform]: Bitcoin Network (IPv4 Nodes Active Within the Last 48 hours) Global: 7623; TRB-Compatible: 31; TRB: 13
crawlerbot[asciilifeform]: TRB-Compatible by Country: United States: 7; Germany: 4; Romania: 3; Singapore: 3; Russia: 3; France: 2; Spain: 2; Canada: 1; Ukraine: 1; Italy: 1; Algeria: 1; Sweden: 1; Belgium: 1; Bulgaria: 1;
crawlerbot[asciilifeform]: TRB by Country: United States: 7; Canada: 1; Romania: 1; Lithuania: 1; France: 1; New Zealand: 1; Norway: 1;
billymg[asciilifeform]: !c trb-status
crawlerbot[asciilifeform]: 75.106.222.93 (Alive), h=742714, v=99999, United States - peers: 130 - last probed: 38m ago
crawlerbot[asciilifeform]: 71.114.46.117 (Alive), h=742715, v=99999, United States - peers: 90 - last probed: 35m ago
shinohai[asciilifeform]: oh hey a Norwegian trb node
crawlerbot[asciilifeform]: 54.38.94.63 (Alive), h=742714, v=88888, France - peers: 68 - last probed: 40m ago
crawlerbot[asciilifeform]: 82.79.58.192 (Alive), h=742714, v=99999, Romania - peers: 64 - last probed: 39m ago
crawlerbot[asciilifeform]: 205.134.172.4 (Alive), h=742714, v=70001, United States - peers: 56 - last probed: 39m ago
crawlerbot[asciilifeform]: 85.164.243.42 (Alive), h=742715, v=99999, Norway - peers: 62 - last probed: 34m ago
crawlerbot[asciilifeform]: 94.176.238.102 (Alive), h=742357, v=99999, Lithuania - peers: 59 - last probed: 39m ago
crawlerbot[asciilifeform]: 205.134.172.28 (Alive), h=742714, v=99999, United States - peers: 39 - last probed: 40m ago
crawlerbot[asciilifeform]: 103.6.212.28 (Could not connect!), h=742599, v=99999, New Zealand - peers: 15 - last probed: 37m ago
crawlerbot[asciilifeform]: 205.134.172.26 (Alive), h=742714, v=99999, United States - peers: 26 - last probed: 39m ago
crawlerbot[asciilifeform]: 54.39.156.171 (Alive), h=742714, v=99999, Canada - peers: 45 - last probed: 39m ago
crawlerbot[asciilifeform]: 205.134.172.6 (Alive), h=742714, v=99999, United States - peers: 14 - last probed: 40m ago
crawlerbot[asciilifeform]: 205.134.172.27 (Alive), h=742714, v=99999, United States - peers: 1 - last probed: 39m ago
billymg[asciilifeform]: i think it's jonsykkel's
awt[asciilifeform]: While debugging I noticed that one thing that might be slowing down the initial sync is that almost every GETDATA response necessarily triggers a db query.
awt[asciilifeform]: Most of the slowness is likely due to blatta being single threaded, however. For ex., I can see message handling visibly pause while output is being dumped to the IRC client.
signpost[asciilifeform]: yeah, I bumped into python's threading limitations with ocpy too (that onlinecodes encoder)
signpost[asciilifeform]: you might be able to make a clean separation of the pest protocol handling and IRC handling using multiprocessing.
signpost[asciilifeform]: huck messages back and forth between them over Queues or something.
awt[asciilifeform]: Another win might be to separate all db writing/reading ops into another process
asciilifeform: signpost: imho ideally ircism in entirely separate proggy eventually
dulapbot: (asciilifeform) 2022-06-28 asciilifeform: the irc frontend is very much, imho, a straightjacket
signpost[asciilifeform]: multiprocessing's pretty much that, fires up separate processes and gives ya a python-shaped hole to yap between them.
signpost[asciilifeform]: but yeah, w/e makes the irc part easiest to cleave off.
awt[asciilifeform]: Ok so best post mortem I can come up with right now for the dupe GETDATA requests is: occasionally stations with very old netchain send a message. When doing initial sync, these are were encountered faster than the order buffer was getting dumped.
awt[asciilifeform]: PROD will fix this. Also, increasing the length of time outgoing GETDATA requests are retained in memory helps.
awt[asciilifeform]: Also planning to add a knob to prevent dumping messages older than x into the IRC client.
awt[asciilifeform]: As already discussed, in the future, loggers should read directly from the station log. I may write a console tool to dump the logs in order in a readable format to verify syncs are completing successfully.
signpost[asciilifeform]: imho I wouldn't even bother avoiding dumping old messages to the IRC console at this point.
signpost[asciilifeform]: that'd be another example of letting accidental complexity creep in from the temporary measure of using IRC for UI.
signpost[asciilifeform]: later "oops, special case that avoided sending this message to IRC also avoided sending it to db"
signpost[asciilifeform]: giving that as an example of the kinds of bugs this backflow of temporary measures into core design can cause.
signpost[asciilifeform]: I don't think anybody's terribly inconvenienced atm, and I think the line oughta be held on hacks.
signpost[asciilifeform]: log-reader oughta read the db, yes, not the IRC adapter. db is the source of truth on message chain state, not the UI.
signpost[asciilifeform]: my 2c
asciilifeform agrees
awt[asciilifeform]: signpost: makes sense