spyked: http://logs.ossasepia.com/log/ossasepia/2020-04-03#1023625 <-- nah, it's no bother really, it takes a few minutes to update some feeds. so rather I didn't rush to solve a problem that didn't exist
ossabot: Logged on 2020-04-03 16:38:29 diana_coman: well, requests for changes are indeed unlikely to come all that often but it's still at best a workaround than a solution and moreover, a rather poor one in that it basically banks on the idea of .. little use of the bot? lolz; not like I mind the operator-maintained version but I'd have thought you rather mind it, tbh; anyways, no rush or anything.
spyked: but the pseudo-wot/access control/alias thingie (I still don't know how to label it yet, lol) sounds like a pretty quick job, so I'll prolly add it to the bot anyway
ossabot: Logged on 2020-04-03 16:30:29 diana_coman: spyked: but is the feed-in-chan a separate/anomaly thing currently, then? basically it needs a sort of alias #ossasepia = diana_coman
diana_coman: http://logs.ossasepia.com/log/ossasepia/2020-03-30#1023308 - dorion, there's no comment that can possibly take 1 week of being worked on, seriously now.
ossabot: Logged on 2020-03-30 16:21:15 dorion: http://logs.ossasepia.com/log/ossasepia/2020-03-29#1023189 - thanks. I'm giving it a second read now and working on a comment.
diana_coman: lobbes: what's your plan with the mp-wp log/bot anyway? and in general actually, are you back to saltmine-eating-my-time-wut-can-do?
feedbot: http://ossasepia.com/2020/04/04/cyli-nder-and-other-meshes-on-a-stick/ << Ossa Sepia -- Cyli N'Der and Other Meshes on a Stick
feedbot: http://thetarpit.org/2020/life-during-the-time-of-covid-the-great << The Tar Pit -- Life during the time of covid the Great
whaack: diana_coman: Following up on your most recent comment http://ztkfg.com/2020/04/thefleet-next-steps-with-data-collection/#comment-245, I want to make sure I have a few things straight. First, as you said the problem with my article where I posted the raw data was that I did not provide sufficient context. I want to clarify that my understanding that context =/= analysis. My understanding is the
whaack: context for the data is a description of how the data was collected, and what the data is. The analysis is in practice a visualization of the results of salient queries to the data. At the very least I can publish an article with the results of queries written to answer the "how much activity is where?" question. http://logs.ossasepia.com/log/ossasepia/2020-03-30#1023333
ossabot: Logged on 2020-03-30 16:29:04 diana_coman: do some proper stats first of all; how many users per logged chan; how many of those said how many lines per interval etc; start with the basic question which was "how much activity is where"
whaack: dorion and jfw: How is it going bunkered up with the family? It is sad to say that had the recent shennanagins not been going on, we would likely be meeting for a drink today. I hope you two are able to make it back shortly, and when you do you should pay a visit to billymg and me. We've had success mixing different fruit with rum.
ossabot: Logged on 2020-02-05 23:36:38 whaack: jfw and dorion: i'm booking flights to Panama for April 2nd - April 9th, i hope you two around around then :)
diana_coman: whaack: context is indeed how and what data was collected from where and under what specific conditions/constraints/known issues.
diana_coman: data analysis is a tricky one to define generically like that but at this stage it's really a very basic thing, mainly calculating a set of basic stats that can provide some information relevant to the question that started the whole thing: where and how much activity is there
diana_coman: note that you might still need to do some data cleaning first though, that is usually pre-analysis anyway
diana_coman: so you will need to actually look at your collection process and at at least some randomly sampled data in more detail
diana_coman: whaack: re "what analysis", start with the main question and then identify a set of concrete measures that are relevant; talk/ask for feedback on a draft of such list if you aren't sure, not a big issue.
jfw: whaack: thanks, and I'll certainly go for that visit when I'm able. It's been great here, really; I've been able to work with minimal disruption and not worry too much.
diana_coman: jfw: why worry anyway?
jfw: There's chaos afoot for sure for those dependent on the local money-drips, but so far its observable effects on the home have been minimal.
jfw: diana_coman: you would say that, heh. No point worrying if you won't do anything about it I suppose
jfw: I hear the government is Doing Something About It though, so no worries indeed!
diana_coman: jfw: ahaha, now that's something to protect against rather than worry about, lolz
diana_coman: but yeah, no point nor need nor anything really to worry, it's a sort of ...blockage only.
diana_coman: then again, it's a very cheap "Doing Something About It" too, so possibly why so easily chosen really; (see, the government could also just worry about it in public - it might even be more useful, setting up worry-points for interested people)
diana_coman: "come here to worry together" ,lolz
jfw: as far as what to protect against, so the local levels are mostly screaming for everyone 'non-essential' to stay in while vying for federal aid, while the federal level trips over itself to hand that out as fast as it can print
jfw: so what to do, idk... buy bitcoin?
diana_coman: ah, so it *does* the public worry (with the screaming); the printing well, finally found a reason for it, but anyways.
diana_coman: jfw: now that's a question each gets to answer for themselves; how would I know what makes most sense for you? for all I know - buy yourself caviar with it better; or bitcoin, sure; or smoke it, what.
jfw: I also gather the left hand is urging people to sew make-believe masks at home while the right hand "pays you a visit" if you're found stockpiling the real thing
diana_coman: ahahaha
diana_coman: see, troll them with apparent stockpiles ?
jfw: ha!
diana_coman: lots of way more interesting (and enjoyable for that matter) things to do than worrying.
jfw: supermarkets must be those public worry points, six-foot distances marked out and all
diana_coman: fwiw I'm quite enjoying the sudden almost ~total quiet (feels like countryside all of a sudden) + all the chirping birds coming over, can't even complain really; had fun discussing the plague in public too (company noted "they are looking at you"; noted back that "well, from 2m what else could they do anyway?"
diana_coman: jfw: neah, should be made properly, with a big arrow: worry HERE.
jfw: I take it "they" being people rather than birds - musta been an interesting discussion!
diana_coman: badges too, I suppose;
diana_coman: jfw: yeah, people; birds are innocent in this (and rather too busy to stare)
jfw: if I were way more bored I'd make that sign and arrow and post it in the toilet paper aisle.
diana_coman: jfw: well, next time you find yourself worrying, make the sign instead!
jfw: diana_coman: I shall.
jfw: !!key jfw_unchecked
trinque: huh, blank eh?
trinque will take a look
jfw: ty trinque. diana_coman: I'll get it on my site too, thanks for noting.
jfw: trinque: is there a particular update interval for the wot.deedbot www?
trinque: jfw: iirc it's once per hour, but the keys update immediately when requested
diana_coman: jfw: np; fwiw in principle getting it from deedbot is preferable to getting it from your site but anyways.
trinque: http://paste.deedbot.org/?id=ci1q << note that the paste you gave !!register is also empty
trinque: which is super weird
trinque: do you recognize that fp?
jfw: o.O yes it's the correct fp
diana_coman: trinque: fwiw at least some ratings from 4days ago haven't yet made it on wot.deedbot
ossabot: Logged on 2020-03-30 19:09:57 diana_coman: !!v 76EC35E7E4726FADE9C743B2D1097962734974B08E50572BAE9996C77BEAEC3C
trinque: diana_coman: looks like I boned up the crontab for that. oughta be up to date in a few minutes.
diana_coman: trinque: ah, cool; np.
trinque: still don't know wtf's going on with this other thing
jfw: one possible weird is there's no subkey on it (doesn't explain the empty paste though)
trinque: ah empty paste is just the db having been zilched after a reboot (it uses a tempfile) a few days ago, and my not handing back the proper 404 response.
trinque: jfw: can I get a new paste of that key, and I'll fiddle with this until sorted?
trinque: I will confirm what ends up on that nick has the fp we expect.
jfw: trinque: http://paste.deedbot.org/?id=-SVW
trinque: cool ty
jfw: np.
trinque: !!key jfw_unchecked
trinque: k, imported the key manually, which means I probably have a problem with key import at !!register
whaack: jfw: http://logs.ossasepia.com/log/ossasepia/2020-04-04#1023647 good to hear you've been able to concentrate and I look forward to it, however long it takes
ossabot: Logged on 2020-04-04 16:49:32 jfw: whaack: thanks, and I'll certainly go for that visit when I'm able. It's been great here, really; I've been able to work with minimal disruption and not worry too much.
trinque: hm wacky, can't reproduce it. jfw I wonder if you did that during my server migration, because key import is working fine atm.
diana_coman: oh hey, looks good, billymg !
billymg: diana_coman: sorry that it took so long, this past month since moving here has been pretty crazy but i think i'm back to a more normal schedule and state of mind now
diana_coman: billymg: no worries; what's your plan re mpwp?
billymg: diana_coman: i'm going to continue on it, i have a clear vision for it in my head and want to see it realized
diana_coman: billymg: glad to hear it!
billymg: namely i want to trim it down to 10-20% of its current size and give it an admin interface that isn't terrible (and terribly ugly)
whaack: diana_coman: Can you shed some light on how I should clean the data? Currently the sql dump contains logs for channels I partially logged, one idea is to remove those and then also filter out the logs that do not fall within a specific 2 week window
diana_coman: esp the trim sounds wonderful to me!
diana_coman: billymg: and feel free to discuss any of it in here if it's any help, ofc.
billymg: diana_coman: awesome, ty
diana_coman: whaack: that is more filtering of data than cleaning and yes, if needed, sure; on top of that, there might be borked/bogus data - you don't really want that counted as the rest since well, rubbish in, rubbish out
billymg: http://logs.ossasepia.com/log/ossasepia/2020-03-25#1022914 << trinque, jfw: i'm going to look into this now if no one's already beat me to it
ossabot: Logged on 2020-03-25 17:09:37 trinque: jfw: or obviously, just stick in getFooter instead
diana_coman: spyked: and now I am already asking for a change to that feed, lolz: would you add to it the feed for billymg.com, please?
jfw: billymg: hey, great to hear you'll continue there, an' I promise not to fragment anything by rewriting the lamp in lisp any time soon, as if I had time, lol
jfw: and yeah nobody's done anything toward getting the selection code out of the theme - but why don't I paste my particular patch that puts it in the theme in case it's of use.
billymg: jfw: lol, sounds good. and yeah re: selection-in-theme patch, it would be helpful to see it all in one place at least rather than spread between blog comments and log lines
whaack: diana_coman: Alright, so to look for borked/bogus data I take a random sample and examine it? For example, should I read the logs of randomly selected channels? I did this before and noticed that messages were missing from a convesation. The reason was the irc protocol is not case sensitive when it comes to channel names, so different clients can use different strings to represent the channel
whaack: name. (i.e. "#Ossasepia" vs. "#ossasepia") So when you select for a channel you need to make sure you use a query such as "select message from irclog where lower(networkname)='freenode' and lower(target)='#ossasepia'" instead of just "... where networkname='freenode' and target='#ossasepia'"
jfw: billymg: come to think of it I should just blog all my patches; though some are oudated or underinformed there's still some more good stuff. Will do by Monday night but likely sooner.
trinque: billymg: sounds good; I haven't touched it
jfw: whaack: yikes, it didn't occur to me that servers might not normalize channel names when forwarding actually, nice find. IRC case mapping is a pain because those norwegians (or was it finns) originally did it in not-quite-ASCII; some servers indicate their casemapping in the welcome messages but the format of these also isn't standard.
diana_coman: whaack: well, in practice (like with a lot of things), it can end up with a few iterations too because yes, you'll need to first have at least a look around just to figure out what you have in there but otherwise there might still be problems you notice later - at which point there's no choice than to go back, cleanup, run the stuff again; hence why ...make scripts for whatever analysis you do; specifically for the example you gave, ...
diana_coman: ... the cleanup would be to make all channel names lower case or something ie to avoid this sort of error; do make any cleanings via scripts too so you can reproduce and otherwise on a copy of the data...
billymg: jfw: cool, looking forward to it
billymg: trinque: ack, ty
diana_coman: whaack: in principle you'd figure out what errors through a combination of methods including at the very least: known issues due to your code/collection method; potential/known issues due to environment (such as the one you noticed, in principle); those observed through initial random sampling and at later stages when there's any anomaly/not fitting
diana_coman: and that's why usually you'd want to have more than just one measure for any thing - ideally anything you conclude relies on some triangulation of data (among other reasons to stand at least a chance of *noticing* discrepancies)
whaack: diana_coman: ack on keeping scripts for everything. The main concern I have is I don't have a great way to verify that my program didn't miss messages. Since I have a VM sitting idle I am going to have it log the same channels as another VM to compare the two DBs for the second batch of logs.
diana_coman: whaack: basically approach the data like everything else - NOT trusting it, quite the opposite, assuming it's rubbish mostly so trying to find counterevidence for any "finding" it might throw at you; for one thing, given enough data, you can "find" anything if you only...try hard enough (hence the classical lies, big lies and statistics, too)
diana_coman: whaack: so all that sort of thing should also be written at analysis report time, even if just as such - potential problems/unknown & unchecked issues; for that specific issue I suppose you could in principle look at intervals and all that but it's certainly overkill atm so yeah, write it down as "limitation" and otherwise live with it for this data set, nothing to do about it; (and no, nothing is ever perfect either, so there ...
diana_coman: ... *should* be limitations clearly stated too)
whaack: diana_coman: alright ty, my path forward is more clear; i'll heavily scrutinize the data.
jfw: http://logs.ossasepia.com/log/ossasepia/2020-04-04#1023703 - it was 2020-03-28 02:51:32 UTC.
ossabot: Logged on 2020-04-04 17:37:14 trinque: hm wacky, can't reproduce it. jfw I wonder if you did that during my server migration, because key import is working fine atm.
trinque: yeah, was definitely moving things around throughout that day.
jfw: trinque: good timing then, since it got me a reminder about assuming a thing worked just because it said it did
trinque: quis custodiet ipsos custodes
trinque: ave1: http://paste.deedbot.org/?id=wS80 << ever seen that?
trinque: !!up ave1
deedbot: You may not !!up yourself.
trinque: !!up #ossasepia ave1
deedbot: ave1 voiced for 30 minutes.
trinque: possibly it's cranky about the gnat version I'm using on the host? GNAT 4.9.2
trinque: ftr yes, has everything to do with that.
trinque: (which ave1 called out on his blog, thanks!)
feedbot: http://younghands.club/2020/04/05/wh-review-of-week-25-mar-30th-april-5th/ << Young Hands Club -- WH Review of Week 25 (Mar 30th - April 5th)
feedbot: http://trilema.com/2020/yet-i-do-not-repent-me-or-semana-santa-en-costa-rica/ << Trilema -- Yet I do not repent me, or Semana Santa en Costa Rica
feedbot: http://younghands.club/2020/04/05/wh-plan-for-week-26-april-6th-april-12th/ << Young Hands Club -- WH Plan For Week 26 (April 6th - April 12th)