trinque: diana_coman: UUIDs are cheaply globally unique for tiny datasets like human-produced logs
trinque: I didn't want to be in the business of worrying about renumbering log lines if I merged logs from two sources, and had previously said all manner of things about that log-line by foreign-key reference
whaack: diana_coman: EOD Report: I missed the deadline for my article today. I worked on TheFleet all day again to try to get the bots online before I'm going to be away from my terminal. I obtained the max-chan-per-nick for my networks, setup join/part logging as well as kicked-from-chan logging, ^ switched the unique ids to pids + changed the db schema to have a specific fleetbotlogs table, created a deploy script, and then ran it. I ran
whaack: out of RAM on my VM so I am only connected to 8 networks right now. I need to fix my script so that it takes good care to keep its memory usage minimal. There also seems to be an error with the reconnect functionality.
whaack: and by 'connected to 8 networks' i mean i have 8 sbcl processes running, since a random few of them got cleaved due to running out of memory and since my disconnect logging is not working, i am not exactly certain which networks i'm connected to atm.
diana_coman: whaack: well, a test of sorts it sounds like, at least; that out of memory thing sounds gross though.
diana_coman: http://logs.ossasepia.com/log/ossasepia/2020-01-03#1014544 - I can see that; otoh though, how would you order them? I can see the case perhaps for both UUIDs and pids but having just one still keeps a set of problems around.
ossabot: Logged on 2020-01-03 23:05:12 trinque: I didn't want to be in the business of worrying about renumbering log lines if I merged logs from two sources, and had previously said all manner of things about that log-line by foreign-key reference
dorion_road: http://logs.ossasepia.com/log/ossasepia/2020-01-03#1014478 - it actually exceeded expectations on pretty much every front. I took a lot of pictures the past 2 days and will have an article to publish, so will spare details now. I disembark tomorrow morning local time, then have back to back red eye flights with stop over in las vegas and arrive to panama Monday morning. it'll be quite the relief
dorion_road: to get my routine and rhythm in motion again.
ossabot: Logged on 2020-01-03 05:34:05 diana_coman: dorion_road: how's the cruise?
diana_coman: dorion_road: oh hey, sounds good; though on the article front, dunno, does it beat hunting then or what?
whaack: diana_coman: re memory there is an easy fix i can do, right now each process unnecessarily keeps in memory the entire list of networks + all their channels. that is at least 5mb per process right from the start
diana_coman: whaack: aha, no need to keep that in memory really, no.
trinque: diana_coman: what problems do UUIDs cause?
trinque: arguably they're a waste of space, in that all that might ever be said would fit in an int32
trinque: at any rate, what makes a good key for a row of data is an interesting problem
trinque: I've long thought that eventually all this data in people's bots would be chattered among the bots to each other.
diana_coman: trinque: I didn't say they were a waste a space, no; and ftr I can state it also: no, they are not a waste of space.
trinque: just asking for your opinion of the problems they cause, not saying you're wrong.
diana_coman: I just think that both UUIDs and PIDs are needed because they serve different purpose really
trinque: sure, UUID doesn't tell you which you saw first, for instance
diana_coman: trinque: exactly; and this matters for log lines because timestamp is such a mess that you can't rely on it.
diana_coman: UUIDs otoh are useful precisely for identifiers though one can argue that you can always calculate on the fly some hash
trinque: yep, hash is perhaps much more appropriate a (somewhat) universal reference
diana_coman: the way I see it, as long as it's one db, might as well go with auto-increment since there's no loss as such; if/when mixing dbs, can identify same/different via a hash and otherwise if/when needed add perhaps another level of ref (e.g. provenance ) or otherwise still just calc and add hash at that time
trinque: renumbering is quite the bitch, can be made easier by "on update cascade"
diana_coman: anyways, I was honestly curious as to why you chose that (and I hadn't caught /recalled what you said re log-line by foreign-key ref http://logs.ossasepia.com/log/ossasepia/2020-01-03#1014544 )
ossabot: Logged on 2020-01-03 23:05:12 trinque: I didn't want to be in the business of worrying about renumbering log lines if I merged logs from two sources, and had previously said all manner of things about that log-line by foreign-key reference
diana_coman: yes, renumbering is a bitch; otoh database design has anyway some tensions as soon as the whole thing is worth anything so I doubt there's a one/single/ideal/perfect solution there.
trinque: for sure, tradeoffs in all directions
whaack: i will likely have to merge dbs further down the line in this project. i need to route through multiple ips to bypass the max-connections-per-host limit. i was originally planning to still have everything running on one server and just go through proxies, but i realize that trying to log absolutely all irc network activity is likely an exercise in DDOSing myself
diana_coman: whaack: you need to produce first a clear report re all those numbers, you know? what decision can one make on this sort of shaky sands of there-is-this-limit and that-limit but dunno exactly .
diana_coman: don't attempt to build the whole thing out of incremental blind adjustments; it'll end up a mess quite predictably.
diana_coman: blind & undocumented*
diana_coman: whaack: anyway, each chan should be in a single db though, right? I don't see why would a single chan end up split across several dbs so not sure there's much actual trouble merging otherwise.
whaack: diana_coman: okay i will try to be more forward thinking with my design. i am still repeating that bad habit of "incremental adjustments"
whaack: diana_coman: yes a single channel would never be split accross dbs
diana_coman: whaack: hm, did that MIT ever mention development methodologies at all? lolz
diana_coman: whaack: anyway, if you have a use case for those UUIDs, you should have argued it and kept them in; if you don't, then they are not in.
whaack: diana_coman: i blame myself for not taking advantage of MIT. but the default software system design class that I took was a how-to-java class
diana_coman: whaack: there is some space and even need for incremental, sure, but it's not all-there-is.
diana_coman: so then programming language class was what?
whaack: diana_coman: what do you mean programming language class? I didn't take any class on language design or any class that *intended* to teach to a specific language
diana_coman: whaack: software system design means > how-to-any-language
diana_coman: ie by the name it actually intended to teach something other than a language; and otherwise hm, no..introductory programming course or something lower level (that I could see focusing on how-to-x-language)?
whaack: http://stellar.mit.edu/S/course/6/fa14/6.005/materials.html "Software Construction" was the name of the class
diana_coman: ahahaha; why not "nuts and bolts that java holds"
diana_coman: whaack: anyways, I guess I still haven't fully grokked the current MIT; don't take it to heart.
whaack: diana_coman: thanks lol. I would not major in CS if I went there again, my advisor warned me against it and i did not listen.
diana_coman: whaack: what did they advise for?
diana_coman: "against it" is easier than "for it", lolz.
whaack: diana_coman: He said anything else I was interested in. His argument was not that the CS curriculum was bad, but that I would probably learn CS on my own time, and that I wouldn't be taking full advantage of resources at mit by majoring in cs. (i.e. why not major in mechanical engineering and learn how to use a $1 mil lathe?)
diana_coman: good point really; so... why didn't you?
whaack: diana_coman: I had decided that for sure I wanted to major in cs and stubbornly closed myself to his input
diana_coman: (normally there would have been a retort to that "point" of your advisor but possibly not by that time anymore so they were right either because they knew CS well enough or because they didn't know it well enough, lol)
diana_coman: ah, so ~the mech eng is not fashionable enough and lathe is not lisp.
whaack: the intro class freshman year was also fun and imo well run. we programmed little vehicles that used sonar sensors to get information about their surroundings to solve mazes
whaack: diana_coman: ^ pretty much
diana_coman: btw the moment one says that "you'll learn X on your own time", that *means* the curriculum is bad; because if you can indeed learn it just the same on your own time then sure, wtf "university"
diana_coman: and the sane retort to what I suspect is his different background simply rather than much more than that would have been that ahem, you can also learn to use $1 mil lathe on your employer's money later on as needed but you'll never get access again to far more valuable resources such as Minsky,
diana_coman: as in "working with Minsky"; but myeah.
whaack: they do a career fair at the beginning of the year too, and it's a big event for the SV circus show, all the cs majors are getting $35-$60/hour internships for the summer while the other majors generally struggle to find jobs
diana_coman: I can imagine.
diana_coman: lobbes: re your search for bugs, the whole discussion can easily go through CS, Maths, Logics and Philosophy in a single paragraph too, lol; might fit better in a conversation or as an article if you want to look at that bit in more depth.