asciilifeform: now, 1 more observation, the table scheme described here, can be safely parallelized, for so long as you ensure that the readers are never writing THE same place being elsewhere written.
asciilifeform: for the ultimate variant, where we don't even store 'blocks' as such, but merely sequence of specially-marked tx, you ~will~ need whole disk.
asciilifeform: trb will not even need to know whether it was given a real disk, or flat file.
asciilifeform: then it can be readily mapped to physical disk.
asciilifeform: at any rate imho the flat-file variant oughta be tried 1st.
asciilifeform: Framedragger: there is a udev trick, iirc, for giving particular users access to a given raw block dev
Framedragger: also, wonder if there could be a relevant linux cap'ability for allowing raw access to given /dev/block. but maybe not.
Framedragger: ah, that's the q i guess
asciilifeform: ( i wonder if there are any traditional fs that will actually give you 16G of contiguous blocks, if asked nicely )
Framedragger: asciilifeform: yeah! i may have time / want to do this in ~april. just to be clear, there's really no way to have underlying userland fs allocate a contiguous block, right?
jhvh1: asciilifeform: The operation succeeded.
asciilifeform: !~later tell Framedragger the table system can be easily prototyped , without writing any kernel code, by using 16G file on ordinary disk. (at the obvious speed penalty.)
asciilifeform: at any rate this is a 'theoretical' problem, imho 'not burning'.
asciilifeform: ( and ~yes~ right now it is very easy to see 'who wired to whom' by looking at ssh, but in the future we will presumably route multi-hop, via a gossiptron, and it will be less obvious )
asciilifeform: BUT this would make it very easy for the enemy to determine who is 'wired' to whom
asciilifeform: (can, say, ban them from talking on 8333 to each other in iptables, but this is stupid because then they will not share the perfectly-good ip with OTHER peers)
asciilifeform: (right now, e.g., zoolag and dulap happily also find each other over ordinary peer table and connect plaintext)
asciilifeform: rather than 'let's put cock into gas tank because it fits'
asciilifeform: i meant 'cheats' that use the machine as-prescribed and save multiple GB.
asciilifeform: you don't need any moar kludges, the scheme described, will work.
asciilifeform: ( however it will be intuitively obvious, to anyone familiar with x86/x64 and similar archs, that this is physically possible. i.e. you never actually need to store a contiguous empty GB in memory, if you can manipulate the page table )
asciilifeform: but THIS i will leave as an exercise for the reader.
asciilifeform: the ~other~ thing i'll point out, is that you do not in fact need 16GB of ram, to implement that table , if you can do it in the kernel
asciilifeform: the scheme 'degrades gracefully'.
asciilifeform: depending on the collision rate.
asciilifeform: ought to help those folx for whom kindergarten happened long, long ago, to picture the subj of today's thread.
Framedragger: sure, yeah! nice illustrations there
Framedragger: aha, so there's high probability that there will *be* a collision across the entire space.
asciilifeform: Framedragger: http://preshing.com/20110504/hash-collision-probabilities << re earlier. quick likbez on collision probability calculation using 'birthday theorem'
a111: Logged on 2017-03-10 18:25 Framedragger: hard to believe, but then, i should read logs at least
asciilifeform: whereas to someone without (e.g. the folx here) it is ~infinite.
asciilifeform: because the cost, to someone who ALREADY has fab capability , is miniscule
asciilifeform: Framedragger: the other thing is that it is quite impossible to come up with meaningful cost figure
Framedragger: hard to believe, but then, i should read logs at least
asciilifeform: Framedragger: there was old thread with mircea_popescu , where he stated that usg and china attempted it at same time, and perma-deadlocked
Framedragger: while at it, and i'm guessing it's in the logs.., i wonder when was the last time some tried to come up with an approximation what the costs to get to >50% network hashing power would be
asciilifeform: (enemy is welcome to bash his skull to a pancake against the cement wall)
asciilifeform: at any rate, working around this idiocy would be very cheap -- say we index by top32 of keccak(txid) instead of plain txid. in the event of.
asciilifeform: so that they collide
asciilifeform: there is 1 potential nuance,
asciilifeform: if you start seeing them ~regularly~ (say, trb makes it to 50 yrs from nao) you put in a l2 table.
Framedragger: and the slow-table.. hm
asciilifeform: if you get moar collisions, you simply end up in the slowtable moar often.
asciilifeform: there is not.
Framedragger: there's an assumption as max num of collisions here, of course, but obvs in practical terms it's a very safe assumption...
asciilifeform: ( i can already picture mircea_popescu spitting out his breakfast, 'modern hdd dun have cylinder, you nut' -- except, it in effect DOES, fetches massive blox , whether mechanical or ssd, by design, for ages now )
asciilifeform: disklookup is the 'go to nth cylinder and before you get to n+1st cylinder, THERE's yer tx' from earlier still.
asciilifeform: 'linearlookup' is the slow-table from earlier
asciilifeform: Framedragger, ben_vulpes : i predict that there will be few enough collisions that you can shunt ~all~ collision cases off to an O(N) lookup table, for next decade, with ~0 measurable loss of avg.case performance.
asciilifeform: idiot put them ~everywhere
asciilifeform: (some time grep for the sleeps, it is instructive)
asciilifeform: trinque: i've found that 'kill', which syncs the db, followed by kill -9, which nukes shitoshi's pointless wait idiocy, worx 100%.
trinque: god forbid the disk always be in a coherent state, eh?
Framedragger: hmh, at least functions make up not the *worst* interface seen, but still lotsa work and weird mutable shit sprayed all around, i imagine
asciilifeform: and reduce all of the bdbisms to calls to table ops.
asciilifeform: the gnarly part is to dissect all of shitoshi's idiot special cases (e.g. 'ReadOwnerTxes')
Framedragger: yeah that's one reason i'm not too attracted to trb, tbh, the amount of sewage gruntwork required to decouple shit from the monolith.
asciilifeform: imho the 'hard part' is not even to implement this table, it is freshman homework, but to unravel the liquishit in trb and learn where to even put the lookup/write !
Framedragger: i guess one can imagine a single sequence of tx then, simply.
asciilifeform: the table can index individual tx.
Framedragger: it's a really Good Thing that the hashing function which spits out transaction hashes gives *uniform distribution*. no congestion / too many collisions expected, and this scheme leverages that.
asciilifeform: if you want to consider 'what would give best performance on the iron we have, with minimal moving parts, with the trb we have' this is what comes out.
Framedragger: (i see how good it is to be aware of how actual disks read data here. some theoretician would propose a pointer-exact-location scheme instead...)
asciilifeform: so in practice, you can have O(1) seeks ~while~ storing the blocks back-to-back in a classical trb.
asciilifeform: that is, you don't need 8 bytes to say which 4kB slice has the beginning
asciilifeform: because you can point inside'em in the index.
asciilifeform: now you can store them back-to-back
asciilifeform: this also abolishes the cost of 'budget 1MB per block'
asciilifeform: see, we store the blocks linearly
asciilifeform: Framedragger: the 1024bt thing works in all cases, it simply means that you don't have to point to beginnings of blocks, but to a place very near the tx
asciilifeform: this form will almost certainly suffice for another decade of trb
Framedragger: right! ahh that's nice. (so just to clarify, the 1024 byte block trick wouldn't work if there's a collision (unless additional budget / w/e))
asciilifeform: the most basic setup has an l1 which immediately proceeds to the 'slow' table (where you have to look at multiple offsets in case of collision, to find your tx)
asciilifeform: 'look at next' can also mean 'go to l3', if you're willing to budget another 16G.
asciilifeform: and if set, you gotta look at the next machine word (each entry has a certain amount of space for collisions)
asciilifeform: where if top bit of the entry is not set, you have your where-to-go-to-find-tx-on-disk result
Framedragger: bear with my slowness, can you clarify how it looks like if there's a collision in the initial lookup?
asciilifeform: Framedragger: alignment in this case means that we can point ~inside~ blocks, supposing they are stored linearly on disk
asciilifeform: so we dispense with the O(N) dumb search inside-block
asciilifeform: or, rather, to where-to-start-looking-for-tx
asciilifeform: then a 31bit (remember, highest bit means 'go to l2 table') offset gives you a 2^^41 bit space in which your table entry can point to the beginning of a block.
asciilifeform: (and thereby a 16GB l1 table would give you wholly in-memory index for ~all tx lookups)
asciilifeform: ok, now question for ben_vulpes , trinque , mircea_popescu , et al : anybody got a quick c proggy that will eat blkcut's blocks and produce a linear list of tx ? i'd like to actually calculate the current 32bit tabular collision rate.
asciilifeform: from the one you are running from
asciilifeform: ...rather than to have any serious number crunching in kernelspace
asciilifeform: the Right Thing would probably be to have a very simple kernel driver that takes a specially-marked disk partition and gives userland trb linear use of it, as plain array
asciilifeform: who the fuck wants trb running as root
asciilifeform: i dare say the tabular algo is simple enough to actually be implementable.
Framedragger: at least i have the excuse of not having looked at the bdb problem / staying away from trb for the time being :p
asciilifeform: if you want to actually have the contiguous 16GB
Framedragger: is this the first time you articulated this approach here? i think that's the best on can have for fs-tx-db
asciilifeform: Framedragger: it's good for many, many times the current cumulative-tx
asciilifeform: if you're willing to blow another 16GB, you can have an l2 table
Framedragger: oh i finally understood, literally all there is when one seeks to location 3ec455a2 is a list of block numbers. (or single block number.)
Framedragger: why the need for "the machine might have to try 2 or 3 blocks before it finds tx" then? and if so, then no guarantee of only 1 seek?
Framedragger: asciilifeform: wait, what is "block index"? just the integer denoting block number?
asciilifeform: also on a reasonable box you can keep the l1 table in ram
asciilifeform: Framedragger: it is literally the smallest number of moving parts i can think of , that'd do the job.
Framedragger: trinque: it's just a kindergarten way of wrapping up some syscalls. will obviously benchmark outside it later. i wasn't completely certain that my tool wouldn't trash the host fs. :)
asciilifeform: however it does have the advantage of not using 10,000,001 lines of ext4 open sores crapolade for anything.
asciilifeform: but, thing is, it won't work any better on mechanical disk than the old bdb. and imho it is quite impossible to make variable-length tx bitcoin work well on mechanical disk, with modern spamola level.
asciilifeform: anyway this is the simplest physically-possible scheme.
asciilifeform: how to ~write~ to this data structure is, i think, obvious to the reader, i do not need to describe.
asciilifeform: and there we find ~lists of block indices~, so the machine might have to try 2 or 3 blocks before it finds tx.
asciilifeform: BUT if there are collisions, the upper bit of the 32bit entry, is set. and then we know that the lower 31, are an index into another contiguous array
asciilifeform: after which finding the actual tx is an O(N) operation in the block, with small N (the tx in the block)
asciilifeform: if there are no collisions (i.e. there is no other tx on the disk whose id begins with 3ec455a2) then that entry in the array IS the block index where the tx lives.
asciilifeform: now you store the table as follows: the top 32 bits (e.g., 3ec455a2 from above) are an array index into this table
Framedragger: (yeah btw, just ftr, symlink *creation* under populated dir structure (`ln -s files_f1/block35461.txt dc/dc89c1f2b58909d3814b250a731a9b9b791b092759553e3ba6579ffaad3a7565`) is slow. however, the creation was done using shellscript, need to move to c to be able to actually profile with precision.)
asciilifeform: btw i will also put down in the log, one very simple possible algorithm for a 'txidx-fs' :
asciilifeform: and likewise when a new block comes, all of the tx in it, get indexed this way.
asciilifeform: let's put down in the log, what exactly ye olde bdb does, that eats 99+% of trb's wall clock:
asciilifeform: does this include the actual read ?
asciilifeform: that's slightly slower, looks like, than the old db..
asciilifeform: trinque: you gotta log in as shown in the recipe
mircea_popescu: i suppose that's why they have that "excluding present company" device.
mircea_popescu: yeah, it's not lost on me that i'm the loudest preacher of continence in a cloister of monks muchly more restrained and disciplined than myself.
asciilifeform: 'if not you, then who' (tm) (r)
asciilifeform: somehow, long ago internalized mircea_popescu's above point re the model airplanes.
a111: Logged on 2017-03-10 15:02 mircea_popescu: so all is not lost! stupor abounds and thanks to the ever advancing technologee is now cheaper than ever!
asciilifeform: http://btcbase.org/log/2017-03-10#1624188 << i built myself a 'games box' , not long ago, 'to unwind', but turned out that the part of my head that was able to enjoy this, had atrophied, found myself constantly 'if i'm this awake, oughta be running $experiment, finishing up $unfinisheds', etc. ended up giving the (princely) fuckofftron to brother to play with.
mircea_popescu: so all is not lost! stupor abounds and thanks to the ever advancing technologee is now cheaper than ever!
mircea_popescu: but, as the eternally mentioned bottle well informs : the above has any merit only inasmuch it rankles the subject himself. otherwise it's nothing ; and besides there's many ways to, in brick pollitt's words, "hear that little click"
mircea_popescu: adolescence is the perceiving of boundless possibilities and maturity comes with the comprehension of the bounds.
Framedragger: i like my rc airplanes. "the will of history necessitates you to X" has a marx'ified hegelian vibe :p
mircea_popescu: not to mention of all the truly priceless fishing and whatnot else, history hasn't recorded.
mircea_popescu: well, anyway, i suppose it is a good thing the intelligent people involved could you know, pursue their own self-determined will and etcetera. who knows how many glorious flights of amateur rc airplanes this obviously reasonable arrangement gave the world!
asciilifeform: ( the sane folx, who had alphas with tru64, openvms, etc, went 'wtf why would i even think of winblowz' -- which wouldn't do. whole edifice 'musted die' )
asciilifeform: one perverse fact, is that microshit had a winblowz ecosystem for it, ready to go, it could have been he 'designated winner' in intel's place. except -- no, because gotta target the konsoomer chump junk ...
asciilifeform: re alphas, i am rather fond of the alpha, i have several here, possibly mentioned before
asciilifeform: afaik only known working examples are in usg heathen pits
mircea_popescu: lasted about as their professional lives did, also.
asciilifeform: amd hired the designers away
a111: Logged on 2017-03-10 12:25 mircea_popescu: phf the story of alpha is perhaps one of the best illustrations of the fundamentally anti-intellectual stance and calling of the female state. 1980s true 64 bit architecture, well supported (openvms started there, ffs, go revolutionize shit in 2015 with the remnants of 1980s tech). "sold" to fiorina's company, never heard from again, because gotta prop up intel.
asciilifeform: http://btcbase.org/log/2017-03-10#1624146 << alpha was already half-dead when hp swallowed compaq, in '02. iirc compaq shitburied it in '01 and sold all rights to intel, who proclaimed the arch now known as 'itanic' as its replacement.
a111: Logged on 2017-03-10 10:43 phf: i don't quite remember, what goes where, but i'm pretty sure the emulator.c that you linked is a frankenstein that primarily emulates alpha, though it has some signifcant knowledge about what the alpha emulator does, so it's not quite qemu, but something a lot scarrier
a111: Logged on 2017-03-10 10:39 phf: asciilifeform: if you spent time with it though, you'll realize that it's an assembly dump of the original emulator written for alpha, and the "emulator" part is actually a combination of c and lisp necessary to ~translate original alpha assembly into portable-ish c~
asciilifeform: http://btcbase.org/log/2017-03-10#1624127 << the c src i linked to does not appear to match this description, seems to be honest emulator
a111: Logged on 2017-03-10 10:39 phf: asciilifeform: if you spent time with it though, you'll realize that it's an assembly dump of the original emulator written for alpha, and the "emulator" part is actually a combination of c and lisp necessary to ~translate original alpha assembly into portable-ish c~
asciilifeform: http://btcbase.org/log/2017-03-10#1624127 << the classic alpha ver you described was in one of the subdirs, aha. and a recent g5 one (that dks mentioned to me in 2010 when he delivered the long-awaited genera alpha to my rupturefarm)
mircea_popescu: phf the story of alpha is perhaps one of the best illustrations of the fundamentally anti-intellectual stance and calling of the female state. 1980s true 64 bit architecture, well supported (openvms started there, ffs, go revolutionize shit in 2015 with the remnants of 1980s tech). "sold" to fiorina's company, never heard from again, because gotta prop up intel.
davout: other than that nailed it
mircea_popescu: o shit did i actually get the verb righ for once ?
phf: there's a translation code (in lisp) that takes that alpha body and spits out c equilvalent of the BIS ... SUBL ... etc. sequence
phf: https://github.com/hanshuebner/vlm/blob/master/alpha-emulator/ifunlist.as#L52 e.g. lisp machine instruction DoAssoc, but the code below, all those BIS, SUBL, SUBQ are alpha instruction set
deedbot: http://qntra.net/2017/03/park-geun-hye-impeachment-upheld-by-south-korean-court-shes-out-of-there/ << Qntra - Park Geun-hye Impeachment Upheld By South Korean Court – She's Out Of There
phf: i don't quite remember, what goes where, but i'm pretty sure the emulator.c that you linked is a frankenstein that primarily emulates alpha, though it has some signifcant knowledge about what the alpha emulator does, so it's not quite qemu, but something a lot scarrier
phf: asciilifeform: if you spent time with it though, you'll realize that it's an assembly dump of the original emulator written for alpha, and the "emulator" part is actually a combination of c and lisp necessary to ~translate original alpha assembly into portable-ish c~
phf: asciilifeform: there's several sources of vlm floating around, some of them improved upon by independent hackers (in the direct meaning of the word improve)
trinque: in fact, I'll go ahead and get the logbot-service thing released now, since it's been running the wot & voicing for maybe a month now
lobbes: sure, but from what I can observe, your own deedbot (and others' log-o-trons) appear to operate in a more sane manner. I guess I'm just saying I have much to learn
lobbes: I'm still in the process of climbing out of the primordial soup, so to speak. 'had to start somewhere, etc'; Though I realize I will need to flamethrow my own creations soon enough and rebuild once I 'evolve' a bit more. But all in due time
a111: Logged on 2017-03-09 18:23 trinque: I said to him "students may learn swordplay but may not tell us the fork in their hands is a sword"
lobbes: http://btcbase.org/log/2017-03-09#1623875 << I agree and will admit that my bot and log-o-tron are/were largely forged from used dildos I found laying in the forest. I also don't pretend that they are anything but
trinque: I'm sure they have a cloud API for that
trinque: mod6: the thing should really have one of those google AIs to figure out which goes with which
mod6: a cautionary note to anyone who is going to use my V to press with wires_rev1 (http://therealbitcoin.org/ml/btc-dev/2017-February/000251.html), be sure to name the seal as such or it won't get picked up in the flow (in the new, forthcoming version 99994) as such: asciilifeform_wires_rev1.vpatch.asciilifeform.sig
mircea_popescu: aaand in other kid-and-his-chemiferous-garage news, http://68.media.tumblr.com/7c9c5e795cd0594b0b4ebb887243e652/tumblr_oetvs7QpQd1u4w44io1_500.gif
mod6: and would have the same input hashes.
mod6: in the spirit of experimentation, it makes sense that one experiment would not necessairly contain the same changes as a different experiment.
asciilifeform: mod6: you can safely consign these to 'experimental tree' or wherever.
mod6: no worries. shinohai was building, asked me to double check just to be sure. thought it was worth the mention to future spelunkers.
mod6: which is fine, if you only use one of the three above, but not good if you try to use >1 of them.
mod6: asciilifeform_blackhole_odometer.vpatch, asciilifeform_blocktimer.vpatch, and asciilifeform_goodbye_pingers_fixed.vpatch all have the same input hash.
mod6: alf was it mentioned that some of these recent submission need regrinding?
BingoBoingo: <mircea_popescu> (he means republic of china, yes, not the communist fake state ?) << Well they both lost
mircea_popescu: sad testament for the cockroach race.
mircea_popescu: (he means republic of china, yes, not the communist fake state ?)
deedbot: http://qntra.net/2017/03/us-sportscaster-tout-south-korean-victory-over-china-the-republic-of-in-world-baseball-classic-ignores-actual-losses-by-peoples-republic/ << Qntra - US Sportscaster Tout South Korean Victory Over China (The Republic of) in World Baseball Classic, Ignores Actual Losses By People's Republic
asciilifeform: https://stallman.org/articles/texas.html << further lulz re rms guest lecture at mcc.
mircea_popescu: dude these women presidents aren't doing so well after all ? brazil, korea... who has one left ? impeach the queen ?
asciilifeform: 'For a long while I was doing research in software productivity. We began asking, ‘What is wrong? And why is it so difficult? Why is it so costly? What is complexity?’ It led me to interesting research, but nothing happened. We did not discover how to formulate or mathematically express the idea of program complexity. Not program complexity in the sense of algorithmic complexity—NP-complete problems and all that jazz—but the
Framedragger: mircea_popescu: basically, and that's strictly it - because i couldn't intuitively wrap my head around the fact that average number of nodes per specific folder would be _really_ low if depth is say more than 3. still weird in my head, but yeah.
asciilifeform: apparently was a consortium of mainframe makers, serious r&d corps, etc., worked on cad. utterly thermonuked by lisp winter and the microshitization of computing.
asciilifeform: 'MCC was part of the Artificial Intelligence boom of the 1980s, reportedly the single largest customer of both Symbolics and Lisp Machines, Inc. (and like Symbolics, was one of the first companies to register a .com domain). ' << aaaah apparently THAT's how.
mircea_popescu: it helps if it exists in the first place.
asciilifeform: holy fuck how can something the size of mcc vanish without ANY trace
mircea_popescu: is there such a thing as an indian who isn't a total shitbag ?
asciilifeform: https://groups.google.com/forum/#!topic/comp.lang.lisp/BebxJD27sao << ancient lulz found when trying to find what the fuck 'mcc' was
asciilifeform: and apparently responsible singlehandedly , or so himself claims, for one of the more infamous fake wotrons.
trinque: sandpaper on the cock will do that.
trinque: BingoBoingo: the guy looks miserable
Framedragger: (deeper path => slower traversal but fewer nodes per folder, up to the point where e.g. 'fast symlinks' can be used by ext4 (http://lxr.free-electrons.com/source/fs/ext4/inode.c#L148) - maybe; etc.)
Framedragger: << (obviously these'd be more useful with actual empirical numbers of average/median seek times, writes, seek/write as things get congested, etc.)
Framedragger: re. fs nodes, couldn't sleep + not sure if this makes sense, so just throwing these out - barebones super simplistic (function is `n_objects_to_store ^ 1 / folder_depth`) plots showing expected average number of nodes per folder (assumptions are no bias in hashspace and also equal share of hash bits per folder level) - it may not be intuitive how low the averages are until you look:
jhvh1: asciilifeform: The operation succeeded.
asciilifeform: !~later tell phf https://github.com/hanshuebner/vlm << sent in by a reader. somehow this has been just sitting there since '09, without my noticing
Framedragger: ^ the above is plain-obvious, but just ftr.
Framedragger: for symlink fs testers (or maybe selfnote for later): note that if you allow for sufficient folder tree depth, the "1000s of symlinks per dir" won't realistically happen when storing, say, bitcoin transaction hashes. the latter have 256 bits => 64 hex chars. if you allow for depth of 8 where last level (8) is symlink itself, you get 32 bits per folder level.
mircea_popescu: the quality of service in this restaurant is unparalleled. but yes, vupen.
asciilifeform: (the 'anonymized' vendor is, most likely, vupen)
asciilifeform: mircea_popescu: http://wotpaste.cascadianhacker.com/pastes/94ar2/?raw=true << the possibly interesting bit
mircea_popescu: kinda lost me in the first paragraph.
asciilifeform: in other lulz, http://www.rand.org/content/dam/rand/pubs/research_reports/RR1700/RR1751/RAND_RR1751.pdf >> http://wotpaste.cascadianhacker.com/pastes/JUw3R/?raw=true
deedbot: http://danielpbarron.com/2017/the-unreasonable-update/ << Daniel P. Barron - The Unreasonable Update
mircea_popescu: this is like the superlative of "what do i do now ?" "drink."
ben_vulpes: hey trinque didja hear that one about cups and the symlinks?
mircea_popescu: asciilifeform well, tradeoffs. he wants to store the block in parts, he wants to store it whole, what can i say
a111: Logged on 2017-03-09 20:54 mircea_popescu: nothing generally forces you to keep a block other than a collection of transaction-files, for instance. the exact implementation is up in the air for exactly such reason\
mircea_popescu: Framedragger yes, the fs is a major pile of dubious, as asciilifeform well points out.
diana_coman: needs to reimplement to get out on the other side really (I did NOT dig deep into Blender but I wouldn't be surprised if it were terribly bloated at the very least)
diana_coman: http://btcbase.org/log/2017-03-09#1623846 <-- fwiw the only way I can see this dependency on Python going anywhere would be if someone makes a sane replacement basically - however, artists need it but won't do it and otherwise people who are able to do it have a huge list of *other* things that need to be done as far as I can see; moreover (and as usual already), the whole steaming pile is deep so I can't even say how much one
Framedragger: also, """But I appear to have a lingering effect that seems to have started from the time my /tmp directory had the millions of files in it.
mircea_popescu: nothing generally forces you to keep a block other than a collection of transaction-files, for instance. the exact implementation is up in the air for exactly such reason\
mircea_popescu: and if fixed widths, you would actually "use" offsets in the sense of making files.
mircea_popescu: you have to load the block anyway.
mircea_popescu: you could use them to connect transactions to blocks, addresses to blocks, whatever index you're trying to keep
Framedragger: would this be performant enough even theoretically, given no way to use offsets?
Framedragger: mircea_popescu: re. http://btcbase.org/log/2016-12-22#1588180 , for my elucidation, so would the symlinks just point to a particular 1MB block file?
Framedragger: but i do hope you're doing the former, i mean i assumed so. that's the lowest-hanging fruit re. 'how do i do batch writes to db'
Framedragger: asciilifeform: on top of 'transactions', postgres has 'checkpoint' parameter. but you probably won't like it because of the whole 'not turning off fsync' thing
asciilifeform: shinohai: bitwise, the way you got'em, no browser cut'n'paste crapolade .
mircea_popescu: there's always that.
asciilifeform: later tonight, will begin reading the actual src
asciilifeform: mircea_popescu: i did go and sift through the docs, found 0 mention of write aggregation as an option
mircea_popescu: how large a part is an open question, but it obviously comes after establishing whether the would-be airplane is even larger than a breadbox in the first place
mircea_popescu: a large part of the "not writing properly" can actually be fixed via configs for the fs, rather than using the "middle of the road" stuff shipped with eg ubuntu.
a111: Logged on 2017-02-26 19:27 mircea_popescu: the other problem is that a good db fix is a very large project, because bitcoin is written insanely. and our fs db isn't moving, last i heard a month ago someone was going to try and profile an extx
asciilifeform: mircea_popescu: aite, meanwhile i found the prev thread: http://btcbase.org/log/2017-02-26#1618705
mircea_popescu: you repeat halfway of the discussion, "here is what i said lalala i can't hear anyone"
asciilifeform: and incidentally this will also change the semantics of the block-saver, unless somehow kludged around (e.g. via locking)
asciilifeform: (ext4 , asked to write 10,000 files/symlinks, will do the journal dance ~each~ time, rather than once-at-the-end)
asciilifeform: the more worrisome bit is that there is NO write aggregation (iirc i mentioned this before)
Framedragger: instead, it's "just" a matter of having a however-deep directory tree with symlinks as the leaves.
mircea_popescu: ah. no, that is correctly within the limit, as designed we'd have under 8 anyway
Framedragger: for a minute i thought (don't know why) that what is *additionally* needed is the capability to have paths of /symlinks/to/symlinks/.
Framedragger: oh wait, i phrased this incorrectly while at the same time horribly mis-reading: sorry, this is about max depth of path composed of symlinks.
Framedragger: well there ya go. :)
mircea_popescu: WHO THE FUCK THINKS LIKE T?HIS