#therealbitcoin | 2020-05-15

jfw: asciilifeform: mind expanding on what you mean by leave it the fuck alone and why?

snsabot: Logged on 2020-05-11 13:23:37 asciilifeform: asciilifeform's pov re trb worx ftr

jfw: you think it's good enough? readable? works well? "close enough for government work" as the contractors said back home?

asciilifeform: jfw: i'll elaborate (in ~15 m, tied up in meatspace atm)

jfw: no rush.

asciilifeform: jfw: are you familiar with how i stepped into that pile o'shit to begin with, in '14 ?

asciilifeform: my notion was 'we'll rewrite the client'. but 1st thought 'let's find what original does', as it was never 100% documented anywhere. mp contributed a copy of 0.5.3. asciilifeform then cut quarter MB of obvious rubbish from it, and signed (this was pre-'v'.) various other folx started reading, cutting.

jfw: not entirely, I know you were btc-skeptical at first, and MP invited you to review his exchange

asciilifeform: asciilifeform's orig. notion was to produce 'the book', i.e. something like lion's unix 6th ed. for bitcoin .

asciilifeform: jfw: this was already 1+y after asciilifeform was persuaded to join mp et al.

asciilifeform: the orig. threads on tbf ml.

jfw: right, there was a 'gypsy chicken recipe' involved back then

asciilifeform: right, that was where i took the 0.5.3 and cut off the mswin, gui, buncha cruft

asciilifeform: phf's patch viewer still best imho illustration of what/when cut, by whom

jfw: I've read or at least skimmed all the vpatches ftr, signed & hosted at http://fixpoint.welshcomputing.com/v/bitcoin/

asciilifeform: for long time, project was stalled -- 0.5.3 would sync up to certain block and mysteriously grind to halt. then in fact mp dug up the pill, was bug in the orig. bdb. then became possible to run working nodes .

jfw: insufficient magic numbers in the bdb locking system config

asciilifeform: aha

asciilifeform: so, to return to jfw's orig. q: to date, ~errything that's been done to trb, consisted of a) cuts of dross (the irc seed finder, mswin liquishit, gui, etc) b) bug fixes (most recently : the 'getinfo' wedge)

jfw: (to be pedantic, 'at least skimmed' does not apply to genesis, though I've started on a gradual full readthrough.)

asciilifeform: the thing is defo not pleasant to read, or in any way sane example of programming , by even microshit standards

asciilifeform: asciilifeform cannot speak for all of the participants, but what he wanted it for, was as ~reference~

asciilifeform: i.e. so can write new client that's guaranteed to be 100% compat (i.e. won't spontaneously end up forked off at any pt)

asciilifeform: any major change to the orig. item, potentially destroys this function (i.e. potentially changes the semantics of the thing in the face of new inputs) . mp called this the 'grandfather's pistols' problem .

asciilifeform: so no one made any substantial change. ( asciilifeform at one time glued on a scheme interpreter, and 'let's rewrite pieces in that', but this didn't catch on, and abandoned. arguably it was ill-conceived idea from start. )

jfw: yes, that's a challenge; otoh, I find it ~impossible to reason about what wtf those pistols currently do anyway

asciilifeform: to give very concrete example : the recent 'wedge' was on acct of the nonsensical ocean of semaphore locks in trb. and would seem 'why not remove'em' . but go and try.

asciilifeform: there's ~no way to guarantee exactly same behaviour as the orig. cuz orig author simply sprinkled locks around like a child.

jfw: perhaps to make matters worse, I'm not sure there's any guarantee that original has same behavior as itself on different occasions.

asciilifeform: exactly; the 'getinfo' wedge was specific instance of just this.

asciilifeform: trb is literally the shoddiest piece of ??? asciilifeform ever had the misfortune to work with. and asciilifeform maintains legacy winblows crud for a living.

jfw: the bdb locks too; I imagine it'd depend on how exactly your b-trees ended up layed out due to past history as to how many pages it needed to lock and thus what exact block it would stick on.

asciilifeform: exactly. and think, wtf does the thing need a general-purpose db, when ~old blocks don't change~ ~

asciilifeform: block >20 or so deep, will never change ! till sun burns out. why need b-trees ?!

jfw: well for starters it needs to update an index of where txns are spent, no?

asciilifeform: answer is, orig author was lazy ignoramus. and, on top of this, never finished the item.

asciilifeform: jfw: this doesn't require a changeable db, w/ trees. i outlined a simple o(1) scheme some yrs ago, where one actually stores backlinks to the tx being spent.

jfw: does that require changing the block format though?

asciilifeform: nope

asciilifeform: entirely client-size (i.e. follows traditional protocol when speaking)

asciilifeform: jfw: posted some of the example coad a while ago .

asciilifeform: ( see 'type Tx...' )

asciilifeform: idea is, when you eat a block, you store the tx in such a way that they actually point to the ones being spent. in mmap'd array. the respective scripts are kept in another, so that they don't fuck your page alignments. (ea. tx points back to the orig. script. so can reconstitute orig. tx / block on demand )

jfw: how do you find where the ones being spent are, if you don't have an index by txid?

asciilifeform: in yet a 3rd mmap'd array you keep hash table. so you can take e.g. 1st 32bits of txid, and deref array, and in that place either a 0, or the index of where the tx lives in the tx array.

asciilifeform: you do index by txid. and with ea. tx, keep the indices of any it spends, and any that in turn later spent it.

asciilifeform: this would be very difficult if you had to handle reorgs. asciilifeform's discovery was that you don't have to, if you keep e.g. last 100blox in 'nursery' (something similar to the old db) and only >100-deep blox, in the static arrays.

asciilifeform: then you don't need to support reorg, for the latter.

jfw: but tx index is a tree then, no? (or hash table which isn't guaranteed access time)

asciilifeform: guaranteed o(1) access.

asciilifeform: i'll elaborate if it aint clear

asciilifeform: say you want to look up a tx by id.

asciilifeform: you take 1st 32bits of the id.

asciilifeform: then you look at what's in index(first_32_of_id). it'll be a e.g. 48bit word. if it's 0, there's no tx with that id.

asciilifeform: if it aint 0, that's an index into the blocks array .

asciilifeform: both mmaped.

asciilifeform: this requires a large, but, interestingly, roughly same mass of disk as the old db.

jfw: and in case of collision of first 32 bits?

asciilifeform: and because the index is a sparse array, mmap Does Right Thing, so it doesn't eat up all yer ram

asciilifeform: jfw: in case of collision, the 1st of those 48 is set, and the pointed address contains the full list of all id's having those given 32bits in start.

asciilifeform: my other suggestion , for minimizing collisions, was to pre-hash the txids with a machine-local salt, to keep people from 'mining' colliding txid's and grinding erryone down

asciilifeform: but in fact you do have to handle collisions, elementarily, given as txid is quite bit longer than 32b

asciilifeform: ftr i thought all of this was clear when said 'hash table...'

jfw: sounds the same as any other hash table, so not sure why the objection to btree, but ok, maybe implementation ends up simpler.

asciilifeform: cuz it doesn't need to change!

asciilifeform: there's no reason to have to balance a tree for this.

asciilifeform: simply suffices to lay down the tx as the come in, in the next available free space. and update the index appropriately.

asciilifeform: (likewise gotta track which block # given tx belongs to. but this is imho obv. detail.)

jfw: anyway, mind returning upstack for now? I'll assume this scheme works fine; where does that leave TRB work?

asciilifeform: jfw: after described this, went to write a mmap lib for ada. this was where stopped.

snsabot: (trilema) 2019-01-22 a111: Logged on 2017-06-06 19:40 asciilifeform: mod6, phf , et al : http://nosuchlabs.com/pub/ada/horsecocks.tar.gz << i dun recall posting this before, so here it will live, for nao : unofficial release of mmaptron

asciilifeform: aite

asciilifeform: jfw: imho trb is a 'life support' item. i.e. one needs a working client, not only to study 'let's rewrite', but to actually use bitcoin .

asciilifeform: most of what i've done to the thing since '15, was from that pov. i.e. 'people report it takes 8 weeks to sync, let's see why'

asciilifeform: 'nodes seem to all be wedged, let's find why.' etc

asciilifeform: taking, for instance, sync (iirc jfw once said he was interested in why sync slow)

asciilifeform: in '18 i profiled the thing, for months. found that spends ~100% of time ~waiting for someone to give new block~

asciilifeform: cuz orig author was a moron, and made sync something that happens only briefly ~at boot time~

jfw: re sync, quite; for instance I presently can't transact because I've had a node sitting on desk for >month that spending most of its time doing nothing but receiving bastard blocks until killed & restarted.

asciilifeform: wrote 'aggression' patch. which seemed to help (changed logic to 'ask for blocks' at time new peer connects, rather than the idiot bootup )

asciilifeform: but this also not cure. because if entire cluster of nodes not gets new blocks for while, but connections still alive, they happily sit there like idiots and get 0 new blocks.

asciilifeform: ben_vulpes attempted a more aggressive sync, but iirc wasn't quite enuff .

asciilifeform: afaik no one has since experimented with this bit ( it only comes into play when node falls severely behind, is possibly why )

asciilifeform: jfw: 'bastard blocks' are simply blocks for which the antecedent is not known. it'll be 100% of incoming blocks , from a node's pov, if it lacks even one antecedent for'em.

asciilifeform: i introduced the term in this patch .

jfw: right.

asciilifeform: ( orig. 0.5.3 had a data structure called 'orphanage' , which stored ~arbitrary~ incoming turds claiming to be blocks, and ate ~100% of memory )

jfw: and I've nfi why trb peers send me non-current blocks for which I lack antecedents

asciilifeform: cuz they don't know that you lack the antecedent .

jfw: they just guess at what block numbers I need?

asciilifeform: default behaviour is, for instance, to retransmit ~all~ incoming new blocks that got accepted by node.

asciilifeform: they don't even guess. simply blindly relay.

asciilifeform: and, moreover, even a node that is syncing, will relay. i'ma work an example :

jfw: perhaps it's that - they're relaying new blocks only, and problem is that the sync 'ran out of steam' after leaving 'bootup' phase.

asciilifeform: you have a new node, it's at 0. mr. x has another young node, it's at 100k. he just got 100,000 .. 100,500. his node peers with yours. yer gonna receive those 100,000 .. 100,500 , whether you want'em or not

asciilifeform: whether you know what to do with'em, or not

asciilifeform: cuz mr. x's node doesn't ~know~ the 'current block height' of whole net. it's an abstraction. it only knows that it just learned 500 'new' blox.

asciilifeform: so 'helpfully' sends'em to you.

jfw: sure, s/new/new-to-them/ then

asciilifeform: i oversimplify slightly, the orig client has a hardcoded turd that tries to guess if it's 'severely behind' based on the block heights reported by peers. but it doesn't, as you might expect, work reliably to prevent this kinda spam.

asciilifeform: the nodes default to sending 'more than you want', cuz that's how the blocks/tx propagate. it's primarily a 'push' mechanism.

jfw: Now - perhaps this is where I interrupted re mmap db - is it your view that efforts at this point are best spent on rewrite while problems with the ref. impl. are handled reactively?

asciilifeform: 1s

asciilifeform: jfw: correct

jfw: asciilifeform: alright, thanks.

jfw: jurov, trinque: I'm curious whether you concur with the above. Gotta run now but will check log.

asciilifeform: jfw: i'm not somehow opposed to efforts to civilize trb 'in sections'. but imho suffers from same problem as ab initio rewrite (i.e. not guaranteed to preserve semantics) while still retains the idjit 'serializations' etc

asciilifeform: see also .

snsabot: (asciilifeform) 2020-05-03 asciilifeform: the ~whole calamity of trb is that the internal representations for ~everything are thoroughly braindamaged. if you fix this, you get ~0 of the orig. trb remaining, i.e. same as total rewrite. if you don't, you get 100k loc of glue.

asciilifeform: for my part, i'ma rip out the ssl crapola an' replace w/ ffa. (dunno if anyone ~else~ wants such a trb, but i'm personally tired of sslism)

asciilifeform: this strictly from pov of ' asciilifeform's personal nodes ' , but of course will publish .

asciilifeform: http://logs.nosuchlabs.com/log/therealbitcoin/2020-05-15#1000138 << btw i notice the wedge pill aint in there. may be 1 reason why jfw's box can't sync .

snsabot: Logged on 2020-05-15 14:09:11 jfw: I've read or at least skimmed all the vpatches ftr, signed & hosted at http://fixpoint.welshcomputing.com/v/bitcoin/

trinque: jfw: mmmm it's hard to say; the thing's a hellish tarpit for sure.

trinque: were I taking it on, I'd probably pare the thing down a bit yet before rewriting

trinque: I have a patch on my desk that cleaves off the entire wallet and stuff only used by wallet

trinque: you're welcome to it if it's helpful

asciilifeform: trinque: iirc jfw made something similar , prior to tuning in

trinque: ah, cool then

trinque: it does cut the weight a bit

trinque: (not nearly enough but)

asciilifeform: reduces the thickness of the 3ring you need to keep the thing in, for sure

asciilifeform: but does 0 against 'serializations', the locking horror, etc

asciilifeform: they're pretty firmly 'glued with broken glass'

trinque: probably wireshark protocol analysis is a better path forward eh?

asciilifeform: trinque: oddly enuff, the protocol i found to be quite simple. e.g. 'watchglass' in fact nao speaks 100% of it

trinque: mhm

asciilifeform: ( and indeed i had to wireshark , it aint 100% documented afaik anywhere )

trinque: honestly if the piece of shit relies on implicit behavior, better out with it now than later

asciilifeform: for double lulz, 'wireshark' comes w/ what claims to be bitcoin protocol decoder . it only quasi-worx.

trinque: would be neat to see an item produced that horks and sprays blocks, and doesn't worry about whether it's doing it in exactly the same way as trb.

asciilifeform: for instance, 1 find.

snsabot: (asciilifeform) 2020-03-01 asciilifeform: shinohai: this btw confirmed. i did experiment, if one sends that extra byte, prb noades do in fact send addrs. (and yes some of'em ipv6, have to be thrown out)

asciilifeform: ( re protocol corners. )

asciilifeform: trinque: indeed would. and one could jettison some of the weird special cases (e.g. support for replaceable tx's) in the orig., if you cement in hashes for 1st 600k blox etc

asciilifeform: ( iirc i described this in the past. cement, not a la prb's 'dun need to verify these', but rather 'oh and btw the 1st N blox have THESE hashes, and it someone brings in a 599,999 that doesn't, he's trying to fuck you )

asciilifeform: the demo for this, on the ml.

asciilifeform will add : imho the main reason why trb seems to consist 80+% by mass of kilometres of pointless abstractions standing in each others' (and reader's) way -- is that this style is forced on cpp victims by the language; it is how they avoid coarse errors of pilotage (buffer off-by-1s and other consequences of c/cpp's idiot lack of ranged types, bounds checks, built-in array moves )

asciilifeform: microshit's coad looks ~entirely like trb. (but -- a little cleaner, they have 1000 monkeys on erry piece of shit, sweeping up )

asciilifeform: this is how 'oop'ism became a thing.

asciilifeform: 'oh noez, if you expose the raw bytes, someone could off-by-1...'

asciilifeform: for illustration, compare this to that .

asciilifeform: ( for the innocent: the item on left , and the 1 on right, do nominally 'same thing' ... )

asciilifeform would link to a cpp example, from heathendom, but folx will lose their lunch, so won't

shinohai: asciilifeform: Do you, offhand, know the link to the thread re: bdb bug ?

asciilifeform: shinohai: it's in phf's logs , defo not in mine, being 'era 1' (pre-apr of '16)

asciilifeform: there .

shinohai: oh fuck NOW I remember that. ty asciilifeform

asciilifeform: np.

jurov: asciilifeform: i don't have node atm, the hdd died and waits for reinstall. no idea whose is the radiolan.sk one

jurov: will ingest the rest of log tmrw

asciilifeform: ah hm jurov i always assumed it was yours

asciilifeform wonders who the seekrit lurker slovak were...