The general unifying vision I keep angling toward is a distributed data publish-subscribe-compute network roughly like Ted Nelson’s Xanadu. (With the strong caveat that I’m very concerned about the privacy danger of distributed computation: any practical distributed-computation fabric must have the ability to restrict computation to trusted locales in that network).
For the primitive data element, I want something like S-expressions extended to allow relations and data types: that’s why I’m interested in something like term-expressions as a way of extending S-expressions in an unambiguous, user-extendable way.
For computation, I want something pure-functional; preferably based on something like Lisp’s fexprs, so that any generalised expression can be computed and a generalised relational data-representation language, something like Prolog, built on top. This is why I’ve been chasing the idea of variable-free relational programming for many years, mostly unsuccessfully; it feels like a way of building relations out of function-like operations. One of the stumbling blocks to me – other than the issues around multivalued logic, the handling of change and negation in logic programming, remains the question of just what a logic variable and the unique ‘unknown’ value associated with one, might represent in a functional paradigm. They appear to be concepts which simply (mostly) don’t map to the functional world. (Except for Scheme’s concept of variable location; which exists only because of mutation, which is not a pure-functional concept.) This seems very odd, since functional programming derived from logic. Presumably this is because it’s a subset of logic.
(I feel strongly against ‘types’ as they exist in current functional languages; for a number of reasons, but largely that they restrict expressiveness and cannot possibly be a fundamental language element. How could you describe a type annotation, except as a smaller, more primitive assertion in a more fundamental data-representation language? Then give me that more fundamental language and we can begin to talk seriously about all the data I need to represent – not just the computations I wish to perform on that data, which is a much smaller set. What is the type of a file, for example? What’s the type of a JSON node? What if your entire filesystem and memory storage was made of something like JSON nodes, and that was all there was? What could the ‘type’ of a node possibly mean then, other than another node attached to it – as a Lua metatable is, perhaps? So give me the ‘attachment’ mechanism you would use to implement your ‘types’; remember that you will be me, and I may be you; the roles of ‘user’, ‘programmer’ and ‘language designer’ are fluid and will change over time).
For I/O, I want something like functional reactive programming, so that we preserve pure-functionality and the UI can be simply described as a function over a stream of input and output events. I’m still waiting for FRP concepts to stabilize though. I like Elm as a proof of concept, but I dislike its typefulness for all the reasons above.
For the network component, I want something like content-centric networking, where data packets or blocks are coded uniquely and cached persistently for as long as possible, moving through the network to wherever demand is. With connections flowing between them as changes occurred.
So I’m wondering how something like email – or ‘message passing’ generally – might appear in a publish-subscribe format. In pub-sub or FRP, we would not really think in terms of ‘message sending’: rather, we’d make small local changes or assertions to a local object or relation or data entity. Then, should others choose to observe us, they would pick up our changes, but only if they were observing.
For a lot of purposes – think Facebook or Twitter – what we want to do isn’t send point-to-point messages. We very often want to broadcast updates or posts. We’d like to keep then the ability to smoothly change between broadcasts and narrowcasts.
One way might be: we create outgoing mailboxes. Imagine we have a folder structure to our sea-of-nodes. So Alice’s public feed might have \alice\public\out\bob\ .. and Bob would have an \bob\private\in\alice\ (Handwaving over just what public and private might mean in a distributed network: let’s say only public gets automatically exported out of a local LAN-like physical network, via something like a high-level firewall). And Bob’s definition of \in\alice is something like a function: ‘include \alice\public\out\bob’ – and anything then that Alice creates under \out\bob gets automatically transcluded (in the Ted Nelson sense) into that folder.
How does the information get from Alice to Bob? It seems like there must be an intermediary, a routing node where he could find \router\alice\ . And that’s maybe where we would need to reinvent 50+ years of TCP/IP and SMTP routing theory, since there would have to be some kind of overlapping hierarchy/heterarchy of such routing nodes. And we would need to deal with how to send connect/disconnect messages between them, how to keepalive and restore broken connections and garbage-collect dropped ones.
A bigger question though: how does Alice ‘push’ to Bob, assuming this is something we want? (We don’t always; that’s the spam problem. The cool thing about Facebook/Twitter is it’s mostly NOT push but pub-sub). How might Bob discover Alice for the first time? He hasn’t subscribed to her by creating a function. How does he know she wants to connect? Perhaps we need another channel for initial notifications: \bob\connect\in\… , \alice\connect\out… and they both subscribe to their local router, which subscribes to its upstream routers… and…
This is where I start to get lost. There must be a simple way of thinking about the bootstrap problem between routers, and also the problem of how to only request the information for which active subscriptions exist. Do we need at least the primitive concept of ‘send a subscription / notification / poll message’, or can we even reduce those to pure-functional observations?
It seems like we ought to be able to: after all, the lowest level of a network is a wire (or radio wave) between two devices. No device ever actually sends a message: they just put a formatted packet on the wire and pull off all packets with their address on it, ignoring the others. So what might the lowest ‘subscription message’ look like if it’s not a message but a sort of broadcast?
Ie at the low level you might /ethernet/packet which is the current packet addressed to this machine…. and then something like an ARP table as /machines/ips/192/0/0/1… but it’s what might go above that that gets tricky.