Keywords

1 Introduction

The developer of a distributed system rarely implements it from scratch, as a monolithic program. Instead, a common approach is to compose independent components, either off-the-shelf or bespoke. For instance, a sharded key-value store might be composed of shard servers (a and b in Fig. 1a), with a router to direct client requests to the correct shard.

The composed system should both be safe and have good performance. This requires the developer to be able to: (1) formalize the individual components; (2) specify how they communicate [17, 25, 30]; (3) reason over both the static effects of the composed object [25], and its dynamic effects; and, (4) control and other non-functional and performance-related properties, such as co-location or inlining.

The current approach to compositional programming is ad-hoc and mostly manual. It consists of running components as processes that send messages to each others’ API [30]. This satisfies in part Requirements 1 and 2 above, but does not express high-level safety [17, 30], placement, or performance constraints.

An improvement is to use an orchestration engine, such as Docker Compose, Kubernetes or OpenStack [3, 4, 15] to automate deployment and to control the topology. This addresses Requirement 4 only. Alternatively, a protocol language or an orchestration language can express some of the semantics. However, current languages do not satisfy Requirements 3 and 4, as we detail in Sect. 4.

Fig. 1.
figure 1

(a) A key-value store (S-KV) composed of two shards and a router. (b) Adding causally consistent communication (cc) layer to our key-value store: with two client (c1, c2), one KVServer (kv) and three CC-sidecars (s1, s2, s3). m denotes a message and ts denotes a vector clock.

Full size image

We address these issues with Varda, our framework for compositional distributed programming. A system developer specifies the architecture of a distributed system in the Varda language. This enables to formally define the components of a system, their interface, their interconnection, and their placement. In particular, our orchestration sublanguage prescribes the run-time interactions between the components. Based on this specification, the Varda compiler performs static and run-time checks, and generates the interaction code between components, called the glue.

Note that an architecture description abstracts over issues not related to distribution. In particular, the individual components are imported and linked into the generated glue, but assumed implemented outside of our framework (written for instance in Java).

We claim the following contributions for this work:

  • A language for expressing the component architecture and the orchestration of a distributed system (Sect. 2).

  • A general interception mechanism for imposing orchestration logic and other transformations onto components (Sect. 3).

  • As an example, we show how to impose transparently a common pattern: sharding (Sect. 3).

This paper does not yet provide an experimental evaluation, as the implementation is progress.

2 Programming Model

2.1 Concepts

Let us first explain the Varda concepts and terminology, based on the example in Fig. 1a. It represents the architecture of a key-value store. Its components are a client on the left, and the key-value store proper on the right, itself composed of a router in the centre and two servers on the middle right. The router forwards client requests to the appropriate server.

The two servers are distinct activations of the same schema; these concepts are somewhat to instantiations and classes in object-oriented languages respectively.

A schema, written in the Varda language, describes the component’s interactions with other components. In our example, a server schema accepts get and put invocations, which it executes against its storage backend. The router schema accepts the same get/put signature as a server, but its behaviour is different: based on the arguments, it forwards the invocation to the appropriate server activation, awaits the response, and forwards the response to the client.

An activation can link to an implementation, a black-box executable component exported as a library. In the figure, the implementation of Activation  stores its data in a Redis implementation [36], whereas that of Activation  uses a custom storage logic.

Finally, the figure shows places, i.e., physical or logical locations in the distributed system. In this example, the client is in its own place, and Server  is in the same place as its implementation. Placement is an important consideration, for instance for performance, fault tolerance or physical security.

2.2 Components

Recall that a schema is the code for a class of components. The Varda schema code has several parts, each described in an appropriate sublanguage. Its signature declares the names and types of messages it can send and receive, using both classical (declarative) types and (imperative) safety assertions. Its protocol describes the sequencing of such messages, expressed in the language of session types [14]. Ports are communication entry and exit points; a port is described by its name, signature, and protocol.

The orchestration logic describes how the component behaves, in a Turing-complete imperative sublanguage. It can specify a callback method to be invoked when a given type of message is received. It also includes specific methods for creating and destroying an activation of the component (called and respectively). Orchestration logic can maintain local state, can send messages, and can invoke the implementation.

The binding between the component, and its implementation written in some external programming language, is expressed using imperative templates that embed fragments of the external language.Footnote 1

A component schema may contain sub-components. The scope of a sub-component is the enclosing component, i.e., a sub-component cannot be invoked from the outside.

An instance of a schema at run time is called an activation. The activation is the smallest grain of distribution and concurrency. Computation within an activation is sequential. Receiving a message, instantiating the activation or terminating it run the corresponding callback method. A method executes until it terminates, or until it waits for an asynchronous invocation.

2.3 Interaction Interface

This subsection details how two components interact. Activations communicates by sending messages to each other. Programmers group message into protocols. A protocol describes the type and the order of events. Session types [14] directly inspire protocols. Between activations, those messages are flowing through channels. A channel interconnects ports of multiple components. A programmer formalizes components interface by defining ports: , to communicate with the outside, and , to listen for incoming messages.

The interaction should both be safe and have good performance. This requires the developer to be able to: (a) constraints the communication topology to explicitly specify which component is talking to whom; (b) interacting component have to agree on the order and the type of messages they exchange to perform lightweight verification and to drive the code-generation of the networking interfaces; (c) represents the underlying network layer to do specialize the code-generation and to represents assumption on the underlying network in the architecture description; and, (d) (weakly) isolate component functionalities from each other.

Events. To communicate, activations exchange events. Each event is strongly typed and can carry a payload. Its payload should be serializable.

A programmer can manually define an event key carrying a payload with . Otherwise, a programmer can send classical serializable types without defining events. The Varda compiler auto-box those types into events and un-box them at reception. Event auto-boxing alleviate the programmer from the burden of defining events for base types (e.g., ).

Varda type system supports type evolution of event through subtyping. Subtyping define a relation of substitutability between data types. Substitutability is a property where code written to operate on the supertype can safely be substituted for any of the subtypes in the subtyping relationship [27].

A component can send a message with more information than expected. For instance, lets assume than Activation b expects messages of type record: . An Activation a can send to b a message of type: . At reception, b considers only the field .

Protocols. Protocols address Requirement b. It constraints communication between activations: programmer attaches a protocol to each channel and each port. A (binary) session represents one instance of a protocol. An Activation i can creates a session with Activation j by calling . Let’s assume that the protocol of the port is . Where key and value are event types. The session type implicitly bound to guarantees that a communication through s is as follows: i starts by sending a message of type and ends by receiving a message of type .

Varda exposes classical communication primitives managing session [14]: asynchronous message sending , asynchronous receiving using callback (ports) or primitive, non-deterministic branching and recursive protocol. Each of this operations returns a new session types with the protocol of the continuation and preserves the session identity.

Channels. Channels address Requirement c and their types solves the static part of Requirement a. A channel can interconnect multiple activations, of different component schemas. A channel can represents different communication guarantees, provided by the underlying network primitives. For instance, a channel can be protected by TLS encryption or can guarantee point to point FIFO communication, which is the default guarantee. A channel is compiled directly to network layer code to preserve performance.

A channel definition is asymmetric for communication establishment to statically constrain communication topology. A channel of type guarantees that only activations of type can initiate a request to activations of type . Bidirectional channels can be constructed using union type: .

Ports. The set of ports of a schema defines its interaction signature. Ports solves Requirement d. Each port define a functionality of a component: A port only accept communication that follow a given session type. Moreover, ports reduce the complexity of the component code: Ports abstract away the communication interconnection between the component inner logic (statically defined) and the activations interactions over channels (dynamic bindings). For instance, sessions primitives take ports as arguments and not channels.

Ports are static since they define the signature of a component schema: new ports can not be added nor removed at runtime. However, bindings between ports and channels depend of activation identity. Those bindings can evolve dynamically, and transparently for the inner activation logic. Operationally, a programmer binds a channel with port using the initial knowledge provided at activation creation (thanks to parameters) or by exchanging channel identity over existing sessions.

2.4 Orchestration Logic

The objective of the orchestration is to write executable code doing dynamic interaction whereas the interaction interfaces describes what messages can be exchanged between components.

The main work of orchestration is to spawn activations and to interconnect ports using channels. Inside a component schema, the orchestration logic is in charge of doing the bindings between communication interfaces with procedural ones. For instance, this is the only work of the method of Listing 2. A programmer can also write the core behaviour of orchestration schemas (e.g., ) using the Varda orchestration logic in order to be completely agnostic to the underlying language. The Varda compiler generates the effective implementation.

In addition to component schema description language, Varda proposes a small imperative and Turing-complete language to write the orchestration logic. Varda language contains classical language constructs (e.g., binders, expression, function, control-flow statement and inductive type); communication primitives to exchange between activations using sessions; and, activation creation primitive: .

2.5 Example: A Minimal Key-Value Store

Listing 1 presents the architecture of a warm-up case study: a key-value store composed of one server and one client. This warm-up example is a piece of Fig. 1a: a ( ), without sharding, that serves requests of one client such that the server use a Redis backend and is collocated with it. This example assumes that the Redis server is already running before spawning a . serves as a proxy to the Redis server.

figure ah
figure ai
figure aj

In Listing 2, specifies the interface of a Redis server. Conversely, specifies of the interface of an application using the key-value store. exposes a communication interface, composed of its port and the communication handling logic method; a procedural interface composed of two abstract methods and bound the black-box service (resp. implementation) and no orchestration logic. The compiler specializes the two abstract methods and during code generation according to implementation bindings (Listing 3). Moreover, the communication handling logic (here the method) is in charge of doing the binding between the communication interface and the procedural interface.

A channel , guaranteeing FIFO delivery for point-to-point communication, interconnects the client with the server (Listing 1). Both client and server discover as an argument. is asymmetric and constrains the communication topology: the left hand side of the channel type (e.g., ) initiate the communication, the right hand side can not. Moreover, communication follows the protocol (technically, a session type [14]): a client can choose between two operations or . Once client chooses the (resp. ) case, the communication must follow the pattern: client sends a and expects to receive a before the session is closed (resp. put).

3 Interception

At this point, one major remaining question is how to easily and safely enrich (or trim) system’s functionalities. For instance, manually sharding the minimal key-value (Listing 1) would be time consuming: a programmer needs to manually (1) create the sharding logic (the router); (2) creates new channels to interconnect the shards (resp. the clients) with the sharding logic; (3) instantiate a router with correct channels interconnections; and (4) for each shards, bind correctly the new channels.

We propose that Steps (2), (3) and (4) should be automatically handled during compilation while followings this requirements: (a) impose arbitrary interception, orthogonal from placement and communication topology, and prevent intercepted activation to bypass the interception mechanism (b) be non invasive and transparent to avoid to the programmer to edit the whole architecture; (c) be generic and modular (d) should be executed efficiently to preserve performance; and, (e) be preserved by composition: multiple alterations could be nested to modularly build a major functionalities.

To address this problem, Varda leverages the interception mechanism as the core Varda primitive used to uniformly apply the orchestration logic. Developers write the interception logic at same abstraction level, and in the same language as the orchestration logic. Then, the Varda transparently and statically rewrite the architecture by adding proxies [38] in between groups of activations.

Varda interception is an architecture construction. This helps preserving the preexisting semantics and formalizing the new architecture. Other approaches work on the network layer and do dynamic interception, as we describe it in Sect. 4.4.

In the following, we review what a programmer can achieve using interception:

  • Message redirection can be achieved by using the same interception instrumentation as sharding, with a custom routing policy (e.g., round-robin for loadbalancing and broadcasting for replication).

  • Dynamically constraining topology (e.g., access control) can be done as long as dropping communication take place at session establishment since sessions can not be discard arbitrarily due to session type guarantees.

  • Encapsulating messages or piggy-packing metadata between activations can also be done even if it is a bit more tricky: the programmer needs to introduce a new intermediate protocol without breaking transparency.

  • Changing the communication behaviour can be performed by intercepting the communication and implementing the communication behaviour inside the interception logic. For instance, programmers can transparently replace a point to point communication by a broadcast.

  • Any combinations of those patterns can be achieved using nested interception contexts.

3.1 What is Interception?

The interception concernes a group of channels in between an internal group of activations and the external one composed of all the remaining activations. In Varda, the programmer has only to enclose the creations of activations, she want to intercept, into an interception scope (using a statement). The interception scope is part of the orchestration code. Therefore, applying interception is orthogonal to defining the logic of the both groups, their interactions and their placement. This solves Requirement a. Whereas, the interception behaviour can depend on those three elements.

Interception concerns both the session establishment and the messages exchanged inside the session. Interception give the ability to the programmer to alter arbitrarily the communications between two groups of activations: message value and session can be alter or delayed. However, the type of the protocol can not be altered arbitrarily, this point will be discussed when detailing transparency.

What is not Interception? Interception is not designed for ensuring security isolation. Interception can no prevent malicious activations to communicate with the external worlds. Indeed, interception works with Varda communication primitives whereas a malicious activation could bypass it from below by using arbitrary communication primitives provided by external code (e.g., sockets). Even if activations only communicate with Varda communication primitives, interception isolation could also be breached by above if an intercepted protocol allows channel exchange (recall that channels are first-class value) and if the intercepted activation dynamically binds this received channel to one of its ports. Varda compiler does not prevent this: breaching interception could be used to removed interception at some point to preserved performance, for instance once an activation has migrated. However, this kind of breaches can by either forbidden: by disallowing channel transmission in the protocol definition: or, mitigated: by checking the identity of forwarded channels inside the interception logic.

figure bg

3.2 Example: A Sharded Key-Value Store

With Varda, transforming the simple key-value store example into a sharded version is a matter of transparently created an interception context containing two , Listing 4. Such that the interception logic, defined as a component called , implements the sharding strategy. The instantiates a singleton activation for the whole interception context. The postpones the establishment of a session between the interceptor and a until the client give enough knowledge (e.g., the key) to select the right . Delaying messages can be tricky, since arbitrary long delay between messages of the same session could be trigger a timeout depending of session implementation.

3.3 Expressing Interception

To setup interception context, programmers have three things to do: (1) define the interception logic by providing an interceptor component schema (e.g., ); (2) delimit the interception scope using a intercept block statement and (3) describe what interceptor activation is in charge of which intercepted activations thanks to an interception policy.

Interception Logic. The interception logic is in charge of processing (alteration, delaying and forwarding) session establishments and messages between internal and external activations. The interception logic is defined as annotated methods to remains generic and not to be specific to a given interception context. That way, programmers do not have to take care of creating the communication interface of the interceptor which depends on the interception context. The compiler is in charge of specializing the interceptor component schema, for each context, in order to create the needed ports according to the intercepted bridges. It binds the annotated methods with generated ports based on methods signature: intercepted session type (and message type for ) and the topology of the communication (defined by and schemas).

Varda provides three methods annotations: , and . methods are triggered at the creation of an intercepted activation. Interceptor needs onboarding to distinguish internal activations from externals one. To preserve transparency, onboarding must be hidden to the intercepted activation (resp. external activations). Hence, it is up to the activation running the interception context to trigger the onboarding. Interception logic can access the set of in the set of . methods are triggered when a session is established, conversely methods are triggered when a message cross the interception border.

Interception Context. Programmers define interception context inside the orchestration logic using a syntactic scope introduce by the statement (Listing 4). Activations spawned inside this scope are intercepted, the others are not. Interception context does not behave like a classical syntactic scope. Indeed, to make the interception fully transparent in term of variable bindings, the interception scope exposes its binders. The parent scope, of the , contains the variables bound inside the interception context. Activation and channels must process with special care not to break isolation, we only describe the activation case for brevity. Activations variables are exposed with the same type but, outside the interception context, they are references to their interceptor activation. This work transparently since the compiler specialize the interceptor schema into a subtype of any intercepted schema, i.e., communication interfaces are equivalent. Moreover, exposed activations may need to embed additional identity information. There are two different use cases: (1) for sharding, identity ( and ) are not exposed because a does not need to distinguish between intercepted activation; whereas (2) to achieve access control with interception, the intercepted activation identity must be exposed since sending a request to differs from sending one to . Identity exposition is managed by using the modifier of the statement: preserve identity whereas erase identity of intercepted activations.

User Defined Interception Policy. Neither the interception logic nor the interception context can expressed how and where interceptor activations are spawned and what is the relation between intercepted activation and interceptor activation (e.g., one to one or many to one). To achieve this, the statement expect a user defined function called . Listing 5 defines a singleton interceptor activation in charge of all intercepted .

figure co

Programmers can use the interception policy to (1) define the relation between intercepted activations and interceptor activation by splitting intercepted activation in groups managed by an interceptor (according to their place, schema and identity); (2) to reuse interceptor activation(s) between interception context; (3) to choose where to place interceptors; and, (4) to customize interceptor arguments in a per context basis.

The policy is called at each spawn of an intercepted activation and it attributes an interceptor activation to each spawned activations. The arguments denotes the schema of the intercepted activation and denotes its place. To make policy generic and strongly typed, the compiler does not pass arguments of the intercepted spawn to the policy.

To relieve programmers of binding the generated ports, of the specialized interceptor, with intercepted bridges (remember that both depends on the context and not only of the interceptor schema). The function spawns interceptor’s activations to relieve programmers of binding the generated ports, of the specialized interceptor, with intercepted bridges (remember that both depends on the context and not only of the interceptor schema). The compiler provides and specializes a factory function for each context.

4 Related Work

4.1 Programming Languages

Classical programming models for distributed computing are actor model [6, 8, 13], service oriented computing [26], dataflow [9] or reactive programming and tierless programming [10]. Recent evolutions tend to focus on easing specific distribution features by incorporating them into programming languages like consistency handling [18, 29, 32, 33], placement aware-computation [37, 40] and builtin fault-tolerance with [13, 23, 34] or without manual control [13]. However, they are not designed to compose black boxes easily and transparently while preserving programmer control on low-level details. This has a high cognitive cost for the programmer and a performance overhead.

4.2 Interface Description Languages

Interface description languages permits to formalize API to some extent and often to derive serialization mechanism and interfaces skeleton. Google’s Protocol Buffer [21] and Apache’s Thrift [19] provide basic typed specification of exchange messages. Hagar [1] extends the type system with polymorphism and generics. However, all of them tend to be limited on what they can specify: they can not reason on values; and, they must be used manually in combination with other tools to build a system which implies that they can not capture the orchestration nor the non-functional requirements.

4.3 Composition Framework

Currently composition mostly rely on interconnecting containerized application [5, 12, 16] or even serverless approach [7, 20, 22, 31]. However, composition frameworks do not achieve safe composition [17, 30]. They mostly work at the network layer which hamper reasoning on the semantics of the composition and of working on non-functional requirements. At a higher level of abstraction, CORBA [39] permits to transparently compose heterogeneous components with well-defined interfaces. They all deport the dynamic interconnections description and management into each component implementation without any general plan, except in English written documents. Regis [28] models communications and dynamic interconnection logic. However, this work do not address non-functional aspect of composition and they do not provide the ability to transform the architecture (like our interception mechanism) which means that every patterns must be established by hand.

4.4 Dynamic Interception

Other approaches providing interception mostly focus on dynamic interception. The use network based interception mechanisms: firewall-like features (e.g., iptables [2], mesh-services [11, 24]) or service workers [35] embedded in browsers. They all lack the ability to describe the effects of the interceptions on the system’s behaviour.

5 Conclusion

We present Varda, an architectural framework designed to build performant and safe distributed systems by composing heterogeneous components. Furthermore, it discharges the programmer from bridging the gap between implementation and design architecture; and that simplifies the writing of classical distribution patterns using a language-based interception mechanism. Varda model rests on three principles: (1) strict separation of concern between architecture and component implementation: one architecture can be used to generated multiple distributed systems; (2) interception is the core primitive to uniformly and transparently apply distribution patterns using static architecture rewriting; and, (3) preserve programmer control on distribution by incorporating dynamic aspects of the architecture (orchestration logic) and by embedding low-level details as first class value (e.g., place, bridges).

We are currently working on the evaluation of Varda: we are investigating the cognitive cost of the model and the performance overhead of the generated glue. Futures works can be divide in two branches: a) the first one targets performance, for instance optimizing the architecture using rewriting (e.g., merging components to avoid context switching); whereas, b) the second one explores how to improve the dependability of distributed system using Varda (e.g., enriching the type system or adding dynamic contracts).