Announcing Marble.js 3.0 - a marbellous evolution

tl;dr

1. Brand new HTTP engine — better performance, new possibilities.

2. Official support for microservices is here. @marblejs/messaging —a new module for event-based communication.

3. Async readers — resolve asynchronous dependencies on app startup.

4. Logging — pluggable, out-of-the-box solution for server/service logging

5. fp-ts 2.0

Reworked HTTP request processing

The basic HTTP/1.1 protocol can be tricky, especially when trying to place it in frames of proper event-based thinking. In its very basic version, every request should come with a corresponding response. It is not a full-duplex communication — first a client makes the request, then a server can respond to it. That means you cannot in an easy way selectively omit/filter/combine incoming requests because every incoming event has to be consumed. The processing enforced by the protocol frames makes things very limited. Some time ago Ben Lesh had a talk about just fledgling Marble.js v0.x. He mentioned some potential use cases that theoretically you can do with Observable of requests, like for example waiting with processing till some other request won’t be triggered. Up till now, Marble.js 2.x reflected every incoming request as a separate stream which prevented the developer from doing a so-called “Rx-magic”.

Version Marble.js 3.0 introduces a brand new, reworked and optimized HTTP request processing engine which can be easily adapted to developer needs. This is a huge milestone for the framework, taking the potential possibilities to the next level. Marble.js 3.0 refines an event-based request processing that the reactive paradigm deserves. The brand new HTTP/1.1 route resolving mechanism can work in two modes — continuous and non-continuous. The second way is the default one which applies to 99% of possible use cases that you can model with REST APIs. Under the hood, it applies a default error handling for each incoming request making the request processing safe but tangled to disposable stream. The last 1% is for all the crazy ones. The new continuous modeallows you to process the stream of incoming requests in a fluent, non-detached way. Having that you can, e.g. buffer/throttle/group incomingrequests, opening the possibilities for future integration with HTTP/2 and HTTP/3 protocols. By treating each incoming request as an event that always has to be consumed, continuous mode involves the necessity of adding a custom error handling mechanism for every processed event and mapping the response with its corresponding request.

const foo$ = r.pipe(
  r.applyMeta({ continuous: true }),
  r.matchPath('/'),
  r.matchType('GET'),
  r.useEffect((req$, ctx) => {
    const reqBus$ = useContext(HttpRequestBusToken)(ctx.ask);
    const terminate$ = reqBus$.pipe(filter(req => req.url === '/flush'));

    return req$.pipe(
      bufferWhen(() => terminate$),
      mergeMap(buffer => from(buffer)),
      mergeMap(request => of(request).pipe(
        mergeMap(processData),
        map(body => ({ body, request })),
        catchError(error => of({
          request,
          status: HttpStatus.BAD_REQUEST,
          body: { error: { message: error.message }}
        })),
      )),
    );
  }));

It wouldn’t be possible without the fact that from now each Effect is eagerly bootstrapped at the app startup, which means that each route is initialized with its own Observable subject. Again, it gives you another set of possibilities in terms of optimization, so e.g. you can resolve all required context readers during the app startup.

Microservices

As mentioned at the beginning, Marble.js defines a uniform interface for asynchronous processing of incoming events. Its interface is very similar to other popular libraries that you can find in the frontend, like — redux-observable or ngrx/effects.

Both libraries had a huge influence on design and architectural decisions. When looking at Marble.js Effect type definition, you can easily notice that it is just a function that processes incoming data in the form of Observable-like streams, which makes event processing easy to adapt to different forms of transport protocols.

interface Effect<I, O, Client> {
  (input$: Observable<I>, ctx: EffectContext<Client>): Observable<O>;
}

interface HttpEffect<
  I = HttpRequest,
  O = HttpEffectResponse,
> extends Effect<I, O, HttpServer> {}

interface WsEffect<
  T = Event,
  U = Event,
> extends Effect<T, U, WebSocketClientConnection> {}

interface MsgEffect<
  I = Event,
  O = Event,
> extends Effect<I, O, MsgClient> {}

But let me come back to the popular buzzword in software development…

… microservices.

They need a lightweight approach for communicating with one another. HTTP/1.1 is certainly a valid protocol but there are better options — especially when considering performance. While you might have used REST as your service communications layer in the past, more and more projects are moving to an event-driven architecture. When a service performs some piece of work that other services might be interested in, that service produces an event — a record of the performed action. Other services consume those events so that they can perform any of their own tasks needed as a result of the event.

Events can be handled in a variety of ways. For example, they can be published to a queue that guarantees delivery of the event to the appropriate consumers, or they can be published to a “pub/sub” model stream that publishes the event and allows access to all interested parties. In either case, the producer sends the event, and the consumer receives that event, reacting accordingly.

Marble.js v3.0 takes the best parts of previously mentioned RxJS libraries and puts them to the frames of the backend world via providing a generic interface for event-based processing despite the underlying transport layer. @marblejs/messaging defines the concept of transport layers, similar to NestJS. They are taking the responsibility of transporting messages between two parties. The messaging module abstracts the implementation details of each layer behind a generic Effect interface. For the sake of beginning, Marble.js 3.0 implements the AMQP (RabbitMQ) and Redis Pub/Sub transport layers. Till the next minor release, the second transport will available under “beta access” restrictions. Additionally, it includes the third “local” layer for EventBus messaging, which can be used for CQRS patterns. Yup, all it is possible with a consistent, uniform interface. More transport layers are supposed to come in future minor releases, e.g. NATS, MQTT or GRPC.

RabbitMQ:

// 📄 publisher - fib.effect.ts

const fib$ = r.pipe(
  r.matchPath('/fib/:number'),
  r.matchType('GET'),
  r.useEffect((req$, ctx) => {
    const client = useContext(ClientToken)(ctx.ask);

    return req$.pipe(
      validateRequest,
      map(req => req.params.number),
      mergeMap(number => forkJoin(
        client.send({ type: 'FIB', payload: number + 0 }),
        client.send({ type: 'FIB', payload: number + 1 }),
        client.send({ type: 'FIB', payload: number + 2 }),
      )),
      map(body => ({ body })),
    );
  }),
);

// 📄 consumer - fib.effect.ts

const microservice = createMicroservice({
  transport: Transport.AMQP,
  options: {
    host: 'amqp://localhost:5672',
    queue: 'fib_queue',
  },
  listener: messagingListener({
    effects: [fib$],
  }),
});

const fib$: MsgEffect = event$ =>
  event$.pipe(
    matchEvent('FIB'),
    act(eventValidator$(t.number)),
    act(event => of(event).pipe(
      map(event => fib(event.payload)),
      map(payload => reply(event)({ payload })),
    )),
  );

CQRS:

// 📄 postOfferFile.effect.ts

const postOffersFile$ = r.pipe(
  r.matchPath('/offers/:id/file'),
  r.matchType('POST'),
  r.useEffect((req$, ctx) => {
    const eventBusClient = useContext(EventBusClientToken)(ctx.ask);

    return req$.pipe(
      validateRequest,
      map(req => req.params.id),
      tap(id => eventBusClient.emit(OfferCommand.generateOfferFile(id)),
      mapTo({ status: HttpStatus.ACCEPTED }),
    );
  }));

// 📄 generateOfferFile.effect.ts

const generateOfferFile$: MsgEffect = (event$, ctx) =>
  event$.pipe(
    matchEvent(GenerateOfferFileCommand),
    act(eventValidator$(GenerateOfferFileCommand)),
    act(event => of(event).pipe(
      map(event => event.payload.offerId),
      // ...
      map(id => OfferFileCreatedEvent.create(id)),
    )),
  );



// 📄 sendOffer.effect.ts    

const sendOffer$: MsgEffect = (event$, ctx) =>
  event$.pipe(
    matchEvent(OfferFileCreatedEvent),
    act(eventValidator$(OfferFileCreatedEvent)),
    act(event => of(event).pipe(
      map(event => event.payload.offerId),
      // ...
      map(id => OfferSentEvent.create(id)),
    )),
  );

Async readers

Context API is the most important thing introduced in the previous major release. Another cool new feature I cannot omit in an introduction to the newest version are “async readers”.

Sometimes there is a need to suspend the application startup until one or more asynchronous tasks or jobs are fulfilled. For example, you may want to wait with starting up your server before the connection with a database has been established. The updated syntax of context readers handles Promises or async/await syntax out of the box in the reader factory. The context container (including Marble app factory) will await a resolution of the promise before instantiating any reader that depends on (injects) async reader.

bindEagerlyTo(Token)(async () => 'bar');

const foo = useContext(Token)(ask);   // foo === 'bar'

// but...

bindTo(Token)(async () => 'bar');

const foo = useContext(Token)(ask);   // foo === Promise<'bar'>

To make the dependency binding more explicit, version 3.0 introduces a new set of factory functions for binding asynchronous and synchronous dependencies — bindEagerlyTo, bindLazilyTo (an alias to bindTo).

Logging

The design decision in version 2 assumed that everything that developers would like to be interested in should be accessible via exposed API in form of observable-like events. Well… the framework evolves and sometimes it’s better to have something working out of the box instead of repeating the same logging boilerplate over and over again every time you create a new service or server.

// HTTP server logger output

λ - 47700 - 2020-01-27 22:04:21 - http [Context] - Registered: "HttpRequestBusToken"
λ - 47700 - 2020-01-27 22:04:21 - http [Context] - Registered: "HttpServerClientToken"
λ - 47700 - 2020-01-27 22:04:21 - http [Context] - Registered: "HttpServerEventStreamToken"
λ - 47700 - 2020-01-27 22:04:21 - http [Context] - Registered: "LoggerToken"
λ - 47700 - 2020-01-27 22:04:21 - http [Router] - Effect mapped: /api/:version GET
λ - 47700 - 2020-01-27 22:04:21 - http [Router] - Effect mapped: /api/:version/user GET
λ - 47700 - 2020-01-27 22:04:21 - http [Router] - Effect mapped: /api/:version/user POST
λ - 47700 - 2020-01-27 22:04:21 - http [Router] - Effect mapped: /api/:version/user/:id GET
λ - 47700 - 2020-01-27 22:04:21 - http [Router] - Effect mapped: /api/:version/user/:id/buffered GET
λ - 47700 - 2020-01-27 22:04:21 - http [Router] - Effect mapped: /api/:version/static/upload POST
λ - 47700 - 2020-01-27 22:04:21 - http [Router] - Effect mapped: /api/:version/static/stream/:dir* GET
λ - 47700 - 2020-01-27 22:04:21 - http [Router] - Effect mapped: /api/:version/static/:dir* GET
λ - 47700 - 2020-01-27 22:04:21 - http [Router] - Effect mapped: /api/:version/error GET
λ - 47700 - 2020-01-27 22:04:21 - http [Router] - Effect mapped: /api/:version/(.*) *
λ - 47700 - 2020-01-27 22:04:21 - http [Router] - Effect mapped: /(.*) *
λ - 47700 - 2020-01-27 22:04:21 - http [Server] - Server running @ http://127.0.0.1:1337/ 🚀

The upcoming version defines a pluggable Logger dependency which is registered by default to the context of every server factory. You can override the Logger binding and map the stdout IO operations to a different destination (e.g. file or external service) when needed. Having that you can inspect things like, registered context dependencies, mapped HTTP routes, request/response logs, input/output event tracing, etc.

fp-ts v2

Marble is the first Node.js framework that tries to spread the word about functional programming to the wider audience. It doesn’t follow FP in its pure form but rather tries to incorporate selected patterns and concepts which can fit well in the framework ecosystem. The version 3.0 finally adds official support to `fp-ts@2.xmaking it as a required peer dependency that has to be installed together next torxjs`.

In `fp-ts@2.xdata types are no longer implemented with classes; the biggest change resulting from this is that the chainable API has been removed. As an alternative, apipe` function is provided, along with suitable data-last top level functions (one for each deprecated method).

Breaking changes

It is hard to build a framework that preserves the backward compatibility in every aspect from the very first iteration, especially when you want to make it better and better. As an opensourcer I’m still learning from release to release and want to deliver the best developer experience as possible without breaking changes… but as you know, it is not so simple. Sometimes you just find a better way of approaching the problem. Your point of view/perspective shifts as you mature as a developer — striving for perfection.

The newest iteration comes with some API breaking change, but don’t worry, these are not game-changers, but rather convenient improvements that open the doors to new future possibilities. During the development process, the goal was to notify and discuss incoming changes within the community as early as possible. You can check here what has changed since the latest version.

Field tested

Marble 3.0 wasn’t released within a planned six-month interval, but there is a reason for that. The middle and end of 2019 were very important for the whole project. For me, as the framework author, it was a chance to validate the functional/reactive concepts against much real, bigger scaled problems. It was the first time when I had a chance to see how people who don’t know reactive and functional paradigms are working with the framework, solving nontrivial problems by showing their own way of thinking and reasoning. It was a big challenge. Since then I can clearly see that the uniqueness of Marble, the reactive manner with functional sugar on top, can also be it’s the worst nightmare, especially if the development team comes from a solid, object-oriented way of programming. It was a hard crossing, especially since that at some point I almost gave up on the whole reactive idea, trying to push the thinking barriers too much forward.

Since the first public release, the framework goal is still unchanged — spread the functional thinking to an as much bigger audience as possible. Functional programming is not scary, but requires a different way of approaching problems, especially when combined with reactive philosophy on top.

Marble.js is the most ambitious project that I’ve ever made. The framework is growing from version to version. Trends show that more people are interested in functional or reactive programming topics. The project requires a solid maintenance, community support, documentation, and clear processing frames. At this point the community support is priceless. Dear supporters and contributors, I would like to officially thank you and encourage to take a part in discussions that for sure will have a real impact on the future of Marble.js platform. Your voice is the most precious thing. Thank you!

Keep an eye on our GitHub Space Flight Center, Twitter, and docs.marblejs.com in order to grab the latest information and updates from the space!

Happy hacking and be prepared for the next Big Bang! 🚀

Thanks to Dawid Yerginyan.