Things are certainly moving fast. However, we've clearly been lacking in the blog department. To remedy this here's a summary of all the stuff we would have blogged about if we had more time and bother.

Community Growth​

Benthos has now surpassed 150 contributors, and between our discord and slack communities we're steadily approaching 1,000 gossips, at which point we'll be too mainstream to be considered cool and hip. This site has daily traffic in the thousands and Benthos itself is downloaded around 2,000 times per day. The Jeffail youtube channel, which features all of our Benthos and stream processing related video content, is also growing steadily in both content and precious subscribers. I'd promise none of this traffic is bots but I actually don't know.

Another major milestone reached is that I watched witlessly as my wife gave birth to a human child, she's never done that before. In order to support the incoherent squawker (and our new child) I took a month away from the project. If you're a keen follower of the open source work going on in Benthos land then you'll be aware that it has historically been a mostly one-person show (me), but I'm thrilled (perhaps a bit sour) to report that during my absence the project carried on at pretty much the same intense pace.

These are all signs that our codebase, community support and developer ecosystems are becoming more decentralised. This trend in growth is entirely organic (besides a few stickers sent to conferences) and mostly based on the volunteer effort of professionals that use Benthos in their daily work. We're also blessed with a number of hobbyists getting involved just because they enjoy it, we've tried sapping the fun out of the project but they just won't go away. For now I'm still retaining absolute control over what gets merged but the bus factor is certainly becoming less significant.

The trends I'm outlining here should be soothing for Benthos fans, we're growing, we're managing the growth just fine, and you can safely bet on the project continuing on that path. Over the years some users (active or prospective) have expressed concern that Benthos does not have an organisation around it with a pot of money, their worry being the longevity of the project is at risk. I've obviously not agreed as I haven't yet sought after such a pot of money (more on that later in this post). Over time it has become easier to shrug those concerns off as we continue to move at a pace that'd easily beat the expectations of any funded operation.

Benthos Studio​

I've been quietly working on Benthos Studio pretty much since the early years of Benthos itself. The idea being a visual editing application to complement the development and running of Benthos configs. What I like about having this exist as an entirely separate application is that the main project is still incentivised to make configuration as easy and intuitive as possible, we can't just rely on visual tooling to plug gaps in ergonomics or observability. This allows Studio to focus on lifting that experience up a few notches as an optional extra.

If you aren't familiar with it then here's a quick video introduction:

Studio is currently offered as an open beta with new stuff coming in constantly. The main directions we're heading in are:

• More storage flexibility, including allowing you to use Studio to view and configure streams running in your own local deployments
• More execution visibility, better visuals and ability to dig into what's happening within a Benthos configuration
• A smoother configuration experience for beginners, including wizards for building new stream pipelines

You may also have noticed that Benthos Studio is not open source. This is because I believe Studio is a good bet at putting together a scalable monetisation strategy that complements the goals of the open source work rather than causing friction against it, a sort of golden path for building a business around an existing open source ecosystem. It will take a while to work out which features should be paid for, but there will always be a free tier that provides all the interesting bits.

Monetisation​

Uh oh! He said monetisation, bag him! Monetising an open source project is a hefty topic and gets lots of people mad and sweaty, one far end of the spectrum believing it's entirely counter to the open source movement, the other end refusing to take seriously any project that isn't monetised.

The reality is obviously that neither extreme holds merit. People need to eat and when projects reach a certain size the time required to maintain them goes well beyond the scope of a fun hobby, but conversely there's nothing preventing a non-funded project from outperforming a funded one even with enterpisey things like support and product stability. A common feedback we get is that our stability and support goes well beyond what people are used to even with paid products. So, given that and all the stuff I outlined at the beginning of this post, do we even need funding?

Well yes, we do. In fact, as the main driving force behind Benthos I've been funded since day one, just not directly. I get returns from my Benthos work in the form of job opportunities that feed back into the project, support contracts, issue bounties and sponsorships dotted throughout. This is a similar story for many open source maintainers, where if you keep yourself from burning out then a project can get very far indeed without needing to commit to a more scalable business model.

This set up has been working out just fine but it's far from ideal. I spend a lot of energy keeping these gigs going which could be much better spent growing a team around a paid product. This team would then steward and give back to the open source community and we would all win, the only losers being people that hate teams, products or the abstract concept of winning, or me as a person.

Many may have jumped straight into the deep end and asked for VC funding up front. If/when we choose to go down that route we're locked in for the whole ride, where in abstract terms if that aforementioned golden path of monetisation gives us problems then we're going to be pressured into following a new path that prioritises the venture and not the open source. Well, that's life, every decision has risks and in reality we'd still do well as a community in any outcome, but for now I've been exploring that golden path and seeing how far I can get on my own.

So that's how we got to here, we have a ship that seems pretty much sea worthy and without any obligations or milestones to deliver. Dare I remain bootstrapped? Perhaps I should join a fleet? Do we finally rocket boost these salty decks and aim for the stars? Is that a dumb metaphor? Well keep checking this blog to find out a few months after all the Discord and Slack users do.

Supporting the Project​

The longer this project remains sustainable without obligation the more we can experiment freely and independently. If you want to help stretch that progress further and maybe help keep us on the golden path then get involved in the project, get your organisation to sponsor my work, or consider some of the paid support options we're currently offering.

We're now also at a point where the breaking changes that we want to include in v4 have been completed on a branch, and a migration guide is ready to use which describes how to prepare for v4. Therefore, it seems to me, that we're ready to actually prepare the release.

Why​

Benthos follows semantic versioning, which means a major version release is not necessarily an indication of significant improvements to the product, but an indication that things have changed and therefore upgrading should be done with care (and likely some effort). As such, all the major improvements that could have been part of this release (config templates, new plugin APIs, etc) are already part of version 3 and being actively used.

The vast majority of these improvements have been an evolution or consolidation of prior features, which have since become deprecated (and hidden from the documentation). Doing things this way has enabled you to slowly learn about and adopt the nice new goodies whilst also running the old stuff. However, the reality is that the old deprecated features are a maintenance burden. Not only do we need to ensure that a growing number of potentially unused features remain bug free, but these features often frustrate or entirely prevent us from making improvements elsewhere.

As such, after keeping v3 going for more than two years, it feels like it's about the right time for spring cleaning. The absolute top priority of this release is to remove deprecated components, which for people keeping up with the newest features would in theory mean no breaking changes at all. However, since major releases are a rare event (this could possibly be the last one) it also makes sense to use this as an opportunity to change things about Benthos that we feel aren't intuitive.

The biggest of these intentionally breaking changes is the way in which metrics work. Benthos exposes a huge amount of metrics but the names can often be confusing and difficult to understand, and some components expose inconsistent or undocumented metrics. For v4 they have been massively simplified and made consistent, leaning on labels/tags for presenting (optionally) granular series, and allowing full customisation for all metric destination types with a generic mapping field.

The full list of changes we've decided to add are outlined in the the migration guide, which is an attempt at finding a good balance between keeping to a minimal impact for the majority of users, and tackling the biggest culprits of confusion in Benthos land. It's extremely important to stress that at this stage nothing is absolute, we can pull these changes back if they seem to heavy, and likewise we can push further if there's hunger for more. Ultimately, we need feedback in the coming weeks in order to get this right.

Timescales​

None of this is set in stone but it currently looks as though we will have a release candidate available for v4 in the coming weeks. You can expect edge tagged docker images to include v4 changes soon, followed by an image tagged 4.0.0-rc1 and so on, at the same time there will be artifacts you can obtain from the github releases page: https://github.com/Jeffail/benthos/releases.

At this stage we'll be looking for general testing as well as feedback on the migration process. If you find yourself in a situation where a feature you rely upon in v3 is gone and you can't find an alternative in v4 then we want to hear about it as soon as possible. There will be a temporary website up with v4 documentation but the URL will change each time the docs are updated, so we'll post them in several of the community spaces, make sure you're in one if you're interested in testing.

We'll likely be playing with release candidates for a few weeks, in which time any official releases we put out will be v3 patches for any bug fixes we've added in the meantime. Once we're happy with release candidates the official v4 tag will be pushed, built and the documentation site will be bumped to represent v4. When this happens the domain v3.benthos.dev will be set up to contain the v3 documentation.

Preparing​

If you're interested in minimizing the effort to migrate to v4 right now then the first thing worth doing is moving away from any deprecated components in your v3 configs. You can detect them by pulling the latest version of v3 and running the linter with the --deprecated flag:

benthos lint --deprecated ./configs/*.yaml

This will report any deprecated components and fields that you still have in your configs.

Next, if you're using plugins or have a custom build of Benthos then you need to ensure you're no longer using any of the packages within ./lib, all of those packages are being removed in v4 in favour of the new ./public packages. The plugin APIs within ./lib all have a newer (and better) alternative, but if you were using some of the misc packages as helper libraries then copy them from the v3 branch into your codebase.

The metrics changes are likely to affect almost every deployment, it is recommended that you make the best use of the new labels and rework your dashboards. However, if in your case the dashboards are difficult to change then it is likely easier to use a mapping in your new v4 config to reproduce the metrics series your dashboards currently rely upon. In order to prepare for this it's worth making an inventory of each series you use, the labels they have, and preparing the mapping you'll need. The documentation for the new metrics system can be found here: https://github.com/Jeffail/benthos/blob/main/website/docs/components/metrics/about.md

Finally, it's worth getting familiar with the changes outlined in the migration guide. There are breaking changes in some components, mostly default values being changed in order to make them more intuitive, but they should all be understood before upgrading in order to avoid unintended consequences.

Please Help​

If you're interested in helping out then we'd love to get you involved and the best way is to read the migration guide, try out the release candidates, and give us feedback in any of the community spaces.

The major concern that we're looking for feedback on is whether the breaking changes are manageable for you, if you have identified things that will significantly hinder your migration then it's important we hear about it as soon as possible. This allows us to triage whether it's worth building tooling to help with the transition, or to potentially roll-back those changes if necessary.

The full API docs can be found at pkg.go.dev/github.com/Jeffail/benthos/v3/public, and there's an example repository demonstrating a few different component plugin types at github.com/benthosdev/benthos-plugin-example.

Now that most of those features are ready to test, namely a new plugins API and config templating, I'm looking for people to try them out and give feedback. Please read on if that sounds like fun to you, or also if it doesn't sound fun but you intend to do it anyway.

Config Templates​

The new config templates functionality allows you to define parameterised templates for Benthos configuration snippets. These templates can then be imported with a cli flag and used in Benthos configs like native Benthos components.

This is going to be super useful in situations where you have commonly used configuration patterns with small differences that prevent you from using resources.

The current state of templates is that they'll be included in the next release as an experimental feature, meaning any aspect of this functionality is subject to change outside of major version releases. This includes the config spec of templates, how they work, and so on.

Defining a template looks roughly like this:

name: log_message
type: processor
summary: Print a log line that shows the contents of a message.

fields:
  - name: level
    description: The level to log at.
    type: string
    default: INFO

mapping: |
  root.log.level = this.level
  root.log.message = "${! content() }"
  root.log.fields.metadata = "${! meta() }"
  root.log.fields.error = "${! error() }"

And you're able to import templates with the -t flag:

benthos -t ./templates/foo.yaml -c ./config.yaml

And using it in a config looks like any other component:

pipeline:
  processors:
    - log_message:
        level: ERROR

To find out more about configuration templates, including how to try them out, check out the new templates page. More importantly, you can give feedback on them in this Github discussion.

The V2 Go Plugins API​

Benthos has had Go plugins for a while now and they're fairly well received. However, they can sometimes be confusing as they expose Benthos internals that aren't necessary to understand as plugin authors.

It was also an issue for me as a maintainer that the current plugin APIs hook directly into Benthos packages that have no business being public. This makes it extra difficult to improve the service without introducing breaking changes.

The new APIs are simpler, more powerful (in the ways that matter), add milk after the water, and most importantly are air-gapped from Benthos internals so that they can evolve independently. Here's a sneaky glance of what a processor plugin looks like:

type ReverseProcessor struct {
	logger *service.Logger
}

func (r *ReverseProcessor) Process(ctx context.Context, m *service.Message) ([]*service.Message, error) {
	bytesContent, err := m.AsBytes()
	if err != nil {
		return nil, err
	}

	newBytes := make([]byte, len(bytesContent))
	for i, b := range bytesContent {
		newBytes[len(newBytes)-i-1] = b
	}

	if bytes.Equal(newBytes, bytesContent) {
		r.logger.Infof("Woah! This is like totally a palindrome: %s", bytesContent)
	}

	m.SetBytes(newBytes)
	return []*service.Message{m}, nil
}

func (r *ReverseProcessor) Close(ctx context.Context) error {
	return nil
}

func main() {
	err := service.RegisterProcessor(
		"reverse", service.NewConfigSpec(),
		func(conf *service.ParsedConfig, mgr *service.Resources) (service.Processor, error) {
			return &ReverseProcessor{logger: mgr.Logger()}, nil
		})
	if err != nil {
		panic(err)
	}

	service.RunCLI()
}

You can play around with these APIs right now by pulling the latest commit with:

go get -u github.com/Jeffail/benthos/v3@master

And you can find more examples along with the API docs at pkg.go.dev.

The package will remain in an experimental state under public/x/service for a month or so, and once it's "ready" (I'm personally happy with it) then it'll be moved to public/service and will be considered stable.

The goal is to allow everyone to migrate to the new APIs whilst still supporting the old ones, and then when Benthos V4 is tagged the old ones will vanish and we're no longer blocked on them.

Similar to the templates there is a Github discussion open for feedback. Be honest, be brutal.

Join the Community​

I've been babbling on for months so if this stuff is news to you then you're clearly out of the loop. Worry not, for you can remedy the situation by joining one or more of our glorious community spaces.

Combining the power of Redpanda and Benthos for your streaming needs is so simple that this blog post is almost over already.

Benthos is an open source stream processor that provides data mapping, filtering, hydration and enrichment capabilities across a wide range of connectors. It is driven by a minimal, declarative configuration spec, and with a transaction based architecture it eliminates the development effort of building resilient stream processing pipelines.

Likewise, with its simplicity and high performance, Redpanda eliminates the operational effort of data persistence and availability by providing a Kafka-compatible streaming platform without the moving parts.

With so much taken care of you're well in for a boring, uneventful time when you combine the two. Make sure you've grabbed a copy of both services, full instructions can be found in the getting started guide for Benthos and the Redpanda docs. In this post we'll be running them with Docker so we'll start by pulling both images:

docker pull vectorized/redpanda:latest
docker pull jeffail/benthos:latest

We can then create a new network for the services to connect with:

docker network create -d bridge redpandanet

Next, run Redpanda in the background, we'll go with a single node for now:

docker run -d \
  --network redpandanet \
  --name redpanda \
  -p 9092:9092 \
  vectorized/redpanda redpanda start \
  --reserve-memory 0M \
  --overprovisioned \
  --smp 1 \
  --memory 1G \
  --advertise-kafka-addr redpanda:9092

In order to send data to Redpanda with Benthos we'll need to create a config, starting off with a simple Stdin to Kafka pipeline, copy the following config into a file producer.yaml:

input:
  stdin: {}

output:
  kafka:
    addresses: [ redpanda:9092 ]
    topic: topic_A

Pro tip: You can also use Benthos itself to generate a config like this with docker run --rm jeffail/benthos create stdin//kafka > ./producer.yaml.

And now run Benthos by adding the config as a Docker volume, along with a pseudo-TTY for writing our messages:

docker run --rm -it \
  --network redpandanet \
  -v $(pwd)/producer.yaml:/benthos.yaml \
  jeffail/benthos

This will open an interactive shell where you can write in some data to send. Benthos will gobble up anything you throw at it, try mixing structured and unstructured messages, ending each message with a newline:

{"id":"1","data":"a structured message"}
but this here ain't structured at all!
[{"id":"2"},"also structured in a different (but totally valid) way"]

When you're finished hit CTRL+C and it'll exit.

Next, let's try reading that data back out from Redpanda, this time let's also add a processor in order to mutate our data, copy the following into a file consumer.yaml:

input:
  kafka:
    addresses: [ redpanda:9092 ]
    topics: [ topic_A ]
    consumer_group: example_group

pipeline:
  processors:
    - bloblang: |
        root.doc = this | content().string()
        root.length = content().length()
        root.topic = meta("kafka_topic")

output:
  stdout: {}

And run it with our new config, and without the pseudo-TTY this time:

docker run --rm \
  --network redpandanet \
  -v $(pwd)/consumer.yaml:/benthos.yaml \
  jeffail/benthos

Now you should see it print mutated versions of the messages you sent to Stdout:

{"doc":{"data":"a structured message","id":"1"},"length":40,"topic":"topic_A"}
{"doc":"but this here ain't structured at all!","length":38,"topic":"topic_A"}
{"doc":[{"id":"2"},"also structured in a different (but totally valid) way"],"length":69,"topic":"topic_A"}

The Bloblang processor in our consumer config has remapped the original message to a new field doc, first attempting to extract it as a structured document, but falling back to a stringified version of it when it's unstructured. We've also added a field length which contains the length of the original message, and topic which contains the Kafka topic the message was consumed from.

That's it for now, if you're still hungry for more then check out the Benthos website at https://www.benthos.dev, and you can learn more about the Benthos mapping language Bloblang in this guide.

However, eventually it would be nice to cut a new major release and prune all of the dead weight that has accumulated during this time. Since major version releases don't come often I wanted to be sure that we've considered and planned any other potential breaking changes that could be bundled along with it.

Up until now Benthos has never had a roadmap or really any plan beyond just building what we want to use or want to build, this is known in the industry as attention-span-driven development. Alas, if we're going to get mileage out of version 4 then some planning is necessary, and I figured we might as well put together our very first roadmap.

A few months ago I asked for feedback, I already had my own wish list of things to change in the next major release but I wanted to give you all an opportunity to factor in your own use cases. I've attempted to capture all of the feedback and create issues for the stuff that's achievable, then I marked the issues that require breaking changes and added them to my roadmap plans. I think it's currently in a state that works for me and is something deliverable, therefore I think it's now worth sharing and allowing you all to help shape it further.

Benthos is blessed with a decent and growing number of contributors. However, it's still clear that if I personally were to burn out then the project would pretty much grind to a temporary halt, and therefore my sanity is a higher priority than committing to a rigid plan. Here's a few things to clarify about this roadmap before you get too excited:

1. This isn't final, it's going to mutate over time in order to flex around "everything else" going on.
2. This isn't everything. The only items included in this roadmap are items that I consider required to have ready for v4. Any features that can definitely be implemented without breaking changes are not included and can be worked on at any time, including right now.
3. There is no timeline or estimate for this work (by design). If you are blocked on any of the items on this roadmap and aren't able to contribute then please still make sure I'm aware and I'll factor that in, but do not expect promises or commitments (unless you're paying for them).

With that made clear and everyone sufficiently bored let's get into the planned work as it currently stands. I've created an issue for every item here where you can read more details beyond my elevator pitch.

Improved plugin APIs​

Click here to access the issue.

This is by far the biggest item of work I want to establish before v4. The plugin APIs are currently heavily tied into the same component interfaces that are used internally. This means that it's not possible for me to modify the signatures of internal components without breaking the plugin APIs. This has historically put us in awkward positions where in order to make a change that's backwards compatible with both our configuration spec and the plugin APIs we have to implement nasty tricks.

If we're instead able to isolate the plugin APIs with an air gap then it will allow us to iterate on the internal components without impacting the APIs used for plugins.

The plan is to fully implement an isolated (and nicer) plugin API, give everyone a lot of time to try it out, provide feedback, and migrate, all within good time before v4 so that I don't pull the rug out from under current plugin users.

Streams Mode API for Resources​

Click here to access the issue.

This one's pretty simple, we want to expand the streams mode APIs to allow the mutation of resources. This is blocked behind a breaking change (to the plugin APIs) as it would require sweeping changes to how resources are accessed.

Input Scheduling Capabilities​

Click here to access the issue.

Sometimes it's nice to slow things down, this issue would allow us to configure inputs that are triggered in scheduled bursts rather than realtime streams in order to have them behave similar to batch processors. Implementing this will require a minor review of the input initialization flow, which could potentially lead to breaking changes to the internal API.

Configuration Templating​

Click here to access the issue.

This would allow you to create reusable, parameterized, configuration templates and have them natively supported within Benthos. This issue is pretty dope but also a significant amount of work, it could easily result in breaking changes being required and so I'd like to have this at least planned out and understood before v4.

Improved Logging​

Click here to access the issue.

As Benthos has evolved it has gained a few oddities in how logging works. This issue adjusts logging to lean more into structured logging fields and update the configuration defaults to be more sensible. This will mostly impact internal components that create logs, and therefore depends on having the isolated plugin APIs.

Improved Metrics​

Click here to access the issue.

Similar to the logging issue, metrics in Benthos are a bit wonky due to the collision between targets that do and don't support labels/tagging. Since Prometheus and other tag based metrics types seem to be winning out nowadays I think we can flip the defaults to favour tags over long metric names.

Configuration File Reloading​

Click here to access the issue.

Pretty much self explanatory. I believe this can be implemented without any breaking changes, but it would be good to have it understood (or finished) before v4 just in case.

Tracking these Features​

There's a project on Github containing all of these issues, but the way that I've configured it is unique as issues aren't necessarily tracked by their progress. Issues in the "Blocked" column are unable to progress without a breaking change and therefore are blocked on v4. Issues in the "Unblocked" column are features that can be worked on, and will either become done if they were able to be completed without breaking changes, or will be put back into "Blocked" once they reach a point where breaking changes are needed.

Once the "Unblocked" column has been emptied, and all of our v4 issues are either blocked or done, that will indicate that we are ready to commit to a new major version release, at which point a v4 branch will be created and that work can be started.

I'm hoping that this will make it easier for me to minimize disruption. Ideally, I want the process of implementing Benthos v4 to be a simple case of deleting old deprecated stuff, and then removing flags/feature toggles in order to make new breaking features the default, having already been implemented and tested. There should be no green field work as part of the new v4 branch.

What's Next​

Make sure you get your thoughts and opinions added to the issues you're interested in. I'm also going to try and open up mini forums over our Discord server to get feedback on the plans. If any of these issues are something you'd personally like then please add a thumbs up emoji to it, as that helps me prioritize them.

If you're interested in getting involved then make sure you've joined one or more of our glorious community spaces.

The Motivation​

After similar efforts to improve the mapping story in Benthos it seemed sensible to target workflows. Specifically, I've added a new branch processor for wrapping child processors in request/result maps, and have reworked the workflow processor to use them.

If you haven't used workflows in Benthos then there's a section in the new workflow processor page outlining why they're useful. In short, when performing multiple integrations within a pipeline such as hitting HTTP services, lambdas, caches, etc, it's best to perform them in parallel when possible in order to reduce the processing latency of messages, organizing these integrations into a topology with a workflow makes it easier to manage their interdependencies and ensure they're executed in the right order.

In the old world you could use the process_dag processor which has child process_map processors, where the mappings were a series of clunky to/from dot paths, separated into optional and non-optional mappings. There was no way to manually specify the dependency tree, and conditional flows required a separate list of conditions which didn't factor into dependency resolution.

Having such complex and brittle mapping capabilities meant these processors were difficult to document and more so to understand and use.

Leaning into Bloblang​

Thankfully, with Bloblang now finished it was pretty easy to replace most of the complexity of the workflow mappings for the language itself.

For example, when mapping the request payload for an integration you can express a bunch of different patterns...

Empty request body:

request_map: root = ""

Sub-object (foo) as request body, if the sub-object doesn't exist (or is null) the integration is abandoned:

request_map: root = this.foo.not_null()

Sub-object as request body which can be obtained from one of a number of possible paths:

request_map: root = this.(foo | bar | baz).doc.not_null()

Conditional integration applies when the type is foo, with an unmodified message as request body:

request_map: |
  root = if this.type != "foo" {
    deleted()
  }

Conditional integration applies when the type is foo, with a sub-object as the request body:

request_map: |
  root = if this.type == "foo" {
    this.foo.not_null()
  } else {
    deleted()
  }

Similarly, it's possible to express a bunch of things in the result mapping...

Discard the result (the original message is unchanged):

result_map: ""

Place the entire result at a path:

result_map: root.foo = this

Place the result in a metadata field:

result_map: meta foo = this

If you want to see what it looks like there is an enrichment cookbook that demonstrates workflows in action, but there are also smaller examples on the workflow page such as the following snippet:

pipeline:
  processors:
    - workflow:
        meta_path: meta.workflow
        branches:
          foo:
            request_map: 'root = ""'
            processors:
              - http:
                  url: TODO
            result_map: 'root.foo = this'

          bar:
            request_map: 'root = this.body'
            processors:
              - lambda:
                  function: TODO
            result_map: 'root.bar = this'

          baz:
            request_map: |
              root.fooid = this.foo.id
              root.barstuff = this.bar.content
            processors:
              - cache:
                  resource: TODO
                  operator: set
                  key: ${! json("fooid") }
                  value: ${! json("barstuff") }

Conclusion​

The docs have been updated to use these new goodies. Obviously the old processors are still being maintained but in a mostly dormant state. The workflow and branch processors are currently labelled as beta, but their general behavior is stable with the only exceptions being odd edge cases that might arise.

With the behavior of these processors being dramatically simplified I've also been able to simplify the documentation for them, which also means using more space on the page for example configs.

If you have feedback then get the absolute heck in the chat you utter recluse.

The Motivation​

In the last post I outlined my motivations for experimenting with a mapping language. Words are stupid and boring and so to illustrate why a mapping language kicks ass here's a config example using the old processors compared to the new one. Keep in mind that the new version is simpler and performs better.

Using old processors:

pipeline:
  processors:
  - metadata:
      operator: set
      key: bar
      value: ${!json_field:foo.bar} 

  - json:
      operator: delete
      path: foo.bar

  - json:
      operator: set
      path: foo.topic
      value: ${!metadata:topic} 

  - metadata:
      operator: delete
      key: topic

  - conditional:
      condition:
        jmespath:
          query: "foo.baz == 'thing'"
      processors:
      - json:
          operator: set
          path: foo.thing_id
          value: ${!uuid_v4}

Using Bloblang:

pipeline:
  processors:
  - bloblang: |
      root = this

      foo.topic = meta("topic")
      meta topic = deleted()

      meta bar = foo.bar
      foo.bar = deleted()

      foo.thing_id = match {
        foo.baz == "thing" => uuid_v4()
      }

My ultimate intention is to completely eradicate the need for a json, metadata and text processor, as well as a range of others. However, I'll need as much help as possible to get the language right, so please consider testing and feeding back on Github, the Gitter channel, or event @ me on Twitter for the good of blobkind.

Why​

My life has no meaning. Also, mapping is one of the most common boring tasks in stream and event processing. Given Benthos is meant to specialise in the boring and mundane it makes sense to treat mapping as a first class citizen.

Up until now the story for mapping documents in Benthos has been to use JMESPath, AWK or a string of the general purpose JSON processors. Time and time again it has been made apparent that it ain't good enough for many use cases.

I should mention at this point that there's also the option of IDML, and although Benthos hasn't supported it internally there is a solution to running it in your pipeline.

For the last few years I've been helping users adopt these options and each time they fall short I've taken note of where the gaps are. This is an important part of the "research" phase for a language, but I also don't want to dwell on it. Here's an insultingly terse summary of what we currently have within Benthos.

JMESPath​

The spiritual cousin of jq, JMESPath is a great spec for mapping JSON documents, especially so when your intention is to outright replace the original document.

However, when our goal is to preserve the majority of the existing document, and we only wish to express isolated mutations within the structure, it becomes ugly and risky. For example, changing just foo.bar.baz to this value looks like this:

merge(@, {
  "foo": merge(foo, {
	  "bar": merge(bar, {
	    "baz": "this value"
	  })
  })
})

Hopefully you don't add a typo there or miss on a merge, otherwise you're scrapping a large chunk of your original document!

Expressing your entire map in one single object also scales pretty poorly as the mapping grows in complexity.

A final and Benthos specific issue is that JMESPath only supports mapping the content of Benthos messages, without the ability to modify or reference the metadata of a message or other messages of a batch, which would be great for windowed processing.

AWK​

Benthos has an AWK processor, and since this is a proper programming language it has uses far beyond mapping. However, this also makes it riskier to use for large and complex maps. More opportunities to write bugs, more opportunities to break your program, more opportunities to regress.

A simpler language specifically designed for mappings is a much more scalable solution as it reduces the opportunities for mistakes as both maps and teams grow. Although, risk aside, the major problem with using AWK within Benthos is the performance hit.

JSON Processor​

The JSON processor is pretty flexible and would be the highest performer of all options here. However, beyond one or two mutations a mapping becomes an absolute mess of YAML, and if we need to add conditional maps into the mix it becomes much worse.

It has been clear to me for a while that this processor is so quickly and easily outgrown by a typical user config that it perhaps ought to be entirely replaced with a real mapping solution.

IDML​

If I could run IDML natively from Benthos then Bloblang wouldn't be happening. In my opinion IDML is a criminally underused technology and absolutely nails the issue of mapping data at scale.

Similar to JMESPath the language itself doesn't have a concept of metadata, or querying across multiple documents (a batch). The issue I had here was that if I were going to go through the trouble of implementing IDML in Go I might as well add metadata and cross-batch querying, making it a different language anyway.

However, I'm definitely writing Bloblang with IDML in mind, and if I manage to reach feature parity with IDML then I intend to break it out into its own lib and offer it to the org, with my Bloblang extensions as Benthos specific plugins.

Features​

So with that in mind what does Bloblang look like? Right now we only have queries, which is the "right hand side" of a mapping. These queries support literals:

"string literal"
true
93435.45

And arithmetic:

50 + 34
("this" == "that") || ("that" == "that")

And functions:

json("foo.bar.baz")
meta("kafka_key")
timestamp_unix()

And methods, which are attached to a function or value:

json("foo.bar.baz").from_all().sum()

And path literals with coalescing:

json().foo.(bar | something_else).baz

Next Steps​

In terms of core syntaxes Bloblang is basically complete. It's implemented using parser combinators, and is very easy for me to extend with new functions and methods. Soon I'll expand Bloblang to support left hand query targets, which is when it really becomes a mapping language. It'll look something like this:

pipeline:
  processors:
  - bloblang:
      mapping: |
        json.foo.bar = json().(something + another.thing)
        json.and_this = meta("kafka_key").base64()

And I'll also add a condition type for expressing logic as a Bloblang query:

pipeline:
  processors:
  - filter_parts:
      bloblang:
        query: |
          (meta("kafka_topic") == "junk") &&
            json().foo.(bar | baz.quz).id.contains("blah")

Until I'm allowed to practice with my professional rock paper scissors team again I'm sure each weekend will deliver something new to the world of Bloblang.

Too lazy to read? You can find a video equivalent of this post at: https://youtu.be/Ilah_Y0uMk4. If you prefer to dig straight into code then you should check out the benthos-plugin-example repo.

Plugins allow you to embed your code within Benthos as a component. Processors are the most common type of component to get plugged, which is what we're going to do in this post. If you want to run non-Go code from Benthos then you still have options, such as the subprocess, http or lambda processors.

Roleplay​

Imagine you are a competent engineer. You wrote a function to detect sarcasm in internet posts on a linear scale of 0 to 100:

// HowSarcastic TOTALLY detects sarcasm EVERY time.
func HowSarcastic(content []byte) float64 {
	if bytes.Contains(content, []byte("/s")) {
		return 100
	}
	return 0
}

You are confident that HowSarcastic is 100% accurate and wish to apply it to a continuous stream of data by deploying it within a stream processing solution.

You want this service to be resilient with at-least-once delivery guarantees, scalable both horizontally and vertically, and able to expose various metrics about the health of the data stream.

You have decided to use Benthos for this service because you love the logo.

Stuff you don't need to care about yet​

Since you are using Benthos you don't need to choose a queue system, metrics aggregator or deployment platform yet, those items can be configured.

You don't even need to know what format the data comes in or how it needs to look when it leaves your service, as Benthos has plenty of processors for configuring that stuff on the fly.

Getting Started​

You're going to use Go modules for this one, make a directory and create a go.mod file:

mkdir foo && cd foo
go mod init github.com/bar/foo

Next, you need to pull in your only dependency, Benthos:

go get github.com/Jeffail/benthos/v3
# Look! Now you have more dependencies than friends!

That'll automatically add the dep to your go.mod file at the latest v3 tag. Next, you're going to write your stream processor service. Write this to the file main.go:

package main

import (
    "github.com/Jeffail/benthos/v3/lib/service"
)

func main() {
    service.Run()
}

That's it, you've got a full Benthos. If you want to verify then you can run it:

go run ./main.go --help

Write Your Plugin​

Now you will write the actual plugin that executes your function. Processor plugins implement types.Processor and have the signature:

func ProcessMessage(msg types.Message) ([]types.Message, types.Response)

A message can have multiple parts (synonymous with a batch) and we are allowed to return either one or more messages or a response which is either an ack or noack.

A message part has both content and any number of metadata key/value pairs. It is therefore up to you as to whether you modify the contents of messages or whether the sarcasm level is added as metadata.

Thankfully you don't need to make that decision now. Instead, you're going to expose it as a config field and support both. The config field will be called metadata_key, and if left empty the contents of messages will be replaced entirely with the sarcasm level.

There won't be much code needed so for brevity you are going to write this straight into your main.go file:

// SarcasmProc applies our sarcasm detector to messages.
type SarcasmProc struct {
	MetadataKey string `json:"metadata_key" yaml:"metadata_key"`
}

// ProcessMessage returns messages mutated with their sarcasm level.
func (s *SarcasmProc) ProcessMessage(msg types.Message) ([]types.Message, types.Response) {
	newMsg := msg.Copy()

	newMsg.Iter(func(i int, p types.Part) error {
		sarcasm := HowSarcastic(p.Get())
		sarcasmStr := strconv.FormatFloat(sarcasm, 'f', -1, 64)

		if len(s.MetadataKey) > 0 {
			p.Metadata().Set(s.MetadataKey, sarcasmStr)
		} else {
			p.Set([]byte(sarcasmStr))
		}
		return nil
	})

	return []types.Message{newMsg}, nil
}

// CloseAsync does nothing.
func (s *SarcasmProc) CloseAsync() {}

// WaitForClose does nothing.
func (s *SarcasmProc) WaitForClose(timeout time.Duration) error {
	return nil
}

Let's break this down. You have a struct called SarcasmProc, which contains a configuration field MetadataKey. The functions CloseAsync and WaitForClose can be ignored as your processor doesn't contain any state that requires termination.

Within your function ProcessMessage you iterate all the payloads within the message batch and calculate the sarcasm level with your function HowSarcastic. The result is converted into a string and, depending on whether a metadata key has been set, replaces the contents with the result or sets a new metadata value on the payload.

That's your processor completed. Now you need to register the plugin before calling service.Run. Since this is a processor plugin you're going to call processor.RegisterPlugin:

func main() {
	processor.RegisterPlugin(
		"how_sarcastic",
		func() interface{} {
			s := SarcasmProc{}
			return &s
		},
		func(
			iconf interface{},
			mgr types.Manager,
			logger log.Modular,
			stats metrics.Type,
		) (types.Processor, error) {
			return iconf.(*SarcasmProc), nil
		},
	)

	service.Run()
}

The first argument is a string that identifies the type of this plugin, that's the string used to specify it within a Benthos config file.

The second argument is a function that creates our config structure, this will be embedded within the Benthos config specification. In this case our processor implementation is the same type as the configuration struct, but you can separate them if you prefer.

The third argument is the generic function that constructs our processor. In this case we've already constructed it as our configuration type and so we can simply cast it and return it.

Now you're going to build your custom Benthos with:

go build -o benthos

Run Your Plugin​

In order to execute your plugin with Benthos you need a config. Write the following to a file config.yaml:

pipeline:
  processors:
  - type: how_sarcastic

And run it:

./benthos -c ./config.yaml

Your config hasn't specified an input or output so they will default to stdin and stdout. Write the line 'this is not sarcastic', followed by the line 'this is sarcastic /s'. Benthos should print 0 and 100 respectively.

Cool, but this config is pretty useless, good job idiot. Now you're going to fix your mistake. Let's imagine you are processing a stream of JSON documents of the form {"id":"fooid","content":"this is the content"} and you want to add a field sarcasm containing the sarcasm level of content. You can do that purely through config by using the json and process_field processors:

pipeline:
  processors:
  - json:
      operator: copy
      path: content
      value: sarcasm
  - process_field:
      path: sarcasm
      result_type: float
      processors:
      - type: how_sarcastic

Run that config with some JSON documents:

echo '{"id":"foo1","content":"this is totally sarcastic /s"}
  {"id":"foo2","content":"but this isnt sarcastic at all"}' |
  ./benthos -c ./config.yaml

You'll see some log events but also you should see your two modified documents:

{"content":"this is totally sarcastic /s","id":"foo1","sarcasm":100}
{"content":"but this isnt sarcastic at all","id":"foo2","sarcasm":0}

That's much more useful, but this is just barely scratching the surface of what Benthos can do. For example, here's a config that calculates sarcasm with your processor and removes anything with a sarcasm level at or above 80:

pipeline:
  processors:
  - type: how_sarcastic
    plugin:
      metadata_key: sarcasm
  - filter_parts:
      metadata:
        operator: less_than
        key: sarcasm
        arg: 80

Note that it makes use of your metadata_key field in order to filter the documents without changing their content.

Try experimenting with other Benthos processors, you can find the documentation at benthos.dev/docs/components/processors/about.

Next Steps​

After playing around with Benthos processors you should check out the various inputs, outputs, metrics aggregators and tracers that it's able to hook up with.

For example, here's a modified version of the previous config where we write from Kafka to an S3 bucket, sending our metrics to Prometheus:

http:
  address: 0.0.0.0:4195

input:
  kafka:
    addresses:
    - localhost:9092
    consumer_group: foo_consumer_group
    topics:
    - foo_stream

pipeline:
  processors:
  - type: how_sarcastic
    plugin:
      metadata_key: sarcasm
  - filter_parts:
      metadata:
        operator: less_than
        key: sarcasm
        arg: 80

output:
  s3:
    bucket: foo_bucket
    content_type: application/json
    path: ${!metadata:kafka_key}-${!timestamp_unix_nano}-${!count:files}.json

metrics:
  # Endpoint hosted at both :4195/stats and :4195/metrics
  type: prometheus

I'm sure you'll make great use of Benthos plugins with your extremely important work /s.

Benthos Lab is a website where users of the Benthos stream processor can write, format, execute and share their pipeline configurations. This was made possible by compiling the entire service (written in Go) into Web Assembly so that it can run directly in your browser.

Here's a video of it in action: https://youtu.be/1ZN-42A0sJU.

Some technical details about how this was achieved can be found in a previous post of mine. The repo can be found at: https://github.com/benthosdev/benthos-lab, feel free to clone it, hack it and host your own version.

Like every good laboratory it is scrappy and sometimes explodes. You need a modern browser version and it doesn't currently support mobile. Regardless, I think it's a good time to invite you try it out. Don't worry, I'll be watching from a safe distance.

Also, if it does break on you then please remember that you don't pay me for this, you vile parasite.

Why​

At Meltwater we have many distributed teams using Benthos for a wide and ever increasing list of use cases, which results in a lot of remote collaboration.

For the slower moving, asynchronous types of work we are usually fine with ye olde git. However, sometimes it's nice to quickly hash stuff out, at which point things would suddenly devolve into a barrage of files getting chaotically thrown at a slack channel.

By contrast, the lab is a civilised place where one can quickly and easily create a pipeline concept, including sample input data, and share it directly with a group. Opening up a shared lab provides all that same context, allows you to make your own revisions to it, and even lets you execute it in order to see the results it produces.

For our teams this app dramatically reduced the time taken to create, prove and demonstrate a pipeline concept. It's also much quicker for me to help teams that have issues with their configs as they can make sure I have all the information needed up front with a single URL.

Use it to build your own web app​

There's also plenty of unintended use cases for Benthos Lab as it basically allows you to build your own custom web applications. Here's a session that lower cases and normalises a JSON document, computes its sha256 hash and then hex encodes the hash: https://lab.benthos.dev/l/N-3sss3WPjj#input.

Pro tip: if you add the anchor #input to the end of the session URL then it opens up at the input tab for quickly inserting stuff.

Since there's such a vast catalogue of Benthos processors available I've already found myself bookmarking a few lab sessions as general utilities.

Next Steps​

With this running within the sandbox of your browser there's lots of missing functionality. For example, you can't create TCP/UDP connections (as of right now) and you can't access a file system.

However, I don't plan to address this any time soon as the intention of the app is to test snippets of config, not to execute your whole damn streaming pipeline. You ought to learn to manage your expectations.

What I will do though is continue to improve the UI. If a feature you want is missing (or broken, obviously) then please open an issue.

The Build​

The first thing I did and the first thing you ought to do if you are targeting wasm yourself is skim through this section of the Go wiki.

In short, I wrote a Go file:

package main

import (
	"syscall/js"

	"github.com/Jeffail/benthos/lib/config"
	"gopkg.in/yaml.v3"
)

func normalise(this js.Value, args []js.Value) interface{} {
	var configStr string
	if len(args) > 0 {
		configStr = args[0].String()
	}

	conf := config.New()

	// Ignoring errors for brevity
	yaml.Unmarshal([]byte(configStr), &conf)

	sanit, _ := conf.Sanitised()
	sanitBytes, _ := yaml.Marshal(sanit)

	return string(sanitBytes)
}

func main() {
	c := make(chan struct{}, 0)
	js.Global().Set("benthosNormaliseConfig", js.FuncOf(normalise))
	<-c
}

And compiled it:

GOOS=js GOARCH=wasm go build -o main.wasm

I was pretty sure that this would be the end of the road for me. Benthos uses a vast swathe of dependencies for its various connectors and so I was sure that I would be immobilised with errors. However, to my surprise there were only three (formatted for brevity):

lib/util/disk/check.go:29:11: undefined: syscall.Statfs_t
github.com/edsrzf/mmap-go@v1.0.0/mmap.go:77:9: undefined: mmap
github.com/lib/pq@v1.0.0/conn.go:321:13: undefined: userCurrent

Which involved some calls for a buffer implementation using a memory-mapped file library and the PostgreSQL driver for the SQL package. The errors themselves are basically "this thing doesn't exist in Web Assembly", which usually means the library has a feature behind build constraints but doesn't support wasm yet.

The solution for these problems in my case was as simple as to not to do the call, and perhaps document that the feature doesn't work with a wasm build.

Obviously, we only want to disable these calls specifically when targeting wasm. In Go that's easy, stick a cheeky build constraint on there. Here's the actual commit: 9903b3d5d8519fcf7ecbce94c336e7f054a75942, note that you can't just constrain the feature, you also need to add an empty stub that has the opposite constraint in order to satisfy your build.

Executing Go From JavaScript​

The Go Wiki shows you how to actually execute your wasm build and I won't repeat it here, but I followed the steps and it was pretty straight forward.

There was, however, one issue I came across. Some functions that I was calling from JavaScript were causing my wasm runtime to panic and stop. The functions all had channel blocking in common, something like this:

func ashHasACoolBlog(this js.Value, args []js.Value) interface{} {
	someChan <- args[0].String()
	return <-someOtherChanIHateNamingThings
}

The function would sometimes execute successfully. Other times, specifically for longer running calls, I would get a deadlock panic:

fatal error: all goroutines are asleep - deadlock! wasm_exec.js:47:6
wasm_exec.js:47:6
goroutine 1 [chan receive]: wasm_exec.js:47:6
main.main() wasm_exec.js:47:6
	/home/ash/tmp/wasm/main.go:20 +0x7

Which was odd as they would be occasions where I would not expect a real deadlock. I then found the relevant docs in the syscall/js package:

Blocking operations in the wrapped function will block the event loop. As a consequence, if one wrapped function blocks, other wrapped funcs will not be processed. A blocking function should therefore explicitly start a new goroutine.

The consequences of blocking sound pretty harmless here, but in reality it seemed to be the cause of my deadlock crash. I assume the odd error message is a result of some nuanced mechanics within the wasm runtime.

I didn't investigate this crash any further as I was a lazy idiot back in those dark days. I simply stopped writing blocking functions, and instead spawned goroutines everywhere like they were losers at a Nickelback concert:

func iJustWantToClarify(this js.Value, args []js.Value) interface{} {
	go func() {
		someChan <- args[0].String()
		otherThing := <-someOtherChanIHateNamingThings

		js.Global().Get("thatActually").Set(
			"textContent",
			"I quite enjoy and respect Knickelback as artists... " + otherThing,
		)
	}()
	return nil
}

Other Issues​

There weren't any.

Final Words​

It took a day for me to get a working application together and soon I'll be blogging about the resulting product. Web assembly with Go is dope.

Kudos to both the W3C and the Go team for taking their time to build something to completion without rushing the conclusion. Yes, I'm still bitter about Game of Thrones.