Planet Jabber

September 21, 2018

JC Brand

Converse 4 released

After more than 7 months of active development, Converse 4 has finally been released.

Converse is an open source XMPP-based chat client written in JavaScript and which runs in your browser.

This release contains lots of highlights, including rewriting the UI to use Bootstrap 4, support for OMEMO Encryption of private messages, message corrections and file-sharing via HTTP file upload.

XMPP is an IETF standardized messaging and presence protocol with multiple independent server and client implementations.

Unlike other popular open source teamchat applications like Mattermost and Rocket.chat, Converse doesn't depend on any particular server (e.g. backend) application. Any XMPP server which supports the relevant extensions (aka XEPs) will do.

Converse is 100% front-end JavaScript and CSS. The only backend you need is an XMPP server, which you can either set up and host yourself or you can sign up on an existing one.

XMPP's killer feature

Picture of the USS Enterprise, defending the federation, I think... I'm not really a trekkie

XMPP allows for something called federation. Even if you've never heard the word being used in this context, you already know what it means if you've used email.

When you sign up for an email account from one provider (for example Gmail, Fastmail or Kolab), you can still send emails to people who have accounts at other providers. That's because Email is also federated.

A federated protocol allows you to communicate with user-accounts on foreign hosts. The network is decentralized between many providers (who all adhere to a common protocol), instead of being controlled by a single centralized behemoth (like Facebook, Twitter or LinkedIn).

Federation is getting quite a lot of renewed attention lately, in large part due to the hype around the federated social networking site Mastodon which is based on the recently published ActivityPub standard.

You're not forced to federate when running an XMPP server for your organisation, friends or family, and sometimes it makes sense not to.

However it's in my opinion the killer feature of XMPP and one of the main reasons why I'm working on Converse.

Federation implies user-freedom and user-choice, and it's one thing that Slack or Discord simply cannot do. Not because it's not technically feasible, but because their business models, based on user lock-in and control, don't allow it.

It's also something which the open source Slack-alikes like Mattermost and Rocket.chat don't have.

I believe people should have the freedom to use the chat app of their choice, while still being able to effectively communicate with their contacts who prefer to use a different chat app.

XMPP and federation can make this a reality.

In a world of federated chat apps, some people will use a commandline client like Poezio, a desktop client like Dino or a mobile client like Conversations (or a combination of all three).

There will also be a growing need for web-based chat clients, and that's were Converse comes in.

Federation not only provides choice between chat clients, it can also foster inter-departmental or inter-organisational communication.

In Germany, where I currently live, there is a lot of talk of digitization of government. Government bureaucracy often consists of various different departments who need coordinate and send resources around.

The same of course applies to large enterprises.

A decentralized and federation communications protocol can play a large role here, and I'm not just talking about chat apps.

The Salut à Toit project has shown how you can create a decentralized and federated issue tracker, while Movim has blogging and microblogging capabilities, all based on XMPP.

Converse's history

Converse was originally developed at Syslab.com as part of a web-based intranet for the Star Alliance airline group.

Back then it was developed as a Gtalk-like popover chat, like you still see on LinkedIn.

After open sourcing it, my initial goal was to make it as configurable and capable as possible, so that it could be used in any website in almost any conceivable manner.

A powerful plugin architecture and API was added, which allows you to add or remove features and change just about any behaviour.

Over time, as the project matured and people started using it, it became clear that there are many more usecases for a webchat than simply popover chats.

Full-page teamchat apps like Slack also became spectacularly successful.

I subsequently updated Converse to have different "view modes", which enable you to use it in different ways, depending on your needs.

The focus also shifted slightly from presenting it merely as something to be integrated into an existing website, to showing that it can be an independent chat client in its own right.

There is the "overlay" mode, which matches the original UX design:

A screenshot of Converse with overlayed chatboxes

A screenshot of Converse with overlayed chatboxes

Then there are the "embedded" and "mobile" modes:

A screenshot of Converse in embedded mode

A screenshot of Converse in embedded mode

A screenshot of Converse in mobile mode

A screenshot of Converse in mobile mode

And lastly there's the "fullscreen" mode, more similar in UX to teamchat applications like Slack and Discord.

A screenshot of Converse in fullpage mode

A screenshot of Converse in fullpage mode

All these modes use the same JavaScript and CSS, and you can switch between the view modes. Currently it's not possible to do so without reloading the page, but we'll work on making that possible as well.

Working on Converse fulltime

In the beginning of this year, I decided to drop other work commitments and to dedicate this year fully to working on Converse and to see where that leads me.

This was in many ways a daunting leap into the unknown, because I knew I would now have to find more ways to try and earn a living off this work.

I've been doing consulting and custom development around Converse for a few years, but never exclusively. I also wasn't sure what business model I should adopt.

Consulting? Proprietary plugins? Hosting? Sponsorships?

I decided to first just focus on building the product, to make Converse better, and not to worry about making money just yet.

In part this is due to my bias as engineer, and being out of my comfort zone when trying to monetize something.

But I also believe that a compelling product, that users love, can go a long way to helping you come up with a sustainable and feasible business model. I hoped that along the way people might appear who would help me realize this vision.

It's the "if you build it, they will come" approach.

During this time, various people did in fact reach out to me and some of them funded further development.

I also got enough consulting and development gigs to keep the lights on and to keep me going.

Marc Laporte from Wikisuite funded many hours of work moving the UI to Bootstrap 4, and then some more.

Converse will become part of Wikisuite, together with the OpenFire XMPP server.

He gave me valuable advice on how to make a living off open source software and encouraged me to make the leap.

I also got a call from Happy Dev, who needed a federated webchat solution for their Startin' Blox project (more on that in future posts). We're doing some interesting work for them, which addresses some pain points with XMPP and will be of benefit to the larger community.

Nicolas Vèrité, of Nayego funded responsiveness improvements while individuals like Christian Weiske and Rafael Munoz generously contributed funds as well.

Some people also backed me on Patreon and Liberapay.

Every gig or donation I got was for me a vote of confidence, and gave me the opportunity to stay the course and continue down this path.

Besides monetary contributions, many people contribute features and bugfixes, and hang out in the Converse chatroom, answering questions, providing their expertise and generally making it a more lively, interesting place.

In many ways this release would not have been possible without all of their contributions, and I therefore dedicate it to everyone who has contributed money, time or energy (all interchangeable actually) to this project and therefore has contributed to its success. Thank you!

The archetypal nature of free software communities

Over the last year I've witnessed the Converse project becoming a community and not just a collection of software. It's a subset of the larger XMPP community and one that overlaps with various other XMPP and free software related communities.

This has been for me one of the most magical and satisfying things about being involved in open source.

Community is archetypally feminine [1], in the sense that it's something that cannot be enforced, demanded or created out of domination, control or rule-making (which are archetypally masculine attributes). Instead, it's something that arises seemingly spontaneously, after an unknown period of gestation and only after the groundwork has been laid and perhaps after enough loving care and attention has been provided. Its organic, emergent nature means that it doesn't strictly adhere to top-down schedules and deadlines.

My wife and I, both interested and fascinated by Jungian archetypes, are amazed at the archetypally feminine aspects of her pregnancy, which is soon coming to an end (and new beginning).

As much as we as a society try to control, monitor and manage a pregnancy, if you decide to have a natural birth, then you're waiting for it to happen on its own accord, and aren't able to impose your will and deadlines on it.

In some ways the same is true for a software commons. A commons is a shared resource which no-one has exclusive control or ownership of. This lack of ownership and exclusive control makes people resilient against the software being used as a tool for domination and exploitation. It also means that the project (i.e. the software and its surrounding community) is more organic.

This coming together between software as inanimate artifact, and community as a living breathing organic hive-mind, brings to mind a Taoist union of Yin and Yang, the archetypal feminine and masculine.

Incidentally, the Converse logo is a stylized Yin Yang symbol. Many people don't know this because I did such a poor job of designing it. I'm still looking for a better logo design which encapsulates these principles of harmony between community and software.

What does the future hold for Converse?

This has turned out to be a relatively strange blog post, where I wanted to originally write about the features of the 4.0 release and instead took an ever more esoteric detour.

In any case, let's try to salvage what's left of the original intent by closing off with what's in the pipeline for future releases.

Message markup

Converse still doesn't support markdown-like syntax highlighting. This is a top-priority for future releases.

Voice messages

With voice messages I mean sending voice recordings, not audio streaming (although that might also come sometime).

My wife, friends and family use voice messages (via Conversations) a lot and I believe they'd be a useful addition to Converse as well.

Email notifications

One area where XMPP needs to improve is sending (email) notifications when you're offline and mentioned in a groupchat. XMPP's groupchat presence rules have made this difficult. Together with Happy Dev and Matthew Wild from Prosody I'm working on a potential solution.

Some of the code for this solution is already in Converse, and we'll submit a specification for standardization (a so-called XEP) once the time is right.

Making Converse a progressive web-app

There are various features of so-called progressive web apps which will make Converse a much better client, such as:

  • Push notifications even when Converse is not open.
  • Using IndexedDB for caching, thereby avoiding storage limitations of localStorage and sessionStorage.
  • Offline support for editing and sending messages.

Persistent login

Other webchat clients like Mattermost and Rocket.chat can keep you constantly logged in via cookies or perhaps JWT.

XMPP-authentication isn't cookie-based and it's not safe to store user credentials in the browser cache. This means that users often have to login anew when they open Converse in a browser tab.

The Credential Management API might help here, but it's still Chrome-only and I haven't studied it enough to know whether it's a solution.

Another approach is to allow Converse to also use authentication cookies by adding support for them to XMPP servers.

Matthew Wild has already added support for cookie authentication to Prosody but more work needs to be done to get this working on conversejs.org.

OAuth-based login

Converse already supports OAuth-based login, but it's not yet deployed to conversejs.org and more work can be done to fetch and show your user profile from the OAuth provider.

Let's build this together

There are many more features and improvements planned, too many to list here.

The health and sustainability of this project depends in large part on its community of users and contributors.

The community is invaluable in finding bugs, making feature suggestions and doing quality control that would otherwise not be possible given the circumstances.

Another important requirement for the long-term health and viability of the project is funding.

I and others are spending much of our own time working on Converse, but to make this project sustainable we need more funding. Either through donations (e.g. Patreon), funded feature requests or by using my services as an XMPP and Converse consultant and developer.

Another option is allowing an employee to spend some of their paid time on helping the project with bugfixes and other contributions.

This applies in general to open source projects. Companies derive immense value from them, and the vast majority don't have wealthy corporate backers like Facebook and Google.

I also intend to soon start offering hosted solutions for organisations who want to use Converse as their preferred teamchat solution.

Please reach out to me here if you're interested.

References

[1]

I've been thinking a lot about archetypes (and Jungian-analysis) the last while, in part due of various books I've read that deal (partly) with the topic.

Recommended reading, if you're interested:

September 21, 2018 08:30

September 20, 2018

Jérôme Poisson

SàT news: control your media player from SàT + OMEMO

Time is running so fast that I can barely find some to write about the advancement of Salut à Toi, and there's a lot to say.

I'll be short this time (no more long list of features like alpha release ;) ).

If you are following SàT for long time, you may remember the experiment to have an universal remote control, and the demonstration with VLC.

Well, this is now implemented in a cleaner way, and it's user friendly. With Cagou running on your mobile device, it's quite useful as you can see in the video below.

Note: I'm testing Peertube to diffuse SàT demo videos, you can also see the video on the instance I'm trying.

If you have already SàT running on your desktop, all you have to do is run your media player (VLC for instance). Then on Cagou click on the new "remote control" icon, and you should see an icon for your media player. Note that this is working only on GNU/Linux (and probably *BSD) as it is relying on D-BUS which is not available everywhere.

The other thing to mention is that OMEMO support is now available in SàT (thanks to the work of Syndace on python-omemo, and his prompt reactions to feedback). It's not fully finished yet: fingerprint management is missing, encryption is only available on one2one conversations, and it's not yet available on Android, but the biggest part is done and working on desktop.

Note: this video is also available on Peertube

That's it. I'll publish a new alpha release in a couple of days. The list of features to implement before release become narrow, stabilisation phase should start soon.

Once again help is needed for packaging (Debian, Arch, or other GNU/Linux distributions, Docker, etc.), testing/packaging on many platforms (*BSD, Windows and Mac for instance), and would be welcome for development too. Also we have a very low visibility, don't hesitate to shout out loud about the project :).

by goffi at September 20, 2018 18:37

Erlang Solutions

MongooseIM 3.1 - Inbox got better, testing got easier

This summer the MongooseIM team have not had a second to be lazy - the development of MongooseIM 3.1 release took over what was supposed to be our downtime, when all major activities driven throughout the year slow down. We’re happy to say that in this time of leisure, the new release is packed with important features, improvements and excitement. Take a look at the version highlights and be part of this major step in creating a world-class communication server.

This time, a large part of the changes are focused on development efficiency, but rest assured, we’ve added new items you are going to appreciate.

In the “Test Runner” section, you get to learn all about a little tool we provided so that you can easily set the development environment up and run tests locally without Travis CI.

Our Inbox extension has got three big updates that push the boundaries of what’s currently offered in XMPP world. As a rule, we want to lead by example; less talk and more action. That is why working in cooperation with Forward, we decided to put forward an implementation of a popular use case as a unique feature in this software sector.

This release is also an important lesson for us. A lesson about edge cases and concurrency in integration testing. You don’t necessarily have to be an Erlang developer to benefit from the third section, but reading it allows you to learn with us.

The “Honorable Mentions” section may seem minor, but for some projects the items listed there can indeed make a difference! It’s a candy mix of different changes, so read carefully not to miss your favourite flavours!

Obviously, a single blog post is too small a space to tell a profound story about every new item in the changelog, so we encourage you to check it out. You can find a link at the bottom of this blog post.

Test Runner

The Travis CI is our main verification tool for the Pull Requests we create and merge. Whilst being convenient and critical for our process, it is not very handy for day-to-day development. It is very common to frequently execute limited subset of tests to ensure that a new piece of code we wrote works perfectly. However, waiting for Travis results every time would extend implementation time excessively as it executes multiple presets, while being a shared resource at the same time.

The test runner is a script that helps to set the development environment up, and run tests on a developer machine locally. The test runner shares a lot of code with our Travis CI build scripts, which ensures that test results are consistent between local and CI runs.

The test runner allows to choose which tests to run. It can be useful, when one of the tests is failing and we want to rerun it.

Since MongooseIM supports different database backends, the test runner is able to set a selected database up for you and configure MongooseIM. Simply put, it prepares the local environment before executing actual test suites.

The test runner supports a lot of options to customise your build and test execution scenario. It also supports shell autocompletion for option names and values.

We’ve prepared a recording for you that presents a basic usage of the script.

.

Please note that Asciinema allows you to pause and copy any text fragment you like, so it would be very easy for you to repeat the same steps.

New Inbox Features

Originally sponsored by and created for Forward, Inbox has been available as an open source extension for the last two months already. In MongooseIM 3.1, it returns with a bundle of fresh goodies. The first feature is MSSQL support.

Despite being less frequently used with MongooseIM compared to MySQL or PostgreSQL, it’s still an important piece of the architecture for many projects, especially those running in Azure cloud. We don’t want you to feel excluded, dear Microsoft users!

The second one is the support for classic MUC group chats. MUC Light was never intended as a complete replacement for original XEP-0045 solution. It means that numerous projects exists where mod_muc is still a better match than its younger sibling, and they may now benefit from inbox features as well!

Last but not least is the timestamp support. First of all, they are stored in DB and returned in Inbox query results. For those using mod_inbox from MIM 3.0: you’ll need to update your schemas but don’t worry - it isn’t very complicated. What’s more, a client may now request conversations from a certain time period and sort them by timestamp, both ascending and descending.

This is not our final word on this matter. You may expect further improvements to this extension in upcoming MongooseIM versions!

We’ve prepared a demo of the Inbox feature. It shows both the backed and the frontend side of it. The application used in the demo has been designed by Forward.

.

Lessons learnt regarding CI

OK, these are short and sweet but nevertheless important:

  1. Avoid RPC in integration tests. They tend to time out in slow CI environments (such as Travis).
  2. When test users exchange messages, always wait until they are received to ensure proper synchronisation.
  3. On a slow machine, MSSQL SELECT query may return more than one row (even when retrieving by the exact primary key value) as a consequence of the transaction deadlock.
  4. When you can’t use any other means of server state synchronisation, don’t use hardcoded sleep periods; replace them with an incremental backoff and verification in a loop. Sometimes you can’t predict whether a server state is updated properly in 500ms, 1000ms or 3000ms. Adding 5s waits everywhere may cause test suites to run veeery long.
  5. Be careful about leaking presences between cases. This applies to XMPP testing. Best practice is to generate fresh user accounts for every scenario.
  6. Some databases don’t support transactions so the new data may not be instantly available. For example, in the case of Riak (its Search function in particular) a delay between data insert and query is required.
  7. Sometimes creating a schema in a DB may fail for the first time due to timing issues, so implement a retry loop in a DB provisioning scripts. This also applies to Riak.
  8. Did I mention creating new user accounts for every test case? It actually applies not only to XMPP. With this practice, you won’t have to worry about possible leftovers of a user’s state.

Honorable mentions

ElasticSearch backend for Message Archive

Almost every MongooseIM module supports more than one type of backend. Usually it’s Mnesia, RDBMS and sometimes Riak. Message Archive Management is a noteworthy exception, as we’ve implemented RDBMS, Riak and Cassandra support for this module. Or “modules” actually, as it consists of over 30 Erlang files already.

It is a very special feature as it processes a vast amount of data and sometimes executes expensive queries. In order to ensure performance and match a project’s architecture, wide range of supported DB backends is essential.

It is our pleasure to announce that yet another backend has joined the family: ElasticSearch.

OTP 21 support

OTP 21.0 has been released ~1 month ago and we’ve added support for this version less than a week after! This is great news for all projects sticking to the most recent Erlang technology as pioneers in BEAM world. The new platform version brought not only improvements in regards to performance but also some incompatibilities that we’ve resolved, so MongooseIM still remains at a technological peak.

As a tradeoff, we’ve dropped official support for OTP 18.x. It should still be possible to compile 3.1 with this version with some minor code modifications, but we’re definitely moving forward. It has allowed us to get rid of non-typed maps specifications as an example. As a reminder, bear in mind, that MongooseIM always supports two most recent, stable OTP branches (currently these are 20.x and 19.x and one being under an active development, 21.x).

Jingle/SIP tutorial

SIP is a common choice for VoIP applications but certain XMPP features may be a very good match for such software. MongooseIM is able to liaise between these two worlds and now it’s easier than ever with significantly extended documentation (compared to the level in 3.0) and a tutorial on mod_jingle_sip usage.

Worker pool unification

Every developer writes a custom worker pool at some point of their career. Everyone. Certain MongooseIM components (the ones that use connection pools) were created with different preferred library in mind. As a result, we’ve ended up with many kinds of worker pools: cuesport, worker_pool and poolboy. It wasn’t only a matter of maintenance difficulty, but performance as well. As an example, cuesport supports only a simple round-robin job assignment algorithm, which is not optimal in every case. It also lacks inspection of any kind.

Given our experience gathered over the years, we’ve selected worker_pool as our primary library. It is very flexible and exposes a dedicated stats API. It was originally created by Inaka and it is actively maintained by its former developers at this present time.

For now, the changes are purely internal. Some projects may observe better performance but the primary goal was to prepare for a second round of unification. Stay tuned for more details in near future.

Changelog

Please feel free to read the detailed changelog. Here, you can find a full list of source code changes and useful links.

Contributors Special thanks to our contributors: @SamuelNichols @Beisenbek @GalaxyGorilla!

Test our work on MongooseIM 3.1 and share your feedback

Help us improve the MongooseIM platform:

  1. Star our repo: esl/MongooseIM
  2. Report issues: esl/MongooseIM/issues
  3. Share your thoughts via Twitter: @MongooseIM
  4. Download Docker image with new release
  5. Sign up to our dedicated mailing list to stay up to date about MongooseIM, messaging innovations and industry news.
  6. Check out our MongooseIM product page for more information on the MongooseIM platform.

September 20, 2018 14:23

September 11, 2018

ProcessOne

First ever MQTT and XMPP dual-protocol server: ejabberd Business Edition

ProcessOne is proud to announce that ejabberd Business Edition now supports MQTT 5. This is a whole new protocol added – as an addition to XMPP (and SIP) – to our server. Our enterprise customers will be able to use both MQTT and XMPP in their real-time solutions.

ejabberd is the world’s leading XMPP server, both in performance and market share. With MQTT support, ejabberd brings equally incredible performance to this popular protocol, and true flexibility to real-time solutions.

What is MQTT?

MQTT stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks. The design principles are to minimise network bandwidth and device resource requirements while also attempting to ensure reliability and some degree of assurance of delivery.

These principles also turn out to make the protocol ideal for the emerging “machine-to-machine” (M2M) or “Internet of Things” world of connected devices, and for mobile applications where bandwidth and battery power are at a premium.

Why is it a big deal?

It is the first time that a single platform is able to support the major reference real-time standard protocols.

XMPP is an IETF standard for instant messaging communication. It is also used for Internet of Things in use cases where powerful devices need to be connected to the internet, especially with their own presence signal and control interface. XMPP offers more features than MQTT, but the extra flexibility and complexity comes at a cost. Some less powerful devices cannot leverage XMPP and need to resort to MQTT, which is designed with that small devices use cases in mind. MQTT is a standard from OASIS.

Supporting both standards opens a whole set of new use cases, making it possible to connect people, applications, small IoT devices and larger IoTs on the same platform.

Moreover, from the very beginning ejabberd Business Edition provides full support for the brand new MQTT 5, being ready to support the most advanced use cases offered by this protocol. It is one of the very first production-ready MQTT 5 servers, with compliance to the brand new specification released in May 2018.

MQTT and XMPP

Thanks to combined support of XMPP and MQTT, ejabberd Business Edition is the first truly multi-protocols real-time messaging servers. Your solutions can use the best of both worlds, and your implementations can be flexible and scalable at the same time.

By using a multi-protocol server, your deployment can benefit from the ecosystems of both protocols, and be compatible with vast number of standard tools and software.

Does it mean, we consider XMPP obsolete? Absolutely not. XMPP and MQTT are designed with different use cases in mind. However, we think that in many cases, a real-time project will need to leverage both XMPP and MQTT during its lifetime, making humans, things and application collaboration possible. This is our vision.

What are the main benefits?

ejabberd Business Edition has been much refined during recent years. ProcessOne has patiently engineered a server framework that is stable and highly available thanks to a fast and flexible clustering engine, and versatile thanks to its plugin API.

MQTT protocol in ejabberd Business Edition has been reimplemented from scratch, but the server infrastructure leverages the brand new modular architecture of eBE. It means it uses the same highly scalable clustering engine, the same authentication infrastructure, can provide virtual hosting, reuse the logging modules, and use the same API.

For the users, the benefits are clear: you can reuse the same server, infrastructure, authentication mechanism, security policy for your XMPP clients and MQTT clients. You can invest in a single platform and be sure to be future-proof and address both your current and future needs in term of real-time technologies.

You can have a part of your devices connected to XMPP and another part connected on MQTT, without the need to manage two separate platforms. And most of all, you can design your system for use cases mixing XMPP strong points (human collaboration, chat bots for automation) with MQTT (connection to millions of very lightweight devices).

The bottom line is simple:

  1. You can switch between XMPP and MQTT as you wish, even use both protocols on the same infrastructure.
  2. You will save on infrastructure, given the high-performance of the platform.
  3. You get support on solution design for real-time infrastructure and can get help choosing between XMPP and MQTT, from a vendor that has no interest in selling one protocol more than another.

How can I test it?

If you are a developer and would like to try connecting your client to our MQTT server, we got you covered.

ProcessOne has deployed a public MQTT broker with anonymous access. This is one of the first MQTT 5 public brokers available.

The broker is deployed at mqtt.fluux.io on port 1883 and 8883 (TLS).

You can, for example, test our broker with mosquitto client. On macOS, you can install mosquitto via Terminal using Homebrew with:

brew install mosquitto

From there, you can subscribe to the topic test/topic with:

mosquitto_sub -h mqtt.fluux.io -p 1883 -V mqttv311 -v -t "test/topic" -q 2

and then (in another Terminal tab or window) you can publish with:

mosquitto_pub -h mqtt.fluux.io -p 1883 -V mqttv311 -t "test/topic" -m "message" -q 2

You should see your message received and displayed below the mosquitto_sub command in your previous Terminal tab or window.

Note: In this example, Mosquitto client is using MQTT v3.1.1, as it doesn’t support version 5.0 at the moment of writing.

Don’t hesitate to send your feedback and findings through our contact page. Happy hacking!

What is next?

From our initial tests ejabberd Business Edition MQTT performance is very impressive, and we will be publishing our benchmarks soon.

Regarding MQTT 5, shared subscriptions are still being developed and will be available in a next release.

Regarding protocol version coverage, our platform supports MQTT 3.1.1 and we plan to add support for MQTT 3.1 as well in a future update.

Most of all, we are planning to increase the interoperability between MQTT and XMPP. The plan is to have both XMPP and MQTT clients be able to leverage the same services provided by ejabberd Business Edition: push notifications, groupchat through Muc/Sub, etc.

The goal is to leverage use cases based on real-time communication between humans, things, and applications. This can only be done by building an ambitious platform that is protocol agnostic and leverages the best of each technology.

We believe the IoT revolution will fuel the growth of both XMPP and MQTT protocols. For this to happen, the development community needs flexible, scalable and – most importantly – trusted tools. For more than 15 years ejabberd has been the best XMPP server. We now want it to be the best MQTT server as well.

by Marek Foss at September 11, 2018 09:00

Real-time Enterprise Issue #14

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning business aspects of real-time enterprise we found interesting in Issue #14. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

Companies are migrating to the cloud for the IoT

Developers’ evolving demands and needs will push IoT platform providers to reshape their products and services in the coming years. Developers’ needs, concerns, and demands will change as their companies build out their IoT projects.

The Internet of Things explained by a professor

The IoT becomes a subject of the degree course on Production and Process Management at Heilbronn University of Applied Sciences in Germany. Prof. Dr. Thomas Pospiech talks about the “IoT box”, an invention he has developed to better convey the concept.

IBM teams with Maersk on new blockchain shipping solution

IBM and shipping giant Maersk having been working together for the last year developing a blockchain-based shipping solution called TradeLens. Today they moved the project from Beta into limited availability.

Risky business? How to manage IoT security in the enterprise

Nearly every week, we hear about a new cyberattack or security breach. In fact, more than 1.7 billion identities have been exposed in data breaches in the past eight years. As the world becomes increasingly interconnected, organizations become susceptible to more risk.

IoT boom will change how data is analysed

As more devices come online, each generating data, the way information is analysed and used to facilitate machine learning will have to change.

Five IoT predictions for 2019

The IoT is growing at an exponential rate. Vehicles, wearable gadgets, RFID sensors and software are advancing past basic function and the network is growing to include even more advancements each day. What will the future bring?

by Marek Foss at September 11, 2018 08:59

September 10, 2018

Monal IM

iOS Share Sheets

Current work in progress share sheet

I have found the lack of a share sheet  in iOS and OSX limits how much I use Monal.  A lot of what I send in chats are links. I suspect that is the case for a lot of other people too. To address that, I’ve added a share sheet to the iOS client. This code should work on OSX too.  Doesn’t quite work yet but when it’s done, will work the way you expect it to. 

by Anu at September 10, 2018 16:37

Jérôme Poisson

Salut à Toi 0.7 alpha: contributors, take your keyboards!

It's my pleasure to announce the release of the first alpha version of Salut à Toi 0.7.

This is the occasion on one hand to show the state, advancement and direction of the project, and on the other hand to launch a call for contributions.

In this article I will show you a part of Salut à Toi capabilities, and explain briefly how to use them.

You'll find a link to an .apk file to test Cagou on Android in the "Test it!" section.

As a reminder, Salut à Toi is a collaborattive ecosystem, which is multi-frontends (web, desktop, mobile devices, terminal, command line), multi-platforms, and multipurpose. It let you handle your communications, organise yourself, share your files, etc. It is also, and above all, an activist project which think about the impact of new technologies on our lives, and which tries to stimulate public debate.

We'll concentrate on the web and desktop/mobile frontends, because those are the ones which will most probably be of interest to the mass.

Web frontend (named "Libervia")

Libervia has seen major changes, with a new web framework.

The idea, which has been explained in a blog post is to offer base components (account authentification, permissions, comments, etc.) ready-to-use and decentralised thanks to XMPP. This way we easily build website which are automatically decentralised and federated.

Please note that everything is prepared for translation, but for now only English is available. Help would be much appreciated to improve on that!

Let's check the available features:

Chat

screenshot of a "chat" discussion in Libervia

Let's start with the most obvious, there is an instant messaging page. To use it, you just have to click on "chat", et to log in if it's not already done.

You'll land on a panel with the rooms in your bookmarks, you just have to click on one of them, or to specify an identifier (jid) of a new room/contact.
It is for now quite basic, but still you can enter "/help" to check textual commands.

Blog

my own blog, powered by Libervia

One of the flagship features of SàT, really rare in XMPP world (the only actively developed other client implenting it being Movim), is the blog (which is then decentralised).

A click on the menu will bring you to a "discover" page where you'll find your contacts with a blog. Click on one of them to read, or specify an identifier to read a blog which is not in your contacts.

Forums

XMPP powered decentralised forums

Unique feature in XMPP world, SàT propose forums, allowing to organise discussions in categories. Because forums are under the hood the same thing as blogs, it is possible to use a blog as forum and vice versa.

The addition of new forums is done by an administrator using command line tool "jp" (see below).

Events

personal list of events

Other feature not seen elsewhere in XMPP (and in decentralised software?), SàT allows you to create events, and of course to invite people.

As with other features, the page is thought to be easy to use: you have a button "create an event" which bring you to a form. A name, location and date later (and optionally a picture to associate), and your event is created. You'll then land to an admin page where you'll have the possibility to invite people.

Events are private for now, only your guests and you have access to it, but public events are also planned.

a guest is answering to an event RSVP

Please note that it's possible (but not yet from the web interface) to add arbitrary fields to your events, for instance to let your guest specify what they'll bring to a picnic.

File sharing

sharing file from browser

Here we are talking about a feature useful daily, and in many circonstances. Salut à Toi has long been able to send files, or to upload them on your server, but we are talking here about sharing one or many directories, or to upload in an advanced way on a serveur component.

All of this is explained in a recent blog post, but let me add 2 use cases to understand well:

  • you are home, you've just taken a video and some photos on your phone that you want to get on your computer. You have installed Cagou, the desktop/Android frontend of SàT on your device and already shared your photo directory. From your desktop computer, in the "file sharing" mode you see your device and you can directly download the video and the photos from there. The connection will be done directly on your local network, from device to device, and your files will not go throught a external server located whoever knows where.

  • you have files that you want to access at any time from anywhere (pictures or working documents for instance). You put them in a virtual directory on your file sharing component (a SàT instance too) and you can access it as simply as for the previous example.

Note that while with Cagou the connection is done directly when possible, it's not the case for Libervia which will first download the file on the server, before taking it throught HTTPS. WebRTC will be needed for the direct connection, and that's planned for 0.8 release.

Permission are handled easily: you specify who can access your files in a couple of clicks.

Files are transfered in clear for now (unencrypted), but this should not be the case anymore once finale release is here.

Photos albums

a photos album with a comment

Based on previous feature, a photos album mode display a directory containing only photos with adapted view, an with comments.

Permissions are really useful here, allowing to let, for instance, only your family see an album.

It's not yet possible to access an album from the interface of Libervia, to test it you'll have to change the URL by hand from the file handling page (by replacing files/list by photos/album).

Tickets

SàT tickets, first ones have been imported from Bugzilla

Developed from our own needs, SàT now handles tickets in pretty much the same way as a code forge. This is classical issue tracking, bugs report, tasks to do, etc.

Fields are free, it's really easy to use this feature for any kind of list: TO-DO tasks, but also, for instance, shopping lists.

Merge requests

a major patch is being reviewed

In the same spirit as tickets, SàT also handle "merge requests". Developed and used primarly for SàT itself, this system has the benefit or not being tied to a specific tool (like Git or Mercurial). Il is currently used with Mercurial, but it is possible to use it with whatever we want, and I'm not only talking about software code here.

Those 2 features means that SàT is a decentralised code forge. Even if it's still relatively basic, it has the benefit of being easy to use, flexible and adaptable. Furthermore there are only a few decentralised code forge at the moment (we can probably count Fossil, maybe some others).

Once again, contributions would accelerate things and would permit to have a very powerful tool.

Application

the dynamic part, which in under minimal maintenance and will be re-written for 0.8

To conclude this small tour of Libervia, we have to mention the application which is an entirely dynamic page, more elaborated but also more complicated to use (one of the main base version goal is to be easy to use).

Libervia web application allows to use widgets that you can drop next to each other, to do end to end encryption (OTR), or to publish blog post in simple or WYSIWYG mode, with the syntax you like (Markdown for instance).

It's the historic application which, for technical reasons, is now under minimal maintainance and have a few issues (like the contacts which can appear 2 times as seen in the screenshot above). It will be re-written for SàT 0.8, but it's let in place in this release for people who want to use it.

Cagou (desktop/Android)

Promised after the success of our crowfunding campaign, which allowed us to get a bit more than 3000 €, Cagou is our new frontend for desktop and Android devices. Its goals are to be modern, easy to use, and powerful.

Its original UI is inspired from Blender, and let split the screen to show several widgets at once. This UI has been thought in a way that it can be used even without knowing this split capacity, while being straightforward to use once the feature has been discovered.

To change mode, you just have to click on the top left icon.

For now, 5 modes are available: chat, contacts, file sharing, widget selector, and settings.

On the very top, you have a menu (only on desktop), and a notification zone, where messages appear for a few seconds and are visible entirely when clicking on the head of the left Cagou. On the other side, an other Cagou head can appear when you get notifications needing a user interaction (new contact for instance, or authentification request – see below –). This way, no popup will steal your focus while you are writing a message or doing something else.

Split

It is possible de split the screen as much as you want by clicking on the 3 dots area (on top and left borders), and then moving your mouse/finger. If the bar is red, this means that you'll close the area.

Here is a small animation to make things more clear:

demo of Cagou's widget split ability

You can organise the way you want, and monitor several things at the same time. In the future, it should be possible to keep handy widgets organisations so you can re-use them easily.

Chat

Cagou "chat" mode screenshot

The chat should be relatively intuitive. The top area show the current room or the identifier (JID) of the person you're talking to. It is possible to enter an other identifier manually to change room. For instance you can enter sat@chat.jabberfr.org to come on Salut à Toi room.

Still on the top area, the "padlock" icon on the right (only visible in 1/1 discussions) allows you to encrypt the conversation end to end, with OTR only for now (see below).

If you have several discussions at the same time, you can switch from one to the other by swiping your finger horizontally. Only discussions not already visible somewhere on the screen will be available, and for now they are in alphanumeric order (most certainly recent discussions will be available first when stable release will be out).

Finally, in the bottom area, in addition to the text input, you can see a + button with a circle which allows you to open a media transfer panel. On the desktop for now you can only select a file from your file-system, but on Android you can select/take a picture/video, a file or record an audio message. In every cases you can choose if you want to upload the file on your server, or send it directly to your contact, without transiting by the server.

file sharing

The principle has already been explained for Libervia, here is an animation showing how to share a directory with Cagou:

sharing a directory from Cagou

As you can see, you just have to do a long click/touch on the directory or file to share, and to select people which will have access to it.

You can click on "share local files" button, in the header zone, to change mode and see the device which are sharing. You'll then see – as with Libervia – 3 areas: the first one with sharing component, if any, the second one with your own devices (your phone or desk computer for instance), and finally the devices of your contacts. Once again, the input area let your filter devices to show.

see devices which can share files

Third party website authentification

SàT is a project which push the use of XMPP in every directions, and this include authentifying on third party websites. The idea is to be able to authenticate yourself without having to create an account on any website just by giving your JID.

Here is a video demo, thanks to the demo website made by jnanar, the screen on the right is a phone (displayed thanks to scrcpy):

super easy authentication with XMPP

You can implement this in your own website either by using Chteufleur component (Chteufleur is actually the original implementer of the feature in SàT, many thanks), either by using a server where it's already available (it's for instance the case on jabberfr, thanks to the admins of this server).

Please note that once it's in place, in your website you just have to do a single HTTPS request to validate authentication.

Contacts

No need to take much time on this mode, but it's good to know that you can add a new contact there (with the top button), or delete an existing one with a long click/touch.

A simple click will launch the chat mode with the selected contact.

Command line (JP)

There are far too many new things in jp to describe everything here.

Some useful informations anyway:

  • in jp we always put commands first, then arguments (jp [command] [subcommand] --arg_1 --arg_2 --etc)
  • jp [command] --help tells you how to use a command
  • jp shell let you use a REPL shell
  • it is possible to have high level and low level commands. For instance with the blog, you can have blog posts in a nice way by using jp blog get -s bloger@somewhere.tld -O fancy, or the raw XML with jp pubsub get -s bloger@somewhere.tld -n urn:xmpp:microblog:0
  • many commands have an output mechanism with --output (or -O). For instance jp blog get -s bloger@somewhere.tld -O json let you find metadata of a blog feed in JSON
  • jp can generate static websites thanks to --output template
  • when the website generator handle it, you can directly use the blog URL to edit it. For instance, I can edit one of my last posts by using jp blog edit -u https://www.goffi.org/b/khii3yEKWDhmNYSDt5GGrY/vers-forge-decentralisee-basee-xmpp where https://www.goffi.org/b/khii3yEKWDhmNYSDt5GGrY/vers-forge-decentralisee-basee-xmpp is the URL of the HTML generated page. I then have my text editor showing the code of my page on one side (in my favorite syntax) and the metadata on the other. Note the this works with webpage generated by Libervia, but also by Movim.

A small demo of the shell to read a blog:

checking a blog from shell (jp)

Test it!

You can test all this, the installation instructions are available (in English only, we are lacking resources to translate) on the wiki by clicking here.

SàT (backend + jp (CLI) + Primitivus (TUI)) is available in Debian experimental (thanks to Robotux, Naha, Debacle and other packagers).

SàT and its frontends (jp, Primitivus, Cagou, Libervia) are available on Arch Linux (thanks to jnanar and Link Mauve)

We got confirmation that SàT and Cagou are working on Mac OS X (thanks to xma), we would like help of people to also test it on Windows, BSD*, or anything else.

You can test on Android by clicking on this link from your device (installation from external resources must be activated from settings).

Note that this is an alpha version, that means:

  • it crashes, the code is not stable
  • all features are not finished, or even here
  • it's not optimised yet
  • it's not polished

Oh, and while we are talking about features not yet implemented, there are strong possibilities that end to end encryption with OMEMO will be implemented before final release, thanks notably to the new Python module available (thanks to Syndace, the author). End to end encryption is already available for years in SàT, but only with OTR so far.

Participate!

A last animation to show you the automatic language detection, an experimental feature:

language detection, then messages filtering

Let's stop here for the features, but we have not seen all of them.

As you can see, there's a lot to play with, and there is enough for every taste in this project.

And that's good, with summer holidays you'll have more time, a perfect occasion to contribute to a unique project, ethics and activist.

Do not hesitate to come on SàT XMPP room (sat@chat.jabberfr.org, also available without account, from the web).

If you are lacking time (picnic on the beach is nice too), support on Liberapay and/or adhering to the association is also helping.

Try to spread the word around you, our project lack visibility and needs you. If you are willing to host an instance, you can start to test (in a "experimental/not stable" part of your website), and give us feedbacks. We are also looking for people helping to package in every place where it's possible.

Thanks in advance, and see you soon!

by goffi at September 10, 2018 06:04

September 09, 2018

Monal IM

iOS has OMEMO working

Just tried out the first beta for iOS 3.1. This is not the most exciting screen shot except for the fact that  the messages are OMEMO messages  from conversations. 

by Anu at September 09, 2018 00:49

September 07, 2018

Paul Schaub

Future of OMEMO

OMEMO is an XMPP extension protocol, which specifies end-to-end encryption for XMPP clients using the double ratchet algorithm of the Signal protocol. Introduced back in 2015 by GSoC student Andreas Straub in the Conversations client, OMEMO had a lot of press coverage and many privacy and security oriented websites praise XMPP clients that do support it. Its beyond debate, that OMEMO brought many new faces to XMPP. For many users, having end-to-end encryption built into their chat client is a must. Today OMEMO is implemented in a range of clients on different platforms. While Conversations, ChatSecure and Dino support it out of the box, there is a series of plugins that teach OMEMO to other clients such as Gajim, Pidgin and Miranda NG.

However, there is quite a lot of controversy around OMEMO. Part of it are technical discussions, others are more or less of a political nature. Let me list some of them for you.

Some users and client developers see no value in OMEMOs forward secrecy (the fact, that messages can only be decrypted once per device, so new devices do not have access to the chat history of the user). That is a fair point. Especially webclients have a hard time implementing OMEMO in a sensible way. Also the average user is probably having a hard time understanding what exactly forward secrecy is and what the consequences are. Communicating to the user, that not having access to past messages is actually a feature might be a hard task for a client developer.

OMEMOs trust management (still) sucks. One architectural key feature of OMEMO is, that every device does have its own identity key. If a user wants to send a message to one of their contacts, they’re presented with a list of all of their identity keys. Now they have to decide, which keys to trust and which not by comparing fingerprints (seamingly arbitrary strings of 64 characters). This is not a very comfortable thing to do. Some clients encourage the user to verify your contacts devices by scanning QR-Codes, which is way more comfortable, users do however have to meet up in person or share the QR code on another channel.
But what if you get a new device or just reinstall your chat application? Suddenly all your contacts have to decide whether to trust your new fingerprint or not. In the long run this will lead to the user just being annoyed and blindly accepting new fingerprints, ruining the concept of end-to-end encryption.

Daniel Gultsch introduced the concept of BTBV (Blind Trust Before Verification) which can be summed up as “do not bother the user with encryption and hope everything goes well until the user explicitly states that they are interested in having good security”. The principle is, that clients blindly trust any OMEMO identity keys until the user commits to verifying them manually. This makes OMEMO easy to use for those who don’t really care about it, while offering serious users who depend on it the needed security.

But what do you do, if suddenly a rogue fingerprint appears? Do you panic and message all your contacts not to trust the stranger key? In theory any device which has access to the users account (the users server too) can just add another OMEMO identity key to the users list of devices and there is not really anything the user can do about it. OMEMO does not have a “blacklist”-feature or a signed list of trusted keys. It would however be possible to implement such thing in the future by combining OMEMO with OpenPGP for example. Of course, if some stranger has access to your account, it is time to change the password/server anyways.

Another weakness of OMEMO is, that it is currently only usable for encrypting the messages body. Since XMPP is an extensible protocol with other use cases than messaging, it would be nice to have support for arbitrary extension element encryption. There is however the extension protocol “OX” (XEP-0373: OpenPGP for XMPP), which has such capabilities. This feature can be extracted from OX and reused in OMEMO relatively easy.

Lets now focus on the “political” controversies around OMEMO.

In 2016/2017 there has been a lot of discussions, whether or not OMEMO should become a standard in the first place. The problem is, that in it’s current form (which has not really changes since its introduction), OMEMO depends on the wire format used by libsignal (the Signal protocol library used by conversations). That library however is licensed under the GPLv3 license, preventing permissively licensed and closed source applications from implementing OMEMO. While the Signal protocol itself is openly documented, the wire format used by libsignal is not, so any implementations which want to be compatible to current OMEMO clients must implement the same wire format by looking into the libsignal source code, which in turn makes the implementation a derivative of libsignal, which must be licensed under the GPL as well. There has been a pull request against the OMEMO XEP which addressed this issue by specifying an independent wire format for OMEMO, however that pull request was more or less rejected due to inactivity of the author.

During the phases of hot debates around OMEMO, it was discussed to base the protocol on Olm instead of the Signal protocol. Olm is the encryption protocol used by matrix.org. However, up to this point there is no Olm based OMEMO implementation, that I know of, neither have there been any experiments in that direction from people that proposed the idea (again – not that I know of).

Another idea was to completely redesign OMEMOs double ratchet specification as OMEMO-NEXT from ground up without even mentioning a specific library as the foundation. This would obviously be a way more complex XEP, as all the cryptographic operations and primitives which are currently abstracted and hidden away behind the Signal protocol, would have to be written down in that XEP. However, recently the IETF announced that work is done to create MLS (Message Layer Security), which does exactly that. It specifies a completely open version of the double ratchet algorithm along with infrastructure to share key material and so on. I’m not sure whether this is a coincidence, or if some of those who proposed OMEMO-NEXT are directly involved with the development of MLS. We’ll see, when MLS is ready and whether it will compete against OMEMO. I’d really love to see a cross-protocol encryption algorithm btw ;) #bridges #federateEverything

Now lets talk about the biggest problem of OMEMO. Currently the XEP does not have an active “legal guardian”. The author has been inactive for an extended period of time, ignoring requests for comments, causing a total halt in the development of the standard (making changes to the XEP needs the authors approval). Things like specifying a new wire protocol are possible and reasonably easy to do. However not having changes written down in the XEP makes it nearly impossible to make coordinated changes. I’m sure there is a ton of potential for OMEMO and it is sad to see its (protocol-) development having come to a halt.

I’m sure that many of its current issues can be addressed by changes to the XEP. This is what I think needs to be done to OMEMO to make it more usable:

  • Specify a new wire protocol: This could make OMEMO accesible for commercial applications and allow independent implementations -> broader deployment
  • Specify a general payload encryption scheme: This could benefit other encryption protocols as well and would make it possible to apply end-to-end encryption to a wider variety of use cases.
  • Reuse the payload encryption scheme in OMEMO: Utilize OMEMO for things besides body encryption.
  • Specify a way to sign device lists with “persistent key” algorithms like OpenPGP: This could simplify trust management.
  • Specify a way to backup the identity key: This could reduce the identity key chaos, since the key could be reused on new devices/installations. However clients would have to make it clear to the user, not to use the same key on multiple devices.
This have been my thoughts about OMEMOs current state. What do you think about it?
Happy Hacking!

by vanitasvitae at September 07, 2018 12:49

September 05, 2018

Peter Saint-Andre

Perspectives on the Path

An essential lesson of philosophy is the ancient Greek formula "know yourself". It sounds easy, but it's immensely difficult. One model of self-knowledge I'm exploring is to look at your path in life from six different perspectives: beginning, end, above, below, side, within....

September 05, 2018 00:00

August 30, 2018

The XMPP Standards Foundation

The XMPP Newsletter, 31 August 2018

Welcome to the XMPP newsletter.

If you have an article, tutorial or blog post you'd like us to include in the newsletter, please submit it on the XMPP wiki.

This month's newsletter is a bit thinner than usual, no doubt due to the summer holidays in the northern hemisphere.

News

Maxime “pep.” Buquet has written a great report on the progress made at the recent XMPP sprint held at the Collabora offices in Cambridge.

The XMPP server compliance tester, started by Daniel Gultsch from Conversations.im, has been improved by Rishi Raj for Google Summer of Code and has a new website.

You can now test how compliant your server is at compliance.conversations.im.

The JSXC project has made a managed XMPP server available which is now also 100% compliant with the compliance tester.

Also from JSXC, XMPP Cloud Authentication is now available as Debian package.

The Business Bot platform now also supports XMPP chatbots.

ProgrammableWeb published a four part series on push protocols. In part 4, they mention XMPP and XMPP-IoT.

Software releases

Clients

Libraries

by jcbrand at August 30, 2018 22:00

August 28, 2018

Monal IM

New OMEMO alpha

If you downloaded the earlier OEMEO OSX build, I recommend you grab the newer build. It should resolve the crashing

by Anu at August 28, 2018 23:02

Monal OSX 2.2 With OMEMO (ALPHA)

This is a very early version of the Mac client with OMEMO. This is definitely not meant for general use.  I am really putting this out here because I’ve tested with Gajim, Chatsecure and Conversations and I’m curious to see where things break as more use it.  When I say alpha here, I mean it.  Download here

The UI is minimal and as a result everything is trusted. There is no UI to inspect/approve/revoke keys yet.  To use this, select a recipient, click the text input then click the lock icon on the toolbar (there isn’t any feedback other than the lock closing) and send some messages. You should see locks on the icons of the messages you send. This is really intended for testing, so you will have to re-enable sending with OMEMO when you go to another thread and back. It can receive OMEMO messages and will automatically show a lock next to them as well.

 

by Anu at August 28, 2018 02:53

August 24, 2018

ProcessOne

Real-time Stack Issue #14

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning tech aspects of real-time development we found interesting in Issue #14. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

RTB, an open source real-time protocols benchmark tool

The idea of RTB is to provide a benchmarking tool to stress test XMPP and MQTT servers with the minimum configuration overhead. Also, at least in the XMPP world, a “golden standard” benchmark is highly demanded in order to compare server implementations, avoiding disambiguations as much as possible.

Portable cloud programming with Go Cloud

The Go team at Google is releasing a new open source project, Go Cloud, a library and tools for developing on the open cloud. With this project, they aim to make Go the language of choice for developers building portable cloud applications.

Understanding and using the JavaScript MQTT client with websockets

Web browsers use the http protocol and modern ones can also use websockets. To publish and subscribe to an MQTT broker with a browser you will need to use JavaSript MQTT over websockets client.

Scalable IoT ML platform with Apache Kafka + deep learning + MQTT

The public cloud is used for training analytic models at extreme scale (e.g. using TensorFlow and TPUs on Google Cloud Platform (GCP) via Google ML Engine. The predictions (i.e. model inference) are executed on premise at the edge in a local Kafka infrastructure (e.g. leveraging Kafka Streams or KSQL for streaming analytics).

Thingstream: the world’s largest IoT network

Zurich based Thingstream claims to be able to provide IoT connectivity in 190 countries using the MQTT protocol over the networks of 600 mobile network operator partners.

Five biggest challenges facing embedded software developers

Developing embedded software is not as simple as it used to be. Creating a standalone device was and still often is challenging for many development teams. In the IoT age, in which we are starting to connect absolutely everything—including toothbrushes—system complexity is skyrocketing.

by Marek Foss at August 24, 2018 14:09

Real-time Enterprise Issue #13

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning business aspects of real-time enterprise we found interesting in Issue #13. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

Announcing RTB, an open source real-time protocols benchmark

ProcessOne is excited today to announce the first release of RTB (Real-Time Benchmark), an open source benchmarking tool to stress-test real-time protocols. The aim of this project is to define an easy benchmark for XMPP and MQTT servers, with minimal configuration overhead.

The fairy tale about a single standard for the Internet of Things

Standards are good. They allow the prices of products to go down quickly, they eliminate the fear of buying something that will not be compatible soon, and they lower the price of services. In a nutshell, standards speed up the adoption of new technology and lower its price. This is crucial.

Demystifying WebSockets for real-time web communication

WebSocket is a protocol for bidirectional communication (client-server and server-client), which is part of the HTML 5 API. In simple terms, it enables the interaction between a web client and a web server with lower overheads, facilitating data transfer in real-time.

Understanding push & streaming protocols: XMPP, CoAP, MQTT, and more

Even though Webhooks, PubSubHubbub, and WebSockets are really popular, they aren’t the only options at your disposal. You might find that you need some other alternative that better meets a particular need.

Supercharging Kafka to enable real-time web streaming

Apache Kafka is the new hotness when it comes to adding realtime messaging capabilities to your system. At its core, it is an open source distributed messaging system that uses a publish-subscribe system for building realtime data pipelines.

Chat apps interface: a case of Matrix/Riot redesign

With the redesign Jouni Helminen tried to focus on clarity, better hierarchy, contrast, colour and spacing to help with visual skimming — as well as establish design patterns to make life easier for people starting to use Riot.

by Marek Foss at August 24, 2018 14:01

Real-time Stack #13

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning tech aspects of real-time development we found interesting in Issue #13. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

Poezio

Poezio is a free console XMPP client. Its goal is to let you connect very easily (no account creation needed) to the network and join various chatrooms, immediately. It tries to look like the most famous IRC clients (weechat, irssi, etc).

Dis`root´

Disroot is a platform providing online services based on principles of freedom, privacy, federation and decentralization. The project is based in Amsterdam, it is maintained by volunteers and depends on the support of its community.

Building Confidence as a CTO

Matt Aimonetti writes: “I’ve been the CTO of Splice for more than 5 years now, from early prototypes to today’s 100+ employees. When Steve and I started Splice, I never thought we would get this big. What I didn’t know was that one of the challenges of being a CTO in a post product-market fit would be to build internal and external confidence around the engineering group.”

Microsoft Buys GitHub: The Linux Foundation’s Reaction

Jim Zemlin writes: “This week Microsoft announced that it is purchasing GitHub for $7.5 billion in stock. I waited for a few days to write up my thoughts because this is something that deserves some thoughtful reflection. The bottom line: This is pretty good news for the world of Open Source and we should celebrate Microsoft’s smart move.”

Designing Open Standards with the Minority Rule

This controversial text by Ben Sima about open standards is not easy to access to begin with – you need a browser in private mode to safely accept a certificate expiration exception. And while his analogies may be considered politically incorrect, his arguments are something to think about.

by Marek Foss at August 24, 2018 13:48

August 05, 2018

Ignite Realtime Blog

Smack 4.3.0 released

@Flow wrote:

The Ignite Realtime developer community is happy to announce the availability of Smack 4.3.0.

This is the first release of the new Smack 4.3 series, which brings new features and bug fixes.

Compared to the API changes between Smack 4.1 and 4.2, the changes of the Smack 4.3 are far less intrusive. Remember that the 4.2 API changes where mostly caused by the introduction of the Jid type, replacing String which was previously used to represent a JID.

We like to thank everyone who contributed by reporting bugs or giving suggestions. A special thank goes to people who contributed code:

$ git shortlog -sn 4.2.4..4.3.0
   227  Florian Schmaus
    22  vanitasvitae
    14  Jesus Fuentes
     6  Paul Schaub
     3  Guus der Kinderen
     2  Timothy Pitt
     2  Yash Thakkar
     1  Boris Grozev
     1  Cem Yabansu
     1  Daniel Hintze
     1  Fernando Ramirez
     1  Mohsen Hariri
     1  Rajat Kumar Gupta
     1  Son Goku
     1  Thomas
     1  akrherz

More information about the 4.3.0 release can be found one the release’s Readme.

Posts: 1

Participants: 1

Read full topic

by @Flow Florian Schmaus at August 05, 2018 15:54

August 02, 2018

The XMPP Standards Foundation

The XMPP Newsletter, 03 August 2018

Welcome to the latest edition of the XMPP newsletter!

If you have an article, tutorial or blog post you'd like us to include in the newsletter, please submit it on the XMPP wiki.

News

Slack has acquired HipChat's (and Stride's) intellectual property from Atlassian. HipChat was a proprietary service based on non-federated XMPP.

Prosody has added various PEP improvements which will be included in a future 0.11 release.

The Java XMPP library Smack now has OpenPGP support.

The Freedombone project now supports automatic XMPP notifications in addition to email notifications. Since publication of this post, Bob Mottram has informed us that notifications are now sent out as "headline" messages.

Dino has received a "filter expressions" feature as part of a Google Summer of Code project, allowing advanced search through messages.

Skype for Business Server will deprecate XMPP Gateways in 2019.

Guides and Howtos

In XMPP With Node.js, Jeremy Jackson documents how he sets up a development environment for writing an XMPP echo bot in server-side JavaScript.

Divya Prajapati has published a two-part guide on how to create an XMPP Android client: Part 1: OpenFire server installation Part 2: Creating an Android XMPP client

Videos

Daniel Gakwaya has made a video showing how to send and receive files with XMPP using HTTP-upload.

Software releases

Servers

  • Jackal 0.3
    • Jackal is a relatively new XMPP server written in Go. This release adds server-to-server federation support.
  • MongooseIM 3.1
    • Amongst other things, the Inbox feature has received new features, such as MSSQL support and support for MUC groupchats.

Clients

Libraries

Other

New Services

  • Muclumbus
    • A search engine for public Multi-User Chat Rooms

by jcbrand at August 02, 2018 22:00

August 01, 2018

Prosodical Thoughts

PEP improvements in trunk

Our next trunk nightly release uses a new implementation of PEP (XEP-0163) by default. PEP is used for a number of things, primarily sharing with your contacts: Extended statuses (e.g. sharing the user’s current playing music, their mood, activity, etc.) Avatars OMEMO keys Chatroom bookmarks While PEP started out as a simplified form of ‘pubsub’ and a way to share information with your contacts, it quickly became clear that it was also a neat mechanism for sharing public info with non-contacts (such as OMEMO keys) or sharing private data (such as saved chatrooms) with other clients on your account.

by The Prosody Team at August 01, 2018 21:54

July 30, 2018

Paul Schaub

Summer of Code: Smack has OpenPGP Support!

I am very proud to announce, that Smack got support for OpenPGP for XMPP!

Today the Pull Request I worked on during my GSoC project was merged into Smacks master branch. Admittedly it will take a few months until smack-openpgp will be included in a Smack release, but that gives me time to further finalize the code and iron out any bugs that may be in there. If you want to try smack-openpgp for yourself, let me know of any issues you encounter :)

(Edit: There are snapshot releases of Smack available for testing)

Now Smack does support two end-to-end encryption methods, which complement each other perfectly. OMEMO is best for people that want to be safe from future attacks, while OpenPGP is more suited for users who appreciate being able to access their chat history at any given time. OpenPGP is therefore the better choice for web based applications, although it is perfectly possible to implement web based clients that do OMEMO (see for example the Wire web app, which does ratcheting similar to OMEMO).

What’s left to do now is updating smack-openpgp due to updates made to XEP-0373 and extensive testing against other implementations.

Happy Hacking!

by vanitasvitae at July 30, 2018 12:03

July 27, 2018

Paul Schaub

Summer of Code: Finalizing the PR

Quick update:

Only a few days are left until the last and final Evaluation Phase.

I spent the week opening my pull request against Smacks master branch and adding a basic trust management implementation. Now the user is required to make decisions whether to trust a contacts key or not. However, the storage implementation is kept very modular, so an implementor can easily create a trust store implementation that realizes custom behaviour.

Smack-openpgp now allows users which did not subscribe to one another to exchange encryption keys quite easily. If a user receives an encrypted message, the implementation automatically fetches the senders keys to allow signature verification.

Furthermore there are more JUnit tests now, so that Smacks total test coverage actually increases when my PR gets merged :D

Happy Hacking!

by vanitasvitae at July 27, 2018 10:54

July 23, 2018

Peter Saint-Andre

Intensive Reading

As previously mentioned, I have in mind to write a novel about the ancient Greek philosopher Pyrrho and his journey to central Asia and northwestern India with the army of Alexander the Great. As part of my research I'm reading a great deal of philosophy, history, and fiction. It's been fascinating to return to the reading of fiction after a lapse of many years - I feel like I have fresh eyes for everything. My readings in the genre of philosophical fiction have ranged all over the map - during a business trip last week I devoured The Name of the Rose by Umberto Eco, before that I re-read The Left Hand of Darkness by Ursula K. Le Guin, and I'm currently reading both The Picture of Dorian Gray by Oscar Wilde and The Awakening by Kate Chopin. Alongside these works of fiction I'm reading books of philosophical scholarship (the latest were a full-length study of Pyrrho and an exploration of epistemology after Protagoras) and I'm starting to explore the historical background to the massive cultural change from classical to hellenistic Greece. Umberto Eco's book in particular has given me a new ambition of combining the story of Pyrrho's intellectual development with a mystery novel about the still unexplained death of Alexander the Great! It turns out the Pyrrho was probably not quite the radical skeptic he's usually taken to be (although he did believe that the entities we encounter in reality are indefinite and thus hard to pin down in many ways). Combine this with political intrigue on an imperial scale and perhaps regicide (indeed, there's an ancient legend that Aristotle himself was somehow involved in the assassination of his former pupil Alexander, perhaps in revenge for Alexander's execution of Aristotle's nephew Kallisthenes), and we might have the makings of large-scale novel that hinges in part on the validity of the law of non-contradiction! This is all quite speculative at this point because I have over 200 more books to read during the research phase, and I would also need to intensively study how the better authors construct their novels (an art of which I am mostly ignorant). Although I'm not sure if I have a book this big in me, I'm enjoying the exploration of possibilities. And all this reading has kept me from blogging, thus my absence here. Hopefully the results will be worth waiting for......

July 23, 2018 00:00

July 19, 2018

Paul Schaub

Summer of Code: First PGPainless Release!

I’m very happy and proud to announce the first alpha release of PGPainless!

PGPainless 0.0.1-alpha1 is the first non-snapshot release and is available from maven central. It was an interesting experience to go through the process of creating a release and I’m looking forward to have many more releases in the future :)

The current release contains a workaround for the bug I described in an earlier blog post. The issue was, that bouncycastle wouldn’t mark the public sub keys of a secret key ring as sub keys, which results in loss of keys if the user tries to create a public key ring from the exported public keys. My workaround fixes the issue by iterating through all sub keys of an existing key ring and converting the key packages of subkeys to subkey packages. The code is also available as a gist.

Ironically I had some issues related to OpenPGP during the release process. Releases to maven central have to be signed with an OpenPGP key, so I created a dedicated signing key pair using GnuPG, which I wanted to put into a separate GPG key box. After creating the key, I exported it using

gpg --armor --export-secret-keys [key-id] > pgpainless-singing-key.asc

imported it into a dedicated key box using

gpg --no-default-keyring --keyring pgpainless.gpg --import pgpainless-signing-key.asc

and deleted the key from my old key box, as well as the .asc-file. But when I tried to sign my release, I got the error, that a secret key would be needed. After checking the key box, I noticed, that only a public key was present.

Unfortunately that key had already been published to the key servers and I have no way to revoke it, due to lack of a secret key. I have no idea, what exactly happened or how it could happen, but its too late to recover the key.

Edit: I found my error: Importing a secret key is only possible with the flag `–allow-secret-key-import`, eg

gpg --import --allow-secret-key-import key.asc

So in the end I had to create a new OpenPGP key, which I now carefully backed up on a freshly bought USB stick which will be locked away for the event that I lose the copy on my work station. Better safe than sorry.

Happy Hacking!

by vanitasvitae at July 19, 2018 15:55

July 17, 2018

Christian Schudt

Babbler 0.8.0 released

After a longer period of development, Babbler 0.8.0 has just been released today.

The new major features are certainly

Further information about the release can be found in the release notes and in the changelog.

by Christian Schudt (noreply@blogger.com) at July 17, 2018 15:29

July 16, 2018

ProcessOne

Announcing RTB, an open source real-time protocols benchmark tool

ProcessOne is excited today to announce the first release of RTB (Real-Time Benchmark), an open source benchmarking tool to stress-test real-time protocols. The aim of this project is to define an easy benchmark for XMPP and MQTT servers, with minimal configuration overhead.

Why

To compare server implementations and be sure they are ready for massive traffic, there is a need for a “golden standard” benchmarking tool that is easy to use and avoids ambiguity due to configuration or overhead – each test should have the same baseline and capabilities for the results to be comparable.

We believe RTB fits such a role, because it has sane defaults (gathered from statistics of real world servers) and is able to cover all the test features defined in the XMPP Compliance Suite 2018.

Status

RTB is in early stages of development: currently, only XMPP protocol is implemented, and support for Multi-User Chat (MUC) and Personal Eventing Protocol (PEP) is lacking.

However, “sane” defaults and what should be considered a “golden standard” for this benchmark is open for discussion with the XMPP community – we welcome issues and pull requests. The tool has already been battle-tested: we used RTB to stress test our ejabberd SaaS deployments with more than 2 000 000 concurrent connections.

RTB includes a script to populate your server’s database with user accounts and rosters, so it’s easy to create test pools. When you start a test with RTB, you can monitor a local website on port 8080 with live graphs like the one below:

Requirements

We want RTB to be straightforward and easy to deploy. Currently, only Unix-based operating systems are supported. To compile, you just need gcc, make, Erlang, expat, yaml, openssl, zlib and gnuplot. There’s a default YAML configuration file to get you started and only 5 mandatory parameters before you can lanuch your first benchmark.

RTB is available on GitHub under Apache License 2.0 and we are waiting for your feedback!

by Marek Foss at July 16, 2018 12:59