This summer the MongooseIM team have not had a second to be lazy - the development of MongooseIM 3.1 release took over what was supposed to be our downtime, when all major activities driven throughout the year slow down. We’re happy to say that in this time of leisure, the new release is packed with important features, improvements and excitement. Take a look at the version highlights and be part of this major step in creating a world-class communication server.

This time, a large part of the changes are focused on development efficiency, but rest assured, we’ve added new items you are going to appreciate.

In the “Test Runner” section, you get to learn all about a little tool we provided so that you can easily set the development environment up and run tests locally without Travis CI.

Our Inbox extension has got three big updates that push the boundaries of what’s currently offered in XMPP world. As a rule, we want to lead by example; less talk and more action. That is why working in cooperation with Forward, we decided to put forward an implementation of a popular use case as a unique feature in this software sector.

This release is also an important lesson for us. A lesson about edge cases and concurrency in integration testing. You don’t necessarily have to be an Erlang developer to benefit from the third section, but reading it allows you to learn with us.

The “Honorable Mentions” section may seem minor, but for some projects the items listed there can indeed make a difference! It’s a candy mix of different changes, so read carefully not to miss your favourite flavours!

Obviously, a single blog post is too small a space to tell a profound story about every new item in the changelog, so we encourage you to check it out. You can find a link at the bottom of this blog post.

Test Runner

The Travis CI is our main verification tool for the Pull Requests we create and merge. Whilst being convenient and critical for our process, it is not very handy for day-to-day development. It is very common to frequently execute limited subset of tests to ensure that a new piece of code we wrote works perfectly. However, waiting for Travis results every time would extend implementation time excessively as it executes multiple presets, while being a shared resource at the same time.

The test runner is a script that helps to set the development environment up, and run tests on a developer machine locally. The test runner shares a lot of code with our Travis CI build scripts, which ensures that test results are consistent between local and CI runs.

The test runner allows to choose which tests to run. It can be useful, when one of the tests is failing and we want to rerun it.

Since MongooseIM supports different database backends, the test runner is able to set a selected database up for you and configure MongooseIM. Simply put, it prepares the local environment before executing actual test suites.

The test runner supports a lot of options to customise your build and test execution scenario. It also supports shell autocompletion for option names and values.

We’ve prepared a recording for you that presents a basic usage of the script.

.

Please note that Asciinema allows you to pause and copy any text fragment you like, so it would be very easy for you to repeat the same steps.

New Inbox Features

Originally sponsored by and created for Forward, Inbox has been available as an open source extension for the last two months already. In MongooseIM 3.1, it returns with a bundle of fresh goodies. The first feature is MSSQL support.

Despite being less frequently used with MongooseIM compared to MySQL or PostgreSQL, it’s still an important piece of the architecture for many projects, especially those running in Azure cloud. We don’t want you to feel excluded, dear Microsoft users!

The second one is the support for classic MUC group chats. MUC Light was never intended as a complete replacement for original XEP-0045 solution. It means that numerous projects exists where mod_muc is still a better match than its younger sibling, and they may now benefit from inbox features as well!

Last but not least is the timestamp support. First of all, they are stored in DB and returned in Inbox query results. For those using mod_inbox from MIM 3.0: you’ll need to update your schemas but don’t worry - it isn’t very complicated. What’s more, a client may now request conversations from a certain time period and sort them by timestamp, both ascending and descending.

This is not our final word on this matter. You may expect further improvements to this extension in upcoming MongooseIM versions!

We’ve prepared a demo of the Inbox feature. It shows both the backed and the frontend side of it. The application used in the demo has been designed by Forward.

.

Lessons learnt regarding CI

OK, these are short and sweet but nevertheless important:

1. Avoid RPC in integration tests. They tend to time out in slow CI environments (such as Travis).
2. When test users exchange messages, always wait until they are received to ensure proper synchronisation.
3. On a slow machine, MSSQL SELECT query may return more than one row (even when retrieving by the exact primary key value) as a consequence of the transaction deadlock.
4. When you can’t use any other means of server state synchronisation, don’t use hardcoded sleep periods; replace them with an incremental backoff and verification in a loop. Sometimes you can’t predict whether a server state is updated properly in 500ms, 1000ms or 3000ms. Adding 5s waits everywhere may cause test suites to run veeery long.
5. Be careful about leaking presences between cases. This applies to XMPP testing. Best practice is to generate fresh user accounts for every scenario.
6. Some databases don’t support transactions so the new data may not be instantly available. For example, in the case of Riak (its Search function in particular) a delay between data insert and query is required.
7. Sometimes creating a schema in a DB may fail for the first time due to timing issues, so implement a retry loop in a DB provisioning scripts. This also applies to Riak.
8. Did I mention creating new user accounts for every test case? It actually applies not only to XMPP. With this practice, you won’t have to worry about possible leftovers of a user’s state.

Honorable mentions

ElasticSearch backend for Message Archive

Almost every MongooseIM module supports more than one type of backend. Usually it’s Mnesia, RDBMS and sometimes Riak. Message Archive Management is a noteworthy exception, as we’ve implemented RDBMS, Riak and Cassandra support for this module. Or “modules” actually, as it consists of over 30 Erlang files already.

It is a very special feature as it processes a vast amount of data and sometimes executes expensive queries. In order to ensure performance and match a project’s architecture, wide range of supported DB backends is essential.

It is our pleasure to announce that yet another backend has joined the family: ElasticSearch.

OTP 21 support

OTP 21.0 has been released ~1 month ago and we’ve added support for this version less than a week after! This is great news for all projects sticking to the most recent Erlang technology as pioneers in BEAM world. The new platform version brought not only improvements in regards to performance but also some incompatibilities that we’ve resolved, so MongooseIM still remains at a technological peak.

As a tradeoff, we’ve dropped official support for OTP 18.x. It should still be possible to compile 3.1 with this version with some minor code modifications, but we’re definitely moving forward. It has allowed us to get rid of non-typed maps specifications as an example. As a reminder, bear in mind, that MongooseIM always supports two most recent, stable OTP branches (currently these are 20.x and 19.x and one being under an active development, 21.x).

Jingle/SIP tutorial

SIP is a common choice for VoIP applications but certain XMPP features may be a very good match for such software. MongooseIM is able to liaise between these two worlds and now it’s easier than ever with significantly extended documentation (compared to the level in 3.0) and a tutorial on mod_jingle_sip usage.

Worker pool unification

Every developer writes a custom worker pool at some point of their career. Everyone. Certain MongooseIM components (the ones that use connection pools) were created with different preferred library in mind. As a result, we’ve ended up with many kinds of worker pools: cuesport, worker_pool and poolboy. It wasn’t only a matter of maintenance difficulty, but performance as well. As an example, cuesport supports only a simple round-robin job assignment algorithm, which is not optimal in every case. It also lacks inspection of any kind.

Given our experience gathered over the years, we’ve selected worker_pool as our primary library. It is very flexible and exposes a dedicated stats API. It was originally created by Inaka and it is actively maintained by its former developers at this present time.

For now, the changes are purely internal. Some projects may observe better performance but the primary goal was to prepare for a second round of unification. Stay tuned for more details in near future.

Changelog

Please feel free to read the detailed changelog. Here, you can find a full list of source code changes and useful links.

Contributors Special thanks to our contributors: @SamuelNichols @Beisenbek @GalaxyGorilla!

Test our work on MongooseIM 3.1 and share your feedback

Help us improve the MongooseIM platform:

1. Star our repo: esl/MongooseIM
2. Report issues: esl/MongooseIM/issues
3. Share your thoughts via Twitter: @MongooseIM
4. Download Docker image with new release
5. Sign up to our dedicated mailing list to stay up to date about MongooseIM, messaging innovations and industry news.
6. Check out our MongooseIM product page for more information on the MongooseIM platform.

ProcessOne is proud to announce that ejabberd Business Edition now supports MQTT 5. This is a whole new protocol added – as an addition to XMPP (and SIP) – to our server. Our enterprise customers will be able to use both MQTT and XMPP in their real-time solutions.

ejabberd is the world’s leading XMPP server, both in performance and market share. With MQTT support, ejabberd brings equally incredible performance to this popular protocol, and true flexibility to real-time solutions.

What is MQTT?

MQTT stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks. The design principles are to minimise network bandwidth and device resource requirements while also attempting to ensure reliability and some degree of assurance of delivery.

These principles also turn out to make the protocol ideal for the emerging “machine-to-machine” (M2M) or “Internet of Things” world of connected devices, and for mobile applications where bandwidth and battery power are at a premium.

Why is it a big deal?

It is the first time that a single platform is able to support the major reference real-time standard protocols.

XMPP is an IETF standard for instant messaging communication. It is also used for Internet of Things in use cases where powerful devices need to be connected to the internet, especially with their own presence signal and control interface. XMPP offers more features than MQTT, but the extra flexibility and complexity comes at a cost. Some less powerful devices cannot leverage XMPP and need to resort to MQTT, which is designed with that small devices use cases in mind. MQTT is a standard from OASIS.

Supporting both standards opens a whole set of new use cases, making it possible to connect people, applications, small IoT devices and larger IoTs on the same platform.

Moreover, from the very beginning ejabberd Business Edition provides full support for the brand new MQTT 5, being ready to support the most advanced use cases offered by this protocol. It is one of the very first production-ready MQTT 5 servers, with compliance to the brand new specification released in May 2018.

MQTT and XMPP

Thanks to combined support of XMPP and MQTT, ejabberd Business Edition is the first truly multi-protocols real-time messaging servers. Your solutions can use the best of both worlds, and your implementations can be flexible and scalable at the same time.

By using a multi-protocol server, your deployment can benefit from the ecosystems of both protocols, and be compatible with vast number of standard tools and software.

Does it mean, we consider XMPP obsolete? Absolutely not. XMPP and MQTT are designed with different use cases in mind. However, we think that in many cases, a real-time project will need to leverage both XMPP and MQTT during its lifetime, making humans, things and application collaboration possible. This is our vision.

What are the main benefits?

ejabberd Business Edition has been much refined during recent years. ProcessOne has patiently engineered a server framework that is stable and highly available thanks to a fast and flexible clustering engine, and versatile thanks to its plugin API.

MQTT protocol in ejabberd Business Edition has been reimplemented from scratch, but the server infrastructure leverages the brand new modular architecture of eBE. It means it uses the same highly scalable clustering engine, the same authentication infrastructure, can provide virtual hosting, reuse the logging modules, and use the same API.

For the users, the benefits are clear: you can reuse the same server, infrastructure, authentication mechanism, security policy for your XMPP clients and MQTT clients. You can invest in a single platform and be sure to be future-proof and address both your current and future needs in term of real-time technologies.

You can have a part of your devices connected to XMPP and another part connected on MQTT, without the need to manage two separate platforms. And most of all, you can design your system for use cases mixing XMPP strong points (human collaboration, chat bots for automation) with MQTT (connection to millions of very lightweight devices).

The bottom line is simple:

1. You can switch between XMPP and MQTT as you wish, even use both protocols on the same infrastructure.
2. You will save on infrastructure, given the high-performance of the platform.
3. You get support on solution design for real-time infrastructure and can get help choosing between XMPP and MQTT, from a vendor that has no interest in selling one protocol more than another.

How can I test it?

If you are a developer and would like to try connecting your client to our MQTT server, we got you covered.

ProcessOne has deployed a public MQTT broker with anonymous access. This is one of the first MQTT 5 public brokers available.

The broker is deployed at mqtt.fluux.io on port 1883 and 8883 (TLS).

You can, for example, test our broker with mosquitto client. On macOS, you can install mosquitto via Terminal using Homebrew with:

brew install mosquitto

From there, you can subscribe to the topic test/topic with:

mosquitto_sub -h mqtt.fluux.io -p 1883 -V mqttv311 -v -t "test/topic" -q 2

and then (in another Terminal tab or window) you can publish with:

mosquitto_pub -h mqtt.fluux.io -p 1883 -V mqttv311 -t "test/topic" -m "message" -q 2

You should see your message received and displayed below the mosquitto_sub command in your previous Terminal tab or window.

Note: In this example, Mosquitto client is using MQTT v3.1.1, as it doesn’t support version 5.0 at the moment of writing.

Don’t hesitate to send your feedback and findings through our contact page. Happy hacking!

What is next?

From our initial tests ejabberd Business Edition MQTT performance is very impressive, and we will be publishing our benchmarks soon.

Regarding MQTT 5, shared subscriptions are still being developed and will be available in a next release.

Regarding protocol version coverage, our platform supports MQTT 3.1.1 and we plan to add support for MQTT 3.1 as well in a future update.

Most of all, we are planning to increase the interoperability between MQTT and XMPP. The plan is to have both XMPP and MQTT clients be able to leverage the same services provided by ejabberd Business Edition: push notifications, groupchat through Muc/Sub, etc.

The goal is to leverage use cases based on real-time communication between humans, things, and applications. This can only be done by building an ambitious platform that is protocol agnostic and leverages the best of each technology.

We believe the IoT revolution will fuel the growth of both XMPP and MQTT protocols. For this to happen, the development community needs flexible, scalable and – most importantly – trusted tools. For more than 15 years ejabberd has been the best XMPP server. We now want it to be the best MQTT server as well.

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning business aspects of real-time enterprise we found interesting in Issue #14. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

Companies are migrating to the cloud for the IoT

Developers’ evolving demands and needs will push IoT platform providers to reshape their products and services in the coming years. Developers’ needs, concerns, and demands will change as their companies build out their IoT projects.

The Internet of Things explained by a professor

The IoT becomes a subject of the degree course on Production and Process Management at Heilbronn University of Applied Sciences in Germany. Prof. Dr. Thomas Pospiech talks about the “IoT box”, an invention he has developed to better convey the concept.

IBM teams with Maersk on new blockchain shipping solution

IBM and shipping giant Maersk having been working together for the last year developing a blockchain-based shipping solution called TradeLens. Today they moved the project from Beta into limited availability.

Risky business? How to manage IoT security in the enterprise

Nearly every week, we hear about a new cyberattack or security breach. In fact, more than 1.7 billion identities have been exposed in data breaches in the past eight years. As the world becomes increasingly interconnected, organizations become susceptible to more risk.

IoT boom will change how data is analysed

As more devices come online, each generating data, the way information is analysed and used to facilitate machine learning will have to change.

Five IoT predictions for 2019

The IoT is growing at an exponential rate. Vehicles, wearable gadgets, RFID sensors and software are advancing past basic function and the network is growing to include even more advancements each day. What will the future bring?

Current work in progress share sheet

I have found the lack of a share sheet  in iOS and OSX limits how much I use Monal.  A lot of what I send in chats are links. I suspect that is the case for a lot of other people too. To address that, I’ve added a share sheet to the iOS client. This code should work on OSX too.  Doesn’t quite work yet but when it’s done, will work the way you expect it to. 

Just tried out the first beta for iOS 3.1. This is not the most exciting screen shot except for the fact that  the messages are OMEMO messages  from conversations. 

OMEMO is an XMPP extension protocol, which specifies end-to-end encryption for XMPP clients using the double ratchet algorithm of the Signal protocol. Introduced back in 2015 by GSoC student Andreas Straub in the Conversations client, OMEMO had a lot of press coverage and many privacy and security oriented websites praise XMPP clients that do support it. Its beyond debate, that OMEMO brought many new faces to XMPP. For many users, having end-to-end encryption built into their chat client is a must. Today OMEMO is implemented in a range of clients on different platforms. While Conversations, ChatSecure and Dino support it out of the box, there is a series of plugins that teach OMEMO to other clients such as Gajim, Pidgin and Miranda NG.

However, there is quite a lot of controversy around OMEMO. Part of it are technical discussions, others are more or less of a political nature. Let me list some of them for you.

Some users and client developers see no value in OMEMOs forward secrecy (the fact, that messages can only be decrypted once per device, so new devices do not have access to the chat history of the user). That is a fair point. Especially webclients have a hard time implementing OMEMO in a sensible way. Also the average user is probably having a hard time understanding what exactly forward secrecy is and what the consequences are. Communicating to the user, that not having access to past messages is actually a feature might be a hard task for a client developer.

OMEMOs trust management (still) sucks. One architectural key feature of OMEMO is, that every device does have its own identity key. If a user wants to send a message to one of their contacts, they’re presented with a list of all of their identity keys. Now they have to decide, which keys to trust and which not by comparing fingerprints (seamingly arbitrary strings of 64 characters). This is not a very comfortable thing to do. Some clients encourage the user to verify your contacts devices by scanning QR-Codes, which is way more comfortable, users do however have to meet up in person or share the QR code on another channel.
But what if you get a new device or just reinstall your chat application? Suddenly all your contacts have to decide whether to trust your new fingerprint or not. In the long run this will lead to the user just being annoyed and blindly accepting new fingerprints, ruining the concept of end-to-end encryption.

Daniel Gultsch introduced the concept of BTBV (Blind Trust Before Verification) which can be summed up as “do not bother the user with encryption and hope everything goes well until the user explicitly states that they are interested in having good security”. The principle is, that clients blindly trust any OMEMO identity keys until the user commits to verifying them manually. This makes OMEMO easy to use for those who don’t really care about it, while offering serious users who depend on it the needed security.

But what do you do, if suddenly a rogue fingerprint appears? Do you panic and message all your contacts not to trust the stranger key? In theory any device which has access to the users account (the users server too) can just add another OMEMO identity key to the users list of devices and there is not really anything the user can do about it. OMEMO does not have a “blacklist”-feature or a signed list of trusted keys. It would however be possible to implement such thing in the future by combining OMEMO with OpenPGP for example. Of course, if some stranger has access to your account, it is time to change the password/server anyways.

Another weakness of OMEMO is, that it is currently only usable for encrypting the messages body. Since XMPP is an extensible protocol with other use cases than messaging, it would be nice to have support for arbitrary extension element encryption. There is however the extension protocol “OX” (XEP-0373: OpenPGP for XMPP), which has such capabilities. This feature can be extracted from OX and reused in OMEMO relatively easy.

Lets now focus on the “political” controversies around OMEMO.

In 2016/2017 there has been a lot of discussions, whether or not OMEMO should become a standard in the first place. The problem is, that in it’s current form (which has not really changes since its introduction), OMEMO depends on the wire format used by libsignal (the Signal protocol library used by conversations). That library however is licensed under the GPLv3 license, preventing permissively licensed and closed source applications from implementing OMEMO. While the Signal protocol itself is openly documented, the wire format used by libsignal is not, so any implementations which want to be compatible to current OMEMO clients must implement the same wire format by looking into the libsignal source code, which in turn makes the implementation a derivative of libsignal, which must be licensed under the GPL as well. There has been a pull request against the OMEMO XEP which addressed this issue by specifying an independent wire format for OMEMO, however that pull request was more or less rejected due to inactivity of the author.

During the phases of hot debates around OMEMO, it was discussed to base the protocol on Olm instead of the Signal protocol. Olm is the encryption protocol used by matrix.org. However, up to this point there is no Olm based OMEMO implementation, that I know of, neither have there been any experiments in that direction from people that proposed the idea (again – not that I know of).

Another idea was to completely redesign OMEMOs double ratchet specification as OMEMO-NEXT from ground up without even mentioning a specific library as the foundation. This would obviously be a way more complex XEP, as all the cryptographic operations and primitives which are currently abstracted and hidden away behind the Signal protocol, would have to be written down in that XEP. However, recently the IETF announced that work is done to create MLS (Message Layer Security), which does exactly that. It specifies a completely open version of the double ratchet algorithm along with infrastructure to share key material and so on. I’m not sure whether this is a coincidence, or if some of those who proposed OMEMO-NEXT are directly involved with the development of MLS. We’ll see, when MLS is ready and whether it will compete against OMEMO. I’d really love to see a cross-protocol encryption algorithm btw #bridges #federateEverything

Now lets talk about the biggest problem of OMEMO. Currently the XEP does not have an active “legal guardian”. The author has been inactive for an extended period of time, ignoring requests for comments, causing a total halt in the development of the standard (making changes to the XEP needs the authors approval). Things like specifying a new wire protocol are possible and reasonably easy to do. However not having changes written down in the XEP makes it nearly impossible to make coordinated changes. I’m sure there is a ton of potential for OMEMO and it is sad to see its (protocol-) development having come to a halt.

I’m sure that many of its current issues can be addressed by changes to the XEP. This is what I think needs to be done to OMEMO to make it more usable:

• Specify a new wire protocol: This could make OMEMO accesible for commercial applications and allow independent implementations -> broader deployment
• Specify a general payload encryption scheme: This could benefit other encryption protocols as well and would make it possible to apply end-to-end encryption to a wider variety of use cases.
• Reuse the payload encryption scheme in OMEMO: Utilize OMEMO for things besides body encryption.
• Specify a way to sign device lists with “persistent key” algorithms like OpenPGP: This could simplify trust management.
• Specify a way to backup the identity key: This could reduce the identity key chaos, since the key could be reused on new devices/installations. However clients would have to make it clear to the user, not to use the same key on multiple devices.

An essential lesson of philosophy is the ancient Greek formula "know yourself". It sounds easy, but it's immensely difficult. One model of self-knowledge I'm exploring is to look at your path in life from six different perspectives: beginning, end, above, below, side, within....

Welcome to the XMPP newsletter.

If you have an article, tutorial or blog post you'd like us to include in the newsletter, please submit it on the XMPP wiki.

This month's newsletter is a bit thinner than usual, no doubt due to the summer holidays in the northern hemisphere.

News

Maxime “pep.” Buquet has written a great report on the progress made at the recent XMPP sprint held at the Collabora offices in Cambridge.

The XMPP server compliance tester, started by Daniel Gultsch from Conversations.im, has been improved by Rishi Raj for Google Summer of Code and has a new website.

You can now test how compliant your server is at compliance.conversations.im.

The JSXC project has made a managed XMPP server available which is now also 100% compliant with the compliance tester.

Also from JSXC, XMPP Cloud Authentication is now available as Debian package.

The Business Bot platform now also supports XMPP chatbots.

ProgrammableWeb published a four part series on push protocols. In part 4, they mention XMPP and XMPP-IoT.

Subscribe to receive the next edition in your inbox

Software releases

Clients

• Kontalk 4.2.0-beta1
• Monal OSX 2.2 With OMEMO (ALPHA)

Libraries

• Smack 4.3.0
• XMPP Cloud Authentication 2.0.2

If you downloaded the earlier OEMEO OSX build, I recommend you grab the newer build. It should resolve the crashing

This is a very early version of the Mac client with OMEMO. This is definitely not meant for general use.  I am really putting this out here because I’ve tested with Gajim, Chatsecure and Conversations and I’m curious to see where things break as more use it.  When I say alpha here, I mean it.  Download here

The UI is minimal and as a result everything is trusted. There is no UI to inspect/approve/revoke keys yet.  To use this, select a recipient, click the text input then click the lock icon on the toolbar (there isn’t any feedback other than the lock closing) and send some messages. You should see locks on the icons of the messages you send. This is really intended for testing, so you will have to re-enable sending with OMEMO when you go to another thread and back. It can receive OMEMO messages and will automatically show a lock next to them as well.

 

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning tech aspects of real-time development we found interesting in Issue #14. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

RTB, an open source real-time protocols benchmark tool

The idea of RTB is to provide a benchmarking tool to stress test XMPP and MQTT servers with the minimum configuration overhead. Also, at least in the XMPP world, a “golden standard” benchmark is highly demanded in order to compare server implementations, avoiding disambiguations as much as possible.

Portable cloud programming with Go Cloud

The Go team at Google is releasing a new open source project, Go Cloud, a library and tools for developing on the open cloud. With this project, they aim to make Go the language of choice for developers building portable cloud applications.

Understanding and using the JavaScript MQTT client with websockets

Web browsers use the http protocol and modern ones can also use websockets. To publish and subscribe to an MQTT broker with a browser you will need to use JavaSript MQTT over websockets client.

Scalable IoT ML platform with Apache Kafka + deep learning + MQTT

The public cloud is used for training analytic models at extreme scale (e.g. using TensorFlow and TPUs on Google Cloud Platform (GCP) via Google ML Engine. The predictions (i.e. model inference) are executed on premise at the edge in a local Kafka infrastructure (e.g. leveraging Kafka Streams or KSQL for streaming analytics).

Thingstream: the world’s largest IoT network

Zurich based Thingstream claims to be able to provide IoT connectivity in 190 countries using the MQTT protocol over the networks of 600 mobile network operator partners.

Five biggest challenges facing embedded software developers

Developing embedded software is not as simple as it used to be. Creating a standalone device was and still often is challenging for many development teams. In the IoT age, in which we are starting to connect absolutely everything—including toothbrushes—system complexity is skyrocketing.

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning business aspects of real-time enterprise we found interesting in Issue #13. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

Announcing RTB, an open source real-time protocols benchmark

ProcessOne is excited today to announce the first release of RTB (Real-Time Benchmark), an open source benchmarking tool to stress-test real-time protocols. The aim of this project is to define an easy benchmark for XMPP and MQTT servers, with minimal configuration overhead.

The fairy tale about a single standard for the Internet of Things

Standards are good. They allow the prices of products to go down quickly, they eliminate the fear of buying something that will not be compatible soon, and they lower the price of services. In a nutshell, standards speed up the adoption of new technology and lower its price. This is crucial.

Demystifying WebSockets for real-time web communication

WebSocket is a protocol for bidirectional communication (client-server and server-client), which is part of the HTML 5 API. In simple terms, it enables the interaction between a web client and a web server with lower overheads, facilitating data transfer in real-time.

Understanding push & streaming protocols: XMPP, CoAP, MQTT, and more

Even though Webhooks, PubSubHubbub, and WebSockets are really popular, they aren’t the only options at your disposal. You might find that you need some other alternative that better meets a particular need.

Supercharging Kafka to enable real-time web streaming

Apache Kafka is the new hotness when it comes to adding realtime messaging capabilities to your system. At its core, it is an open source distributed messaging system that uses a publish-subscribe system for building realtime data pipelines.

Chat apps interface: a case of Matrix/Riot redesign

With the redesign Jouni Helminen tried to focus on clarity, better hierarchy, contrast, colour and spacing to help with visual skimming — as well as establish design patterns to make life easier for people starting to use Riot.

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning tech aspects of real-time development we found interesting in Issue #13. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

Poezio

Poezio is a free console XMPP client. Its goal is to let you connect very easily (no account creation needed) to the network and join various chatrooms, immediately. It tries to look like the most famous IRC clients (weechat, irssi, etc).

Dis`root´

Disroot is a platform providing online services based on principles of freedom, privacy, federation and decentralization. The project is based in Amsterdam, it is maintained by volunteers and depends on the support of its community.

Building Confidence as a CTO

Matt Aimonetti writes: “I’ve been the CTO of Splice for more than 5 years now, from early prototypes to today’s 100+ employees. When Steve and I started Splice, I never thought we would get this big. What I didn’t know was that one of the challenges of being a CTO in a post product-market fit would be to build internal and external confidence around the engineering group.”

Microsoft Buys GitHub: The Linux Foundation’s Reaction

Jim Zemlin writes: “This week Microsoft announced that it is purchasing GitHub for $7.5 billion in stock. I waited for a few days to write up my thoughts because this is something that deserves some thoughtful reflection. The bottom line: This is pretty good news for the world of Open Source and we should celebrate Microsoft’s smart move.”

Designing Open Standards with the Minority Rule

This controversial text by Ben Sima about open standards is not easy to access to begin with – you need a browser in private mode to safely accept a certificate expiration exception. And while his analogies may be considered politically incorrect, his arguments are something to think about.

@Flow wrote:

The Ignite Realtime developer community is happy to announce the availability of Smack 4.3.0.

This is the first release of the new Smack 4.3 series, which brings new features and bug fixes.

Compared to the API changes between Smack 4.1 and 4.2, the changes of the Smack 4.3 are far less intrusive. Remember that the 4.2 API changes where mostly caused by the introduction of the Jid type, replacing String which was previously used to represent a JID.

We like to thank everyone who contributed by reporting bugs or giving suggestions. A special thank goes to people who contributed code:

$ git shortlog -sn 4.2.4..4.3.0
   227  Florian Schmaus
    22  vanitasvitae
    14  Jesus Fuentes
     6  Paul Schaub
     3  Guus der Kinderen
     2  Timothy Pitt
     2  Yash Thakkar
     1  Boris Grozev
     1  Cem Yabansu
     1  Daniel Hintze
     1  Fernando Ramirez
     1  Mohsen Hariri
     1  Rajat Kumar Gupta
     1  Son Goku
     1  Thomas
     1  akrherz

More information about the 4.3.0 release can be found one the release’s Readme.

Posts: 1

Participants: 1

Read full topic

Welcome to the latest edition of the XMPP newsletter!

If you have an article, tutorial or blog post you'd like us to include in the newsletter, please submit it on the XMPP wiki.

News

Slack has acquired HipChat's (and Stride's) intellectual property from Atlassian. HipChat was a proprietary service based on non-federated XMPP.

Prosody has added various PEP improvements which will be included in a future 0.11 release.

The Java XMPP library Smack now has OpenPGP support.

The Freedombone project now supports automatic XMPP notifications in addition to email notifications. Since publication of this post, Bob Mottram has informed us that notifications are now sent out as "headline" messages.

Dino has received a "filter expressions" feature as part of a Google Summer of Code project, allowing advanced search through messages.

Skype for Business Server will deprecate XMPP Gateways in 2019.

Subscribe to receive the next edition in your inbox

Guides and Howtos

In XMPP With Node.js, Jeremy Jackson documents how he sets up a development environment for writing an XMPP echo bot in server-side JavaScript.

Divya Prajapati has published a two-part guide on how to create an XMPP Android client: Part 1: OpenFire server installation Part 2: Creating an Android XMPP client

Videos

Daniel Gakwaya has made a video showing how to send and receive files with XMPP using HTTP-upload.

Software releases

Servers

• Jackal 0.3
  • Jackal is a relatively new XMPP server written in Go. This release adds server-to-server federation support.
• MongooseIM 3.1
  • Amongst other things, the Inbox feature has received new features, such as MSSQL support and support for MUC groupchats.

Clients

• Salut à Toi 0.7 alpha
  • Features include forums, photo albums, file sharing and issue tracking.

Libraries

• Babbler 0.8.0
  • Adds support for XEP-0390: Entity Capabilities 2.0 and XEP-0392: Consistent Color Generation.
• aioxmpp 0.10 An XMPP library for use with Python 3.4+ asyncio.

Other

• XMPP Cloud Auth v1.1.0
  • Allows you to use Nextcoud as authentication service for XMPP servers.
• Real-Time Benchmark
  • An open source benchmarking tool to stress-test real-time protocols, such as XMPP.
• XMPP Echo bot written in SED
  • This project received some admiration in this Hacker News discussion

New Services

• Muclumbus
  • A search engine for public Multi-User Chat Rooms

Our next trunk nightly release uses a new implementation of PEP (XEP-0163) by default. PEP is used for a number of things, primarily sharing with your contacts: Extended statuses (e.g. sharing the user’s current playing music, their mood, activity, etc.) Avatars OMEMO keys Chatroom bookmarks While PEP started out as a simplified form of ‘pubsub’ and a way to share information with your contacts, it quickly became clear that it was also a neat mechanism for sharing public info with non-contacts (such as OMEMO keys) or sharing private data (such as saved chatrooms) with other clients on your account.

I am very proud to announce, that Smack got support for OpenPGP for XMPP!

Today the Pull Request I worked on during my GSoC project was merged into Smacks master branch. Admittedly it will take a few months until smack-openpgp will be included in a Smack release, but that gives me time to further finalize the code and iron out any bugs that may be in there. If you want to try smack-openpgp for yourself, let me know of any issues you encounter

(Edit: There are snapshot releases of Smack available for testing)

Now Smack does support two end-to-end encryption methods, which complement each other perfectly. OMEMO is best for people that want to be safe from future attacks, while OpenPGP is more suited for users who appreciate being able to access their chat history at any given time. OpenPGP is therefore the better choice for web based applications, although it is perfectly possible to implement web based clients that do OMEMO (see for example the Wire web app, which does ratcheting similar to OMEMO).

What’s left to do now is updating smack-openpgp due to updates made to XEP-0373 and extensive testing against other implementations.

Happy Hacking!

Quick update:

Only a few days are left until the last and final Evaluation Phase.

I spent the week opening my pull request against Smacks master branch and adding a basic trust management implementation. Now the user is required to make decisions whether to trust a contacts key or not. However, the storage implementation is kept very modular, so an implementor can easily create a trust store implementation that realizes custom behaviour.

Smack-openpgp now allows users which did not subscribe to one another to exchange encryption keys quite easily. If a user receives an encrypted message, the implementation automatically fetches the senders keys to allow signature verification.

Furthermore there are more JUnit tests now, so that Smacks total test coverage actually increases when my PR gets merged

Happy Hacking!

As previously mentioned, I have in mind to write a novel about the ancient Greek philosopher Pyrrho and his journey to central Asia and northwestern India with the army of Alexander the Great. As part of my research I'm reading a great deal of philosophy, history, and fiction. It's been fascinating to return to the reading of fiction after a lapse of many years - I feel like I have fresh eyes for everything. My readings in the genre of philosophical fiction have ranged all over the map - during a business trip last week I devoured The Name of the Rose by Umberto Eco, before that I re-read The Left Hand of Darkness by Ursula K. Le Guin, and I'm currently reading both The Picture of Dorian Gray by Oscar Wilde and The Awakening by Kate Chopin. Alongside these works of fiction I'm reading books of philosophical scholarship (the latest were a full-length study of Pyrrho and an exploration of epistemology after Protagoras) and I'm starting to explore the historical background to the massive cultural change from classical to hellenistic Greece. Umberto Eco's book in particular has given me a new ambition of combining the story of Pyrrho's intellectual development with a mystery novel about the still unexplained death of Alexander the Great! It turns out the Pyrrho was probably not quite the radical skeptic he's usually taken to be (although he did believe that the entities we encounter in reality are indefinite and thus hard to pin down in many ways). Combine this with political intrigue on an imperial scale and perhaps regicide (indeed, there's an ancient legend that Aristotle himself was somehow involved in the assassination of his former pupil Alexander, perhaps in revenge for Alexander's execution of Aristotle's nephew Kallisthenes), and we might have the makings of large-scale novel that hinges in part on the validity of the law of non-contradiction! This is all quite speculative at this point because I have over 200 more books to read during the research phase, and I would also need to intensively study how the better authors construct their novels (an art of which I am mostly ignorant). Although I'm not sure if I have a book this big in me, I'm enjoying the exploration of possibilities. And all this reading has kept me from blogging, thus my absence here. Hopefully the results will be worth waiting for......

I’m very happy and proud to announce the first alpha release of PGPainless!

PGPainless 0.0.1-alpha1 is the first non-snapshot release and is available from maven central. It was an interesting experience to go through the process of creating a release and I’m looking forward to have many more releases in the future

The current release contains a workaround for the bug I described in an earlier blog post. The issue was, that bouncycastle wouldn’t mark the public sub keys of a secret key ring as sub keys, which results in loss of keys if the user tries to create a public key ring from the exported public keys. My workaround fixes the issue by iterating through all sub keys of an existing key ring and converting the key packages of subkeys to subkey packages. The code is also available as a gist.

Ironically I had some issues related to OpenPGP during the release process. Releases to maven central have to be signed with an OpenPGP key, so I created a dedicated signing key pair using GnuPG, which I wanted to put into a separate GPG key box. After creating the key, I exported it using

gpg --armor --export-secret-keys [key-id] > pgpainless-singing-key.asc

imported it into a dedicated key box using

gpg --no-default-keyring --keyring pgpainless.gpg --import pgpainless-signing-key.asc

and deleted the key from my old key box, as well as the .asc-file. But when I tried to sign my release, I got the error, that a secret key would be needed. After checking the key box, I noticed, that only a public key was present.

Unfortunately that key had already been published to the key servers and I have no way to revoke it, due to lack of a secret key. I have no idea, what exactly happened or how it could happen, but its too late to recover the key.

Edit: I found my error: Importing a secret key is only possible with the flag `–allow-secret-key-import`, eg

gpg --import --allow-secret-key-import key.asc

So in the end I had to create a new OpenPGP key, which I now carefully backed up on a freshly bought USB stick which will be locked away for the event that I lose the copy on my work station. Better safe than sorry.

Happy Hacking!

After a longer period of development, Babbler 0.8.0 has just been released today.

The new major features are certainly

• support for Java NIO (using the Netty framework), which is a corner stone for future development (like an XMPP server or Serverless Messaging),
• improvements to the JID implementation (e.g. it now uses the upgraded PRECIS framework as per RFC 8264)
• and support for XEP-0390: Entity Capabilities 2.0 and XEP-0392: Consistent Color Generation.

Further information about the release can be found in the release notes and in the changelog.

ProcessOne is excited today to announce the first release of RTB (Real-Time Benchmark), an open source benchmarking tool to stress-test real-time protocols. The aim of this project is to define an easy benchmark for XMPP and MQTT servers, with minimal configuration overhead.

Why

To compare server implementations and be sure they are ready for massive traffic, there is a need for a “golden standard” benchmarking tool that is easy to use and avoids ambiguity due to configuration or overhead – each test should have the same baseline and capabilities for the results to be comparable.

We believe RTB fits such a role, because it has sane defaults (gathered from statistics of real world servers) and is able to cover all the test features defined in the XMPP Compliance Suite 2018.

Status

RTB is in early stages of development: currently, only XMPP protocol is implemented, and support for Multi-User Chat (MUC) and Personal Eventing Protocol (PEP) is lacking.

However, “sane” defaults and what should be considered a “golden standard” for this benchmark is open for discussion with the XMPP community – we welcome issues and pull requests. The tool has already been battle-tested: we used RTB to stress test our ejabberd SaaS deployments with more than 2 000 000 concurrent connections.

RTB includes a script to populate your server’s database with user accounts and rosters, so it’s easy to create test pools. When you start a test with RTB, you can monitor a local website on port 8080 with live graphs like the one below:

Requirements

We want RTB to be straightforward and easy to deploy. Currently, only Unix-based operating systems are supported. To compile, you just need gcc, make, Erlang, expat, yaml, openssl, zlib and gnuplot. There’s a default YAML configuration file to get you started and only 5 mandatory parameters before you can lanuch your first benchmark.

RTB is available on GitHub under Apache License 2.0 and we are waiting for your feedback!