For the past week i have been trying to figure out why Monal stopped working in iOS 13. Building with the iOS13 SDK is the only way to get dark mode (and french app store support) . It appears that when I do build with this SDK it disables the push mechanism I use. the replacement does not work well. I wasn’t planning on needing to get it working until mid 2020. I am still allowed to submit builds with the iOS12 SDK. I may have to sacrifice dark mode and other iOS 13 niceties for now until I have the core messaging at parity. I did manage to get it close and the server infrastructure is set up to handle it so I will keep working on that. I’ll probably keep two different test versions going 4.0 for iOS 12 builds that make it to the app store and 4.1 for ios13 builds that use the new mechanism.

Yesterday, Apple unveiled the new iPhones and set September 19 as the date for the release of iOS 13. This new operating system brings lots of new features and changes, but among them there’s one easy-to-miss update that’s very significant for messaging server operators: an update to APNS requests that enable sending push notifications to iOS clients.

We are happy to announce that today we deploy an update to how our services, like Fluux and software like ejabberd Business Edition handles APNS requests, so all our endpoints are compatible and support iOS 13 APNS from day one.

Technical details

This small but significant update means adding support for the new apns-push-type parameter while sending the request to the APNS. And of course, a respective interface for our customers to be able to set and manipulate the said parameter.

It’s significant because, as the Apple docs state, the new parameter is “Required when delivering notifications to devices running iOS 13 and later, or watchOS 6 and later. (…) The value of this header must accurately reflect the contents of your notification’s payload. If there is a mismatch, or if the header is missing on required systems, APNs may delay the delivery of the notification or drop it altogether.”

As you can see, the implementation has to be carefully executed and tested. Thanks to our expert team and proven processes, we were able to prepare the updates to our services and software quickly and deploy them today, ahead of iOS 13 launch.

Important note: If you are using ejabberd Business Edition, you only need to update if you are already using ejabberd with APNS v3. If you are using the legacy APNS v2 service, you do not have to update at the moment.

Photo by Jamie Street, Unsplash

ProcessOne has made a mark on Erlang history, developing reference software in Erlang, providing strong Erlang expertise and helping grow the its ecosystem. Still, ProcessOne is much more than our Erlang fame. We are technology agnostic. We are great at selecting the right tool for the job to build innovative projects for our customers.

TL;DR

ProcessOne is much more than an Erlang company and we can help you develop your projects in Go, Swift and Kotlin.

Our Erlang history

ProcessOne is an Erlang pioneer. The core team have been developing in Erlang since it was released as Open Source in 1998. We have been building the most famous Erlang project, the real-time messaging platform: ejabberd. Consequently, RabbitMQ followed suite because ejabberd led the way.

We used Erlang very early because it was the only way, at that time, to build highly robust and scalable platforms. ejabberd is able to handle a massive number of connections, supports clustering, get hot code upgrades – all because of the underlying Erlang VM.

We have been jumping on the Elixir bandwagon very soon for quite the same reasons. It is a technology that fills a niche, making it possible to build large scale real-time web platforms with a syntax that feels more familiar (especially for Ruby developers).

Innovation is our driver

When we started ProcessOne, there was no Amazon Web Service. Servers were multiple times slower than they are today. There was no Docker, no Kubernetes. Basically, there was no alternative to Erlang VM if you wanted to build a scalable and manageable service. And most of all, there was no iPhone and Android, whose support is now a critical component of many projects.

So, over the years, we had to refine what innovating for our customers meant, integrating new technologies in our skill set and our stack.

In the context of our customers’ projects, Erlang or even Elixir is not always the best answer. It is hard to find Erlang or Elixir skills and training an Erlang or an Elixir team is most of the time not desirable for customers. They often need to hire and train large set of developers and sysadmins. They need to have a path for innovation to fit their corporate culture. And finally, most of all, Erlang and Elixir are not enough to cover the client-side development, especially on mobile devices.

That’s why we had to refine our technology stack with innovation as our main focus.

Expanding our technology stack

Over the years, we had to expand our technology stack to have a good balance between innovation and ability for our customers to adopt it. We ended up developing the following additional skills to handle our customer projects:

• Go to handle server-side services and web applications. Go is very popular and provides good scalability and maintainability. Even if Go seems approachable and easy, writing very good Go code that feels like “native Go” and is maintainable, is hard. However, it is accessible for our customers and we have found it was quite easy to transfer the knowledge to customers’ teams. Once we have laid the basis and delivered a strong Go applications, it is easy for our customers to take over the projects. It works well with our Erlang components, as our customers can use them as black boxes.
• Swift to handle iOS native developments. This is the de facto standard in the Apple ecosystem. Given our high-profile customers target, cross-platform development is generally not enough. That’s why we are investing in providing state-of-the-art native code, following the latest standards on iOS/MacOS/Watch/TvOS.
• Kotlin to handle native Android developments. We target high-end projects that needs native developments with the latest features. Kotlin is the way to deliver future-proof code on Android.

We are still working on Erlang projects and you can expect the same involvement in ejabberd. Still, you can also expect us to talk more about our other skills and see new projects in those languages. Our Open Source software stack will get richer and serve as basic building bricks for our customers’ projects.

Summary

Given our unique Erlang history, we are often still seen as an Erlang company. While we have a unique expertise and a part of our customer-base work with us because of that Erlang or Elixir expertise, limiting ProcessOne to this small set of technologies could not be further from the truth.

We deliver full projects for customers that need to deliver innovative services, in record time. Finally, we hand over that innovation so that it can live for many years after we delivered our result.

Working with ProcessOne is like having your own team of R&D experts, that can bootstrap your project to help your reach the market before the competition, without compromise on the technology.

Do not hesitate to contact us if you need help building your projects, with the confidence that you will be able to integrate the innovation internally.

IOS 13 comes out tomorrow and Monal 4 will be out in the next day or so. I was hoping to be ready day 1 this time and I was for dark mode but the push issues took me by surprise. I expected to have until 2020 to resolve it. Monal as it is in the app store, Monal won’t work with iOS 13, the OS will block the voip pushes. The updated version will work with a new push server that does not send voip pushes anymore. I have posted that to the beta channel today. It just tells you there is a new message and doesn’t have the message text or sender. It’s not great but the alternative is not knowing when you have messages. I will continue to work on the notification extension I am using for these messages and will update it in the coming week hopefully we will get similar behavior to iOS 12 soon.

Have I mentioned how much I hate Facebook messing this up for everyone.

@wroot wrote:

The Ignite Realtime community is happy to announce the release of version 4.4.4 of the Fastpath Servive plugin for Openfire!

This update fixes exception when pressing on workgroup name, makes it possible to build plugin with Maven and adds an ad-hoc command to update workgroup.

Your instance of Openfire should automatically display the availability of the update. Alternatively, you can download the new release of the plugin at the Fastpath Service plugin’s archive page

For other release announcements and news follow us on Twitter

Posts: 1

Participants: 1

Read full topic

We are pleased to announce ejabberd version 19.08. The main focus has been to further improve ease of use, consistency, performance, but also to start cleaning up our code base. As usual, we have kept on improving server performance and fixed several issues.

New Features and improvements

New authentication method using JWT tokens

You can now authenticate with JSON Web Token (JWT, see jwt.io for details).

This feature allows you to have your backend generate authentication tokens with a limited lifetime.

Generating JSON Web Key

You need to generate a secret key to be able to sign your JWT tokens.

To generate a JSON Web Key, you can for example use JSON Web Key Generator, or use your own local tool for production deployment.

The result looks like this:

{
  "kty": "oct",
  "use": "sig",
  "k": "PIozBFSFEntS_NIt...jXyok24AAJS8RksQ",
  "alg": "HS256"
}

Save you JSON Web Key in a file, e.g. secret.jwk.

Be careful: This key must never be shared or committed anywhere. With that key, you can generate credentials for any users on your server.

Configure ejabberd to use JWT auth

In ejabberd.yml change auth_method to jwt and add the jwt_key option pointing to secret.jwk:

auth_method: jwt
jwt_key: "/path/to/jwt/key"

Generate some JWT tokens

See jwt.io for an example of how to generate JSON Web Tokens. The payload must look like this:

{
  "jid": "test@example.org",
  "exp": 1564436511
}

And the encoded token looks like this:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqaWQiOiJ0ZXN0QGV4YW1wbGUub3JnIiwiZXhwIjoxNTY0NDM2NTExfQ.SMfzCVy8Nv5jJM0tMg4XwymIf7pCAzY8qisOSQ5IAqI

Authenticate on XMPP using encoded token as password

Now, the user test@example.org can use this token as a password before 1564436511 epoch time (i.e. July 29, 2019 21:41:51 GMT).

New configuration validator

With ejabberd 19.08, we introduce a new configuration checker, giving more precise configuration guidance in case of syntax errors or misconfiguration. This configuration checker has also been released as an independent open source project: yconf.

The new configuration validator makes it possible to improve the configuration parsing. For example, it supports the following:

Better handling of Erlang atom vs string

There is not need to quote string to express the fact you want an atom in the configuration file: the new configuration validator handles the Erlang types mapping automatically.

More flexible ways to express timeouts

Now, all timeout values can be expanded with suffixes, e.g.

negotiation_timeout: 30s
s2s_timeout: 10 minutes
cache_life_time: 1 hour

If the suffix is not given, the timeout is assumed in seconds

Atomic configuration reload

The configuration will either be fully reloaded or rolled back.

Better, more precise error reporting

Here are a couple of examples of the kind of message that the new configuration validator can produce.

In the following example, the validator will check against a value range:

14:15:48:32.582 [critical] Failed to start ejabberd application: Invalid value of option loglevel: expected integer from 0 to 5, got: 6

More generally, it can check value against expected types:

15:51:34.007 [critical] Failed to start ejabberd application: Invalid value of option modules->mod_roster->versioning: expected boolean, got string instead

It will report invalid values and suggest fixes in case error was possibly due to a typo:

15:50:06.800 [critical] Failed to start ejabberd application: Invalid value of option modules->mod_pubsub->plugins: unexpected value: pepp. Did you mean pep? Possible values are: flat, pep

Prevent use of duplicate options

Finally, it will also properly fail on duplicate options and properly report the error:

15:56:35.227 [critical] Failed to start ejabberd application: Configuration error: duplicated option: s2s_use_starttls

It was a source of error as an option could shadow another one, possibly in an included file.

Improved scalability

We improve scalability of several modules:

Multi-User chat

MUC Room modules is more scalable, allowing supporting more rooms, by hibernating the room after a timeout. Hibernating means removing it from memory when not used and reloading it on-demand.

The MUC messages processing has also been changed to now properly handle all the available CPU cores. MUC room message handling is now faster and support larger throughput on SMP architectures

SQL database handling

We improve the way the SQL pool is managed to better handle high load. We also improved MySQL schema a bit to help with indexing.

Changed implementation of mod_offline option use_mam_for_storage

Previous version was trying to determine the range of messages that should be fetched from MAM by storing time when last user resource disconnected. But that had couple edge cases that could cause problems, for example in case of hardware node crash we could not store information about user disconnect and with that we didn’t have data to initiate MAM query.

The new version doesn’t track user disconnects, but simply ensure that we have timestamp of first message that is gonna be put in storage, after some measurements cost of that check with caching on top is not that costly, and as it is much more robust we decided to move to that safer approach.

New option captcha_url

Option captcha_host is now deprecated in favor of captcha_url. However, it’s not replaced automatically at startup, i.e. both options are supported with ‘captcha_url’ being the preferred one.

Deprecated ‘route_subdomains’ option

This option was introduced to fulfil requirements of RFC3920 10.3, but in practice it was very inconvenient and many admins were forced to change its value to ‘s2s’ (i.e. to behaviour that violates the RFC). Also, it seems like in RFC6120 this requirement no longer presents.

Those admins who used this option to block s2s with their subdomains can use ‘s2s_access’ option for the same purpose.

API changes

Renamed arguments from ‘Server’ to ‘Host’

Several ejabberd commands still used as argument name ‘Server’, instead of the more common ‘Host’. Such arguments have been renamed, and backward support allows old calls to work correctly.

The eight affected commands are:
– add_rosteritem
– bookmarks_to_pep
– delete_rosteritem
– get_offline_count
– get_presence
– get_roster
– remove_mam_for_user
– remove_mam_for_user_with_peer

If you are using these calls, please start updating your parameter names to Host when moving to ejabberd 19.08. You will thus use a more consistent API and be future proof.

Technical changes

Removed Riak support

Reasons:

• Riak DB development is almost halted after Basho
• riak-erlang-client is abandoned and doesn’t work correctly with OTP22
• Riak is slow in comparison to other databases
• Missing key ordering makes it impossible to implement range queries efficiently (e.g. MAM queries)

If you are using Riak, you can contact ProcessOne to get assistance migrating to DynamoDB, an horizontally scalable key value datastore made by Amazon.

Erlang/OTP requirement

Erlang/OTP 19.1 is still the minimum supported Erlang version for this release.

Database schema changes

There is no change to perform on the database to move from ejabberd 19.05 to ejabberd 19.08.
Please, make a backup before upgrading.

It means that an old schema for ejabberd 19.05 will work on ejabberd 19.08. However, if you are using MySQL, you should not that we changed the type of the server_host field to perform better with indexes. The change is not mandatory, but changing it to varchar(191) will produce more efficient indexes.

You can check the upgrade page for details: Upgrading from ejabberd 19.05 to 19.08

Download and install ejabberd 19.08

The source package and binary installers are available at ProcessOne. If you installed a previous version, please read ejabberd upgrade notes.
As usual, the release is tagged in the Git source code repository on Github. If you suspect that you’ve found a bug, please search or fill a bug report in Issues.

SwiftUI is a new application framework from Apple that complements UIKit and AppKit. It is expected to be the future of UI definition on Apple platform, unifying GUI application writing using the same framework on all Apple platforms, from Apple Watch to iOS and MacOS.

A change in architectural patterns: From MVC to MVVM

SwiftUI marks a shift in application architectural pattern. UIKit and AppKit applications are structured using the Model-View-Controller (MVC) paradigm. The Controller is the central abstraction layer used to link the Model, containing the business logic, to the View. This extra layer exists to make the code more maintainable and reusable.

The MVC pattern has been dominant in User Interface frameworks since it was first introduced in Smalltalk in the 80s.

However, it tends to lead to excessive boilerplate code. Moreover, as a programmer you have to fight the tendency to put large part of the application logic into the controller: The controller can become bloated and complex over time in large applications.

With SwiftUI, Apple is moving application development to the Model-View-ViewModel architectural pattern. This pattern was invented at Microsoft, as a variation of the Presentation model by Martin Fowler.
This pattern simplifies the design of event-based applications by offering a clearer dependency models between components and a clearer data flow. In MVVM, the view has a reference to the ViewModel and the ViewModel has a reference to the Model, and that’s it.

The MVVM pattern is used in many popular Javascript frameworks these days (ReactJS, Vue.js, Ember.js, etc.) and is becoming more and more widely used to design user interfaces.

The reference flow is simple but at some point, you will also want to let the Model update the View to reflect state changes.

You can still use the proven delegation approach to let the model update the view, but in the case, you are adding circular references and you are missing a big part of what makes SwiftUI great.

The concept of Binding is used to finally allow the Model to update the View. In SwiftUI, you can pass bindings to link view states to the underlying model. As defined in Apple documentation, the Binding is a “manager for a value that provides a way to mutate it”. It defineds a two way-connection propagating user changes on the application states in the view, but also updating the view when the model changes.

If you want to get further, you can design part of your application logic using Combine. Combine is a reactive framework to help build more complex applications using SwiftUI, taking care of event propagation and asynchronous programming, using a publish and subscribe model. Combine can be used for example to add a networking layer to your application.

A change in View design approach

SwiftUI also change radically how Views are designed and maintained. In the past, Views on iOS have often been designed graphically, using Interface Builder or Storyboard editor. Defining the Views in code was possible, but was a much larger effort. The interface was thus often designed graphically, adding constraints to define how the interface was supposed to change when displayed at different screen sizes. This resulted in interfaces being described in XML files, making teamwork difficult, as merge conflicts were painful to solve.

With SwiftUI, Apple introduced a declarative approach to design UI. The UI is always designed as code, but Apple also provides a canvas, to show a live preview of your view and guide the developer when developing the application. You can also interact directly on the canvas, and the code is updated accordingly.

At this time, the canvas is sometimes a bit unstable, but when it works, the overall process feels quite magical. No more fiddling with lots of panels defining values and constraints. SwiftUI simplifies the process by providing sane defaults for most system-provided views, but also remove the need for constraints by providing containers for your view that define the logical relation between the subviews. For example, a VStack will define a group of view that need to be rendered as a vertical stack on after the other. HStack define views that are laid out horizontally. ZStack defines set of view that are rendered one into the other (merged), etc. By combining high-level containers, you can describe the relationship between the views in your applications and let the system process by itself how to render the view.

Finally, I said previously that SwiftUI complements UIKit and AppKit. This is because SwiftUI works well with both UIKit and AppKit. Apple has defined tools and patterns so that both framework can work together. It means you can integrate with high-level components that are not directly available in SwiftUI, like MapKit or WebKit. It also means you can convert your application incrementally, adding SwiftUI views to an existing UIKit for example.

Get prepared for Apple platform future

For now SwiftUI is still in version 1. It already feels like the future of Apple application design, but you have to be prepared to face little bugs from time to time in Xcode or on-device rendering.

The fact that SwiftUI can only target iOS 13 / MacOS 10.15 / WatchOS 6 / TvOS 13 devices will also limit adoption, as developers often have to target applications running on several different OS versions.

However, it is consistent with the fact that you can expect SwiftUI to improve and mature considerably in the coming months.

With SwiftUI, Apple is attempting to solve several issues:

• They want to unify the UI framework on all their platforms.
• They are simplifying the programming model, offering a path to migrate from MVC to MVVM pattern.
• As a side effect, they are reworking their concurrency pattern, introducing Combine as the underlying framework for implementing reactive programming in Swift-based applications.

If you are developing a brand new application today that is going to be released in a few months from now, it makes sense to consider SwiftUI.

At the very least, you should start learning SwiftUI and Combine now to be ready to adopt it when you feel it is possible for your kind of app.

A word of warning. I started to see this in my logs today. Im am working on a fix for this but I can’t guarantee it will be ready by Sept 19. Pushes will not work with iOS 13 at the moment.

Converse is a JavaScript XMPP client that can be run in a web browser. Unfortunately, it does not support the XMPP extension (XEP) that allows to verify HTTP Requests via XMPP (XMPP): XEP-0070. So I decided to code a small plugin to provide this functionality.

This article follows the serie about this XEP:

• One about Passordless Authentication using XMPP on a Django website.
• A presentation about a demo website providing this functionnality.

How to use it

See the official documentation on how to install this plugin.

If you want to hide the authentication request of your provider, you can set the following option in your converse.initialize function:

hidden: ['auth.example.com']

It will also work with any jid.

You can test the implementation with my demo website: https://demo.agayon.be/

Links

• Code on Gitlab
• XEP-0070
• Django authentification
• Demo
• Authentication component by Chteufleur
• Description of the mechanism (french)

Almost all messaging systems  become useless when you lose access to the internet. What happens when the internet is cut off by a disaster or government?  What if the app is banned  in an app store?  This is something I have been thinking about lately.  Both because of the frequency of natural disasters and also because internet cutoff is a tactic increasingly used by authoritarian governments. The result of this is xmpp+OMEMO (i.e. signal) over airdrop. Since this circumvents the internet and never tries to connect to anything,  it works around national firewalls and internet blocks.  There are multiple layers of encryption here, that offered by Apple with Airdrop as well as application level encryption with OMEMO .

To do this,  I hijacked airdrop’s file transfer and automated the process.  However, it works remarkably well. There have been other attempts to do things like this but these are  usually not open source, not a documented protocol like signal and do not work when the app is closed. Additionally,  there is often a dependence on SMS/phone numbers which is an inherently insecure system.  Using airdrop allows for notifications to show up when Monal is closed without going over Apns (Apple’s push system).  There is no connection to Apple or my own push server. 

For now you need to establish your initial key trust with an xmpp server however after that you can go offline and work via airdrop (I recommend setting airdrop  to contacts only to prevent spam). I hope to have qr code based key exchange some point in the future. So the circle of trust can be established quickly in person. You can use an existing server  using the in app registration flow or just run a simple xmpp server like prosody on a Linux vm or raspberry pi.

This is in the latest beta.  To use airdrop mode go to your account and  toggle use air drop. This will take your account offline.  When you send a message, the airdrop share sheet will appear and you will be asked to select the recipient.  

This is text messages only for now. Airdrop mode will create an XMPP payload with a message encrypted with the signal protocol and send that to its recipient. I suggest you set your airdrop receiving settings to contacts only so only those who are your contacts in your iOS contacts app can send you files (in this case messages).

The primary flaw for airdrop that I have seen is your screen has to be on. I am not sure how to get around that right now. Because this is peer to peer, obviously you will also need to be nearby to message as well. This is really intended for small groups in close proximity that need to communicate off the grid.

Try the Monal iOS beta and let me know how it works.

Since Monal is open source, if Monal is blocked for some reason anyone can grab the source on GitHub and build it for themselves and friends. Any version of Monal whether from app store or self compiled will be compatible for messaging.

Services like Twitter and Slack have functionality that attempts to interpret parts of the plain text of tweets or message as entered by the user. Pieces of the text that look like links, mentions of another user, hash tags, or stock symbols, cause additional meta data to be added to the object representing the message, so that receiving clients can mark up those pieces of text in a special way. Twitter calls this meta data Tweet Entities and for each piece of interpreted text, it includes indices for the start and end of along with additional information depending on the type of entity. A client can then do in-line replacements at the exact character indices, e.g. by making it into a hyperlink. Twitter Entities served as inspiration for XEP-0372: References.

References can be used in two ways: including a reference as a sibling to the body element of a message. The begin and end attributes then point to the indices of the plain text in the body. This would typically be used if the interpretation of the message is done by the sending client.

Alternatively, a service (e.g. a MUC service) could parse incoming messages and send a separate stanza to mark up the original stanza. In this case you need a mechanism for pointing to that other message. There have been two proposals for this, with slightly differing approaches, and in the examples below, I'll use the proto-XEP Message Fastening. While pointing to the stanza ID of the other message, it embeds a reference element in the apply-to element.

Mentioning another user

Let's start out with the example of mentioning another user.

<message from="room@muc.this.example/Kev" type="groupchat">
  <stanza-id id="2019-09-02-1" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>Some rubbish @ralphm</body>
</message>

A client might render this as:

The MUC service then parses the plain-text message, and finds a reference to my nickname prefixed with an @-sign, and sends a stanza to the room that marks up the message Kev sent to me.

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-2" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-1">
    <reference begin="13" end="19" xmlns="urn:example:reference:0">
      <mention jid="room@muc.this.example/ralphm"/>
    </reference>
  </apply-to>
</message>

This stanza declares that it is attached to the previous message by the stanza ID that was included with the original stanza. In its payload, it includes a reference, referring to the characters 13 through 19. It has a mention child pointing to my occupant JID. Alternatively, the room might have linked to my real JID. A client can then alter the presentation of the original message to use the attached mention reference:

The characters referencing @ralphm are now highlighted, hovering the mention shows a tooltip with my full name, and clicking on it brings you to a page describing me. This information was not present in the stanza, but a client can use the XMPP URI as a key to present additional information. E.g. from the user's contact list, by doing a vCard lookup, etc.

Referring to previous messages

<message from="room@muc.this.example/Ge0rG"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-3" by="room@muc.this.example"/>
  <reference begin="0" end="6" xmlns="urn:example:reference:0">
    <mention jid="room@muc.this.example/ralphm"/>
  </reference>
  <reference begin="26" end="32" xmlns="urn:example:reference:0">
    <message id="2019-09-02-1"/>
  </reference>
  <body>@ralphm did you see Kev's message earlier?</body>
</message>

Unlike before, this example does not point to another stanza with apply-to. Instead, Ge0rG's client added references to go along with the plain-text body: one for the mention of me, and one for a reference to an earlier message.

Emoji Reactions

Instead of reacting with a full message, Slack, like online forum software much earlier, has the ability to attach emoji reactions to messages.

<message from="room@muc.this.example/Kev"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
            id="2019-09-02-4" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-3">
    <reactions xmlns="urn:example:reactions:0">
      <reaction label=":+1:">👍</reaction>
    </reactions>
  </apply-to>
</message>

<message from="room@muc.this.example/ralphm"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-6" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-3">
    <reactions xmlns="urn:example:reactions:0">
      <reaction label=":parrot:"
                img="cid:b729aec3f521694a35c3fc94d7477b32bc6444ca@bob.xmpp.org"/>
    </reactions>
  </apply-to>
</message>

These two examples show two separate instances of a person reacting to the previous message by Ge0rG. It uses the protocol from Message Reactions, another Proto-XEP. However, I expanded on it by introducing two new attributes. The label allows for a textual shorthand, that might be typed by a user. Custom emoji can be represented with the img attribute, that points to a XEP-0231: Bits of Binary object.

Ge0rG 13:16

@ralphm did you see Kev's message earlier?

👍 2  1

The attached emoji are rendered below the original message, and hovering over them reveals who were the respondents. Here my own reaction is highlighted by a squircle border.

Including a link

<message from="room@muc.this.example/ralphm" type="groupchat">
  <stanza-id id="2019-09-02-7" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>Have you seen https://ralphm.net/blog/2013/10/10/logitech_t630?</body>
</message>

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-8" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-7">
    <reference begin="14" end="61" xmlns="urn:example:reference:0">
      <link url="https://ralphm.net/blog/2013/10/10/logitech_t630"/>
    </reference>
  </apply-to>
</message>

Here the MUC service marks up the original messages with an explicit link reference. Possibly, the protocol might be extended so that a service can include shortened versions of the URL for display purposes.

ralphm 13:16

Have you seen https://ralphm.net/blog/2013/10/10/logitech_t630?

Logitech Ultrathin Touch Mouse

Logitech input devices are my favorite. This tiny bluetooth mouse is a nice portable device for every day use or while traveling.

The client has used the markup to fetch meta data on the URL and presents a summary card below the original message. Alternatively, the MUC service could have done this using XEP-0385: Stateless Inline Media Sharing (SIMS):

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-8" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-7">
    <reference begin="14" end="61" xmlns="urn:example:reference:0">
      <link url="https://ralphm.net/blog/2013/10/10/logitech_t630"/>
      <card xmlns="urn:example:card:0">
        <title>Logitech Ultrathin Touch Mouse</ulink></title>
        <description>Logitech input devices are my favorite. This tiny bluetooth mouse is a nice portable device for every day use or while traveling.</description>
      </card>
      <media-sharing xmlns='urn:xmpp:sims:1'>
        <file xmlns='urn:xmpp:jingle:apps:file-transfer:5'>
          <media-type>image/jpeg</media-type>
          <name>ultrathin-touch-mouse-t630.jpg</name>
          <size>23458</size>
          <hash xmlns='urn:xmpp:hashes:2' algo='sha3-256'>5TOeoNI9z6rN5f+cQagnCgxitQE0VUgzCMeQ9JqbhWJT/FzPpDTTFCbbo1jWwOsIoo9u0hQk6CPxH4t/dvTN0Q==</hash>
          <thumbnail xmlns='urn:xmpp:thumbs:1'uri='cid:sha1+21ed723481c24efed81f256c8ed11854a8d47eff@bob.xmpp.org' media-type='image/jpeg' width='116' height='128'/>
        </file>
        <sources>
          <reference xmlns='urn:xmpp:reference:0' type='data' uri='https://test.ralphm.net/images/blog/ultrathin-touch-mouse-t630.jpg' />
        </sources>
      </media-sharing>
    </reference>
  </apply-to>
</message>

Editing a previous message

<message from="room@muc.this.example/ralphm" type="groupchat">
  <stanza-id id="2019-09-02-9" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>Some thoughtful reply</body>
</message>

After sending that message, I want to add a bit more information:

<message from="room@muc.this.example/ralphm" type="groupchat">
  <stanza-id id="2019-09-02-10" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-9">
    <external name='body'/>
    <replace xmlns='urn:example:message-correct:1'/>
  </apply-to>
  <body>Some more thoughtful reply</body>
</message>

Unlike XEP-0308: Last Message Correction, this example uses Fastening to refer to the original message. I would also lift the restriction on correcting just the last message, but allow any previous message to be edited.

Upon receiving the correction, the client indicates that the message has been edited. Hovering over the marker reveals when the message was changed.

Editing a previous message that had fastened references

<message from="room@muc.this.example/Kev" type="groupchat">
  <stanza-id id="2019-09-02-11" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>A witty response mentioning @ralphm</body>
</message>

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-12" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-11">
    <reference begin="28" end="34" xmlns="urn:example:reference:0">
      <mention jid="room@muc.this.example/ralphm"/>
    </reference>
  </apply-to>
</message>

After a bit of consideration, Kev edits his response:

<message from="room@muc.this.example/Kev" type="groupchat">
  <stanza-id id="2019-09-02-13" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-11">
    <external name='body'/>
    <replace xmlns='urn:example:message-correct:1'/>
  </apply-to>
  <body>A slighty wittier response mentioning @ralphm</body>
</message>

Upon receiving the correction, the client discards all fastened references. The body text was changed, so the reference indices are stale. The room can then send a new stanza marking up the new text:

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-14" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-11">
    <reference begin="40" end="46" xmlns="urn:example:reference:0">
      <mention jid="room@muc.this.example/ralphm"/>
    </reference>
  </apply-to>
</message>

Closing notes

• Fastening should also gain a way to unfasten explicitly. I think that should use the stanza ID of the stanza that included the earlier fastening. This allows for undoing individual emoji reactions.

• Unfastening should probably not use the proto-XEP on Message Retraction. That is for retracting the entire original message plus all its fastened items, and invalidating all message references pointing to it.

• It might make sense to have a separate document describing how to handle stanza IDs, so that all specifications could point to it instead of each having their own algorithm. In different contexts, different IDs might be used. The other proposal for attachments, XEP-0367: Message Attaching, has a section (4.1) on this that might be taken as a start.

• In the discussion leading up to this post, a large part was about how to handle all these things attached/fastened to messages in message archives. This is not trivial, as you likely don't want to store a sequence of stanzas, but of (original) messages. Each of those message then might have one or more things fastened to it, and upon retrieval, you want these to come along when retrieving a message. Some of these might be collated, like edits. Some might cause summary counts (emoji, simple polls) with the message itself, and require an explicit retrieval of all the reactions, e.g. when hovering the reaction counts.

  Details on message archive handling is food for a later post. I do think that having a single way of attaching/fastening things to messages makes it much easier to come up with a good solution for archive handling.

• I didn't provide examples for stanza encryption, but discussions on this suggested that stanzas with fastened items would have an empty apply-to, including the id attribute, so that message archives can do rudimentary grouping of fastened items with the original message.

• I didn't include examples on Chat Markers, as its current semantics are that a marker sent by a recipient applies to a message and all prior messages. This means the marker isn't really tied to a single message. I think this doesn't match the model for Message Fastening.

There are new betas for mac and iOS that address many OMEMO issues people have seen. The feedback has been positive and will try to get both out to production soon.

I am also trying to figure out why comments are turned off on this blog now. It wasn’t intentional. WordPress isn’t cooperating with me.

Welcome to the September edition of the XMPP Newsletter!

Loads of news for these two months of July and August: two events in Lyon and Stockholm (more to come), new tools, new server releases (Openfire, ejabberd), lots of client releases (Kaidan, Salut à Toi, Xabber Android, Movim, Converse, Beagle IM and Siskin IM). We also experiment a new section about XEP, or specifications.

Please submit your XMPP/Jabber articles, tutorials or blog posts on our wiki.

Articles

Georg Lukas posted an article about the 10 years of yaxim!

Marek Foss of ProcessOne, details the story on how they have deployed an XMPP geocluster for the FIFA World Cup in Brazil.

Cyril Brulebois wrote a short piece on Sending HTML messages with Net::XMPP (Perl).

Events

Maxime “pep.” Buquet reports on the XMPP sprint in Lyon in July held at Wisolv's offices: DOAP - Description Of A Project, Reactions, Occupant-id, and more.

The next XMPP sprint in Stockholm will be held on September 28th and 29th, in the Nacka Municipality.

A few XMPP hackers met in Ziegeleipark, Mildenberg, Germany in August, for the Chaos Communication Camp 2019, an open-air hacker camp and party that takes place every four years, organized by the Chaos Computer Club (CCC).

The regular meetups in Bavaria is held every third Monday every month. Next will take place 2019-09-16.

If you organise an event in your city, please submit your meetups on the wiki.

Software releases

Servers

Prometheus XMPP Blackbox Exporter (Apache license 2.0) allows you to probe XMPP services and export metrics from the probes to Prometheus.

The igniterealtime community has announced multiple releases:

• Openfire 4.4.1
• Monitoring Service 1.8.0 and inVerse 4.2.0.1 plugins
• Thread Dump 1.0.0 plugin

Antonino Siena has announced xmpp-http-upload, a lightweight and efficient HTTP upload server for XMPP (MIT license).

Mickaël Rémond has announced ejabberd 19.08, with JSON Web Token, configuration validator, improved scalability, and more.

Jérôme "Goffi" Poisson has announced SàT PubSub 0.3.0, a server independent PEP/PubSub XMPP service, which aims to be complete and universal.

Clients

Kaidan 0.4.0 and 0.4.1 have been released, and is available to download on Linux, Windows, and macOS (and experimental Android and Ubuntu Touch).

Jérôme "Goffi" Poisson has released Salut à Toi v0.7 « La Commune » with tons of changes (and writes regular SàT progress notes).

Xabber Android 2.6.4 (634) has been released on Google Play, with improved message archive synchronization, improved startup speed, support of references in messages (files, forwards, markup (bold, italic, etc), mentions, quotes), image compression setting, visual changes, and more.

Timothée Jaussoin has released Movim 0.15 – Donati, with reactions, publication sharing, and more.

JC Brand has released Converse, the web based XMPP/Jabber chat client, in versions 5.0.0 and 5.0.1, with lots of improvements.

Wojciech Kapcia, of Tigase, has announced two versions of Beagle IM from macOS and Siskin IM for iOS.

Libraries

Lance "legastero" Stout has released StanzaJS (formely known as Stanza.io), the JS XMPP library with a JSON API, in multiple versions 12.x.

JC Brand has released the strophejs library in version 1.3.4.

Florian "Flow" Schmaus has announced the extension of the jXMPP library by a testframework for “XMPP-Strings”, as well as Smack 4.4.0-alpha2.

Services

Sadly, Arc Games has announced an XMPP sunset: "As of September 19th, 2019, Star Trek Online will no longer be supporting connection to our internal chat system via an XMPP client."

Extensions and specifications

This section of the newsletter is new and experimental: please let us know what you think.

We start with the XEPs in "LAST CALL", since these are the highest priority in terms of feedback. We move on to the new XEPs so you can learn of their existence, and then the obsoleted ones so you phase them out. We finally finish with the updates. On more section: the friendly specifications.

Last Call

XEP-0353: Jingle Message Initiation

Title: Jingle Message Initiation

Abstract: This specification provides a way for the initiator of a Jingle session to propose sending an invitation in an XMPP message stanza, thus taking advantage of message delivery semantics instead of sending IQ stanzas to all of the responder's online resources or choosing a particular online resource.

URL: https://xmpp.org/extensions/xep-0353.html

This Last Call begins today and shall end at the close of business on 2019-08-13.

XEP-0300: Use of Cryptographic Hash Functions in XMPP

Title: Use of Cryptographic Hash Functions in XMPP

Abstract: This document provides a common wire format for the transport of cryptographic hash function references and hash function values in XMPP protocol extensions.

URL: https://xmpp.org/extensions/xep-0300.html

This Last Call begins today and shall end at the close of business on 2019-08-13.

New

XEP-0421: Anonymous unique occupant identifiers for MUCs

Version 0.1.0 of XEP-0421 (Anonymous unique occupant identifiers for MUCs) has been released.

Abstract: This specification defines a method that allows clients to identify a MUC participant across reconnects and renames. It thus prevents impersonification of anonymous users.

Changelog: Accepted by vote of Council on 2019-07-17. (XEP Editor (jsc))

URL: https://xmpp.org/extensions/xep-0421.html

XEP-0420: Stanza Content Encryption

Version 0.1.0 of XEP-0420 (Stanza Content Encryption) has been released.

Abstract: The Stanza Content Encryption (SCE) protocol is intended as a way to allow clients to securely exchange arbitrary extension elements using different end-to-end encryption schemes.

Changelog: Accepted by vote of Council on 2019-06-26. (XEP Editor (jsc))

URL: https://xmpp.org/extensions/xep-0420.html

Obsoleted

XEP-0387: XMPP Compliance Suites 2018

Version 1.0.0 of XEP-0387 (XMPP Compliance Suites 2018) has been released.

Abstract: This document defines XMPP protocol compliance levels.

Changelog: Move to Draft as per Council vote on 2018-01-24. (XEP Editor (jwi))

URL: https://xmpp.org/extensions/xep-0387.html

Updated

• Version 0.3.0 of XEP-0414 (Cryptographic Hash Function Recommendations for XMPP)
• Version 1.1.0 of XEP-0368 (SRV records for XMPP over TLS)
• Version 0.3.0 of XEP-0380 (Explicit Message Encryption)
• Version 1.4.0 of XEP-0184 (Message Delivery Receipts)

Friendly specs

This one may be of interest to this community; it's in Last Call at the IETF currently: https://datatracker.ietf.org/doc/draft-ietf-tls-sni-encryption/

This is it

This edition was long, we hope you loved it.

You can also show your support by relaying it online, on your blogs, newsletters, or social media.

First of all, where is the stuff?

Really briefly, what was this all about again?

Cool, so what has been done?

What didn't made it through?

And what can be improved?

Conclusion

Aknowledgements

ProcessOne curates the Real-time Radar – a newsletter focusing on articles about technology and business aspects of real-time solutions. Here are the articles we found interesting in Issue #25. To receive this newsletter straight to your inbox on the day it is published, subscribe here.

Breaking 25 Trillion Notifications for the BBC

In 2012, BBC needed to add a push notification service to its portfolio of mobile applications, with main focus on BBC News and BBC Sports. At the scale of the BBC, there was several key requirements that made it difficult to use an off-the-shelf application.

Thoughts on Code Style in Library Design

XMPP protocol is flexible and contains many extensions. It can be daunting to get into XMPP so it’s very tempting to add new tools to make it more convenient for newcomers to get into the protocol.

Next Steps Toward Go 2

The release of Go 1.13 is on its way, hopefully in early August of this year. This is the first release that will include concrete changes to the language (rather than just minor adjustments to the spec), after a longer moratorium on any such changes.

Scaling from 2,000 to 25,000 Engineers on GitHub at Microsoft

At Microsoft today there are almost 25,000 engineers participating in the official GitHub organizations for open source, a great number of them contributing to open source communities throughout GitHub.

Air Pollution Tracking with Node.js, Elastic Stack, and MQTT

What can you do with a couple of IoT devices, Node.js, Elasticsearch, and MQTT? You can put together your own Internet of Things setup for measuring air pollution, like I have.

The State of Developer Ecosystem in 2019

Go started out with a share of 8% in 2017 and now it has reached 18%. In addition, the biggest number of developers (13%) chose Go as a language they would like to adopt or migrate to.

Merging Internet of Things and Blockchain in Preparation for the Future

The ‘Big Four’ technologies of the future are beginning to force themselves to the fore as companies start to dabble in the likes of Artificial Intelligence (AI), Blockchain and Internet of Things (IoT), as well as Big Data.

@wroot wrote:

The Ignite Realtime community is happy to announce the release of version 5.0.1.1 of the inVerse plugin for Openfire!

This update brings Converse up to 5.0.1 version, which had a lot of changes and fixes.

Your instance of Openfire should automatically display the availability of the update. Alternatively, you can download the new release of the plugin at the inVerse plugin’s archive page

For other release announcements and news follow us on Twitter

Posts: 1

Participants: 1

Read full topic

ProcessOne curates the Real-time Radar – a newsletter focusing on articles about technology and business aspects of real-time solutions. Here are the articles we found interesting in Issue #24. To receive this newsletter straight to your inbox on the day it is published, subscribe here.

ejabberd & Nintendo Switch NPNS

Last week, Taiyo Watanabe from Nintendo shared some insights about Nintendo Switch Push Notification infrastructure (NPNS) implementation. Here are some details from his presentation at ElixirFest conference. The Nintendo Switch NPNS is a large-scale realtime messaging service based on ejabberd.

80 Mind-Blowing IoT Statistics

In 2016, $737 billion was spent on the IoT. Global spending associated with the IoT in 2016 amounted to $737 billion. According to ACT, an organization that represents more than 5,000 tech companies, the healthcare IoT market alone will be worth more than $117 billion by 2020.

What is MQTT in IoT

Internet of Things (IoT) and machine-to-machine (M2M) technologies need to use a messaging and connectivity protocol in order to exchange information from a remote location.

Modern Data Transmission for Industry 4.0

A fully-networked factory floor is the future of automation and manufacturing. Robots, cobots, and information going into and out of the cloud are helping us create better products faster and with more accuracy.

Starting with MQTT using Docker and Home Assistant

MQTT is a great protocol to have in your automation arsenal. This article explains how to quickly get started with MQTT using Docker and Home Assistant. MQTT, which stands for Message Queuing Telemetry Transport, is often described as the protocol of the Internet of Things (IOT).

Integrating IoT with IEC 61850

Internet of Things continues to be one of the key technology trends in the recent years. According to Gartner’s estimations, IoT network will grow from 8,7 billion in 2017 to over 20 billion connected things by 2020.

Fight Against Unprotected s2s Connections

In the past, the certificates were worth the money. In the past, obtaining certificates was a difficult task. This forced the use of self-signed certificates in connections to preserve the federation. Invalid certificates are dangerous because unencrypted (without e2e) user communications can be intercepted and modified.

This Google Summer of Code was about adding peer-to-peer file transfers to Dino. Dino is an XMPP client, XMPP is decentralized instant messaging standard.

The work was mostly done in two larger pull requests. The first one, dino#577, added a Jingle framework (XEP-0166), the in-band-bytestream transport (XEP-0047, XEP-0261) and the Jingle file transfer protocol (XEP-0234).

Jingle is a protocol for (mostly out-of-band) peer-to-peer connections between XMPP clients. It is designed to support multiple transport types, e.g. in-band bytestreams, connections via SOCKS5 proxies or direct IP connections. It provides support to send different kinds of data between clients, the two major users are file transfers (described below) and audio/video calls. All this is reflected in the code, the Jingle interface was designed to be extensible for more transport methods and different data types.

In-band bytestreams are a simple method of exchanging binary data between XMPP clients by sending base64-encoded data in the XMPP stream via the server. It is a good fallback method in case no other transport works, but adds 33% overhead to the data sent.

Apart from the HTTP file upload specification, the Jingle file transfer protocol is another standard way of exchanging files between XMPP clients. For Dino (and also in Conversations), this method is used transparently (using the same UI) for files larger than the upload limit of the HTTP upload server. This allows the user to benefit from both methods.

The work on this pull request was concluded by some interoperability testing with Conversations and Gajim, two other major XMPP clients.

The second pull request, dino#592 (not merged yet, currently on commit 6028fd1) added the SOCKS5 transport method for Jingle (XEP-0065, XEP-0260).

The SOCKS5 transport method allows XMPP clients to exchange data via a direct connection or via a XMPP-server provided SOCKS5 proxy. The direct transfer only works if one client can see the other client’s IP address and is not firewalled, which is mostly the case if both the clients are on the same private network. It is currently not implemented and is waiting for some libnice support for enumerating the local interfaces (see below). Instead, the data can be exchanged via a proxy reachable by both clients, determined by some candidate exchange.

Interoperability testing for the second pull request uncovered some old and new bugs in both existing clients and my implementation. Thanks to the Conversations developer for taking my bug reports seriously and responding very fast.

There were also a couple of other smaller pull requests which I’ll mention here:

• dino#569. Tiny fix for logging to a file.
• dino#570. Tiny typo fix, sorry for even mentioning. :o
• dino#579. Improvement around the error generation in response to certain XMPP messages.

In the course of the interoperability testing, I also opened a couple of issues on other XMPP clients: Conversations#3478, Conversations#3500, Conversations#3514, Conversations#3515, Gajim#9784.

As the next steps, after Google Summer of Code, I’ll implement the WebRTC transport method to get interoperability with XMPP browser clients that cannot open direct TCP connections to SOCKS5 servers. Additionally, one could implement audio/video calls on top of the current Jingle code.

At this point I’d like to thank my two mentors fiaxh and mar-v-in without whom this wouldn’t have been possible. They were quick to answer any questions that arose, about architectural design of the code, the XMPP standards or even just questions about Vala, the language Dino is written in.

Thanks to my mentoring organization, the XSF (XMPP standards foundation), as well, for the weekly meetings and generally the advice given therein.

A decade ago, today, the first yaxim commit was created, making it now officially half as old as XMPP. Since then, much has happened both in the XMPP ecosystem and on the Android side of things.

2009: The Beginning

Back in 2009, the Android platform was still brand new, and it was lacking a Free IM client. There were rumors and announcements, but nobody had posted working code yet.

The first specific hint was a German slide deck by Sven and Chris, presenting their semester project, YAXIM - Yet Another XMPP Instant Messenger.

Some friendly emails later, a GitHub project got created, and the coding went on. At the end of the year, a lightning talk was given at the 26C3, showing how yaxim looked back then:

In the early days, it was a huge challenge to reliably deliver messages, and things were improving only slowly.

Significant Steps

In 2010, YAXIM got renamed into yaxim, to look more like a name and less like a yelling acronym.

In 2013, Bruno was launched as yaxim’s cute little sibling, to appeal to children and everybody who loves animals. It has attained almost 2000 active fans by now.

Also in 2013, the yax.im XMPP service was launched, mainly to make on-boarding from yaxim and Bruno easier, but also to have a stable and reliable service usable for mobile clients.

Finally, in 2016, yaxim got its current yak-oriented logo.

A Review In Numbers

From day 1, yaxim was more or less a hobby project, with no commercial backing and no full time developers. Over the years, the code base has grown slowly, with 2015 being an especially slow year.

Some contributors were more active than others… 😉

Even though yaxim still has more total installs on Google Play than Conversations, the latter is said to be the go-to client on Android, and very popular among federation nerds. Still, at least in the last three years, there was no decline in the number of devices that yaxim is installed on (Google doesn’t provide stats before 2016):

Current Challenges

The technical foundation of yaxim (Smack 3.x, ActionBarSherlock) became rather dated, and currently much effort is going into making yaxim look great on modern (Material design) Android devices, and to support modern features like interactive permission dialogs and battery saving, as well as the Matrix (with the support being currently dormant.

Please check out the Google Play beta channel to stay up-to-date with the most recent developments, and let’s see how the next decade will develop!

Who will find this interesting

If you’re considering XMPP for your project but you are unsure if it can provide the functionality you need, you’ll eventually end up here:

http://xmpp.org/extensions/

I’m pretty sure you’ll be quite intimidated by such a long list of extensions. In some cases it will be pretty easy to find what you need. If you look for PubSub functionality, you’ll quickly notice “Publish-Subscribe”. Sometimes it’s not so obvious though. XMPP developers already know that in order to synchronise outgoing messages between several devices, they have to enable “Message Carbons”. Not very intuitive, isn’t it?

The aim of this blog post is to guide you towards proper XMPP technologies and solutions, given your specific use cases. I’ve worked with and deployed solutions powered by XMPP, such as MongooseIM, for years; so let me be your personal Professor Oak, providing a perfect “companion(s)” to work with and begin your journey in XMPP world. There are almost 400 XEPs, will you catch them all? ;)

The length of this article is caused not by a complexity of descriptions but by a count of use cases and features. :)

All numbers and information on implementation status are valid for March 2017.

What can you expect here?

For every use case, I will list XMPP features you definitely should consider using. Each one of them will be briefly described. The goal here is to understand the usefulness without reading whole specification. Besides that, each item will include MongooseIM’s module name providing discussed extension and example client implementations.

What you won’t find in this post

This post won’t cover any XMPP basics. It assumes you either know them already (what are JID, C2S, S2S, IQ, stanzas, stream etc.) or you intend to learn them from some other guide, like the excellent (iOS) tutorial written by Andres Canal Part 1, Part 2). It’s more of a cookbook, not Cooking For Dummies.

ToC

1. I’m creating …
   1.1 … a mobile application.
   1.2 … a desktop application.
   1.3 … a web application.
   1.4 … an application that just can’t speak XMPP.
2. I need my application to …
   2.1 … show message status like Facebook does.
   2.2 … provide message archive to end users.
   2.2.1 I’d like to have a full text search feature.
3. … display inbox (a list of conversations with unread count and a last message).
4. … allow file transfers and media sharing between users.
   4.1 P2P
   4.2 File Upload
5. … support groupchats …
   5.1 … and I need precise presence tracking in each group.
   5.2 … and I don’t need to broadcast presence information in each group.
6. … be compatible with other public XMPP setups.
7. … present the same view of each conversation on every user’s device.
8. … allow users to block each other.
9. … support end-to-end encryption.
10. … be a part of Internet of Things.
11. … receive push notifications.
12. … publish messages to groups of subscribers.

1. Creating …

Before we proceed to more specific requirements, it’s important to identify crucial standards based on your application type.

1.1 … a mobile application.

Smartphones are omnipresent nowadays. It’s a fact. The whole software market considered, mobile apps are an important medium between various companies and their customers. Some of them are the actual products (games, communicators, car navigations, etc.), not only a “channel”. If you’re going to develop a mobile application, you will need…

XEP-0198 Stream Management

It’s an extension that provides two features actually. One of them is stanza delivery confirmation (both server and client side), what allows early detection of broken connections or malfunctioning network layer. The other one is stream resumption. It makes reconnection faster by reducing the round-trip count and relieves the client of fetching message archive as pending, unacknowledged messages will be retransmitted from server buffer.

It is enabled by default in MongooseIM and supported by major client libs like Smack or XMPPFramework. From a client developer perspective, it’s pretty transparent because the whole extension is enabled with a single flag or method call.

MUC Light, MIX, Push notifications, HTTP File Upload

These extensions are especially useful in the mobile environment. Why? With MUC Light and MIX you gain control over presence broadcasting - you can spare your users frequent radio wakeups and bandwidth usage. These extensions are a significant improvement over traditional presence-driven group chats.

Virtually every app on our smartphones uses push notifications. Some are useful and some are just annoying commercials. It doesn’t matter - it’s almost certain you’ll want them integrated with your XMPP service.

HTTP File Upload allows asynchronous media sharing, which is much more convenient in the case of group chats and doesn’t require both parties to stay online during the transfer.

These are just brief summaries. You can find more details further in this post.

1.2. … a desktop application.

Despite mobile phones’ expansion and software products exclusive for them (Instagram, Snapchat, Tinder, etc.), nobody can deny the comfort of UI operated with a mouse, keyboard, or tablet. Some apps simply require processing power that portable devices can’t provide. If your code is going to be executed on desktops PCs and laptops, you’ll appreciate…

There are no extensions that are strictly essential for desktop apps. Everything depends on specific applications. Just bear in mind that the standards important for mobile apps are generally useful for desktop ones too, only less critical.

1.3. … a web application.

As the days of heavy browser incompatibility (thank you, standardisation!) and Flash technology abuse are long gone, web applications are a great way to provide cross-platform solutions. It’s not only easier to reach more platforms but also to ensure the users are always running the most up-to-date version.

If you’re a web developer, you’re going to connect to the XMPP server via BOSH or Websockets.

Websockets

Websockets technology allow to upgrade an HTTP connection to an asynchronous, full-duplex, binary one (a bit of simplification but it’s the essence). It means that XMPP stanzas can be exchanged almost as efficiently as over a raw TCP connection (Websockets add small overhead of integer with packet size). It’s the recommended protocol for single-page apps.

Note: You can combine Stream Management’s resumption with Websockets, although it will still be slower than BOSH’s session pause.

Warning: Websockets are not implemented by old browsers. If you have to support any outdated clients, take a look at this table first.

BOSH

Defined in XEP-0124: Bidirectional-streams Over Synchronous HTTP (BOSH) and XEP-0206: XMPP Over BOSH. This protocol encapsulates XMPP stanzas in HTTP requests. It also simulates asynchronous, bidirectional communication by issuing long polling requests from client to the server to retrieve live data. What does it mean in practical terms?

A persistent connection may be maintained but in general BOSH is designed to deal with interrupted connections. It’s a perfect solution for web apps that trigger browser navigation. On such event, all connections made by e.g. JavaScript from browser are closed but the BOSH session survives it on the server side (not forever of course) and the client can quickly and efficiently resume the session after page reload.

The protocol is pretty verbose though, so if you don’t need this feature, go for Websockets.

1.4. … an application that just can’t speak XMPP.

You probably think that I’m crazy; why use XMPP with XMPP-less clients? Let’s change the way we think about XMPP for a moment. Stop considering XML the only input data format the XMPP server accepts. What if I told you that it’s possible to restrict XML to the server’s routing core and just make REST calls from any application? Tempting?

It’s a non-standard approach and it hasn’t been documented by XSF (yet), but MongooseIM already exposes most important functionalities via REST. Check out this and this document to find out more.

2. I need my application to …

Now we continue to more specific use cases.

2.1. … show message status like Facebook does.

By message status we mean following states (plus live notifications):

1. Not sent to server yet.
2. Acknowledged by the server.
3. Delivered to the recipient.
4. Displayed by the recipient.
5. User is composing a message.
6. User has stopped composing a message.

(1) and (2) are handled by Stream Management. It’s pretty obvious - before receiving an ack from the server, you are in (1); and ack confirms the message entered state (2).

We can deal with (3) and (4) by using XEP-0333: Chat Markers. These are special stanzas sent by a recipient to the original sender. There are dedicated markers for received and displayed events.

(5) and (6) are provided by XEP-0085: Chat State Notifications. It is up to a client to send updates like <composing/> and <paused/> to the interlocutor.

2.2. … provide message archive to end users.

Virtually every modern chat application maintains conversation history both for 1-1 communication and group chats. It can remind you of a promise you’ve made, be evidence in a divorce case, or help in police investigation.

XMPP provides two protocols for accessing message archives. The older one, XEP-0136 Message Archiving is used by hardly anyone, because it’s difficult to implement and overloaded with features. It has been superseded by more modern XEP-0313 Message Archive Management, which is the current standard.

There is one caveat though - its syntax changed significantly between versions, so it’s common for libraries and servers to explicitly state what versions are supported by the specific piece of software. These are 0.2, 0.3 and 0.4(.1) and 0.5. MongooseIM supports all of them in mod_mam module. If you choose another server, make sure its MAM implementation is compatible with your client library. Smack and XMPPFramework use 0.4 syntax.

2.2.1. I’d like to have a full text search feature.

Although standard Message Archive Management doesn’t specify any queries for full text search, it remains flexible enough to create such requests on top of the existing ones.

In MongooseIM this feature is still in experimental phase and has been recently merged into master branch. It’s not supported in any client library yet, so you have to construct a custom MAM query to do full text searches. Take a look at the PR description, It’s not that difficult. :)

2.3. … display inbox (a list of conversations with unread count and a last message).

Unfortunately there are no open solutions providing this feature. XMPP community is in the process of discussing and creating the specification of Inbox functionality. Erlang Solutions is designing a XEP proposal, which you can view here.

A quasi-inbox is available as a part of experimental standard Bind 2.0. It doesn’t cover all possible use-cases but a list of unread messages is what you actually need for optimal UX after establishing a connection. This feature is already under development in MongooseIM project.

In the meantime, you can build an inbox view by persisting last known archived message ID or timestamp and query Message Archive Management for all messages that came later. When you fetch them all, you can build an inbox. Unfortunately this is not very efficient and that’s why the community needs a new standard.

2.4. … allow file transfers and media sharing between users.

Almost everyone loves to share cat pictures and every modern IM solution provides means to do this. Various file transfer techniques in the XMPP world can be grouped in two categories: P2P connections and file upload.

The former involves establishing a direct connection between two clients, sometimes with a bit of a help from a TURN server. It ensures that data won’t get stored on any intermediate servers. Obviously, it requires less effort from the service provider because it’s easier and cheaper to set up a TURN service than to maintain a proper media server (or pay for storage in the cloud).

File upload is much more efficient when sharing media with a group. It doesn’t require both parties to remain online for the transfer duration.

2.4.1. P2P

Now, you DO have a choice here. There are a couple of XEPs, describing various P2P transfer initiation methods. XEP-0047 In-Band Bytestreams (IBB) is guaranteed to work in every network, because it sends data (Base64-encoded) via IQs. So if you can reach the XMPP service, you can transfer files. It may be slow and not very convenient but it will work.

Let’s carry on. You can transfer media via bytestreams external to XMPP. The P2P session is negotiated via XMPP but it’s only the “signalling” part. There are quite a few XEPs describing various negotiation and transmission protocols, so I will highlight specific implementations rather than listing all of the names which would only confuse readers who just want to send some bytes.

• XMPPFramework: Look for XMPPIncomingFileTransfer and XMPPOutgoingFileTransfer. They support SOCKS5 and In-Band Bytestreams.
• Smack: Everything begins with FileTransferManager. It supports SOCKS5 and In-Band Bytestreams as well.
  
  

2.4.2. File Upload

Unless you already have a dedicated media server that exposes an API to perform uploads and downloads, you should definitely take a look at XEP-0363 File Upload. It defines standard stanzas to request upload slots and respective download links. It is XMPP server’s responsibility to allocate the slots and return the links to the client.

Unfortunately this extension is not widely supported yet. You can find it in XMPPFramework but not in Smack yet. In the case of MongooseIM, it’s already available with Amazon S3 backend (with more storage plugins to come!).

2.5. … support group chats …

A couple of years ago it was really simple - there was only one kind of group chat supported in the XMPP world. Today we have three standards, two of them being maintained by XSF and one published by Erlang Solutions. MIX (XEP-0369), doesn’t have any implementations yet and as a standard it changes very frequently, so it is not described in this post.

2.5.1. … and I need precise presence tracking in each group.

If you need IRC-like experience where users have certain roles in a room and client disconnection triggers leaving the room, then classic XEP-0045 Multi-User Chat will work for you. It has its disadvantages (frequent presence broadcast may impact UX and consume processing power or connection throughput) but fits the use case, where accurate presence information is important. It is provided by MongooseIM’s mod_muc (other major servers implement it as well) and is supported by all mainstream client libs.

2.5.2. … and I don’t need to broadcast presence information in each group.

Erlang Solutions’ Multi-User Chat Light is a protocol derived from real world use cases, where groups doesn’t care about presences and full member list is always available to room members. It has some strong assumptions (like only 2 types of affiliation or rooms being joinable only by invite) but is designed to reduce round-trips, expose powerful API (e.g. room creation + configuration + adding new members in one request) and be easy to work with. Check it out and see if it fits in your application. Server implementation is currently exclusive to MongooseIM (mod_muc_light) and respective plugins are available in Smack and XMPPFramework.

2.6. … be compatible with other public XMPP setups.

Even some proprietary installations do integrate with open XMPP world (like GTalk and Facebook at some point), so if this is your use case as well, the first important thing to remember is that no custom stanzas may leave your cluster. By custom I mean anything that is not covered by any XSF-approved XEP. Additionally, you will really benefit from using XEP-0030 Service Discovery protocol a lot, because you can never be sure what is the supported feature set on the other end. It is used to query both clients and servers. Virtually every client and server supports it. In case of MongooseIM, the base module is mod_disco.

2.7. … present the same view of each conversation on every user’s device.

I use Facebook messenger on multiple devices and I really expect it to display the same shopping list I got from my wife on both my desktop and my mobile phone. It usually breaks message order but anyway - at least the list is there.

The problem is actually a bit more complex, because you have to take care of synchronising both online and offline devices.

Online devices can ask the server to forward all incoming/outgoing messages, even if they originate from or are addressed to some other resource of the same user. It is achieved by enabling XEP-0280 Message Carbons. On the client side it’s easy - just enable the feature after authenticating and the server will do the rest. It’s supported by MongooseIM in mod_carboncopy module. You can find respective implementations in Smack, XMPPFramework, Stanza.io and many others, since it’s a very simple, yet powerful extension.

If you want to fetch everything that happened while a specific device was offline for a while, just query XEP-0313 Message Archive Management (see “… provide message archive to end users.” section).

2.8. … allow users to block each other.

You just can’t stand your neighbour nagging you via IM to turn down the volume while Kirk Hammett is performing his great solo? Block him. Now. XMPP can help you with it. In two ways actually.

Yes, XMPP features two standards that deal with blocking: XEP-0016 Privacy Lists and the simpler XEP-0191 Blocking Command. The former allows users to create pretty precise privacy rules, like “don’t send outgoing presences to JID X” or “accept IQs only from JIDs in my roster”. If you need such a fine grained control, take a look at MongooseIM’s mod_privacy. On the client side it is supported by the likes of Smack and XMPPFramework.

Blocking Command is much simpler but most setups will find it sufficient. When a client blocks a JID, no stanza will be routed from the blockee to the blocker. Period. MongooseIM (mod_blocking), Smack and XMPPFramework have it.

2.9. … support end-to-end encryption.

When Alice wants to send a message to Bob… no, we’ve all probably seen this classic example too many times already. :)

There is no “one size fits all” when it comes to E2E encryption. The first tradeoff you’ll have to make is to decide whether you want new users devices to be able to decrypt old messages, or do you prefer to have a property of forward secrecy. For a full comparison between available encryption methods, let’s take a look at the table published by OMEMO authors:

It’s difficult to find an open library that supports any of these methods. Gajim communicator has an OMEMO plugin. Smack and XMPPFramework don’t support E2E encryption in their upstream versions. If you’re going to use E2E encryption in your application, most probably you’ll have to implement it on your own. Good thing is there are standards you can base your code on.

2.10. … be a part of Internet of Things.

We are a peculiar bunch. We use semiconductors to build machines that do heavy number crunching for us, deliver messages in a blink of an eye and control robotic arms with precision far beyond ours. A desire has awoken in us to go even deeper and augment everything with electronics. To let everything communicate with each other.

If you’re designing a fridge microcontroller that is supposed to fetch results from bathroom scales and lock the door for 8h for every excessive BMI point, you’ll need…

• XEP-0323 Internet of Things - Sensor Data
• XEP-0324 Internet of Things - Provisioning
• XEP-0325 Internet of Things - Control
• XEP-0326 Internet of Things - Concentrators
• XEP-0347 Internet of Things - Discovery
  
  

Unfortunately there are no public implementations of these standards. I wish it was easier but it seems you just can’t avoid reading these XEPs, picking the most suitable parts and creating your own implementation.

To find out more and become an active member of XMPP IoT community, check out IoT Special Interest Group.

2.11. … receive push notifications.

Push Notifications are (usually) doing a great service to mobile devices’ battery life. It is great indeed that a single TCP connection is maintained by OS, while apps can remain hibernated in the background. It is natural to every chat application to deliver notifications to the end user, even when a smartphone is resting in the pocket. How does XMPP cooperate with popular services like APNS or GCM?

It depends.

Although it’s not difficult to find XEP-0357 Push Notifications, it deserves some explanation. This specification is very generic. It assumes the existence of another XMPP-enabled “App server” that handles push notifications further. Although implementations could be found (e.g. in MongooseIM or Prosody server), it is very common for commercial installations to use custom protocols to provide push tokens and send PN packets directly to the respective services (APNS, GCM, SNS…)

2.12. … publish messages to groups of subscribers.

Publish-subscribe is an ancient and extremely useful pattern. XMPP got its own PubSub specification quite early (first versions were published in 2003) and even though the protocol is pretty verbose (and a bit complicated), for the basic usage you’ll need to learn only the most important aspects: there are nodes in the PubSub service where publishers can publish data. Nodes can group other nodes or remain plain leaves. That’s the whole story. The rest is about configuration, access control, etc.

XEP-0060 Publish-Subscribe is implemented in every major XMPP piece of software. In case of MongooseIM, it’s handled by mod_pubsub. You can find in popular client libraries as well: Smack, XMPPFramework or Stanza.io.

Set sail!

Now, if you feel brave enough, you can dive into this looong, official list of XEPs. These documents are designed to provide a precise information for libs and server developers. In your daily routine you’ll probably won’t care about every server-side edge case or whether some information is encapsulated in element X or Y. These are perfect reference guides for if you’re stuck somewhere or need to tap into lib’s plugin internals.

I’d recommend one more intermediate step though. Browse servers’ and clients’ feature lists published by virtually every project on their webpages. They usually skip (or enumerate them in separate, more detailed docs) minor items and highlight the features that may be important to you. This way you’ll expand your XMPP vocabulary and knowledge, while postponing the stage where reading XEPs is unavoidable, thus making the learning curve less steep.

It won’t be long until you realise that XMPP is your friend, I promise. :)

Stay tuned for Part 2! In the meantime:

1. Brush up on your XMPP basics with our guide to building an iOS app from scratch using XMPPFramework (parts 1 and 2)
2. Learn more about MongooseIM, our XMPP based open source mobile messaging platform.

@Flow wrote:

jXMPP, a FOSS XMPP-base library, has just been extended by a testframework for “XMPP-Strings”. Currently, this is limited to Local-, Domain- and Resourceparts and the various XMPP address types, but may be extended to future Strings found in XMPP-land.

The testframework comes with a corpus of known valid and invalid XMPP addresses (JIDs). I am happy about feedback and contributions to the corpus. Hopefully it becomes a useful resource for XMPP developers. You can find the corpus under

• jxmpp-strings-testframework/src/main/resources/xmpp-strings/jids/valid/main
• jxmpp-strings-testframework/src/main/resources/xmpp-strings/jids/invalid/main

The testframework currently uses the following libraries to prepare and enforce the Strings:

• ICU4J
• GNU libidn
• xmpp.rocks’s PRECIS

a further minimalistic implementation, called “simple”, is part of jXMPP.

You can run the testframework simply by

git clone https://github.com/igniterealtime/jxmpp.git
cd jxmpp
./test-xmpp-strings

if you have gradle and python3 in your path.

Please note that ICU4J and libidn (and “simple”) do not currently provide the required PRECIS profile and hence the older Stringprep profiles are used.

Feel free to use the provided infrastructure to test your own implementation. Thanks to JNI and the polyglot programming feature of the GraalVM, it should be easily possible. Please contact me if you need assistance.

Posts: 1

Participants: 1

Read full topic

ProcessOne curates the Real-time Radar – a newsletter focusing on articles about technology and business aspects of real-time solutions. Here are the articles we found interesting in Issue #24. To receive this newsletter straight to your inbox on the day it is published, subscribe here.

MQTT for system administrators and for the IoT

They say MQTT is a PUB/SUB protocol for the Internet of Things, which it was originally designed for, but it’s also well suited for monitoring machines and services. Presentation given at BSDCan2019 in Ottawa.

Z-Wave to MQTT

Fully configurable Z-Wave to MQTT gateway and control panel. The Z-Wave to MQTT add-on allows you to decouple your Z-Wave network from your Home Assistant instance by leveraging your MQTT broker.

Migrating trashserver.net from Prosody to Ejabberd

Author writes: “Yesterday I moved my old Prosody setup to a new Ejabberd-based XMPP server setup. I’d like to leave you a few notes on why and how I did that. There ware multiple reasons for my decision to give Ejabberd a chance.”

Emoji avatars for personal website

Author writes: “My previous avatar was almost 3 years old, and I was getting tired of it. I decided to replace my avatar on my website for my IndieWebCamp Austin hack day project. But if you know me, you know I can’t do anything the easy way.”

Gotify server

A simple server for sending and receiving messages, written in Go and licensed under the MIT license.

LabPlot 2.6 released with MQTT support

The next release of LabPlot is announced! One big new feature is the support for the MQTT protocol. With LabPlot 2.6 is is possible now to read live data from MQTT brokers. This feature was contributed by Ferencz K. during Google Summer of Code 2018.