Planet Jabber

September 20, 2019

Monal IM

A mad dash to the finish

For the past week i have been trying to figure out why Monal stopped working in iOS 13. Building with the iOS13 SDK is the only way to get dark mode (and french app store support) . It appears that when I do build with this SDK it disables the push mechanism I use. the replacement does not work well. I wasn’t planning on needing to get it working until mid 2020. I am still allowed to submit builds with the iOS12 SDK. I may have to sacrifice dark mode and other iOS 13 niceties for now until I have the core messaging at parity. I did manage to get it close and the server infrastructure is set up to handle it so I will keep working on that. I’ll probably keep two different test versions going 4.0 for iOS 12 builds that make it to the app store and 4.1 for ios13 builds that use the new mechanism.

by Anu at September 20, 2019 02:19

September 19, 2019

ProcessOne

Fluux and ejabberd BE support iOS 13 Apple Push Notification System from day one

Yesterday, Apple unveiled the new iPhones and set September 19 as the date for the release of iOS 13. This new operating system brings lots of new features and changes, but among them there’s one easy-to-miss update that’s very significant for messaging server operators: an update to APNS requests that enable sending push notifications to iOS clients.

We are happy to announce that today we deploy an update to how our services, like Fluux and software like ejabberd Business Edition handles APNS requests, so all our endpoints are compatible and support iOS 13 APNS from day one.

Technical details

This small but significant update means adding support for the new apns-push-type parameter while sending the request to the APNS. And of course, a respective interface for our customers to be able to set and manipulate the said parameter.

It’s significant because, as the Apple docs state, the new parameter is “Required when delivering notifications to devices running iOS 13 and later, or watchOS 6 and later. (…) The value of this header must accurately reflect the contents of your notification’s payload. If there is a mismatch, or if the header is missing on required systems, APNs may delay the delivery of the notification or drop it altogether.”

As you can see, the implementation has to be carefully executed and tested. Thanks to our expert team and proven processes, we were able to prepare the updates to our services and software quickly and deploy them today, ahead of iOS 13 launch.

Important note: If you are using ejabberd Business Edition, you only need to update if you are already using ejabberd with APNS v3. If you are using the legacy APNS v2 service, you do not have to update at the moment.

Photo by Jamie Street, Unsplash

by Marek Foss at September 19, 2019 10:32

We are not an Erlang company

ProcessOne has made a mark on Erlang history, developing reference software in Erlang, providing strong Erlang expertise and helping grow the its ecosystem. Still, ProcessOne is much more than our Erlang fame. We are technology agnostic. We are great at selecting the right tool for the job to build innovative projects for our customers.

TL;DR

ProcessOne is much more than an Erlang company and we can help you develop your projects in Go, Swift and Kotlin.

Our Erlang history

ProcessOne is an Erlang pioneer. The core team have been developing in Erlang since it was released as Open Source in 1998. We have been building the most famous Erlang project, the real-time messaging platform: ejabberd. Consequently, RabbitMQ followed suite because ejabberd led the way.

We used Erlang very early because it was the only way, at that time, to build highly robust and scalable platforms. ejabberd is able to handle a massive number of connections, supports clustering, get hot code upgrades – all because of the underlying Erlang VM.

We have been jumping on the Elixir bandwagon very soon for quite the same reasons. It is a technology that fills a niche, making it possible to build large scale real-time web platforms with a syntax that feels more familiar (especially for Ruby developers).

Innovation is our driver

When we started ProcessOne, there was no Amazon Web Service. Servers were multiple times slower than they are today. There was no Docker, no Kubernetes. Basically, there was no alternative to Erlang VM if you wanted to build a scalable and manageable service. And most of all, there was no iPhone and Android, whose support is now a critical component of many projects.

So, over the years, we had to refine what innovating for our customers meant, integrating new technologies in our skill set and our stack.

In the context of our customers’ projects, Erlang or even Elixir is not always the best answer. It is hard to find Erlang or Elixir skills and training an Erlang or an Elixir team is most of the time not desirable for customers. They often need to hire and train large set of developers and sysadmins. They need to have a path for innovation to fit their corporate culture. And finally, most of all, Erlang and Elixir are not enough to cover the client-side development, especially on mobile devices.

That’s why we had to refine our technology stack with innovation as our main focus.

Expanding our technology stack

Over the years, we had to expand our technology stack to have a good balance between innovation and ability for our customers to adopt it. We ended up developing the following additional skills to handle our customer projects:

  • Go to handle server-side services and web applications. Go is very popular and provides good scalability and maintainability. Even if Go seems approachable and easy, writing very good Go code that feels like “native Go” and is maintainable, is hard. However, it is accessible for our customers and we have found it was quite easy to transfer the knowledge to customers’ teams. Once we have laid the basis and delivered a strong Go applications, it is easy for our customers to take over the projects. It works well with our Erlang components, as our customers can use them as black boxes.
  • Swift to handle iOS native developments. This is the de facto standard in the Apple ecosystem. Given our high-profile customers target, cross-platform development is generally not enough. That’s why we are investing in providing state-of-the-art native code, following the latest standards on iOS/MacOS/Watch/TvOS.
  • Kotlin to handle native Android developments. We target high-end projects that needs native developments with the latest features. Kotlin is the way to deliver future-proof code on Android.

We are still working on Erlang projects and you can expect the same involvement in ejabberd. Still, you can also expect us to talk more about our other skills and see new projects in those languages. Our Open Source software stack will get richer and serve as basic building bricks for our customers’ projects.

Summary

Given our unique Erlang history, we are often still seen as an Erlang company. While we have a unique expertise and a part of our customer-base work with us because of that Erlang or Elixir expertise, limiting ProcessOne to this small set of technologies could not be further from the truth.

We deliver full projects for customers that need to deliver innovative services, in record time. Finally, we hand over that innovation so that it can live for many years after we delivered our result.

Working with ProcessOne is like having your own team of R&D experts, that can bootstrap your project to help your reach the market before the competition, without compromise on the technology.

Do not hesitate to contact us if you need help building your projects, with the confidence that you will be able to integrate the innovation internally.

by Mickaël Rémond at September 19, 2019 09:17

September 18, 2019

Monal IM

Monal 4

IOS 13 comes out tomorrow and Monal 4 will be out in the next day or so. I was hoping to be ready day 1 this time and I was for dark mode but the push issues took me by surprise. I expected to have until 2020 to resolve it. Monal as it is in the app store, Monal won’t work with iOS 13, the OS will block the voip pushes. The updated version will work with a new push server that does not send voip pushes anymore. I have posted that to the beta channel today. It just tells you there is a new message and doesn’t have the message text or sender. It’s not great but the alternative is not knowing when you have messages. I will continue to work on the notification extension I am using for these messages and will update it in the coming week hopefully we will get similar behavior to iOS 12 soon.

Have I mentioned how much I hate Facebook messing this up for everyone.

by Anu at September 18, 2019 23:36

Ignite Realtime Blog

Fastpath Service plugin 4.4.4 released

@wroot wrote:

The Ignite Realtime community is happy to announce the release of version 4.4.4 of the Fastpath Servive plugin for Openfire!

This update fixes exception when pressing on workgroup name, makes it possible to build plugin with Maven and adds an ad-hoc command to update workgroup.

Your instance of Openfire should automatically display the availability of the update. Alternatively, you can download the new release of the plugin at the Fastpath Service plugin’s archive page

For other release announcements and news follow us on Twitter

Posts: 1

Participants: 1

Read full topic

by @wroot wroot at September 18, 2019 19:14

ProcessOne

ejabberd 19.08

We are pleased to announce ejabberd version 19.08. The main focus has been to further improve ease of use, consistency, performance, but also to start cleaning up our code base. As usual, we have kept on improving server performance and fixed several issues.

New Features and improvements

New authentication method using JWT tokens

You can now authenticate with JSON Web Token (JWT, see jwt.io for details).

This feature allows you to have your backend generate authentication tokens with a limited lifetime.

Generating JSON Web Key

You need to generate a secret key to be able to sign your JWT tokens.

To generate a JSON Web Key, you can for example use JSON Web Key Generator, or use your own local tool for production deployment.

The result looks like this:

{
  "kty": "oct",
  "use": "sig",
  "k": "PIozBFSFEntS_NIt...jXyok24AAJS8RksQ",
  "alg": "HS256"
}

Save you JSON Web Key in a file, e.g. secret.jwk.

Be careful: This key must never be shared or committed anywhere. With that key, you can generate credentials for any users on your server.

Configure ejabberd to use JWT auth

In ejabberd.yml change auth_method to jwt and add the jwt_key option pointing to secret.jwk:

auth_method: jwt
jwt_key: "/path/to/jwt/key"

Generate some JWT tokens

See jwt.io for an example of how to generate JSON Web Tokens. The payload must look like this:

{
  "jid": "test@example.org",
  "exp": 1564436511
}

And the encoded token looks like this:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqaWQiOiJ0ZXN0QGV4YW1wbGUub3JnIiwiZXhwIjoxNTY0NDM2NTExfQ.SMfzCVy8Nv5jJM0tMg4XwymIf7pCAzY8qisOSQ5IAqI

Authenticate on XMPP using encoded token as password

Now, the user test@example.org can use this token as a password before 1564436511 epoch time (i.e. July 29, 2019 21:41:51 GMT).

New configuration validator

With ejabberd 19.08, we introduce a new configuration checker, giving more precise configuration guidance in case of syntax errors or misconfiguration. This configuration checker has also been released as an independent open source project: yconf.

The new configuration validator makes it possible to improve the configuration parsing. For example, it supports the following:

Better handling of Erlang atom vs string

There is not need to quote string to express the fact you want an atom in the configuration file: the new configuration validator handles the Erlang types mapping automatically.

More flexible ways to express timeouts

Now, all timeout values can be expanded with suffixes, e.g.

negotiation_timeout: 30s
s2s_timeout: 10 minutes
cache_life_time: 1 hour

If the suffix is not given, the timeout is assumed in seconds

Atomic configuration reload

The configuration will either be fully reloaded or rolled back.

Better, more precise error reporting

Here are a couple of examples of the kind of message that the new configuration validator can produce.

In the following example, the validator will check against a value range:

14:15:48:32.582 [critical] Failed to start ejabberd application: Invalid value of option loglevel: expected integer from 0 to 5, got: 6

More generally, it can check value against expected types:

15:51:34.007 [critical] Failed to start ejabberd application: Invalid value of option modules->mod_roster->versioning: expected boolean, got string instead

It will report invalid values and suggest fixes in case error was possibly due to a typo:

15:50:06.800 [critical] Failed to start ejabberd application: Invalid value of option modules->mod_pubsub->plugins: unexpected value: pepp. Did you mean pep? Possible values are: flat, pep

Prevent use of duplicate options

Finally, it will also properly fail on duplicate options and properly report the error:

15:56:35.227 [critical] Failed to start ejabberd application: Configuration error: duplicated option: s2s_use_starttls

It was a source of error as an option could shadow another one, possibly in an included file.

Improved scalability

We improve scalability of several modules:

Multi-User chat

MUC Room modules is more scalable, allowing supporting more rooms, by hibernating the room after a timeout. Hibernating means removing it from memory when not used and reloading it on-demand.

The MUC messages processing has also been changed to now properly handle all the available CPU cores. MUC room message handling is now faster and support larger throughput on SMP architectures

SQL database handling

We improve the way the SQL pool is managed to better handle high load. We also improved MySQL schema a bit to help with indexing.

Changed implementation of mod_offline option use_mam_for_storage

Previous version was trying to determine the range of messages that should be fetched from MAM by storing time when last user resource disconnected. But that had couple edge cases that could cause problems, for example in case of hardware node crash we could not store information about user disconnect and with that we didn’t have data to initiate MAM query.

The new version doesn’t track user disconnects, but simply ensure that we have timestamp of first message that is gonna be put in storage, after some measurements cost of that check with caching on top is not that costly, and as it is much more robust we decided to move to that safer approach.

New option captcha_url

Option captcha_host is now deprecated in favor of captcha_url. However, it’s not replaced automatically at startup, i.e. both options are supported with ‘captcha_url’ being the preferred one.

Deprecated ‘route_subdomains’ option

This option was introduced to fulfil requirements of RFC3920 10.3, but in practice it was very inconvenient and many admins were forced to change its value to ‘s2s’ (i.e. to behaviour that violates the RFC). Also, it seems like in RFC6120 this requirement no longer presents.

Those admins who used this option to block s2s with their subdomains can use ‘s2s_access’ option for the same purpose.

API changes

Renamed arguments from ‘Server’ to ‘Host’

Several ejabberd commands still used as argument name ‘Server’, instead of the more common ‘Host’. Such arguments have been renamed, and backward support allows old calls to work correctly.

The eight affected commands are:
– add_rosteritem
– bookmarks_to_pep
– delete_rosteritem
– get_offline_count
– get_presence
– get_roster
– remove_mam_for_user
– remove_mam_for_user_with_peer

If you are using these calls, please start updating your parameter names to Host when moving to ejabberd 19.08. You will thus use a more consistent API and be future proof.

Technical changes

Removed Riak support

Reasons:

  • Riak DB development is almost halted after Basho
  • riak-erlang-client is abandoned and doesn’t work correctly with OTP22
  • Riak is slow in comparison to other databases
  • Missing key ordering makes it impossible to implement range queries efficiently (e.g. MAM queries)

If you are using Riak, you can contact ProcessOne to get assistance migrating to DynamoDB, an horizontally scalable key value datastore made by Amazon.

Erlang/OTP requirement

Erlang/OTP 19.1 is still the minimum supported Erlang version for this release.

Database schema changes

There is no change to perform on the database to move from ejabberd 19.05 to ejabberd 19.08.
Please, make a backup before upgrading.

It means that an old schema for ejabberd 19.05 will work on ejabberd 19.08. However, if you are using MySQL, you should not that we changed the type of the server_host field to perform better with indexes. The change is not mandatory, but changing it to varchar(191) will produce more efficient indexes.

You can check the upgrade page for details: Upgrading from ejabberd 19.05 to 19.08

Download and install ejabberd 19.08

The source package and binary installers are available at ProcessOne. If you installed a previous version, please read ejabberd upgrade notes.
As usual, the release is tagged in the Git source code repository on Github. If you suspect that you’ve found a bug, please search or fill a bug report in Issues.


Full changelog
===========

Administration
– Improve ejabberd halting procedure
– Process unexpected Erlang messages uniformly: logging a warning
– mod_configure: Remove modules management

Configuration
– Use new configuration validator
– ejabberd_http: Use correct virtual host when consulting trusted_proxies
– Fix Elixir modules detection in the configuration file
– Make option ‘validate_stream’ global
– Allow multiple definitions of host_config and append_host_config
– Introduce option ‘captcha_url’
– mod_stream_mgmt: Allow flexible timeout format
– mod_mqtt: Allow flexible timeout format in session_expiry option

Misc
– Fix SQL connections leakage
– New authentication method using JWT tokens
– extauth: Add ‘certauth’ command
– Improve SQL pool logic
– Add and improve type specs
– Improve extraction of translated strings
– Improve error handling/reporting when loading language translations
– Improve hooks validator and fix bugs related to hooks registration
– Gracefully close inbound s2s connections
– mod_mqtt: Fix usage of TLS
– mod_offline: Make count_offline_messages cache work when using mam for storage
– mod_privacy: Don’t attempt to query ‘undefined’ active list
– mod_privacy: Fix race condition

MUC
– Add code for hibernating inactive muc_room processes
– Improve handling of unexpected iq in mod_muc_room
– Attach mod_muc_room processes to a supervisor
– Restore room when receiving a message or a generic iq for not started room
– Distribute routing of MUC messages across all CPU cores

PubSub
– Fix pending nodes retrieval for SQL backend
– Check access_model when publishing PEP
– Remove deprecated pubsub plugins
– Expose access_model and publish_model in pubsub#metadata

by Mickaël Rémond at September 18, 2019 06:40

September 16, 2019

ProcessOne

The concepts behind Swift UI: Introduction

SwiftUI is a new application framework from Apple that complements UIKit and AppKit. It is expected to be the future of UI definition on Apple platform, unifying GUI application writing using the same framework on all Apple platforms, from Apple Watch to iOS and MacOS.

A change in architectural patterns: From MVC to MVVM

SwiftUI marks a shift in application architectural pattern. UIKit and AppKit applications are structured using the Model-View-Controller (MVC) paradigm. The Controller is the central abstraction layer used to link the Model, containing the business logic, to the View. This extra layer exists to make the code more maintainable and reusable.

The MVC pattern has been dominant in User Interface frameworks since it was first introduced in Smalltalk in the 80s.

However, it tends to lead to excessive boilerplate code. Moreover, as a programmer you have to fight the tendency to put large part of the application logic into the controller: The controller can become bloated and complex over time in large applications.

With SwiftUI, Apple is moving application development to the Model-View-ViewModel architectural pattern. This pattern was invented at Microsoft, as a variation of the Presentation model by Martin Fowler.
This pattern simplifies the design of event-based applications by offering a clearer dependency models between components and a clearer data flow. In MVVM, the view has a reference to the ViewModel and the ViewModel has a reference to the Model, and that’s it.

The MVVM pattern is used in many popular Javascript frameworks these days (ReactJS, Vue.js, Ember.js, etc.) and is becoming more and more widely used to design user interfaces.

The reference flow is simple but at some point, you will also want to let the Model update the View to reflect state changes.

You can still use the proven delegation approach to let the model update the view, but in the case, you are adding circular references and you are missing a big part of what makes SwiftUI great.

The concept of Binding is used to finally allow the Model to update the View. In SwiftUI, you can pass bindings to link view states to the underlying model. As defined in Apple documentation, the Binding is a “manager for a value that provides a way to mutate it”. It defineds a two way-connection propagating user changes on the application states in the view, but also updating the view when the model changes.

If you want to get further, you can design part of your application logic using Combine. Combine is a reactive framework to help build more complex applications using SwiftUI, taking care of event propagation and asynchronous programming, using a publish and subscribe model. Combine can be used for example to add a networking layer to your application.

A change in View design approach

SwiftUI also change radically how Views are designed and maintained. In the past, Views on iOS have often been designed graphically, using Interface Builder or Storyboard editor. Defining the Views in code was possible, but was a much larger effort. The interface was thus often designed graphically, adding constraints to define how the interface was supposed to change when displayed at different screen sizes. This resulted in interfaces being described in XML files, making teamwork difficult, as merge conflicts were painful to solve.

With SwiftUI, Apple introduced a declarative approach to design UI. The UI is always designed as code, but Apple also provides a canvas, to show a live preview of your view and guide the developer when developing the application. You can also interact directly on the canvas, and the code is updated accordingly.

At this time, the canvas is sometimes a bit unstable, but when it works, the overall process feels quite magical. No more fiddling with lots of panels defining values and constraints. SwiftUI simplifies the process by providing sane defaults for most system-provided views, but also remove the need for constraints by providing containers for your view that define the logical relation between the subviews. For example, a VStack will define a group of view that need to be rendered as a vertical stack on after the other. HStack define views that are laid out horizontally. ZStack defines set of view that are rendered one into the other (merged), etc. By combining high-level containers, you can describe the relationship between the views in your applications and let the system process by itself how to render the view.

Finally, I said previously that SwiftUI complements UIKit and AppKit. This is because SwiftUI works well with both UIKit and AppKit. Apple has defined tools and patterns so that both framework can work together. It means you can integrate with high-level components that are not directly available in SwiftUI, like MapKit or WebKit. It also means you can convert your application incrementally, adding SwiftUI views to an existing UIKit for example.

Get prepared for Apple platform future

For now SwiftUI is still in version 1. It already feels like the future of Apple application design, but you have to be prepared to face little bugs from time to time in Xcode or on-device rendering.

The fact that SwiftUI can only target iOS 13 / MacOS 10.15 / WatchOS 6 / TvOS 13 devices will also limit adoption, as developers often have to target applications running on several different OS versions.

However, it is consistent with the fact that you can expect SwiftUI to improve and mature considerably in the coming months.

With SwiftUI, Apple is attempting to solve several issues:

  • They want to unify the UI framework on all their platforms.
  • They are simplifying the programming model, offering a path to migrate from MVC to MVVM pattern.
  • As a side effect, they are reworking their concurrency pattern, introducing Combine as the underlying framework for implementing reactive programming in Swift-based applications.

If you are developing a brand new application today that is going to be released in a few months from now, it makes sense to consider SwiftUI.

At the very least, you should start learning SwiftUI and Combine now to be ready to adopt it when you feel it is possible for your kind of app.

by Mickaël Rémond at September 16, 2019 12:11

Monal IM

iOS 13 and VOIP

A word of warning. I started to see this in my logs today. Im am working on a fix for this but I can’t guarantee it will be ready by Sept 19. Pushes will not work with iOS 13 at the moment.

by Anu at September 16, 2019 02:18

September 15, 2019

Arnaud Joset

New Converse plugin: Verify HTTP requests with XMPP

Converse is a JavaScript XMPP client that can be run in a web browser. Unfortunately, it does not support the XMPP extension (XEP) that allows to verify HTTP Requests via XMPP (XMPP): XEP-0070. So I decided to code a small plugin to provide this functionality.

This article follows the serie about this XEP:

screenshot

How to use it

See the official documentation on how to install this plugin.

If you want to hide the authentication request of your provider, you can set the following option in your converse.initialize function:

hidden: ['auth.example.com']

It will also work with any jid.

You can test the implementation with my demo website: https://demo.agayon.be/

Links

by Arnaud at September 15, 2019 10:30

September 12, 2019

Monal IM

Xmpp+signal+Bluetooth+p2p WiFi = serverless chat

Almost all messaging systems  become useless when you lose access to the internet. What happens when the internet is cut off by a disaster or government?  What if the app is banned  in an app store?  This is something I have been thinking about lately.  Both because of the frequency of natural disasters and also because internet cutoff is a tactic increasingly used by authoritarian governments. The result of this is xmpp+OMEMO (i.e. signal) over airdrop. Since this circumvents the internet and never tries to connect to anything,  it works around national firewalls and internet blocks.  There are multiple layers of encryption here, that offered by Apple with Airdrop as well as application level encryption with OMEMO .

 

To do this,  I hijacked airdrop’s file transfer and automated the process.  However, it works remarkably well. There have been other attempts to do things like this but these are  usually not open source, not a documented protocol like signal and do not work when the app is closed. Additionally,  there is often a dependence on SMS/phone numbers which is an inherently insecure system.  Using airdrop allows for notifications to show up when Monal is closed without going over Apns (Apple’s push system).  There is no connection to Apple or my own push server. 

For now you need to establish your initial key trust with an xmpp server however after that you can go offline and work via airdrop (I recommend setting airdrop  to contacts only to prevent spam). I hope to have qr code based key exchange some point in the future. So the circle of trust can be established quickly in person. You can use an existing server  using the in app registration flow or just run a simple xmpp server like prosody on a Linux vm or raspberry pi.

This is in the latest beta.  To use airdrop mode go to your account and  toggle use air drop. This will take your account offline.  When you send a message, the airdrop share sheet will appear and you will be asked to select the recipient.  

This is text messages only for now. Airdrop mode will create an XMPP payload with a message encrypted with the signal protocol and send that to its recipient. I suggest you set your airdrop receiving settings to contacts only so only those who are your contacts in your iOS contacts app can send you files (in this case messages).

The primary flaw for airdrop that I have seen is your screen has to be on. I am not sure how to get around that right now. Because this is peer to peer, obviously you will also need to be nearby to message as well. This is really intended for small groups in close proximity that need to communicate off the grid.

Try the Monal iOS beta and let me know how it works.

Since Monal is open source, if Monal is blocked for some reason anyone can grab the source on GitHub and build it for themselves and friends. Any version of Monal whether from app store or self compiled will be compatible for messaging.

by Anu at September 12, 2019 13:57

September 09, 2019

Ralph Meijer

XMPP Message Attaching, Fastening, References

Services like Twitter and Slack have functionality that attempts to interpret parts of the plain text of tweets or message as entered by the user. Pieces of the text that look like links, mentions of another user, hash tags, or stock symbols, cause additional meta data to be added to the object representing the message, so that receiving clients can mark up those pieces of text in a special way. Twitter calls this meta data Tweet Entities and for each piece of interpreted text, it includes indices for the start and end of along with additional information depending on the type of entity. A client can then do in-line replacements at the exact character indices, e.g. by making it into a hyperlink. Twitter Entities served as inspiration for XEP-0372: References.

References can be used in two ways: including a reference as a sibling to the body element of a message. The begin and end attributes then point to the indices of the plain text in the body. This would typically be used if the interpretation of the message is done by the sending client.

Alternatively, a service (e.g. a MUC service) could parse incoming messages and send a separate stanza to mark up the original stanza. In this case you need a mechanism for pointing to that other message. There have been two proposals for this, with slightly differing approaches, and in the examples below, I'll use the proto-XEP Message Fastening. While pointing to the stanza ID of the other message, it embeds a reference element in the apply-to element.

Mentioning another user

Let's start out with the example of mentioning another user.

<message from="room@muc.this.example/Kev" type="groupchat">
  <stanza-id id="2019-09-02-1" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>Some rubbish @ralphm</body>
</message>

A client might render this as:

Kev

Some rubbish @ralphm

The MUC service then parses the plain-text message, and finds a reference to my nickname prefixed with an @-sign, and sends a stanza to the room that marks up the message Kev sent to me.

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-2" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-1">
    <reference begin="13" end="19" xmlns="urn:example:reference:0">
      <mention jid="room@muc.this.example/ralphm"/>
    </reference>
  </apply-to>
</message>

This stanza declares that it is attached to the previous message by the stanza ID that was included with the original stanza. In its payload, it includes a reference, referring to the characters 13 through 19. It has a mention child pointing to my occupant JID. Alternatively, the room might have linked to my real JID. A client can then alter the presentation of the original message to use the attached mention reference:

Kev

Some rubbish @ralphm

The characters referencing @ralphm are now highlighted, hovering the mention shows a tooltip with my full name, and clicking on it brings you to a page describing me. This information was not present in the stanza, but a client can use the XMPP URI as a key to present additional information. E.g. from the user's contact list, by doing a vCard lookup, etc.


Note:

The current specification for References does not have defined child elements, but instead uses a type attribute and URIs. However, Jonas Wielicki Schäfer provided some valuable feedback, suggesting this idea. By using a dedicated element for the target of the reference, each can have their own attributes, making it more explicit. Also, it is a natural extension point, by including a differently namespaced element instead.


Referring to previous messages

<message from="room@muc.this.example/Ge0rG"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-3" by="room@muc.this.example"/>
  <reference begin="0" end="6" xmlns="urn:example:reference:0">
    <mention jid="room@muc.this.example/ralphm"/>
  </reference>
  <reference begin="26" end="32" xmlns="urn:example:reference:0">
    <message id="2019-09-02-1"/>
  </reference>
  <body>@ralphm did you see Kev's message earlier?</body>
</message>

Unlike before, this example does not point to another stanza with apply-to. Instead, Ge0rG's client added references to go along with the plain-text body: one for the mention of me, and one for a reference to an earlier message.

Ge0rG

@ralphm did you see Kev's message earlier?

Emoji Reactions

Instead of reacting with a full message, Slack, like online forum software much earlier, has the ability to attach emoji reactions to messages.

<message from="room@muc.this.example/Kev"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
            id="2019-09-02-4" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-3">
    <reactions xmlns="urn:example:reactions:0">
      <reaction label=":+1:">👍</reaction>
    </reactions>
  </apply-to>
</message>
<message from="room@muc.this.example/ralphm"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-6" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-3">
    <reactions xmlns="urn:example:reactions:0">
      <reaction label=":parrot:"
                img="cid:b729aec3f521694a35c3fc94d7477b32bc6444ca@bob.xmpp.org"/>
    </reactions>
  </apply-to>
</message>

These two examples show two separate instances of a person reacting to the previous message by Ge0rG. It uses the protocol from Message Reactions, another Proto-XEP. However, I expanded on it by introducing two new attributes. The label allows for a textual shorthand, that might be typed by a user. Custom emoji can be represented with the img attribute, that points to a XEP-0231: Bits of Binary object.

Ge0rG

@ralphm did you see Kev's message earlier?

👍 2  1

The attached emoji are rendered below the original message, and hovering over them reveals who were the respondents. Here my own reaction is highlighted by a squircle border.

Including a link

<message from="room@muc.this.example/ralphm" type="groupchat">
  <stanza-id id="2019-09-02-7" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>Have you seen https://ralphm.net/blog/2013/10/10/logitech_t630?</body>
</message>
<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-8" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-7">
    <reference begin="14" end="61" xmlns="urn:example:reference:0">
      <link url="https://ralphm.net/blog/2013/10/10/logitech_t630"/>
    </reference>
  </apply-to>
</message>

Here the MUC service marks up the original messages with an explicit link reference. Possibly, the protocol might be extended so that a service can include shortened versions of the URL for display purposes.

ralphm

Have you seen https://ralphm.net/blog/2013/10/10/logitech_t630?

Logitech Ultrathin Touch Mouse

Logitech input devices are my favorite. This tiny bluetooth mouse is a nice portable device for every day use or while traveling.

The client has used the markup to fetch meta data on the URL and presents a summary card below the original message. Alternatively, the MUC service could have done this using XEP-0385: Stateless Inline Media Sharing (SIMS):

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-8" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-7">
    <reference begin="14" end="61" xmlns="urn:example:reference:0">
      <link url="https://ralphm.net/blog/2013/10/10/logitech_t630"/>
      <card xmlns="urn:example:card:0">
        <title>Logitech Ultrathin Touch Mouse</ulink></title>
        <description>Logitech input devices are my favorite. This tiny bluetooth mouse is a nice portable device for every day use or while traveling.</description>
      </card>
      <media-sharing xmlns='urn:xmpp:sims:1'>
        <file xmlns='urn:xmpp:jingle:apps:file-transfer:5'>
          <media-type>image/jpeg</media-type>
          <name>ultrathin-touch-mouse-t630.jpg</name>
          <size>23458</size>
          <hash xmlns='urn:xmpp:hashes:2' algo='sha3-256'>5TOeoNI9z6rN5f+cQagnCgxitQE0VUgzCMeQ9JqbhWJT/FzPpDTTFCbbo1jWwOsIoo9u0hQk6CPxH4t/dvTN0Q==</hash>
          <thumbnail xmlns='urn:xmpp:thumbs:1'uri='cid:sha1+21ed723481c24efed81f256c8ed11854a8d47eff@bob.xmpp.org' media-type='image/jpeg' width='116' height='128'/>
        </file>
        <sources>
          <reference xmlns='urn:xmpp:reference:0' type='data' uri='https://test.ralphm.net/images/blog/ultrathin-touch-mouse-t630.jpg' />
        </sources>
      </media-sharing>
    </reference>
  </apply-to>
</message>

Editing a previous message

<message from="room@muc.this.example/ralphm" type="groupchat">
  <stanza-id id="2019-09-02-9" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>Some thoughtful reply</body>
</message>
ralphm

Some thoughtful reply

After sending that message, I want to add a bit more information:

<message from="room@muc.this.example/ralphm" type="groupchat">
  <stanza-id id="2019-09-02-10" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-9">
    <external name='body'/>
    <replace xmlns='urn:example:message-correct:1'/>
  </apply-to>
  <body>Some more thoughtful reply</body>
</message>

Unlike XEP-0308: Last Message Correction, this example uses Fastening to refer to the original message. I would also lift the restriction on correcting just the last message, but allow any previous message to be edited.

ralphm

Some more thoughtful reply

Upon receiving the correction, the client indicates that the message has been edited. Hovering over the marker reveals when the message was changed.

Editing a previous message that had fastened references

<message from="room@muc.this.example/Kev" type="groupchat">
  <stanza-id id="2019-09-02-11" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <body>A witty response mentioning @ralphm</body>
</message>
<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-12" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-11">
    <reference begin="28" end="34" xmlns="urn:example:reference:0">
      <mention jid="room@muc.this.example/ralphm"/>
    </reference>
  </apply-to>
</message>
Kev

A witty response mentioning @ralphm

After a bit of consideration, Kev edits his response:

<message from="room@muc.this.example/Kev" type="groupchat">
  <stanza-id id="2019-09-02-13" by="room@muc.this.example"
             xmlns="urn:xmpp:sid:0"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-11">
    <external name='body'/>
    <replace xmlns='urn:example:message-correct:1'/>
  </apply-to>
  <body>A slighty wittier response mentioning @ralphm</body>
</message>
Kev

A slightly wittier response mentioning @ralphm

Upon receiving the correction, the client discards all fastened references. The body text was changed, so the reference indices are stale. The room can then send a new stanza marking up the new text:

<message from="room@muc.this.example"
         type="groupchat">
  <stanza-id xmlns="urn:xmpp:sid:0"
             id="2019-09-02-14" by="room@muc.this.example"/>
  <apply-to xmlns="urn:xmpp:fasten:0"
            id="2019-09-02-11">
    <reference begin="40" end="46" xmlns="urn:example:reference:0">
      <mention jid="room@muc.this.example/ralphm"/>
    </reference>
  </apply-to>
</message>
Kev

A slightly wittier response mentioning @ralphm

Closing notes

  • Fastening should also gain a way to unfasten explicitly. I think that should use the stanza ID of the stanza that included the earlier fastening. This allows for undoing individual emoji reactions.

  • Unfastening should probably not use the proto-XEP on Message Retraction. That is for retracting the entire original message plus all its fastened items, and invalidating all message references pointing to it.
  • It might make sense to have a separate document describing how to handle stanza IDs, so that all specifications could point to it instead of each having their own algorithm. In different contexts, different IDs might be used. The other proposal for attachments, XEP-0367: Message Attaching, has a section (4.1) on this that might be taken as a start.

  • In the discussion leading up to this post, a large part was about how to handle all these things attached/fastened to messages in message archives. This is not trivial, as you likely don't want to store a sequence of stanzas, but of (original) messages. Each of those message then might have one or more things fastened to it, and upon retrieval, you want these to come along when retrieving a message. Some of these might be collated, like edits. Some might cause summary counts (emoji, simple polls) with the message itself, and require an explicit retrieval of all the reactions, e.g. when hovering the reaction counts.

    Details on message archive handling is food for a later post. I do think that having a single way of attaching/fastening things to messages makes it much easier to come up with a good solution for archive handling.

  • I didn't provide examples for stanza encryption, but discussions on this suggested that stanzas with fastened items would have an empty apply-to, including the id attribute, so that message archives can do rudimentary grouping of fastened items with the original message.

  • I didn't include examples on Chat Markers, as its current semantics are that a marker sent by a recipient applies to a message and all prior messages. This means the marker isn't really tied to a single message. I think this doesn't match the model for Message Fastening.

by ralphm at September 09, 2019 14:37

September 05, 2019

Monal IM

Mac and iOS fixes for OMEMO

There are new betas for mac and iOS that address many OMEMO issues people have seen. The feedback has been positive and will try to get both out to production soon.

I am also trying to figure out why comments are turned off on this blog now. It wasn’t intentional. WordPress isn’t cooperating with me.

by Anu at September 05, 2019 02:34

September 03, 2019

The XMPP Standards Foundation

The XMPP Newsletter, 03 September 2019

Welcome to the September edition of the XMPP Newsletter!

Loads of news for these two months of July and August: two events in Lyon and Stockholm (more to come), new tools, new server releases (Openfire, ejabberd), lots of client releases (Kaidan, Salut à Toi, Xabber Android, Movim, Converse, Beagle IM and Siskin IM). We also experiment a new section about XEP, or specifications.

Please submit your XMPP/Jabber articles, tutorials or blog posts on our wiki.

Articles

Georg Lukas posted an article about the 10 years of yaxim!

Marek Foss of ProcessOne, details the story on how they have deployed an XMPP geocluster for the FIFA World Cup in Brazil.

Cyril Brulebois wrote a short piece on Sending HTML messages with Net::XMPP (Perl).

Events

Maxime “pep.” Buquet reports on the XMPP sprint in Lyon in July held at Wisolv's offices: DOAP - Description Of A Project, Reactions, Occupant-id, and more.

The next XMPP sprint in Stockholm will be held on September 28th and 29th, in the Nacka Municipality.

A few XMPP hackers met in Ziegeleipark, Mildenberg, Germany in August, for the Chaos Communication Camp 2019, an open-air hacker camp and party that takes place every four years, organized by the Chaos Computer Club (CCC).

The regular meetups in Bavaria is held every third Monday every month. Next will take place 2019-09-16.

If you organise an event in your city, please submit your meetups on the wiki.

Software releases

Servers

Prometheus XMPP Blackbox Exporter (Apache license 2.0) allows you to probe XMPP services and export metrics from the probes to Prometheus.

The igniterealtime community has announced multiple releases:

Antonino Siena has announced xmpp-http-upload, a lightweight and efficient HTTP upload server for XMPP (MIT license).

Mickaël Rémond has announced ejabberd 19.08, with JSON Web Token, configuration validator, improved scalability, and more.

Jérôme "Goffi" Poisson has announced SàT PubSub 0.3.0, a server independent PEP/PubSub XMPP service, which aims to be complete and universal.

Clients

Kaidan 0.4.0 and 0.4.1 have been released, and is available to download on Linux, Windows, and macOS (and experimental Android and Ubuntu Touch).

Jérôme "Goffi" Poisson has released Salut à Toi v0.7 « La Commune » with tons of changes (and writes regular SàT progress notes).

Xabber Android 2.6.4 (634) has been released on Google Play, with improved message archive synchronization, improved startup speed, support of references in messages (files, forwards, markup (bold, italic, etc), mentions, quotes), image compression setting, visual changes, and more.

Timothée Jaussoin has released Movim 0.15 – Donati, with reactions, publication sharing, and more.

JC Brand has released Converse, the web based XMPP/Jabber chat client, in versions 5.0.0 and 5.0.1, with lots of improvements.

Wojciech Kapcia, of Tigase, has announced two versions of Beagle IM from macOS and Siskin IM for iOS.

Libraries

Lance "legastero" Stout has released StanzaJS (formely known as Stanza.io), the JS XMPP library with a JSON API, in multiple versions 12.x.

JC Brand has released the strophejs library in version 1.3.4.

Florian "Flow" Schmaus has announced the extension of the jXMPP library by a testframework for “XMPP-Strings”, as well as Smack 4.4.0-alpha2.

Services

Sadly, Arc Games has announced an XMPP sunset: "As of September 19th, 2019, Star Trek Online will no longer be supporting connection to our internal chat system via an XMPP client."

Extensions and specifications

This section of the newsletter is new and experimental: please let us know what you think.

We start with the XEPs in "LAST CALL", since these are the highest priority in terms of feedback. We move on to the new XEPs so you can learn of their existence, and then the obsoleted ones so you phase them out. We finally finish with the updates. On more section: the friendly specifications.

Last Call

XEP-0353: Jingle Message Initiation

Title: Jingle Message Initiation

Abstract: This specification provides a way for the initiator of a Jingle session to propose sending an invitation in an XMPP message stanza, thus taking advantage of message delivery semantics instead of sending IQ stanzas to all of the responder's online resources or choosing a particular online resource.

URL: https://xmpp.org/extensions/xep-0353.html

This Last Call begins today and shall end at the close of business on 2019-08-13.

XEP-0300: Use of Cryptographic Hash Functions in XMPP

Title: Use of Cryptographic Hash Functions in XMPP

Abstract: This document provides a common wire format for the transport of cryptographic hash function references and hash function values in XMPP protocol extensions.

URL: https://xmpp.org/extensions/xep-0300.html

This Last Call begins today and shall end at the close of business on 2019-08-13.

New

XEP-0421: Anonymous unique occupant identifiers for MUCs

Version 0.1.0 of XEP-0421 (Anonymous unique occupant identifiers for MUCs) has been released.

Abstract: This specification defines a method that allows clients to identify a MUC participant across reconnects and renames. It thus prevents impersonification of anonymous users.

Changelog: Accepted by vote of Council on 2019-07-17. (XEP Editor (jsc))

URL: https://xmpp.org/extensions/xep-0421.html

XEP-0420: Stanza Content Encryption

Version 0.1.0 of XEP-0420 (Stanza Content Encryption) has been released.

Abstract: The Stanza Content Encryption (SCE) protocol is intended as a way to allow clients to securely exchange arbitrary extension elements using different end-to-end encryption schemes.

Changelog: Accepted by vote of Council on 2019-06-26. (XEP Editor (jsc))

URL: https://xmpp.org/extensions/xep-0420.html

Obsoleted

XEP-0387: XMPP Compliance Suites 2018

Version 1.0.0 of XEP-0387 (XMPP Compliance Suites 2018) has been released.

Abstract: This document defines XMPP protocol compliance levels.

Changelog: Move to Draft as per Council vote on 2018-01-24. (XEP Editor (jwi))

URL: https://xmpp.org/extensions/xep-0387.html

Updated

  • Version 0.3.0 of XEP-0414 (Cryptographic Hash Function Recommendations for XMPP)
  • Version 1.1.0 of XEP-0368 (SRV records for XMPP over TLS)
  • Version 0.3.0 of XEP-0380 (Explicit Message Encryption)
  • Version 1.4.0 of XEP-0184 (Message Delivery Receipts)

Friendly specs

This one may be of interest to this community; it's in Last Call at the IETF currently: https://datatracker.ietf.org/doc/draft-ietf-tls-sni-encryption/

This is it

This edition was long, we hope you loved it.

You can also show your support by relaying it online, on your blogs, newsletters, or social media.

by nyco at September 03, 2019 07:00

September 02, 2019

João Duarte

Final GSoC Report - Prosody Plugin Installer!


Here we are. This post is the conclusion of the official GSoC journey with prosody! Check out below what this project is, what it is currently doing and how things can keep on going!

First of all, where is the stuff?

The code is right here! This is everything that made it to the trunk merge. If you want to see the installer in action and how to use it, you can check this demonstration.  If you want instead to see all of the changesets I've made throughout the months, you can check them here. Finally, here you have the project's link in case you want to check it out!

Really briefly, what was this all about again?

ProsodyIM is a XMPP server. People can set it up with a lot of different functionalities. These are provided by what the community calls modules or plugins. Prosody only comes with a given set of core modules but there are many others that we can use. We aimed at providing an easy way to install those, because previously, the user had to go to a mercurial repository and fetch the sources that he wants to use.

Cool, so what has been done?

Prosody makes use of a command tool, prosodyctl, that allows a user to run a series of commands for a multitude of purposes. The installer is run through it, so I added 3 new commands for installation, removal and listing of modules. The user no longer has to go to the web, fetch the sources and manually place them around. All of that is now taken care of by the install and remove commands. Through the list command, the user can check which third-party modules are currently installed.

What didn't made it through?

Most of the code that as removed was either:
- badly written, and therefore had to be redone in a better way
- experimental, to help me understand how everything was working
- Development code, which i was using to help me accomplishing any given task

Only one feature was trully remove, and that was the --tree flag. It allowed the user to specify a location where the installer would operate. An explanation on why it was removed is available at this this blog post.

And what can be improved?

Here is a list of tiny things that could still be neat to add:
- Update information. Right now, the user can update the current module by simply installing it again, but there is no info being printed about what is changing;
- Automatically configure the module, when possible. Tricky, has different modules have different configuration options. But at least adding the module to the modules enabled or disabled list could be a nice little extra
- List of available modules to install. The user still has to go to the web to see which modules exist and their documentation. It would be neat if that could be done from the command line.

Conclusion

Prosody now has an installer that makes the server a bit more user-friendly, I believe. For a newbie like me, these months have been great, but I also feel underwhelming in a way, at the end. I haven't wrote much, and most of the time I was just questioning things around and diving into train of thoughts, because everything seems so big and complicated at times. But hey, It is done anyway! It is simple, but it works, and it is a step forward! 
I hope this software can become a real improvement. At least for me, It was. I also liked the blogging experience, and I hope I can keep contributing in the future and make my work more consistent and solid.

Aknowledgements

I'd like to thank my mentor, Matthew "Matt" Wild \o, for the patience and guidance! He had a child through the project, so be sure to give him the good wishes =D
Kim "Zash" Alvefur o/ also helped me out all the time, so a bit shout-out to him!
The people at XMPP, who were always friendly. Special mention to Florian "Flow" Schmaus, who was always supportive, and to Paul "vanitasvitae" Schaub, who inspired me about how to blog!
I've also like to say thanks to the folks at Luarocks, particularly Thijs Tieste, who patiently explained me some stuff when I was lurking around their Gitter =D
Need to also appreciate Zerobrane's Paul Kulchenko, who always answered me when I was having trouble using the great Zerobrane's Studio IDE.
And finally, to the other GSoC students at our chatrooms, who sometimes showed me how much of a fool I am and also how people can battle with Pokemon on the web nowadays =P

by João Duarte (noreply@blogger.com) at September 02, 2019 16:00

ProcessOne

Real-time Stack #25

ProcessOne curates the Real-time Radar – a newsletter focusing on articles about technology and business aspects of real-time solutions. Here are the articles we found interesting in Issue #25. To receive this newsletter straight to your inbox on the day it is published, subscribe here.

Breaking 25 Trillion Notifications for the BBC

In 2012, BBC needed to add a push notification service to its portfolio of mobile applications, with main focus on BBC News and BBC Sports. At the scale of the BBC, there was several key requirements that made it difficult to use an off-the-shelf application.

Thoughts on Code Style in Library Design

XMPP protocol is flexible and contains many extensions. It can be daunting to get into XMPP so it’s very tempting to add new tools to make it more convenient for newcomers to get into the protocol.

Next Steps Toward Go 2

The release of Go 1.13 is on its way, hopefully in early August of this year. This is the first release that will include concrete changes to the language (rather than just minor adjustments to the spec), after a longer moratorium on any such changes.

Scaling from 2,000 to 25,000 Engineers on GitHub at Microsoft

At Microsoft today there are almost 25,000 engineers participating in the official GitHub organizations for open source, a great number of them contributing to open source communities throughout GitHub.

Air Pollution Tracking with Node.js, Elastic Stack, and MQTT

What can you do with a couple of IoT devices, Node.js, Elasticsearch, and MQTT? You can put together your own Internet of Things setup for measuring air pollution, like I have.

The State of Developer Ecosystem in 2019

Go started out with a share of 8% in 2017 and now it has reached 18%. In addition, the biggest number of developers (13%) chose Go as a language they would like to adopt or migrate to.

Merging Internet of Things and Blockchain in Preparation for the Future

The ‘Big Four’ technologies of the future are beginning to force themselves to the fore as companies start to dabble in the likes of Artificial Intelligence (AI), Blockchain and Internet of Things (IoT), as well as Big Data.

by Marek Foss at September 02, 2019 08:50

August 30, 2019

Jérôme Poisson

SàT progress note 2019-W35

Hi everybody,

I've skipped last week note because I could not find the time, and anyway I did the same thing as this week, so it was not really needed.

I'm currently working on porting jp (the CLI frontend) to Python's asyncio, and at the same time I'm documenting every command (I need to review them one by one, so it's a good occasion for that). It's long and really boring, but it's a good thing to do.

Let me explain why I'm doing that. SàT and its frontends are historically using D-Bus to communicate (it's an IPC, and we call it internally the bridge). SàT is 10 years old, and the Python implementation that we have used was the reference at the time : Python D-Bus. This implementation has flaws, that they recognise themselves (like trying to guess signature, see " Problems and alternatives " in the page linked). Furthermore, it's the main reason why we use GLib and its loop in SàT, which can cause trouble to install (mainly due to compilation).

Frontends and bridges are using a callbacks mechanism, which can be hard to code and maintain in some cases. Python-dbus can work with callbacks or in a blocking way, and historically we have used this feature in jp. For instance, to get backend version we have a getVersion() method, and to avoid too much callbacks hell, it was used with the bridge in a blocking way (blocking jp is not such a big deal as there is no UI to maintain or things which must run in background). This was wrong and experience shows why, you are either blocking or asynchronous, not a mix. Using D-Bus bridge in a blocking way, even if it's only in a few places, makes the code unusable with other bridges, so jp could only work with D-Bus, and today we have other ones (like Twisted Perspective Broker which is the bridge used on Android).

So in one hand we had the callback issue, and on the other hand some bridge calls using blocking mode to limit callbacks hell, and causing design issues.

The proper solution to that is to use coroutines, which is a way to write asynchronous code nearly like we write blocking code. Twisted has been offering this possibility for years thanks to the inlineCallbacks which is using Python generators in a smart way. It's really usable and indeed used in many parts of the backend: you basically have to use a decorator and yield syntax.
This could have been used in jp but this would have forced the use of Twisted (and we want jp to work with other frameworks if other bridges are used). Furthermore, inlineCallbacks are not easy to debug: in a debugger if you don't add a breakpoint just after the yield, a step over brings you in the execution workflow of the main loop.

And here come asyncio and the great async/await syntax. This syntax is doing basically the same thing as Twisted inlineCallbacks, but in a cleaner and more readable way, in locations where yield is not possible (in a loop for instance), and it is nowadays well handled by debuggers (at least pudb that I'm using). Thanks to that, it is possible to convert easily blocking parts of the code, and to simplify many things.

Doing something like repeating a method on several pages of result (when RSM is used for instance) becomes easy and pleasant. One of the main goal of asyncio was to have a common loop/framework for all asynchronous code, and Twisted made the compatibility layer so we can use Twisted and asyncio at the same time with async/await syntax.

I'm currently finishing that, for the moment I'm keeping 2 versions of the bridge (one callbacks based and one using async/await) to progressively do the conversion, but the goal is to have all frontends using the new syntax soon. It is one of the many things that has been unlocked by the port to Python 3. It's taking lot of time because of the documentation made at the same time, but it is definitely worth it.

That's all for this note, see you next week.

by goffi at August 30, 2019 16:02

August 28, 2019

Ignite Realtime Blog

inVerse Openfire plugin 5.0.1.1 released

@wroot wrote:

The Ignite Realtime community is happy to announce the release of version 5.0.1.1 of the inVerse plugin for Openfire!

This update brings Converse up to 5.0.1 version, which had a lot of changes and fixes.

Your instance of Openfire should automatically display the availability of the update. Alternatively, you can download the new release of the plugin at the inVerse plugin’s archive page

For other release announcements and news follow us on Twitter

Posts: 1

Participants: 1

Read full topic

by @wroot wroot at August 28, 2019 20:32

August 26, 2019

ProcessOne

Real-time Enterprise #24

ProcessOne curates the Real-time Radar – a newsletter focusing on articles about technology and business aspects of real-time solutions. Here are the articles we found interesting in Issue #24. To receive this newsletter straight to your inbox on the day it is published, subscribe here.

ejabberd & Nintendo Switch NPNS

Last week, Taiyo Watanabe from Nintendo shared some insights about Nintendo Switch Push Notification infrastructure (NPNS) implementation. Here are some details from his presentation at ElixirFest conference. The Nintendo Switch NPNS is a large-scale realtime messaging service based on ejabberd.

80 Mind-Blowing IoT Statistics

In 2016, $737 billion was spent on the IoT. Global spending associated with the IoT in 2016 amounted to $737 billion. According to ACT, an organization that represents more than 5,000 tech companies, the healthcare IoT market alone will be worth more than $117 billion by 2020.

What is MQTT in IoT

Internet of Things (IoT) and machine-to-machine (M2M) technologies need to use a messaging and connectivity protocol in order to exchange information from a remote location.

Modern Data Transmission for Industry 4.0

A fully-networked factory floor is the future of automation and manufacturing. Robots, cobots, and information going into and out of the cloud are helping us create better products faster and with more accuracy.

Starting with MQTT using Docker and Home Assistant

MQTT is a great protocol to have in your automation arsenal. This article explains how to quickly get started with MQTT using Docker and Home Assistant. MQTT, which stands for Message Queuing Telemetry Transport, is often described as the protocol of the Internet of Things (IOT).

Integrating IoT with IEC 61850

Internet of Things continues to be one of the key technology trends in the recent years. According to Gartner’s estimations, IoT network will grow from 8,7 billion in 2017 to over 20 billion connected things by 2020.

Fight Against Unprotected s2s Connections

In the past, the certificates were worth the money. In the past, obtaining certificates was a difficult task. This forced the use of self-signed certificates in connections to preserve the federation. Invalid certificates are dangerous because unencrypted (without e2e) user communications can be intercepted and modified.

by Marek Foss at August 26, 2019 14:43

hrxi

End of the Google Summer of Code

This Google Summer of Code was about adding peer-to-peer file transfers to Dino. Dino is an XMPP client, XMPP is decentralized instant messaging standard.

The work was mostly done in two larger pull requests. The first one, dino#577, added a Jingle framework (XEP-0166), the in-band-bytestream transport (XEP-0047, XEP-0261) and the Jingle file transfer protocol (XEP-0234).

Jingle is a protocol for (mostly out-of-band) peer-to-peer connections between XMPP clients. It is designed to support multiple transport types, e.g. in-band bytestreams, connections via SOCKS5 proxies or direct IP connections. It provides support to send different kinds of data between clients, the two major users are file transfers (described below) and audio/video calls. All this is reflected in the code, the Jingle interface was designed to be extensible for more transport methods and different data types.

In-band bytestreams are a simple method of exchanging binary data between XMPP clients by sending base64-encoded data in the XMPP stream via the server. It is a good fallback method in case no other transport works, but adds 33% overhead to the data sent.

Apart from the HTTP file upload specification, the Jingle file transfer protocol is another standard way of exchanging files between XMPP clients. For Dino (and also in Conversations), this method is used transparently (using the same UI) for files larger than the upload limit of the HTTP upload server. This allows the user to benefit from both methods.

The work on this pull request was concluded by some interoperability testing with Conversations and Gajim, two other major XMPP clients.

The second pull request, dino#592 (not merged yet, currently on commit 6028fd1) added the SOCKS5 transport method for Jingle (XEP-0065, XEP-0260).

The SOCKS5 transport method allows XMPP clients to exchange data via a direct connection or via a XMPP-server provided SOCKS5 proxy. The direct transfer only works if one client can see the other client’s IP address and is not firewalled, which is mostly the case if both the clients are on the same private network. It is currently not implemented and is waiting for some libnice support for enumerating the local interfaces (see below). Instead, the data can be exchanged via a proxy reachable by both clients, determined by some candidate exchange.

Interoperability testing for the second pull request uncovered some old and new bugs in both existing clients and my implementation. Thanks to the Conversations developer for taking my bug reports seriously and responding very fast.

There were also a couple of other smaller pull requests which I’ll mention here:

  • dino#569. Tiny fix for logging to a file.
  • dino#570. Tiny typo fix, sorry for even mentioning. :o
  • dino#579. Improvement around the error generation in response to certain XMPP messages.

In the course of the interoperability testing, I also opened a couple of issues on other XMPP clients: Conversations#3478, Conversations#3500, Conversations#3514, Conversations#3515, Gajim#9784.

As the next steps, after Google Summer of Code, I’ll implement the WebRTC transport method to get interoperability with XMPP browser clients that cannot open direct TCP connections to SOCKS5 servers. Additionally, one could implement audio/video calls on top of the current Jingle code.

At this point I’d like to thank my two mentors fiaxh and mar-v-in without whom this wouldn’t have been possible. They were quick to answer any questions that arose, about architectural design of the code, the XMPP standards or even just questions about Vala, the language Dino is written in.

Thanks to my mentoring organization, the XSF (XMPP standards foundation), as well, for the weekly meetings and generally the advice given therein.

by hrxi at August 26, 2019 00:00

August 23, 2019

yaxim

Happy Birthday, yaxim! 10 Years of Android XMPP

A decade ago, today, the first yaxim commit was created, making it now officially half as old as XMPP. Since then, much has happened both in the XMPP ecosystem and on the Android side of things.

2009: The Beginning

Back in 2009, the Android platform was still brand new, and it was lacking a Free IM client. There were rumors and announcements, but nobody had posted working code yet.

The first specific hint was a German slide deck by Sven and Chris, presenting their semester project, YAXIM - Yet Another XMPP Instant Messenger.

Some friendly emails later, a GitHub project got created, and the coding went on. At the end of the year, a lightning talk was given at the 26C3, showing how yaxim looked back then:

26C3 slide: state of affairs

26C3 slide: screenshots

In the early days, it was a huge challenge to reliably deliver messages, and things were improving only slowly.

Significant Steps

In 2010, YAXIM got renamed into yaxim, to look more like a name and less like a yelling acronym.

In 2013, Bruno was launched as yaxim’s cute little sibling, to appeal to children and everybody who loves animals. It has attained almost 2000 active fans by now.

Also in 2013, the yax.im XMPP service was launched, mainly to make on-boarding from yaxim and Bruno easier, but also to have a stable and reliable service usable for mobile clients.

Finally, in 2016, yaxim got its current yak-oriented logo.

yaxim feature graphic

A Review In Numbers

From day 1, yaxim was more or less a hobby project, with no commercial backing and no full time developers. Over the years, the code base has grown slowly, with 2015 being an especially slow year.

yaxim commit history

Some contributors were more active than others… 😉

yaxim Lines of Code history

Even though yaxim still has more total installs on Google Play than Conversations, the latter is said to be the go-to client on Android, and very popular among federation nerds. Still, at least in the last three years, there was no decline in the number of devices that yaxim is installed on (Google doesn’t provide stats before 2016):

3 years of devices

Current Challenges

The technical foundation of yaxim (Smack 3.x, ActionBarSherlock) became rather dated, and currently much effort is going into making yaxim look great on modern (Material design) Android devices, and to support modern features like interactive permission dialogs and battery saving, as well as the Matrix (with the support being currently dormant.

Please check out the Google Play beta channel to stay up-to-date with the most recent developments, and let’s see how the next decade will develop!

August 23, 2019 10:43

Erlang Solutions

XMPP Protocol Use-Cases and Guide | Erlang Solution blog

Who will find this interesting

If you’re considering XMPP for your project but you are unsure if it can provide the functionality you need, you’ll eventually end up here:

http://xmpp.org/extensions/

I’m pretty sure you’ll be quite intimidated by such a long list of extensions. In some cases it will be pretty easy to find what you need. If you look for PubSub functionality, you’ll quickly notice “Publish-Subscribe”. Sometimes it’s not so obvious though. XMPP developers already know that in order to synchronise outgoing messages between several devices, they have to enable “Message Carbons”. Not very intuitive, isn’t it?

The aim of this blog post is to guide you towards proper XMPP technologies and solutions, given your specific use cases. I’ve worked with and deployed solutions powered by XMPP, such as MongooseIM, for years; so let me be your personal Professor Oak, providing a perfect “companion(s)” to work with and begin your journey in XMPP world. There are almost 400 XEPs, will you catch them all? ;)

The length of this article is caused not by a complexity of descriptions but by a count of use cases and features. :)

All numbers and information on implementation status are valid for March 2017.

What can you expect here?

For every use case, I will list XMPP features you definitely should consider using. Each one of them will be briefly described. The goal here is to understand the usefulness without reading whole specification. Besides that, each item will include MongooseIM’s module name providing discussed extension and example client implementations.

What you won’t find in this post

This post won’t cover any XMPP basics. It assumes you either know them already (what are JID, C2S, S2S, IQ, stanzas, stream etc.) or you intend to learn them from some other guide, like the excellent (iOS) tutorial written by Andres Canal Part 1, Part 2). It’s more of a cookbook, not Cooking For Dummies.

ToC

  1. I’m creating …
    1.1 … a mobile application.
    1.2 … a desktop application.
    1.3 … a web application.
    1.4 … an application that just can’t speak XMPP.
  2. I need my application to …
    2.1 … show message status like Facebook does.
    2.2 … provide message archive to end users.
    2.2.1 I’d like to have a full text search feature.
  3. … display inbox (a list of conversations with unread count and a last message).
  4. … allow file transfers and media sharing between users.
    4.1 P2P
    4.2 File Upload
  5. … support groupchats …
    5.1 … and I need precise presence tracking in each group.
    5.2 … and I don’t need to broadcast presence information in each group.
  6. … be compatible with other public XMPP setups.
  7. … present the same view of each conversation on every user’s device.
  8. … allow users to block each other.
  9. … support end-to-end encryption.
  10. … be a part of Internet of Things.
  11. … receive push notifications.
  12. … publish messages to groups of subscribers.

1. Creating …

Before we proceed to more specific requirements, it’s important to identify crucial standards based on your application type.

1.1 … a mobile application.

Smartphones are omnipresent nowadays. It’s a fact. The whole software market considered, mobile apps are an important medium between various companies and their customers. Some of them are the actual products (games, communicators, car navigations, etc.), not only a “channel”. If you’re going to develop a mobile application, you will need…

XEP-0198 Stream Management

It’s an extension that provides two features actually. One of them is stanza delivery confirmation (both server and client side), what allows early detection of broken connections or malfunctioning network layer. The other one is stream resumption. It makes reconnection faster by reducing the round-trip count and relieves the client of fetching message archive as pending, unacknowledged messages will be retransmitted from server buffer.

It is enabled by default in MongooseIM and supported by major client libs like Smack or XMPPFramework. From a client developer perspective, it’s pretty transparent because the whole extension is enabled with a single flag or method call.

MUC Light, MIX, Push notifications, HTTP File Upload

These extensions are especially useful in the mobile environment. Why? With MUC Light and MIX you gain control over presence broadcasting - you can spare your users frequent radio wakeups and bandwidth usage. These extensions are a significant improvement over traditional presence-driven group chats.

Virtually every app on our smartphones uses push notifications. Some are useful and some are just annoying commercials. It doesn’t matter - it’s almost certain you’ll want them integrated with your XMPP service.

HTTP File Upload allows asynchronous media sharing, which is much more convenient in the case of group chats and doesn’t require both parties to stay online during the transfer.

These are just brief summaries. You can find more details further in this post.

1.2. … a desktop application.

Despite mobile phones’ expansion and software products exclusive for them (Instagram, Snapchat, Tinder, etc.), nobody can deny the comfort of UI operated with a mouse, keyboard, or tablet. Some apps simply require processing power that portable devices can’t provide. If your code is going to be executed on desktops PCs and laptops, you’ll appreciate…

There are no extensions that are strictly essential for desktop apps. Everything depends on specific applications. Just bear in mind that the standards important for mobile apps are generally useful for desktop ones too, only less critical.

1.3. … a web application.

As the days of heavy browser incompatibility (thank you, standardisation!) and Flash technology abuse are long gone, web applications are a great way to provide cross-platform solutions. It’s not only easier to reach more platforms but also to ensure the users are always running the most up-to-date version.

If you’re a web developer, you’re going to connect to the XMPP server via BOSH or Websockets.

Websockets

Websockets technology allow to upgrade an HTTP connection to an asynchronous, full-duplex, binary one (a bit of simplification but it’s the essence). It means that XMPP stanzas can be exchanged almost as efficiently as over a raw TCP connection (Websockets add small overhead of integer with packet size). It’s the recommended protocol for single-page apps.

Note: You can combine Stream Management’s resumption with Websockets, although it will still be slower than BOSH’s session pause.

Warning: Websockets are not implemented by old browsers. If you have to support any outdated clients, take a look at this table first.

BOSH

Defined in XEP-0124: Bidirectional-streams Over Synchronous HTTP (BOSH) and XEP-0206: XMPP Over BOSH. This protocol encapsulates XMPP stanzas in HTTP requests. It also simulates asynchronous, bidirectional communication by issuing long polling requests from client to the server to retrieve live data. What does it mean in practical terms?

A persistent connection may be maintained but in general BOSH is designed to deal with interrupted connections. It’s a perfect solution for web apps that trigger browser navigation. On such event, all connections made by e.g. JavaScript from browser are closed but the BOSH session survives it on the server side (not forever of course) and the client can quickly and efficiently resume the session after page reload.

The protocol is pretty verbose though, so if you don’t need this feature, go for Websockets.

1.4. … an application that just can’t speak XMPP.

You probably think that I’m crazy; why use XMPP with XMPP-less clients? Let’s change the way we think about XMPP for a moment. Stop considering XML the only input data format the XMPP server accepts. What if I told you that it’s possible to restrict XML to the server’s routing core and just make REST calls from any application? Tempting?

It’s a non-standard approach and it hasn’t been documented by XSF (yet), but MongooseIM already exposes most important functionalities via REST. Check out this and this document to find out more.

2. I need my application to …

Now we continue to more specific use cases.

2.1. … show message status like Facebook does.

By message status we mean following states (plus live notifications):

  1. Not sent to server yet.
  2. Acknowledged by the server.
  3. Delivered to the recipient.
  4. Displayed by the recipient.
  5. User is composing a message.
  6. User has stopped composing a message.

(1) and (2) are handled by Stream Management. It’s pretty obvious - before receiving an ack from the server, you are in (1); and ack confirms the message entered state (2).

We can deal with (3) and (4) by using XEP-0333: Chat Markers. These are special stanzas sent by a recipient to the original sender. There are dedicated markers for received and displayed events.

(5) and (6) are provided by XEP-0085: Chat State Notifications. It is up to a client to send updates like <composing/> and <paused/> to the interlocutor.

2.2. … provide message archive to end users.

Virtually every modern chat application maintains conversation history both for 1-1 communication and group chats. It can remind you of a promise you’ve made, be evidence in a divorce case, or help in police investigation.

XMPP provides two protocols for accessing message archives. The older one, XEP-0136 Message Archiving is used by hardly anyone, because it’s difficult to implement and overloaded with features. It has been superseded by more modern XEP-0313 Message Archive Management, which is the current standard.

There is one caveat though - its syntax changed significantly between versions, so it’s common for libraries and servers to explicitly state what versions are supported by the specific piece of software. These are 0.2, 0.3 and 0.4(.1) and 0.5. MongooseIM supports all of them in mod_mam module. If you choose another server, make sure its MAM implementation is compatible with your client library. Smack and XMPPFramework use 0.4 syntax.

2.2.1. I’d like to have a full text search feature.

Although standard Message Archive Management doesn’t specify any queries for full text search, it remains flexible enough to create such requests on top of the existing ones.

In MongooseIM this feature is still in experimental phase and has been recently merged into master branch. It’s not supported in any client library yet, so you have to construct a custom MAM query to do full text searches. Take a look at the PR description, It’s not that difficult. :)

2.3. … display inbox (a list of conversations with unread count and a last message).

Unfortunately there are no open solutions providing this feature. XMPP community is in the process of discussing and creating the specification of Inbox functionality. Erlang Solutions is designing a XEP proposal, which you can view here.

A quasi-inbox is available as a part of experimental standard Bind 2.0. It doesn’t cover all possible use-cases but a list of unread messages is what you actually need for optimal UX after establishing a connection. This feature is already under development in MongooseIM project.

In the meantime, you can build an inbox view by persisting last known archived message ID or timestamp and query Message Archive Management for all messages that came later. When you fetch them all, you can build an inbox. Unfortunately this is not very efficient and that’s why the community needs a new standard.

2.4. … allow file transfers and media sharing between users.

Almost everyone loves to share cat pictures and every modern IM solution provides means to do this. Various file transfer techniques in the XMPP world can be grouped in two categories: P2P connections and file upload.

The former involves establishing a direct connection between two clients, sometimes with a bit of a help from a TURN server. It ensures that data won’t get stored on any intermediate servers. Obviously, it requires less effort from the service provider because it’s easier and cheaper to set up a TURN service than to maintain a proper media server (or pay for storage in the cloud).

File upload is much more efficient when sharing media with a group. It doesn’t require both parties to remain online for the transfer duration.

2.4.1. P2P

Now, you DO have a choice here. There are a couple of XEPs, describing various P2P transfer initiation methods. XEP-0047 In-Band Bytestreams (IBB) is guaranteed to work in every network, because it sends data (Base64-encoded) via IQs. So if you can reach the XMPP service, you can transfer files. It may be slow and not very convenient but it will work.

Let’s carry on. You can transfer media via bytestreams external to XMPP. The P2P session is negotiated via XMPP but it’s only the “signalling” part. There are quite a few XEPs describing various negotiation and transmission protocols, so I will highlight specific implementations rather than listing all of the names which would only confuse readers who just want to send some bytes.

  • XMPPFramework: Look for XMPPIncomingFileTransfer and XMPPOutgoingFileTransfer. They support SOCKS5 and In-Band Bytestreams.
  • Smack: Everything begins with FileTransferManager. It supports SOCKS5 and In-Band Bytestreams as well.

2.4.2. File Upload

Unless you already have a dedicated media server that exposes an API to perform uploads and downloads, you should definitely take a look at XEP-0363 File Upload. It defines standard stanzas to request upload slots and respective download links. It is XMPP server’s responsibility to allocate the slots and return the links to the client.

Unfortunately this extension is not widely supported yet. You can find it in XMPPFramework but not in Smack yet. In the case of MongooseIM, it’s already available with Amazon S3 backend (with more storage plugins to come!).

2.5. … support group chats …

A couple of years ago it was really simple - there was only one kind of group chat supported in the XMPP world. Today we have three standards, two of them being maintained by XSF and one published by Erlang Solutions. MIX (XEP-0369), doesn’t have any implementations yet and as a standard it changes very frequently, so it is not described in this post.

2.5.1. … and I need precise presence tracking in each group.

If you need IRC-like experience where users have certain roles in a room and client disconnection triggers leaving the room, then classic XEP-0045 Multi-User Chat will work for you. It has its disadvantages (frequent presence broadcast may impact UX and consume processing power or connection throughput) but fits the use case, where accurate presence information is important. It is provided by MongooseIM’s mod_muc (other major servers implement it as well) and is supported by all mainstream client libs.

2.5.2. … and I don’t need to broadcast presence information in each group.

Erlang Solutions’ Multi-User Chat Light is a protocol derived from real world use cases, where groups doesn’t care about presences and full member list is always available to room members. It has some strong assumptions (like only 2 types of affiliation or rooms being joinable only by invite) but is designed to reduce round-trips, expose powerful API (e.g. room creation + configuration + adding new members in one request) and be easy to work with. Check it out and see if it fits in your application. Server implementation is currently exclusive to MongooseIM (mod_muc_light) and respective plugins are available in Smack and XMPPFramework.

2.6. … be compatible with other public XMPP setups.

Even some proprietary installations do integrate with open XMPP world (like GTalk and Facebook at some point), so if this is your use case as well, the first important thing to remember is that no custom stanzas may leave your cluster. By custom I mean anything that is not covered by any XSF-approved XEP. Additionally, you will really benefit from using XEP-0030 Service Discovery protocol a lot, because you can never be sure what is the supported feature set on the other end. It is used to query both clients and servers. Virtually every client and server supports it. In case of MongooseIM, the base module is mod_disco.

2.7. … present the same view of each conversation on every user’s device.

I use Facebook messenger on multiple devices and I really expect it to display the same shopping list I got from my wife on both my desktop and my mobile phone. It usually breaks message order but anyway - at least the list is there.

The problem is actually a bit more complex, because you have to take care of synchronising both online and offline devices.

Online devices can ask the server to forward all incoming/outgoing messages, even if they originate from or are addressed to some other resource of the same user. It is achieved by enabling XEP-0280 Message Carbons. On the client side it’s easy - just enable the feature after authenticating and the server will do the rest. It’s supported by MongooseIM in mod_carboncopy module. You can find respective implementations in Smack, XMPPFramework, Stanza.io and many others, since it’s a very simple, yet powerful extension.

If you want to fetch everything that happened while a specific device was offline for a while, just query XEP-0313 Message Archive Management (see “… provide message archive to end users.” section).

2.8. … allow users to block each other.

You just can’t stand your neighbour nagging you via IM to turn down the volume while Kirk Hammett is performing his great solo? Block him. Now. XMPP can help you with it. In two ways actually.

Yes, XMPP features two standards that deal with blocking: XEP-0016 Privacy Lists and the simpler XEP-0191 Blocking Command. The former allows users to create pretty precise privacy rules, like “don’t send outgoing presences to JID X” or “accept IQs only from JIDs in my roster”. If you need such a fine grained control, take a look at MongooseIM’s mod_privacy. On the client side it is supported by the likes of Smack and XMPPFramework.

Blocking Command is much simpler but most setups will find it sufficient. When a client blocks a JID, no stanza will be routed from the blockee to the blocker. Period. MongooseIM (mod_blocking), Smack and XMPPFramework have it.

2.9. … support end-to-end encryption.

When Alice wants to send a message to Bob… no, we’ve all probably seen this classic example too many times already. :)

There is no “one size fits all” when it comes to E2E encryption. The first tradeoff you’ll have to make is to decide whether you want new users devices to be able to decrypt old messages, or do you prefer to have a property of forward secrecy. For a full comparison between available encryption methods, let’s take a look at the table published by OMEMO authors:

Legacy Open PGP Open PGP OTR OMEMO
Multiple Devices Yes Yes No Yes
Offline Messages Yes Yes No Yes
File Transfer Yes Non-standard Non-standard Yes
Verifiability No Yes Yes Yes
Deniability Yes No Yes Yes
Forward Secrecy No No Yes Yes
Server Side Archive Yes Yes No No
Per Message Overhead High High Low Medium

It’s difficult to find an open library that supports any of these methods. Gajim communicator has an OMEMO plugin. Smack and XMPPFramework don’t support E2E encryption in their upstream versions. If you’re going to use E2E encryption in your application, most probably you’ll have to implement it on your own. Good thing is there are standards you can base your code on.

2.10. … be a part of Internet of Things.

We are a peculiar bunch. We use semiconductors to build machines that do heavy number crunching for us, deliver messages in a blink of an eye and control robotic arms with precision far beyond ours. A desire has awoken in us to go even deeper and augment everything with electronics. To let everything communicate with each other.

If you’re designing a fridge microcontroller that is supposed to fetch results from bathroom scales and lock the door for 8h for every excessive BMI point, you’ll need…

  • XEP-0323 Internet of Things - Sensor Data
  • XEP-0324 Internet of Things - Provisioning
  • XEP-0325 Internet of Things - Control
  • XEP-0326 Internet of Things - Concentrators
  • XEP-0347 Internet of Things - Discovery

Unfortunately there are no public implementations of these standards. I wish it was easier but it seems you just can’t avoid reading these XEPs, picking the most suitable parts and creating your own implementation.

To find out more and become an active member of XMPP IoT community, check out IoT Special Interest Group.

2.11. … receive push notifications.

Push Notifications are (usually) doing a great service to mobile devices’ battery life. It is great indeed that a single TCP connection is maintained by OS, while apps can remain hibernated in the background. It is natural to every chat application to deliver notifications to the end user, even when a smartphone is resting in the pocket. How does XMPP cooperate with popular services like APNS or GCM?

It depends.

Although it’s not difficult to find XEP-0357 Push Notifications, it deserves some explanation. This specification is very generic. It assumes the existence of another XMPP-enabled “App server” that handles push notifications further. Although implementations could be found (e.g. in MongooseIM or Prosody server), it is very common for commercial installations to use custom protocols to provide push tokens and send PN packets directly to the respective services (APNS, GCM, SNS…)

2.12. … publish messages to groups of subscribers.

Publish-subscribe is an ancient and extremely useful pattern. XMPP got its own PubSub specification quite early (first versions were published in 2003) and even though the protocol is pretty verbose (and a bit complicated), for the basic usage you’ll need to learn only the most important aspects: there are nodes in the PubSub service where publishers can publish data. Nodes can group other nodes or remain plain leaves. That’s the whole story. The rest is about configuration, access control, etc.

XEP-0060 Publish-Subscribe is implemented in every major XMPP piece of software. In case of MongooseIM, it’s handled by mod_pubsub. You can find in popular client libraries as well: Smack, XMPPFramework or Stanza.io.

Set sail!

Now, if you feel brave enough, you can dive into this looong, official list of XEPs. These documents are designed to provide a precise information for libs and server developers. In your daily routine you’ll probably won’t care about every server-side edge case or whether some information is encapsulated in element X or Y. These are perfect reference guides for if you’re stuck somewhere or need to tap into lib’s plugin internals.

I’d recommend one more intermediate step though. Browse servers’ and clients’ feature lists published by virtually every project on their webpages. They usually skip (or enumerate them in separate, more detailed docs) minor items and highlight the features that may be important to you. This way you’ll expand your XMPP vocabulary and knowledge, while postponing the stage where reading XEPs is unavoidable, thus making the learning curve less steep.

It won’t be long until you realise that XMPP is your friend, I promise. :)

Stay tuned for Part 2! In the meantime:

  1. Brush up on your XMPP basics with our guide to building an iOS app from scratch using XMPPFramework (parts 1 and 2)
  2. Learn more about MongooseIM, our XMPP based open source mobile messaging platform.

August 23, 2019 08:22

August 22, 2019

Ignite Realtime Blog

Announcing the XMPP-Strings Testframework

@Flow wrote:

jXMPP, a FOSS XMPP-base library, has just been extended by a testframework for “XMPP-Strings”. Currently, this is limited to Local-, Domain- and Resourceparts and the various XMPP address types, but may be extended to future Strings found in XMPP-land.

The testframework comes with a corpus of known valid and invalid XMPP addresses (JIDs). I am happy about feedback and contributions to the corpus. Hopefully it becomes a useful resource for XMPP developers. You can find the corpus under

The testframework currently uses the following libraries to prepare and enforce the Strings:

  • ICU4J
  • GNU libidn
  • xmpp.rocks’s PRECIS

a further minimalistic implementation, called “simple”, is part of jXMPP.

You can run the testframework simply by

git clone https://github.com/igniterealtime/jxmpp.git
cd jxmpp
./test-xmpp-strings

if you have gradle and python3 in your path.

Please note that ICU4J and libidn (and “simple”) do not currently provide the required PRECIS profile and hence the older Stringprep profiles are used.

Feel free to use the provided infrastructure to test your own implementation. Thanks to JNI and the polyglot programming feature of the GraalVM, it should be easily possible. Please contact me if you need assistance.

Posts: 1

Participants: 1

Read full topic

by @Flow Florian Schmaus at August 22, 2019 15:50

August 21, 2019

Jérôme Poisson

SàT PubSub 0.3.0 has been released

SàT Pubsub is a server independent PEP/PubSub XMPP service, which aims to be complete and universal.

This project was born due to the fact that it is difficult to have feature-full PEP/PubSub on every XMPP servers, and even if it is the case there can be huge delays before seeing new functionalities implementation, or they can be difficult to extend.

The "Salut à Toi" project being using extensively XMPP PubSub functionalities, there was 2 ways to work around the issue:

  • to concentrate on a particular XMPP server, to recommend it, and if possible make it evolve in the wanted direction.
    This would mean being blocked on a specific XMPP server implementation and taking the risk to have incorrectly working (or not working at all) functionalities on other servers

  • create a server independent component, by using XMPP extensions to have a privileged access to server

The latter option has been retained, SàT PubSub uses XEPs (XMPP extensions) Namespace Delegation (XEP-0355) and Privileged Entity (XEP-0356) to be able to offer advanced features and PEP.

Beside the use of "SàT" in the name (which is due to its origins), you don't need to install or use Salut à Toi with this component, and it can be used by any XMPP compatible software.

SàT PubSub was already implementing MAM and RSM (allowing to do searches in archives and to use paginations to retrieve new items), the 0.3.0 version also brings:

  • the presence access model
  • +notify handling
  • notion of administrators, which are identifiers ("JID") with privileges
  • affiliations
  • the possibility to find items sorted by creation or modification date, thanks to Order-By (XEP-0413)
  • the "Node Schema" experimental feature, to associate data types to a node using Data Forms (XEP-0004). This feature is notably used by Salut à Toi for tickets
  • the "Serial IDs" experimental feature, which set new items identifiers using increments (1, 2, 3, etc.) instead of random values. This is notably useful for tickets.
  • the "PubSub Admin" experimental feature, which let administrators publish items by specifying an other publisher. This can be used to restore nodes backups.
  • the "consistent publisher" experimental feature which, once activated in node settings, keep the original publisher when an item is updated by the node owner or an administrator. This permit to update an item without preventing the original publisher do modify it himself or herself (for instance while editing a blog comment or updating a ticket status).
  • a config file can now be used with SàT PubSub, avoiding to have to specify settings – including the password – entirely on the command line. The same file as for Salut à Toi (sat.conf) is used, the settings for SàT PubSub must be set in the [pubsub] section.
  • a new documentation

You'll find more details in the CHANGELOG.

Version 0.4 development has already begun with a working Python 3 port, completing the port of the whole Salut à Toi ecosystem.

To install SàT PubSub, you just have to enter pip install sat_pubsub in a Python 2 virtual environment, check documentation for more details.

by goffi at August 21, 2019 18:12

ProcessOne

Real-time Stack #24

ProcessOne curates the Real-time Radar – a newsletter focusing on articles about technology and business aspects of real-time solutions. Here are the articles we found interesting in Issue #24. To receive this newsletter straight to your inbox on the day it is published, subscribe here.

MQTT for system administrators and for the IoT

They say MQTT is a PUB/SUB protocol for the Internet of Things, which it was originally designed for, but it’s also well suited for monitoring machines and services. Presentation given at BSDCan2019 in Ottawa.

Z-Wave to MQTT

Fully configurable Z-Wave to MQTT gateway and control panel. The Z-Wave to MQTT add-on allows you to decouple your Z-Wave network from your Home Assistant instance by leveraging your MQTT broker.

Migrating trashserver.net from Prosody to Ejabberd

Author writes: “Yesterday I moved my old Prosody setup to a new Ejabberd-based XMPP server setup. I’d like to leave you a few notes on why and how I did that. There ware multiple reasons for my decision to give Ejabberd a chance.”

Emoji avatars for personal website

Author writes: “My previous avatar was almost 3 years old, and I was getting tired of it. I decided to replace my avatar on my website for my IndieWebCamp Austin hack day project. But if you know me, you know I can’t do anything the easy way.”

Gotify server

A simple server for sending and receiving messages, written in Go and licensed under the MIT license.

LabPlot 2.6 released with MQTT support

The next release of LabPlot is announced! One big new feature is the support for the MQTT protocol. With LabPlot 2.6 is is possible now to read live data from MQTT brokers. This feature was contributed by Ferencz K. during Google Summer of Code 2018.

by Marek Foss at August 21, 2019 10:52

August 20, 2019

Tigase Blog

BeagleIM 3.2 and Siskin IM 5.2 released

New versions of XMPP clients for Apple's mobile and desktop platforms have been released.

Keep reading for the list of changes.

by wojtek at August 20, 2019 19:07