Planet Jabber

July 17, 2018

Christian Schudt

Babbler 0.8.0 released

After a longer period of development, Babbler 0.8.0 has just been released today.

The new major features are certainly

Further information about the release can be found in the release notes and in the changelog.

by Christian Schudt (noreply@blogger.com) at July 17, 2018 15:29

July 16, 2018

ProcessOne

Announcing RTB, an open source real-time protocols benchmark tool

ProcessOne is excited today to announce the first release of RTB (Real-Time Benchmark), an open source benchmarking tool to stress-test real-time protocols. The aim of this project is to define an easy benchmark for XMPP and MQTT servers, with minimal configuration overhead.

Why

To compare server implementations and be sure they are ready for massive traffic, there is a need for a “golden standard” benchmarking tool that is easy to use and avoids ambiguity due to configuration or overhead – each test should have the same baseline and capabilities for the results to be comparable.

We believe RTB fits such a role, because it has sane defaults (gathered from statistics of real world servers) and is able to cover all the test features defined in the XMPP Compliance Suite 2018.

Status

RTB is in early stages of development: currently, only XMPP protocol is implemented, and support for Multi-User Chat (MUC) and Personal Eventing Protocol (PEP) is lacking.

However, “sane” defaults and what should be considered a “golden standard” for this benchmark is open for discussion with the XMPP community – we welcome issues and pull requests. The tool has already been battle-tested: we used RTB to stress test our ejabberd SaaS deployments with more than 2 000 000 concurrent connections.

RTB includes a script to populate your server’s database with user accounts and rosters, so it’s easy to create test pools. When you start a test with RTB, you can monitor a local website on port 8080 with live graphs like the one below:

Requirements

We want RTB to be straightforward and easy to deploy. Currently, only Unix-based operating systems are supported. To compile, you just need gcc, make, Erlang, expat, yaml, openssl, zlib and gnuplot. There’s a default YAML configuration file to get you started and only 5 mandatory parameters before you can lanuch your first benchmark.

RTB is available on GitHub under Apache License 2.0 and we are waiting for your feedback!

by Marek Foss at July 16, 2018 12:59

Paul Schaub

Summer of Code: Plan for the grand finale

I passed the second evaluation phase :) Now begins the final spurt, as the last month of GSoC has begun. My main goal can be summarized as follows: Get everything merged!

To get that done, I have to polish up my smack-openpgp branch which has grown to a size of 7000 loc. There are still some minor quirks, but Florian recommended to focus on the big picture instead of spending too much time on small details and edge cases.

I also have to release pgpainless to maven central and establish some kind of release cycle. It will be a future challenge for me personally to synchronize the releases of smack-openpgp and pgpainless.

But now enough talking, I have to get to work :)

Happy Hacking!

by vanitasvitae at July 16, 2018 08:55

July 13, 2018

Paul Schaub

Summer of Code: Second evaluation phase

Quite some time has passed since I bothered you with my last post :) A lot has happened since, I have been making steady process in both smack-openpgp, as well as pgpainless.

One big step that I took was to get rid of smack-openpgp-bouncycastle, which now has been merged into smack-openpgp. Having modular code may be worthwhile, however it poses some big challenges. The biggest problem with having smack-openpgp not depend on smack-openpgp-bouncycastle was, that I could not use classes that represent encryption keys directly in smack-openpgp. Instead I had to create interfaces that encapsule functionality and call those in order to get stuff done from inside smack-openpgp. Last week me and flow decided that it would make my job a lot easier if we just got rid of smack-openpgp-bouncycastle by merging the two modules. In case there will be another implementation at some point, the code would still be modular enough to allow extension by overriding classes and methods.

Now smack-openpgp depends on pgpainless directly, which means that I don’t have to create duplicate code to get bundled information from pgpainless to smack-openpgp for instance. This change gave me a huge performance boost in the development process, as it makes the next steps much more clear for me due to less abstraction.

I rewrote the whole storage backend of smack-openpgp, keeping everything as modular as possible. Now there are 3 different store types. One store is responsible for keys, another one for metadata and a third one for trust decisions. For all of those I created a file-based implementation which just writes information to files. An implementor can for example chose to write information to a database instead. For all those store classes I wrote a parametrized junit test, meaning new implementations can easily be tested by simply inserting an instance of the new store into an array.

Unfortunately I stumbled across yet another bug in bouncycastle, which makes it necessary to implement a workaround in my project until a patched version of bouncycastle is released.
The issue was, that a key ring which consists of a master key and some subkeys was not exported correctly. The subkeys would be exported as normal keys, which caused the constructor of the key ring to skip those, as it expected sub keys, not normal keys. That lead to the subkeys getting lost, which caused smack-openpgp to be unable to encrypt messages for contacts which use a master key and subkeys for OpenPGP.

This bug has been fixed pretty fast by the bouncycastle team and the minimal test I created to illustrate my problem has been incorporated into bouncycastle. Thank you :)

Currently I’m working on a workaround for the bug in smack-openpgp, but that work is already working. Next I will polish up my test client and do some more field testing to iron out all the edge cases I probably overlooked :)

Happy Hacking!

by vanitasvitae at July 13, 2018 11:26

July 05, 2018

Jérôme Poisson

Salut à Toi 0.7 alpha: contributors, take your keyboards!

It's my pleasure to announce the release of the first alpha version of Salut à Toi 0.7.

This is the occasion on one hand to show the state, advancement and direction of the project, and on the other hand to launch a call for contributions.

In this article I will show you a part of Salut à Toi capabilities, and explain briefly how to use them.

You'll find a link to an .apk file to test Cagou on Android in the "Test it!" section.

As a reminder, Salut à Toi is a collaborattive ecosystem, which is multi-frontends (web, desktop, mobile devices, terminal, command line), multi-platforms, and multipurpose. It let you handle your communications, organise yourself, share your files, etc. It is also, and above all, a militant project which think about the impact of new technologies on our lives, and which tries to stimulate public debate.

We'll concentrate on the web and desktop/mobile frontends, because those are the ones which will most probably be of interest to the mass.

Web frontend (named "Libervia")

Libervia has seen major changes, with a new web framework.

The idea, which has been explained in a blog post is to offer base components (account authentification, permissions, comments, etc.) ready-to-use and decentralised thanks to XMPP. This way we easily build website which are automatically decentralised and federated.

Please note that everything is prepared for traduction, but for now only English is available. Help would be much appreciated to improve on that!

Let's check the available features:

Chat

screenshot of a "chat" discussion in Libervia

Let's start with the most obvious, there is an instant messaging page. To use it, you just have to click on "chat", et to log in if it's not already done.

You'll land on a panel with the rooms in your bookmarks, you just have to click on one of them, or to specify an identifier (jid) of a new room/contact.
It is for now quite basic, but still you can enter "/help" to check textual commands.

Blog

my own blog, powered by Libervia

One of the flagship features of SàT, really rare in XMPP world (the only actively developed other client implenting it being Movim), is the blog (which is then decentralised).

A click on the menu will bring you to a "discover" page where you'll find your contacts with a blog. Click on one of them to read, or specify an identifier to read a blog which is not in your contacts.

Forums

XMPP powered decentralised forums

Unique feature in XMPP world, SàT propose forums, allowing to organise discussions in categories. Because forums are under the hood the same thing as blogs, it is possible to use a blog as forum and vice versa.

The addition of new forums is done by an administrator using command line tool "jp" (see below).

Events

personal list of events

Other feature not seen elsewhere in XMPP (and in decentralised software?), SàT allows you to create events, and of course to invite people.

As with other features, the page is thought to be easy to use: you have a button "create an event" which bring you to a form. A name, location and date later (and optionally a picture to associate), and your event is created. You'll then land to an admin page where you'll have the possibility to invite people.

Events are private for now, only your guests and you have access to it, but public events are also planned.

a guest is answering to an event RSVP

Please note that it's possible (but not yet from the web interface) to add arbitrary fields to your events, for instance to let your guest specify what they'll bring to a picnic.

File sharing

sharing file from browser

Here we are talking about a feature useful daily, and in many circonstances. Salut à Toi has long been able to send files, or to upload them on your server, but we are talking here about sharing one or many directories, or to upload in an advanced way on a serveur component.

All of this is explained in a recent blog post, but let me add 2 use cases to understand well:

  • you are home, you've just taken a video and some photos on your phone that you want to get on your computer. You have installed Cagou, the desktop/Android frontend of SàT on your device and already shared your photo directory. From your desktop computer, in the "file sharing" mode you see your device and you can directly download the video and the photos from there. The connection will be done directly on your local network, from device to device, and your files will not go throught a external server located whoever knows where.

  • you have files that you want to access at any time from anywhere (pictures or working documents for instance). You put them in a virtual directory on your file sharing component (a SàT instance too) and you can access it as simply as for the previous example.

Note that while with Cagou the connection is done directly when possible, it's not the case for Libervia which will first download the file on the server, before taking it throught HTTPS. WebRTC will be needed for the direct connection, and that's planned for 0.8 release.

Permission are handled easily: you specify who can access your files in a couple of clicks.

Files are transfered in clear for now (unencrypted), but this should not be the case anymore once finale release is here.

Photos albums

a photos album with a comment

Based on previous feature, a photos album mode display a directory containing only photos with adapted view, an with comments.

Permissions are really useful here, allowing to let, for instance, only your family see an album.

It's not yet possible to access an album from the interface of Libervia, to test it you'll have to change the URL by hand from the file handling page (by replacing files/list by photos/album).

Tickets

SàT tickets, first ones have been imported from Bugzilla

Developed from our own needs, SàT now handles tickets in pretty much the same way as a code forge. This is classical issue tracking, bugs report, tasks to do, etc.

Fields are free, it's really easy to use this feature for any kind of list: TO-DO tasks, but also, for instance, shopping lists.

Merge requests

a major patch is being reviewed

In the same spirit as tickets, SàT also handle "merge requests". Developed and used primarly for SàT itself, this system has the benefit or not being tied to a specific tool (like Git or Mercurial). Il is currently used with Mercurial, but it is possible to use it with whatever we want, and I'm not only talking about software code here.

Those 2 features means that SàT is a decentralised code forge. Even if it's still relatively basic, it has the benefit of being easy to use, flexible and adaptable. Furthermore there are only a few decentralised code forge at the moment (we can probably count Fossil, maybe some others).

Once again, contributions would accelerate things and would permit to have a very powerful tool.

Application

the dynamic part, which in under minimal maintenance and will be re-written for 0.8

To conclude this small tour of Libervia, we have to mention the application which is an entirely dynamic page, more elaborated but also more complicated to use (one of the main base version goal is to be easy to use).

Libervia web application allows to use widgets that you can drop next to each other, to do end to end encryption (OTR), or to publish blog post in simple or WYSIWYG mode, with the syntax you like (Markdown for instance).

It's the historic application which, for technical reasons, is now under minimal maintainance and have a few issues (like the contacts which can appear 2 times as seen in the screenshot above). It will be re-written for SàT 0.8, but it's let in place in this release for people who want to use it.

Cagou (desktop/Android)

Promised after the success of our crowfunding campaign, which allowed us to get a bit more than 3000 €, Cagou is our new frontend for desktop and Android devices. Its goals are to be modern, easy to use, and powerful.

Its original UI is inspired from Blender, and let split the screen to show several widgets at once. This UI has been thought in a way that it can be used even without knowing this split capacity, while being straightforward to use once the feature has been discovered.

To change mode, you just have to click on the top left icon.

For now, 5 modes are available: chat, contacts, file sharing, widget selector, and settings.

On the very top, you have a menu (only on desktop), and a notification zone, where messages appear for a few seconds and are visible entirely when clicking on the head of the left Cagou. On the other side, an other Cagou head can appear when you get notifications needing a user interaction (new contact for instance, or authentification request – see below –). This way, no popup will steal your focus while you are writing a message or doing something else.

Split

It is possible de split the screen as much as you want by clicking on the 3 dots area (on top and left borders), and then moving your mouse/finger. If the bar is red, this means that you'll close the area.

Here is a small animation to make things more clear:

demo of Cagou's widget split ability

You can organise the way you want, and monitor several things at the same time. In the future, it should be possible to keep handy widgets organisations so you can re-use them easily.

Chat

Cagou "chat" mode screenshot

The chat should be relatively intuitive. The top area show the current room or the identifier (JID) of the person you're talking to. It is possible to enter an other identifier manually to change room. For instance you can enter sat@chat.jabberfr.org to come on Salut à Toi room.

Still on the top area, the "padlock" icon on the right (only visible in 1/1 discussions) allows you to encrypt the conversation end to end, with OTR only for now (see below).

If you have several discussions at the same time, you can switch from one to the other by swiping your finger horizontally. Only discussions not already visible somewhere on the screen will be available, and for now they are in alphanumeric order (most certainly recent discussions will be available first when stable release will be out).

Finally, in the bottom area, in addition to the text input, you can see a + button with a circle which allows you to open a media transfer panel. On the desktop for now you can only select a file from your file-system, but on Android you can select/take a picture/video, a file or record an audio message. In every cases you can choose if you want to upload the file on your server, or send it directly to your contact, without transiting by the server.

file sharing

The principle has already been explained for Libervia, here is an animation showing how to share a directory with Cagou:

sharing a directory from Cagou

As you can see, you just have to do a long click/touch on the directory or file to share, and to select people which will have access to it.

You can click on "share local files" button, in the header zone, to change mode and see the device which are sharing. You'll then see – as with Libervia – 3 areas: the first one with sharing component, if any, the second one with your own devices (your phone or desk computer for instance), and finally the devices of your contacts. Once again, the input area let your filter devices to show.

see devices which can share files

Third party website authentification

SàT is a project which push the use of XMPP in every directions, and this include authentifying on third party websites. The idea is to be able to authenticate yourself without having to create an account on any website just by giving your JID.

Here is a video demo, thanks to the demo website made by jnanar, the screen on the right is a phone (displayed thanks to scrcpy):

super easy authentication with XMPP

You can implement this in your own website either by using Chteufleur component (Chteufleur is actually the original implementer of the feature in SàT, many thanks), either by using a server where it's already available (it's for instance the case on jabberfr, thanks to the admins of this server).

Please note that once it's in place, in your website you just have to do a single HTTPS request to validate authentication.

Contacts

No need to take much time on this mode, but it's good to know that you can add a new contact there (with the top button), or delete an existing one with a long click/touch.

A simple click will launch the chat mode with the selected contact.

Command line (JP)

There are far too many new things in jp to describe everything here.

Some useful informations anyway:

  • in jp we always put commands first, then arguments (jp [command] [subcommand] --arg_1 --arg_2 --etc)
  • jp [command] --help tells you how to use a command
  • jp shell let you use a REPL shell
  • it is possible to have high level and low level commands. For instance with the blog, you can have blog posts in a nice way by using jp blog get -s bloger@somewhere.tld -O fancy, or the raw XML with jp pubsub get -s bloger@somewhere.tld -n urn:xmpp:microblog:0
  • many commands have an output mechanism with --output (or -O). For instance jp blog get -s bloger@somewhere.tld -O json let you find metadata of a blog feed in JSON
  • jp can generate static websites thanks to --output template
  • when the website generator handle it, you can directly use the blog URL to edit it. For instance, I can edit one of my last posts by using jp blog edit -u https://www.goffi.org/b/khii3yEKWDhmNYSDt5GGrY/vers-forge-decentralisee-basee-xmpp where https://www.goffi.org/b/khii3yEKWDhmNYSDt5GGrY/vers-forge-decentralisee-basee-xmpp is the URL of the HTML generated page. I then have my text editor showing the code of my page on one side (in my favorite syntax) and the metadata on the other. Note the this works with webpage generated by Libervia, but also by Movim.

A small demo of the shell to read a blog:

checking a blog from shell (jp)

Test it!

You can test all this, the installation instructions are available (in English only, we are lacking resources to translate) on the wiki by clicking here.

SàT (backend + jp (CLI) + Primitivus (TUI)) is available in Debian experimental (thanks to Robotux, Naha, Debacle and other packagers).

SàT and its frontends (jp, Primitivus, Cagou, Libervia) are available on Arch Linux (thanks to jnanar and Link Mauve)

We got confirmation that SàT and Cagou are working on Mac OS X (thanks to xma), we would like help of people to also test it on Windows, BSD*, or anything else.

You can test on Android by clicking on this link from your device (installation from external resources must be activated from settings).

Note that this is an alpha version, that means:

  • it crashes, the code is not stable
  • all features are not finished, or even here
  • it's not optimised yet
  • it's not polished

Oh, and while we are talking about features not yet implemented, there are strong possibilities that end to end encryption with OMEMO will be implemented before final release, thanks notably to the new Python module available (thanks to Syndace, the author). End to end encryption is already available for years in SàT, but only with OTR so far.

Participate!

A last animation to show you the automatic language detection, an experimental feature:

language detection, then messages filtering

Let's stop here for the features, but we have not seen all of them.

As you can see, there's a lot to play with, and there is enough for every taste in this project.

And that's good, with summer holidays you'll have more time, a perfect occasion to contribute to a unique project, ethics and militant.

Do not hesitate to come on SàT XMPP room (sat@chat.jabberfr.org, also available without account, from the web).

If you are lacking time (picnic on the beach is nice too), support on Liberapay and/or adhering to the association is also helping.

Try to spread the word around you, our project lack visibility and needs you. If you are willing to host an instance, you can start to test (in a "experimental/not stable" part of your website), and give us feedbacks. We are also looking for people helping to package in every place where it's possible.

Thanks in advance, and see you soon!

by goffi at July 05, 2018 20:01

July 02, 2018

Paul Schaub

Summer of Code: Checkstyle to the rescue!

Today I added some checkstyle rules to PGPainless.Checkstyle is a gradle plugin, which checks the source code for style violations.

Some say, strict checkstyle rules are unnecessary and that it is annoying to be held back from pushing a commit to the master branch only to fix “style issues” for half an hour. I must say, in the beginning I thought the same way. I was annoyed thinking “why does it matter, if a line comment ends with a period or not?” But after being forced to give it a try when I first became a contributor to the Smack project, I became a fan of it. In the beginning I had to often recommit my changes because they broke the checkstyle rules. For example I often forgot to leave spaces between mathematical operators. I would write “i = 5+5;” instead of “i = 5 + 5;”. But after some amount of time, I got less and less warnings.

I adopted most of the (honestly *very* strict) rules in Smacks rule set to my own coding style. I like how it automatically leads to cleaner, more uniform code (not that it is impossible to write garbage with it of course). For that reason, I decided to put those rules into place in PGPainless today (I only left one rule out, because who the hell cares about the alphabetical sorting of imports???).

At some point, PGPainless will be released as a maven artifact. In preparation for this historical event, I bought the domain pgpainless.org. For now it is just a forwarding to the PGPainless git repository, but I will likely setup a small website with documentation etc. at some point.

During my testing of Smacks OX implementation, I came across an interesting problem. When a user queries a PubSub node in Smack, Smack first does a disco#info query on that node to determine, whether it is a LeafNode or a CollectionNode. This normally works fine. However, it becomes more and more popular to make use of the PubSub access model ‘open’. The open access model makes a PubSub node accessible to entities (like other users) which are not in the contact list of the user. This enables the use of OMEMO in group chats, where not every participant is in your contact list for example.

The problem is that a server which allows access to open PubSub nodes, does not necessarily allow the disco#info query. The question is: Should disco#info queries on open PubSub nodes be allowed or not? An argument against it is, that it might allow “jid-harvesting”. An attacker might use disco#info queries on open PubSub nodes in order to determine, whether the user exists or not. This is a bad thing, because it allows spammers to collect the Jabber IDs of potential victims. On the other hand however, the attacker could simply do a direct PubSub query on the open node and the result would be the same. The benefit of allowing disco#info queries would be, that you can in fact determine the node type.
For now my mail to the standards mailing list remained unanswered, but I think that there should be a well defined expected behavior for this edge case.

For now I worked around the issue by using Javas reflections to access the LeafNode constructor directly, avoiding the disco#info query.

Other than that, I didn’t get a whole lot done this week. Unlike the demotivating week though, this time the reason was primarily exciting new hardware :D

Happy Hacking!

by vanitasvitae at July 02, 2018 20:27

June 28, 2018

The XMPP Standards Foundation

The XMPP Newsletter, 29 June 2018

Welcome to the XMPP newsletter.

If you have an article, tutorial or blog post you'd like us to include in the newsletter, please submit it on the XMPP wiki.

News

Github has been acquired by Microsoft, prompting some FOSS developers with long memories to reconsider whether they still want to host their projects there.

The Salut à Toi project is working on a decentralized and federated solution for issues and merge requests, based on XMPP, and Jérôme Poisson explains in this article their motivation for doing this work.

Every now and then a blog post comparing XMPP and Firebase appears. Generally these posts seem to be poorly disguised attempts at SEO and to promote application development by a particular devshop. This submission seems to follow the trend and is also relatively superficial. It is however interesting to note that they mention multiple times that it's difficult to set up an XMPP server. Perhaps something to take note of and to consider how we could ease setup of an XMPP-based solution.

Monal has received a lot of updates last months and now it also has initial OMEMO support. More work needs to be done, so don't expect to see this in the App Store before August.

Software releases

Servers

The XMPP server written in Go has a new release with support for Websocket (RFC-6395), XEP-0191 (Blocking Command), XEP-0012 (Last Activity) and XEP-0237 (Roster Versioning).

The new release includes improved robustness of external authentication backends, support for REST API custom headers and more.

This release includes the so-called "Inbox" feature which was also discussed at the XSF summit in FOSDEM. It is a list of all ongoing conversations, with excerpts of last messages and unread messages count, to help you gain an overview of what happened while you were gone. There is no XEP for this yet, but Erlang Solutions have promised to submit one.

Clients

  • Monal 3.0.2 has been released.

  • JSXC 3.4.0 released including support for location sharing and improved roster management in our internal Nextcloud backend.

Libraries

Chris Ballinger has released Version 4.0 of XMPPFramework, the XMPP library for iOS and macOS. It brings support for XEP-0048: Bookmarks and XEP-0359: Stanza Ids. It also has improved Swift support with optional Swift-only features.

by jcbrand at June 28, 2018 22:00

ProcessOne

ejabberd 18.06

This ejabberd 18.06 release includes, after two months of development, a lot of bug-fixes and many improvements. There are also a few changes, so please read carefully the release notes.

Noticeable changes

Stop ejabberd initialization on invalid/unknown options

Since some time now, we are warning ejabberd admins in log file about incorrect configuration options. We feel that we are ready to make these hints strong requirements about config file validity.

Starting with ejabberd 18.06, ejabberd will not ignore unknown options and doesn’t allow to have options with malformed values. The rationale for this is to avoid unexpected behaviour during runtime, i.e. to conform to “fail early” approach. We thus hope to be helpful to newcomers by making sure they properly format their config files to get the expected behaviour.

Note that it’s safe to reload a configuration with potentially invalid and/or unknown options:
this will not halt ejabberd, but will only prevent the configuration from loading.


NOTE FOR PACKAGE BUILDERS
This new behaviour should be documented in the upgrade notes.

Improve robustness of external authentication backends

Now all external ports are attached to supervising processes and requests are balanced in round-robin manner until the pool is exhausted.

The commit also deprecates extauth_instances option and introduces extauth_pool_size option instead, with the default value of a number of logical processors (i.e. CPU cores).

Add new options for OOM watchdog

  • oom_watermark: 1..100
    Start OOM watchdog only when system memory usage exceeds this value in percents. When the usage drops below the value, OOM watchdog is stopped. The default is 80 (percents). Note that once OOM watchdog is started, it performs full garbage collection periodically: this can be seen as spikes in CPU utilization and drops in RAM usage. If your system is permanently above the watermark, it may cause significant CPU overhead.

  • oom_queue: positive integer
    Only trigger OOM killer when total amount of messages in all queues of all Erlang processes is above this value. The default is 10000. Note that this value only takes effect when oom_killer is set to true (this is the default). Otherwise, only a warning will be logged.

Add support for REST API custom headers

ext_api_headers can be defined as a single string. Headers are separated by comma. Definition MUST NOT contain spaces. Example “X-MyHead:test,X-Token:082748

Optimize HTTP requests memory usage

Due to historical reasons, ejabberd loads the whole file/data into the memory when serving an HTTP request. This is now improved:

  1. For GET requests ejabberd uses sendfile(2) if the underlying connection is HTTP and falls back to read/write loop with 64kb buffer for HTTPS connections. This type of requests are handled by mod_http_fileserver, mod_http_upload, ejabberd_captcha, etc
  2. POST requests are now limited to 20Mb and are fully downloaded into the memory for further processing (by ejabberd_web_admin, mod_bosh, etc)
  3. PUT requests (e.g. for mod_http_upload) are handled by read/write loop with 64kb buffer

Support SASL PLAIN by xmpp_stream_out

Also, SASL mechanisms chaining is now supported: if several mechanisms are supported and authentication fails, next mechanism in the list is picked, until the list is exhausted. In the case of a failure, the latest SASL failure reason is returned within handle_auth_failure/3 callback.

Drop support of mod_irc

If you need IRC support, we suggest to use biboumi. We will not maintain mod_irc anymore and moved it to ejabberd-contrib repository.

SQL schema change

As mod_irc has been obsoleted, we removed the related SQL table from the schema: irc_custom

Changes

Admin

  • Stop ejabberd initialization on invalid/unknown options
  • Add new options for OOM watchdog: oom_watermark and oom_queue
  • Add ability to modify version string
  • Add option ext_api_headers to define REST API custom headers
  • Fix Erlang limits in ejabberdctl.cfg.example to reflect current situation
  • Make trusted_proxied ejabberd_http option accept ip masks
  • Teach acl ip matching about ipv4 mapped ipv6 addresses
  • Removed watchdog_admins option from config, as has no effect anymore
  • Improve logging of external authentication failures
  • ejabberd_auth: Don’t use cache if the option is disabled
  • Make connected_users_info and user_sessions_info DB-agnostic

Core

  • Support SASL PLAIN by xmpp_stream_out
  • Add Resource Binding support to xmpp_stream_out
  • Improve robustness of external authentication backends
  • Don’t use ‘unsupported-version’ inside SM element
  • Generate SASL failures on unencrypted connections only for s2s
  • Fix reset_stream in websocket using pre-rfc protocol
  • Don’t crash in bosh when we receive request with RID < prev_rid
  • Get rid of all calls to jlib.erl module
  • Support IPv6 connections for PostgreSQL, MySQL and LDAP
  • Fix authentication for usernames containing uppercase characters
  • Optimize HTTP requests memory usage
  • PKIX: Just warn instead of ignore a certificate containing no domain names
  • PKIX: Don't replace valid certificates with invalid ones

Modules

  • Log modules startup
  • mod_disco: Advertise disco#info and disco#items features
  • mod_irc: is moved away from ejabberd repo to ejabberd-contrib
  • mod_mam: Don't replace existing stanza ID
  • HTTP upload: Generate HTTP Upload form using xdata codec
  • HTTP upload: Improve error formatting
  • HTTP upload: Return detailed error if HTTP upload is too large

MUC

  • Always display room's xdata in disco#info
  • Display muc#roomconfig_changesubject in room's disco#info
  • Render roomname, allowinvites and allowpm in room disco#info
  • Support for roomconfig_lang/roominfo_lang
  • mod_muc_sql: Fix export to SQL

Push

  • Omit summary for outgoing messages
  • Further improve handling of carbons
  • Also include sender/body for carbons
  • Include a static body text by default
  • keepalive: Increase default timeout to 3 days
  • SQL: Check 'max_user_sessions' limit

Feedback

As usual, the release is tagged in the Git source code repository on Github.

The source package and binary installers are available at ProcessOne.

If you suspect that you’ve found a bug, please search or fill a bug report on Github.

by Christophe Romain at June 28, 2018 15:05

Monal IM

Monal has OMEMO

OMEMO conversations work in Monal. There is still a lot work to be done here to properly handle keys and make UI elements in OSX and iOS for all the the new interactions.  There is a lot that also needs to be tested before this is in ready for the App Store. Realistically, expect to start seeing this stuff in the binaries I release some point in  August.

I have tested this with gajim on linux and chat secure on iOS. Will have dust off an android device somewhere to try it with Conversations.

by Anu at June 28, 2018 13:56

June 26, 2018

Paul Schaub

Summer of Code: An (almost) three line fix to a three days problem

diff --git a/src/main/java/de/vanitasvitae/crypto/pgpainless/decryption_verification/DecryptionStreamFactory.java b/src/main/java/de/vanitasvitae/crypto/pgpainless/decryption_verification/DecryptionStreamFactory.java
index d651b1b..bca7ba4 100644
--- a/src/main/java/de/vanitasvitae/crypto/pgpainless/decryption_verification/DecryptionStreamFactory.java
+++ b/src/main/java/de/vanitasvitae/crypto/pgpainless/decryption_verification/DecryptionStreamFactory.java
@@ -157,15 +157,13 @@ public class DecryptionStreamFactory {
         PGPPrivateKey decryptionKey = null;
         PGPPublicKeyEncryptedData encryptedSessionKey = null;
         while (iterator.hasNext()) {
-            encryptedSessionKey = (PGPPublicKeyEncryptedData) iterator.next();
-            long keyId = encryptedSessionKey.getKeyID();
+            PGPPublicKeyEncryptedData encryptedData = (PGPPublicKeyEncryptedData) iterator.next();
+            long keyId = encryptedData.getKeyID();
 
             resultBuilder.addRecipientKeyId(keyId);
             LOGGER.log(LEVEL, "PGPEncryptedData is encrypted for key " + Long.toHexString(keyId));
-            if (decryptionKey != null) {
-                continue;
-            }
 
             PGPSecretKey secretKey = decryptionKeys.getSecretKey(keyId);
             if (secretKey != null) {
                 LOGGER.log(LEVEL, "Found respective secret key " + Long.toHexString(keyId));
+                encryptedSessionKey = encryptedData;
                 decryptionKey = secretKey.extractPrivateKey(decryptionKeyDecryptor.getDecryptor(keyId));
                 resultBuilder.setDecryptionKeyId(keyId);
             }
         }
         PublicKeyDataDecryptorFactory keyDecryptor = new BcPublicKeyDataDecryptorFactory(decryptionKey);

The above 3 deletions and 1 addition are the fix for a bug in my decryption routines, which took me 3 days to find. Lets examine the bug in more detail in order to understand what the code does, what went wrong, and why it took me so long to find it.

As I described in an earlier blog post, an encrypted OpenPGP message basically consists of the symmetrically encrypted body and a bunch of public key encrypted session keys. In Bouncycastle, the encrypted session keys are called PGPPublicKeyEncryptedData objects. The are encrypted using the asymmetric public key of a recipient and they contain the symmetric session key which was used to encrypt the actual message. So if Alice writes a message to Bob and Carla, the message will contain at least two PGPPublicKeyEncryptedData elements. One containing the session key encrypted for Bob, the other one for Carla.

In PGPainless, I iterate through all PGPPublicKeyEncryptedData objects of an incoming message in order to get a list of all recipients of the message. I do that because I can imagine it would be cool if Bob can see, that the message was also encrypted for Carla. Also during every iteration I see, if the current PGPPublicKeyEncryptedData object is encrypted for a key of which I do have the secret key available. If so, I extract the private key for later use.

Lets have an example:

Alice encrypts a message to both herself and Bob. Then she sends the message to Bob. Bob tries to decrypt the message.
PGPainless saves the first PGPPublicKeyEncryptedData object in the variable encryptedSessionKey.
This first encrypted session key has Alice’s key ID, so PGPainless adds the id to the list of recipient key ids and goes to the next iteration. Again it stores the encrypted data object in encryptedSessionKey.
This second encrypted session key has Bob’s key ID. PGPainless finds the respective private key, extracts is and saves it to the variable decryptionKey.

Now the iteration ends, as all PGPPublicKeyEncryptedData objects have been processed. Now the decryption can begin. Using the extracted private key, Bob decrypts the encryptedSessionKey to retrieve his session key with which he decrypts the message. Easy Peasy Lemon Squeezy – at least that is what I thought.

I wrote some JUnit tests that encrypt a message for one recipient and those tests work just fine. I used them for example to determine the average message length using different algorithms.

For some strange reason however, during integration testing I noticed that every now and then decryption would fail. I thought it had to do with some race conditions at first. I blamed JUnits @Before and @After annotations which I used to delete the key rings from disk after the test was done to fire too early. Coincidentally after I removed the annotations the test would work flawlessly, which encouraged me and seemed to confirm my hypothesis.

However, the more tests I did, the more confused I became, as I could not find the cause of the failing. Bouncycastle has debugging disabled for their library, so I could not follow the calculations step by step. This is done for security reasons I guess. Fortunately there is a debug version which I discovered today. Using this library, I can see, which lines are responsible for throwing exceptions and step through the execution in great detail.

The final hint however came from a forum post. So lets see what exactly went wrong. For that purpose lets assume the same scenario as above, but with a slight twist.

Again, Alice encrypts a message to herself and Bob, but this time she prepends Bobs session key first.
Bob, when decrypting the message, stores the first PGPPublicKeyEncryptedData object in encryptedSessionKey, notices that this is his own session key and extracts his respective private key into the variable decryptionKey.
In the next iteration (remember, Bob wants to know all recipients), he stores the next PGPPublicKeyEncryptedData object in encryptedSessionKey. This is Alices session key.

Now he proceeds with decrypting the encryptedSessionKey with his decryptionKey – and hell breaks lose. Why? Because at this point in time encryptedSessionKey actually contains Alices session key, not Bobs.

The tricky part about this bug is, that it only happens by chance. I’m not sure in which order session keys are prepended to the message by Bouncycastle, but in certain cases this caused my code to fail – while in certain cases it did not. One benefit that OpenPGP has over OMEMO is, that you can write better tests. In OMEMO keys always change when the ratchet advances, so you cannot really test if certain messages decrypt. At least it is much easier using OpenPGP. But this bug told me a lesson, that you have to be very very careful with your JUnit tests. At some point I replaced randomly generated key pairs with some fixed keys to get more reliable test results and to be able to confirm the result using GnuPG. Ironically I was very lucky that as a result my test would reproduce the second scenario above. If instead it would have produced the first scenario, it would have taken me much much longer to discover the cause of the issue.

Fighting this bug took me 3 days. Luckily I didn’t spend 100% of my time bug hunting. I also wrote another integration test, which covers one very cool feature of OpenPGP for XMPP.

Secret Key Synchronization

In OX it is possible to store the secret OpenPGP key in a private PubSub node. That way the key can be easily transported to other devices, so you can read your encrypted messages in a web browser for example. This is one huge benefit over OMEMO, where you can only read messages received *after* you began using the new device.

During my testing, I also found out, that ejabberd despite announcing support for alternative PubSub access models, does not properly handle some requests.
For normal messaging, OX recommends to publish the public keys of users in a PubSub node configured with the ‘open’ access model. That allows users to access your public key, even if they are not in your roster.

Smack however does a disco info query on a PubSub node prior to fetching it in order to get some metadata about it. It turns out that ejabberd does return an error for such a request, stating that you have to be subscribed to the owner of the node in order to fetch information about it. I’m pretty sure though, that this bug will be fixed soon :)

Happy Hacking!

by vanitasvitae at June 26, 2018 15:57

June 20, 2018

ProcessOne

Real-time Enterprise Issue #12

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning business aspects of real-time enterprise we found interesting in Issue #12. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

Cisco: 90% of internet traffic through Erlang-controlled nodes

A tweet from @guieevc and a subsequent discussion on HN gives us some insights into how internet traffic is routed and what powers all the machines doing the job.

Ruby vs Elixir vs Go: concurrency comparision

Siva Gollapalli writes: “As we all know, for comparison you need more than one language. So, I choose Golang and Elixir along with Ruby. When I search about building highly concurrent applications most of the results would involve either one of these languages and moreover I like these two languages. Golang for its simplicity and Elixir for its Ruby like features.”

Apple introduces the AI phone

This “new AI iPhone” — which, to be clear, is your same ol’ iPhone running a new mobile OS — will understand where you are, what you’re doing and what you need to know right then and there.

Your smartphone is listening and it’s not a paranoia

For your smartphone to actually pay attention and record you, there needs to be a trigger, like Hey Siri or Okay Google for example . Without these triggers, there’s no recording, with just some general metrics being sent to your service provider. This might not seem a cause for an alarm, but when it comes to apps like Facebook, no one knows what the triggers are. In fact, there could be thousands.

Use MQTT to stream real-time data

The ny-power project uses a set of microservices to consume open data about the performance of the NY State power grid, and computes an approximate level of carbon intensity. This computed data is served in a real-time MQTT stream, making it easy to consume in other applications. It also shows how to manage a set of data processing in Kubernetes with Helm.

Mining for Bitcoin vanity addresses with elixir

Let’s take another bite out of Mastering Bitcoin and implement the algorithm described for “mining for vanity addresses”. After we implement the basic algorithm, we’ll add our Elixir special sauce and turn it into a fully parallelized procedure.

Why you should never have a data room  for fund-raising

The data room is where your process goes to die. What happens is 18–20 firms access the data room and download all of your documents. You feel proud because data rooms have tracking on them so you know exactly who did and who did not access your data. So you sit around and wait for the next call. You convince yourself that it should take 1–2 weeks until they have gone through the data and you’ll get a call but it never comes.

by Marek Foss at June 20, 2018 19:44

Real-time Stack Issue #12

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning tech aspects of real-time development we found interesting in Issue #12. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

Exception Management at the Heart of Artificial Intelligence Performance

Science fiction makes us dream about the promises of artificial intelligence, but it creates a mythology, which makes it difficult to understand the real issues at stake. Yet, there’s no magic ingredient in AI, but rather a lot of mathematics.

An AI Created New Doom Levels That Are as Fun as the Game’s Original Ones

The technical skills of programmer John Carmack helped create the 3D world of Doom, the first-person shooter that took over the world 25 years ago. But it was level designers like John Romero and American McGee that made the game fun to play.

Choosing the Right Platform for Real-time Chat Application Development: XMPP vs Firebase

With a variety of chat application frameworks available in the market, these are the two most advanced and high performing protocols: Firebase and XMPP. Here are their pros and cons.

How To use XMPP and I2P to Build Your Own Decentralized Messenger

Building decentralized messenger run by users instead of corporations is definitely not new but thanks to XMPP and I2P (Invisible Internet Protocol) it’s easier than ever before.

AI and the Augmentation Fallacy

The fundamental disruption introduced by AlphaZero’s hyperlearning in the chess world can teach business executives about AI. Many pundits, academics, and economists advise business executives on how artificial intelligence (AI) will augment human performance in the workplace.

XMPP Echo Bot

Do you know that situation, you really really need an XMPP echo bot, but you don’t have access to high-level tools like Python to write one? All you have is openssl, bash, dig, stdbuf and sed? Then this tool is for you. This is an XMPP echo bot written in (mostly) sed.

The Erlang Runtime System

The real goal with this book is to give you all the information, all the gory details, that you need in order to really understand the performance of your Erlang application.

by Marek Foss at June 20, 2018 19:41

Real-time Enterprise Issue #11

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning business aspects of real-time enterprise we found interesting in Issue #11. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

France to build alternative to WhatsApp, Telegram due to foreign spying scare

According to a recent report, the French government is currently developing an end-to-end encrypted alternative to WhatsApp and Telegram that its officials could use without worrying about foreign spying.

BotChain wants to put bot-to-bot communication on the blockchain

Increasingly we are going to be having bots conducting business on a company’s behalf. As that happens, it is going to require a trust mechanism to ensure that bot-to-bot communication is legitimate.

How WhatsApp bot is helping redBus improve customer support

Case study: Many of users who booked a bus ticket on redBus in last few weeks would have received ticket details and lot more on WhatsApp. Some of them might have also conversed with redBus on WhatsApp.

Space communication is stuck in the dial-up age, which means it’s time for more lasers

In space, no one can hear you scream — because sound doesn’t travel in a vacuum, but also because you would need some sort of radio relay to carry the message, what with the distances being so extreme.

Smartphones are killing the planet faster than anyone expected

Before you upgrade your next iPhone, you may want to consider a $29 battery instead. Not only will the choice save you money, it could help save the planet.

Red Hat and Microsoft bring OpenShift to Azure

At Red Hat Summit in San Francisco, Red Hat and Microsoft announced they were bringing Red Hat OpenShift, Red Hat’s Kubernetes container orchestration platform, to Microsoft’s Azure, Microsoft’s public cloud.

by Marek Foss at June 20, 2018 19:30

Real-time Stack Issue #11

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning tech aspects of real-time development we found interesting in Issue #11. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

How to Build a Private One to One Chat App from Scratch?

In this article, the development of one to one chat app from scratch is discussed in detail. The technical stack involved in WhatsApp like chat app are Erlang (Language), Ejabberd (Framework), XMPP protocol, MySQL (Database). The following steps will help you to develop a one to one chat application in an hassle-free manner.

Deploy Bulletproof Embedded Software in Elixir with Nerves

Nerves defines an entirely new way to build embedded systems using Elixir. It is specifically designed for embedded systems, not desktop or server systems. It consists of a minimal Buildroot-derived Linux that boots directly to the BEAM VM, ready-to-go library of Elixir modules to get you up and running quickly and powerful command-line tools.

Connecting an Elixir Node to the Bitcoin Network

Pete Corey writes: “Since I first started diving into the world of Bitcoin development, I’ve wanted to build a simple node that connects to the network. The Elixir programming language gives us some fantastic tools to implement a server in the peer-to-peer network. Let’s see how far they can take us!”

Riot: A Distributed IRC and VOIP Client and Home Server

Riot is a free and open source decentralized instant messaging application that can be considered an alternative to Slack. This article takes a look at features of Riot, installation procedure and usage.

Elixir’s Phoenix Powering a Real-time Web UI

A problem WallarooLabs needed to solve early on was deciding on the tooling that would power our metrics monitoring system. They needed a monitoring solution to provide real-time updates on the several steps a data message may take along the way within a Wallaroo application. This post takes a deeper dive into the monitoring problem and how Phoenix and Elixir helped solve specific issues.

Fortnite: Postmortem of Service Outage at 3.4M CCU

Epic Games posted a postmortem on, among other things, their XMPP outage. They had a situation while mitigating a known instability problem that resulted in overloading a downstream system component and effectively paralyzing presence flow. Without presence, a user who is a friend cannot see that a player is online, breaking most of Fortnite social features including the ability to form parties.

by Marek Foss at June 20, 2018 19:27

June 19, 2018

Paul Schaub

Summer of Code: The demotivating week

I guess in anybodies project, there is one week that stands out from the others by being way less productive than the rest. I just had that week.

I had to take one day off on Friday due to circulation problems after a visit at the doctor (syringes suck!), so I had the joy of an extended weekend. On top that, I was not at home that time, so I didn’t write any code during these days.

At least I got some coding done last week. Yesterday I spent the whole day scratching my head about an error that I got when decrypting a message in Smack. Strangely that error did not happen in my pgpainless tests. Today I finally found the cause of the issue and a way to work around it. Turns out, somewhere between key generation and key loading from persistent storage, something goes wrong. If I run my test with fresh keys, everything works fine while if I run it after loading the keys from disk, I get an error. It will be fun working out what exactly is going wrong. My breakpoint-debugging skills are getting better, although I still often seem to skip over important code points during debugging.

My ongoing efforts of porting the Smack OX code over from using bouncy-gpg to pgpainless are still progressing slowly, but steady. Today I sent and received a message successfully, although the bug I mentioned earlier is still present. As I said, its just a matter of time until I find it.

Apart from that, I created another very small pull request against the Bouncycastle repository. The patch just fixes a log message which irritated me. The message stated, that some data could not be encrypted, while in fact date is being decrypted. Another patch I created earlier has been merged \o/.

There is some really good news:
Smack 4.4.0-alpha1 has been released! This version contains my updated OMEMO API, which I have been working on since at least half a year.

This week I will continue to integrate pgpainless into Smack. There is also still a significant lack of JUnit tests in both projects. One issue I have is, that during my project I often have to deal with objects, that bundle information together. Those data structures are needed in smack-openpgp, smack-openpgp-bouncycastle, as well as in pgpainless. Since smack-openpgp and pgpainless do not depend on one another, I need to write duplicate code to provide all modules with classes that offer the needed functionality. This is a real bummer and creates a lot of ugly boilerplate code.

I could theoretically create another module which bundles those structures together, but that is probably overkill.

On the bright side of things, I passed the first evaluation phase, so I got a ton of motivation for the coming days :)

Happy Hacking!

by vanitasvitae at June 19, 2018 18:47

June 17, 2018

Ignite Realtime Blog

Smack 4.3.0-rc1 and 4.4.0-alpha1 released

@Flow wrote:

The Smack developer community is proud to announce the availability of the first release candidate of Smack 4.3. Users of Smack are encouraged switch to the new 4.3 release family of Smack. The Smack 4.3 API is considered frozen and the API changes between 4.2 and 4.3 are not as significant compared to the changes between Smack 4.1 and 4.2. More information can be found in the Readme of Smack 4.3 (please note that the Readme is work in progress).

Together with the 4.3.0-rc1 release, we have also published the first alpha of Smack 4.4, which includes the updated and improved OMEMO API. Credits for this go to Paul.

As always, all the release artifacts are available on Maven Central.

Posts: 1

Participants: 1

Read full topic

by @Flow Florian Schmaus at June 17, 2018 18:41

June 13, 2018

Monal IM

iOS 3.0.2 is out

I have released 3.0.2 to the iOS App Store.  So far I appear to have resolved the worst crashes.

by Anu at June 13, 2018 12:00

June 12, 2018

Tigase Blog

Tigase services continue to improve working with Amazon Web Services!

Recently, we began utilizing our DualIP variant of our Load Balancing solution (based on see-other-host XMPP semantics) on our servers hosted on AWS and have found out that our load balancing implementation behaves very well with Amazon's ELB environment.

by Daniel at June 12, 2018 23:26

Monal IM

Update on OMEMO

I have an update on the status of OMEMO in Monal. I’ve completed my spike  and have a very rough implementation working. I am able to communicate with Gajim and Chatsecure.  I am actually using a lot of  the same OMEMO code as Chatsecure using Chris’ cocoapods.  The shared code base should reduce duplicated effort and ensure compatibility on the two main Apple platform clients going forward.

The current code isn’t anywhere near production but once I clean it up more, you should start seeing it as an option to turn on in Mac betas in the next month or so.  Below you can see my interactions with Gajim and Chatsecure. 

by Anu at June 12, 2018 02:21

June 11, 2018

Paul Schaub

Summer of Code: Evaluation and Key Lengths

The week of the first evaluation phase is here. This is the fourth week of GSoC – wow, time flew by quite fast this year :)

This week I plan to switch my OX implementation over to PGPainless in order to have a working prototype which can differentiate between sign, crypt and signcrypt elements. This should be pretty straight forward. In case everything goes wrong, I’ll keep the current implementation as a working backup solution, so we should be good to go :)

OpenPGP Key Type Considerations

I spent some time testing my OpenPGP library PGPainless and during testing I noticed, that messages encrypted and signed using keys from the family of elliptic curve cryptography were substantially smaller than messages encrypted with common RSA keys. I knew already, that one benefit of elliptic curve cryptography is, that the keys can be much smaller while providing the same security as RSA keys. But what was new to me is, that this also applies to the length of the resulting message. I did some testing and came to interesting results:

In order to measure the lengths of produced cipher text, I create some code that generates two sets of keys and then encrypts messages of varying lengths. Because OpenPGP for XMPP: Instant Messaging only uses messages that are encrypted and signed, all messages created for my tests are encrypted to, and signed with one key. The size of the plaintext messages ranges from 20 bytes all the way up to 2000 bytes (1000 chars).

Diagram comparing the lengths of ciphertext of different crypto systems

Comparison of Cipher Text Length

The resulting diagram shows, how quickly the size of OpenPGP encrypted messages explodes. Lets assume we want to send the smallest possible OX message to a contact. That message would have a body of less than 20 bytes (less than 10 chars). The body would be encapsulated in a signcrypt-element as specified in XEP-0373. I calculated that the length of that element would be around 250 chars, which make 500 bytes. 500 bytes encrypted and signed using 4096 bit RSA keys makes 1652 bytes ciphertext. That ciphertext is then base64 encoded for transport (a rule of thumb for calculating base64 size is ceil(bytes/3) * 4 which results in 2204 bytes. Those bytes are then encapsulated in an openpgp-element (adds another 94 bytes) which can be appended to a message. All in all the openpgp-element takes up 2298 bytes, compared to a normal body, which would only take up around 46 bytes.

So how do elliptic curves come to the rescue? Lets assume we send the same message again using 256 bit ECC keys on the curve P-256. Again, the length of the signcrypt-element would be 250 chars or 500 bytes in the beginning. OpenPGP encrypting those bytes leads to 804 bytes of ciphertext. Applying base64 encoding results in 1072 bytes, which finally make 1166 bytes of openpgp-element. Around half the size of an RSA encrypted message.

For comparison: I estimated a typical XMPP chat message body to be around 70 characters or 140 bytes based on a database dump of my chat client.

We must not forget however, that the stanza size follows a linear function of the form y = m*x+b, so if the plaintext size grows, the difference between RSA and ECC will become less and less significant.
Looking at the data, I noticed, that applying OpenPGP encryption always added a constant number to the size of the plaintext. Using 256 bit ECC keys only adds around 300 bytes, encrypting a message using 2048 bit RSA keys adds ~500 bytes, while RSA with 4096 bits adds 1140 bytes. The formula for my setup would therefore be y = x + b, where x and y are the size of the message before and after applying encryption and b is the overhead added. This formula doesn’t take base64 encoding into consideration. Also, if multiple participants -> multiple keys are involved, the formula is suspected to underestimate, as the overhead will grow further.

One could argue, that using smaller RSA keys would reduce the stanza size as well, although not as much, but remember, that RSA keys have to be big to be secure. An 3072 bit RSA key provides the same security as an 256 bit ECC key. Quoting Wikipedia:

The NIST recommends 2048-bit keys for RSA. An RSA key length of 3072 bits should be used if security is required beyond 2030.

As a conclusion, I propose to add a paragraph to XEP-0373 suggesting the use of ECC keys to keep the stanza size low.

by vanitasvitae at June 11, 2018 20:41

June 06, 2018

Paul Schaub

Summer of Code: PGPainless 2.0

In previous posts, I mentioned that I forked Bouncy-GPG to create PGPainless, which will be my simple to use OX/OpenPGP API. I have some news regarding that, since I made a radical decision.

I’m not going to fork Bouncy-GPG anymore, but instead write my own OpenPGP library based on BouncyCastle. The new PGPainless will be more suitable for the OX use case. The main reason I did this, was because Bouncy-GPG followed a pattern, where the user would have to know, whether an incoming message was encrypted or signed or both. This pattern does not apply to OX very well, since you don’t know, what content an incoming message has. This was a deliberate decision made by the OX authors to circumvent certain attacks.

Ironically, another reason why I decided to write my own library are Bouncy-GPGs many JUnit tests. I tried to make some changes, which resulted in breaking tests all the time. This might of course be a bad sign, indicating that my changes are bad, but in my case I’m pretty sure, that the tests are just a little bit over oversensitive :) For me it would be less work/more fun to create my own library, than trying to fix Bouncy-GPGs JUnit tests.

The new PGPainless is already capable of generating various OpenPGP keys, encrypting and signing data, as well as decrypting messages. I noticed, that using elliptic curve encryption keys, I was able to reduce the size of (short) messages by a factor of two. So recommending EC keys to implementors might be worth a thought. There is still a little bug in my code which causes signature verification to fail, but I’ll find it – and I’ll kill it.

Today I spent nearly 3 hours debugging a small bug in the decryption code. It turns out, that this code works like I intended,

PGPObjectFactory objectFactory = new PGPObjectFactory(encryptedBytes, fingerprintCalculator);
Object o = objectFactory.nextObject();

while this code does not:

PGPObjectFactory objectFactory = new PGPObjectFactory(encryptedBytes, fingerprintCalculator);
Object o = objectFactory.iterator().next();

The difference is subtle, but apparently deadly.

You can find the new PGPainless on my Gitea instance :)

by vanitasvitae at June 06, 2018 17:21

Erlang Solutions

MongooseIM 3.0.0 - Application turbocharger

MongooseIM 3.0.0 is out and with it come many improvements to our global messaging solution! Over the years we have proven that MongooseIM is the way to go when building a scalable, secure messaging system that never fails. With new features and fixes, our battle tested, highly customisable platform provides an enterprise friendly toolbox everyone can use. Whether you’re an XMPP expert or an entrepreneur looking to bring to life your idea for a community building app, MongooseIM platform helps you build a product tailored to your needs, that will easily grow to match your ambition. Find out what goodies we’ve managed to pack up for you this time and see how we can aid your users’ experience with our truly instant messaging platform.

What is so great about 3.0.0?!

As a team we’ve switched into a faster gear and our latest release is a reflection of that. MongooseIM 3.0.0 is an enterprise ready, stable solution that now works faster then ever; delivering a smooth messaging experience to your users. It features important upgrades that will allow your servers to process even more messages per minute and save memory space for additional user sessions. Everything thanks to a couple of improvements, the new XML parser being the prominent highlight.

Efficiency is not the only reason why we’re so proud of this release. It also features prototype Inbox implementation. It’s an essential extension for virtually every chat application. Our rich experience in this topic allowed us to design a solution that should match most use cases already and will be expended even further!

As usual, there are also other improvements we’d like to share with you. You may find the full list in our changelog but we’ve picked five of them to describe, as we feel you should learn more about them.

Achieve more with the same hardware

Thanks to several important changes and improvements, MongooseIM is now able to process information faster and consume less resources. It means your servers may handle more users and traffic with the MongooseIM upgrade alone.

Depending on your specific application, 25-400% better performance may be expected. Actually, the richer and more complex trafic, the better results you’ll get, compared to previous MongooseIM versions!

Three aspects of MongooseIM have been modified in order to achieve this:

  1. All messages from users are interpreted in a completely new way
  2. More users can connect to a single server per second
  3. All user sessions store as little information as possible when they are idle

Hello inbox

We’ve implemented Inbox features in the past for various projects and the time has come to pour the best ideas and experiences into an extension open for everyone!

A few words of explanation for those not familiar with the Inbox feature. It is the view in a chat application that you see every time you open it. It is a list of all conversations, with excerpts of last messages and unread messages count. Simple as that!

Unfortunately, as there is no official Inbox specification in XMPP yet, we’ve come up with custom protocol (thank you XMPP for your extensiveness!) for this purpose. We’re going to submit it as a XEP (XMPP Extension Protocol) for a review by community but in the meantime you can enjoy its simplicity and intuitiveness. All you need to do is to enable mod_inbox and implement a few simple IQ stanzas in your client application.

Please keep in mind though, this extension is still in experimental stage and will be marked as stable in one of our future releases.

Under the hood

We’ve also added some lower level changes that are going to be useful to developers, CTOs and devops.

Performance bundle: Acceptor pool, session hibernation, new XML parser

We’ve been using expat parser since the very beginning. Recently, our brave C++ warrior in the MongooseIM team thought of replacing it with an alternative - RapidXML. Everything indicated that it might consume less resources if properly used.

And he did use it in an excellent way. The whole C code was rewritten into C++ with the new library integrated. Since RapidXML requires only minimal per-user state (which actually is kept in Erlang terms), much fewer allocations are required and the code is simply cleaner.

What about the performance itself? For smaller stanzas the difference is not drastic, however noticeable. The test involved 100,000 users sending standard, small messages.

Please take a look at CPU usage graphs first:

Fig. 1: MongooseIM 2.2.2 CPU usage over time

Fig. 2: MongooseIM 3.0.0 CPU usage over time

Very similar but new version is a bit better in overall.

Regarding the memory consumption:

Fig. 3: MongooseIM 2.2.2 memory usage over time

Fig. 4: MongooseIM 3.0.0 memory usage over time

Now that’s already impressive. 3.0.0 uses ~2GB of RAM less which is over 25% improvement!

Wait, wait. If that was “impressive” then how do we call this? 1000 users exchanging large, ~36kB messages. CPU goes first again.

Fig. 5: MongooseIM 2.2.2 CPU usage over time

Fig. 6: MongooseIM 3.0.0 CPU usage over time

Well, it’s only 4 times better. :) In terms of memory usage the drop is less significant but still observable.

Fig. 7: MongooseIM 2.2.2 memory usage over time

Fig. 8: MongooseIM 3.0.0 memory usage over time

To wrap it up: all applications will surely benefit from a new parser but it’s especially important for those which process complicated, nested, rich stanzas.

Obviously, rewriting thousands of lines of code in C++ isn’t always the only method of improving performance. Sometimes it’s about small ideas, such as using a pool of acceptors or eager hibernation.

The former involves using more than one process to accept incoming connections from clients. Despite being a fairly cheap operation (it’s only about accepting a connection and creating a client process), this single process became a bottleneck in some applications. In 3.0, 100 acceptors are created by default.

The latter is bit more complicated. For people less familiar with Erlang this might sound similar to Java’s Hibernate framework. Actually, Erlang’s hibernation is not related to database mappings. When a process gets hibernated, its memory is garbage collected and it is removed from a scheduler queue (a bit of a simplification, but you get the idea). We already had such a mechanism in place but the hardcoded timeout for hibernation was 60s. When you think about it, client’s process is idle most of the time, at least in a computers’ world where a single CPU cycle takes less than 1ns. Load tests have proved that hibernating immediately after processing a stanza leads to lower memory usage at minimal cost to CPU time.

The first two graphs show CPU usage in a presence-based test. Most of the time ~6.5k users were connected and sending a presence update every 20 seconds to a roster of 8 friends.

Fig. 9: MongooseIM 2.2.2 CPU usage over time

Fig. 10: MongooseIM 3.0.0 CPU usage over time

Fairly similar I’d say. The second graphs shows memory usage decrease in the same test.

Fig. 11: MongooseIM 2.2.2 memory usage over time

Fig. 12: MongooseIM 3.0.0 memory usage over time

Now, that looks definitely better, right? However, if it turns out that frequent hibernations impact your server’s performance, you may easily tune the timeout in the configuration file (look for the hibernate_after option).

Improved ODBC support

Before 3.0, MongooseIM was using the odbc application from OTP to execute queries via ODBC. Unfortunately, this library is not maintained actively enough to match our requirements. It especially applies to SQL types support. Lucky for us, there is a community-developed repository named eodbc. With its help, MongooseIM’s compatibility with e.g. MSSQL improved significantly. What is more, in order to ensure it, we’ve begun testing ODBC connection with MSSQL on Travis!

A byproduct of this refactor is a completely new escaping API in our RDBMS layer. It’s more intuitive now and much less error prone. Now it’s virtually impossible to use unescaped value in a query and the escaping is always done appropriately for a chosen RDBMS.

Farewell, Message Archive Management v0.2!

Getting rid of MAM 0.2 support means several things, such as easier code maintenance. For example, there were completely separate functions and tests present to handle 0.2 stanzas.

Another important difference is a lack of “archived” element. Please configure MongooseIM to inject “stanza-id” into messages instead.

Despite its importance (MAM 0.2 was the first version supported by MongooseIM), it has become obsolete over time. If your application still uses MAM 0.2, we highly recommend you update your XMPP library and the code using it. Storage backend wise, newer versions are backwards compatible.

Changelog

Please feel free to read the detailed changelog, where you can find a full list of source code changes and useful links.

What’s next?

After introducing big, important changes over past few releases, we’re going to take our time to polish what we already have.

Above all else, the Inbox feature is going to be expanded. We’re planning to support more backends and introduce new functions (e.g. sorting by timestamp). Also, we’d like to propose a proto XEP to the XMPP community, so the conversations list we’ve design may become an official standard common to clients and other servers. After all, MongooseIM is not our only priority, as we care for the state of XMPP as a communication protocol as well!

We’re slowly heading towards a configuration file revolution. It’s highlight will be a new config format, that will be friendlier for everyone not familiar with Erlang syntax. Currently we’re considering YAML, TOML, Cuttlefish and Conform. What is more, the configuration will undergo a major cleanup and flexibility improvements.

The third item I’d like to share with you applies to everyone working with MongooseIM code. One of the main structures in our server, Mongoose Accumulator (or mongoose_acc) is going to be significantly refactored. Our aim is to make it more intuitive and organised. Its content will be richer, with clear contract and scope. We’re redesigning it as you read these words. :)

Test our work on MongooseIM 3.0 and share your feedback

  1. Help us improve the MongooseIM platform:
  2. Star our repo: esl/MongooseIM
  3. Report issues: esl/MongooseIM/issues
  4. Share your thoughts via Twitter: twitter.com/MongooseIM
  5. Download Docker image with new release
  6. Sign up to our dedicated mailing list to stay up to date about MongooseIM, messaging innovations and industry news.
  7. Check out our MongooseIM product page for more information on the MongooseIM platform.

June 06, 2018 10:52

Jérôme Poisson

Decentralized code forge, based on XMPP

With the recent announcement concerning the biggest known centralized code forge owner change, we have seen back here and there discussions about the creation of a similar tool, but decentralized.

I've used this occasion to recall the work done to implement tickets and merge requests in Salut à Toi (SàT), work which was relatively unoticed at the time of writing, about 6 months ago.

Now, I would like to bring some details on why building those tools.

First of all, why not the big forge? After all, a good part of current libre software is already using it! Well first it's not libre, and we commited ourself in our social contract to use libre software as much as possible, infrastructure included. Then because it's centralized, and there too our social contract is pretty clear, even if it's not as important for infrastructure as it is for SàT itself. Finally, because we are currently using Mercurial, and the most famous forge is build around Git.
We do not hide the fact that we already ask ourselves wether to use this platform or not in general assemblee (cf. minutes – in French –), we were mainly interested in the great visibility it can offer.

« It's centralized? But "Git" is decentralized! » is a point we are ofter hearing and it's a true fact, Git (and Mercurial, and some others) is decentralized. But a code forge is not the version control system, it's all the tools arount it: hosting, tickets, merge/pull requests, comments, wikis, etc. And those tools are not decentralized at the moment, and even if they are often usable throught a proprietary API, they are still under centralization rules, i.e. rules of the hosting service (and its technical hazards). This also means that if the service doesn't want a project, it can refuse, delete, or block it.

Centralization is also a technical facility to catalog and search project… which are on the service. Any external attempt will then have more difficulties to be visible and to attract contributors/users/help. This is a situation we know very well with Salut à Toi (we are not present on proprietary and centralized "social networks" for the same reasons), and we find it unacceptable. It goes without saying that concentrating projects on a single platform is the best way to contribute and exacerbate this state of affairs.
Please note, however, that we are not judging or attacking people and projects who made different choices. These positions are linked to our political commitment.

Why, then, not using existing Libre projects, already advanced and working, like Gitlab? Well, first because we are working with Mercurial and not Git, and secondly because we would put ourselves here too in a centralized solution. And there is an other point: there are nearly no decentralized forges (Fossil maybe?), and we already have nearly everything we need with SàT and XMPP. And let's add that there is some pleasure to build the tools we are lacking.

SàT is on the way to be a complete ecosystem, offering most, if not all, the tools needed to organise and communicate. But it is also generic and re-usable. That's why the "merge requests" system is not linked to a specific SCM (Git or Mercurial), it can be used with other software, and it is actually not only usable for code development. It's a component which will be used where it is useful.

To conclude this post, I would like to remind that if we want to see a decentralized, ethical and politically commited alternative to build our code, organise ourself, and communicate, we can make this real by cooperating and contributing, being with code, design, translations, documentation, testing, etc.
We got recently some help for packaging on Arch (thanks jnanar and previous contributors), and there are continuous efforts for packaging in Debian (thanks Robotux, Naha, Debacle, and other Debian XMPP packagers), if you can participate, please contact us (see our official website), together we can make the difference.
If you are lacking time, you can support us as well on Liberapay: https://liberapay.com/salut_a_toi. Thanks in advance!

by goffi at June 06, 2018 05:50

Monal IM

iOS 3.0.2 and OSX 2.1.2 betas out

I am still cleaning up all of the issues people have seen (and some old friends) in the latest releases. There are new betas out.  I will looking for feedback and crash reports. I hope to have the next updates out this week.   I know it has been almost weekly releases since the 3.0 release. Hoping to slow down to a more manageable  release cycle after the code is more stable.

by Anu at June 06, 2018 02:51

June 04, 2018

Tigase Blog