@Flow wrote:

The Ignite Realtime developer community is happy to announce the availability of Smack 4.3.0.

This is the first release of the new Smack 4.3 series, which brings new features and bug fixes.

Compared to the API changes between Smack 4.1 and 4.2, the changes of the Smack 4.3 are far less intrusive. Remember that the 4.2 API changes where mostly caused by the introduction of the Jid type, replacing String which was previously used to represent a JID.

We like to thank everyone who contributed by reporting bugs or giving suggestions. A special thank goes to people who contributed code:

$ git shortlog -sn 4.2.4..4.3.0
   227  Florian Schmaus
    22  vanitasvitae
    14  Jesus Fuentes
     6  Paul Schaub
     3  Guus der Kinderen
     2  Timothy Pitt
     2  Yash Thakkar
     1  Boris Grozev
     1  Cem Yabansu
     1  Daniel Hintze
     1  Fernando Ramirez
     1  Mohsen Hariri
     1  Rajat Kumar Gupta
     1  Son Goku
     1  Thomas
     1  akrherz

More information about the 4.3.0 release can be found one the release’s Readme.

Posts: 1

Participants: 1

Read full topic

Welcome to the latest edition of the XMPP newsletter!

If you have an article, tutorial or blog post you'd like us to include in the newsletter, please submit it on the XMPP wiki.

News

Slack has acquired HipChat's (and Stride's) intellectual property from Atlassian. HipChat was a proprietary service based on non-federated XMPP.

Prosody has added various PEP improvements which will be included in a future 0.11 release.

The Java XMPP library Smack now has OpenPGP support.

The Freedombone project now supports automatic XMPP notifications in addition to email notifications. Since publication of this post, Bob Mottram has informed us that notifications are now sent out as "headline" messages.

Dino has received a "filter expressions" feature as part of a Google Summer of Code project, allowing advanced search through messages.

Skype for Business Server will deprecate XMPP Gateways in 2019.

Subscribe to receive the next edition in your inbox

Guides and Howtos

In XMPP With Node.js, Jeremy Jackson documents how he sets up a development environment for writing an XMPP echo bot in server-side JavaScript.

Divya Prajapati has published a two-part guide on how to create an XMPP Android client: Part 1: OpenFire server installation Part 2: Creating an Android XMPP client

Videos

Daniel Gakwaya has made a video showing how to send and receive files with XMPP using HTTP-upload.

Software releases

Servers

• Jackal 0.3
  • Jackal is a relatively new XMPP server written in Go. This release adds server-to-server federation support.
• MongooseIM 3.1
  • Amongst other things, the Inbox feature has received new features, such as MSSQL support and support for MUC groupchats.

Clients

• Salut à Toi 0.7 alpha
  • Features include forums, photo albums, file sharing and issue tracking.

Libraries

• Babbler 0.8.0
  • Adds support for XEP-0390: Entity Capabilities 2.0 and XEP-0392: Consistent Color Generation.
• aioxmpp 0.10 An XMPP library for use with Python 3.4+ asyncio.

Other

• XMPP Cloud Auth v1.1.0
  • Allows you to use Nextcoud as authentication service for XMPP servers.
• Real-Time Benchmark
  • An open source benchmarking tool to stress-test real-time protocols, such as XMPP.
• XMPP Echo bot written in SED
  • This project received some admiration in this Hacker News discussion

New Services

• Muclumbus
  • A search engine for public Multi-User Chat Rooms

Our next trunk nightly release uses a new implementation of PEP (XEP-0163) by default. PEP is used for a number of things, primarily sharing with your contacts: Extended statuses (e.g. sharing the user’s current playing music, their mood, activity, etc.) Avatars OMEMO keys Chatroom bookmarks While PEP started out as a simplified form of ‘pubsub’ and a way to share information with your contacts, it quickly became clear that it was also a neat mechanism for sharing public info with non-contacts (such as OMEMO keys) or sharing private data (such as saved chatrooms) with other clients on your account.

I am very proud to announce, that Smack got support for OpenPGP for XMPP!

Today the Pull Request I worked on during my GSoC project was merged into Smacks master branch. Admittedly it will take a few months until smack-openpgp will be included in a Smack release, but that gives me time to further finalize the code and iron out any bugs that may be in there. If you want to try smack-openpgp for yourself, let me know of any issues you encounter

(Edit: There are snapshot releases of Smack available for testing)

Now Smack does support two end-to-end encryption methods, which complement each other perfectly. OMEMO is best for people that want to be safe from future attacks, while OpenPGP is more suited for users who appreciate being able to access their chat history at any given time. OpenPGP is therefore the better choice for web based applications, although it is perfectly possible to implement web based clients that do OMEMO (see for example the Wire web app, which does ratcheting similar to OMEMO).

What’s left to do now is updating smack-openpgp due to updates made to XEP-0373 and extensive testing against other implementations.

Happy Hacking!

Quick update:

Only a few days are left until the last and final Evaluation Phase.

I spent the week opening my pull request against Smacks master branch and adding a basic trust management implementation. Now the user is required to make decisions whether to trust a contacts key or not. However, the storage implementation is kept very modular, so an implementor can easily create a trust store implementation that realizes custom behaviour.

Smack-openpgp now allows users which did not subscribe to one another to exchange encryption keys quite easily. If a user receives an encrypted message, the implementation automatically fetches the senders keys to allow signature verification.

Furthermore there are more JUnit tests now, so that Smacks total test coverage actually increases when my PR gets merged

Happy Hacking!

This summer the MongooseIM team have not had a second to be lazy - the development of MongooseIM 3.1 release took over what was supposed to be our downtime, when all major activities driven throughout the year slow down. We’re happy to say that in this time of leisure, the new release is packed with important features, improvements and excitement. Take a look at the version highlights and be part of this major step in creating a world-class communication server.

This time, a large part of the changes are focused on development efficiency, but rest assured, we’ve added new items you are going to appreciate.

In the “Test Runner” section, you get to learn all about a little tool we provided so that you can easily set the development environment up and run tests locally without Travis CI.

Our Inbox extension has got three big updates that push the boundaries of what’s currently offered in XMPP world. As a rule, we want to lead by example; less talk and more action. That is why working in cooperation with Forward, we decided to put forward an implementation of a popular use case as a unique feature in this software sector.

This release is also an important lesson for us. A lesson about edge cases and concurrency in integration testing. You don’t necessarily have to be an Erlang developer to benefit from the third section, but reading it allows you to learn with us.

The “Honorable Mentions” section may seem minor, but for some projects the items listed there can indeed make a difference! It’s a candy mix of different changes, so read carefully not to miss your favourite flavours!

Obviously, a single blog post is too small a space to tell a profound story about every new item in the changelog, so we encourage you to check it out. You can find a link at the bottom of this blog post.

Test Runner

The Travis CI is our main verification tool for the Pull Requests we create and merge. Whilst being convenient and critical for our process, it is not very handy for day-to-day development. It is very common to frequently execute limited subset of tests to ensure that a new piece of code we wrote works perfectly. However, waiting for Travis results every time would extend implementation time excessively as it executes multiple presets, while being a shared resource at the same time.

The test runner is a script that helps to set the development environment up, and run tests on a developer machine locally. The test runner shares a lot of code with our Travis CI build scripts, which ensures that test results are consistent between local and CI runs.

The test runner allows to choose which tests to run. It can be useful, when one of the tests is failing and we want to rerun it.

Since MongooseIM supports different database backends, the test runner is able to set a selected database up for you and configure MongooseIM. Simply put, it prepares the local environment before executing actual test suites.

The test runner supports a lot of options to customise your build and test execution scenario. It also supports shell autocompletion for option names and values.

We’ve prepared a recording for you that presents a basic usage of the script.

.

Please note that Asciinema allows you to pause and copy any text fragment you like, so it would be very easy for you to repeat the same steps.

New Inbox Features

Originally sponsored by and created for Forward, Inbox has been available as an open source extension for the last two months already. In MongooseIM 3.1, it returns with a bundle of fresh goodies. The first feature is MSSQL support.

Despite being less frequently used with MongooseIM compared to MySQL or PostgreSQL, it’s still an important piece of the architecture for many projects, especially those running in Azure cloud. We don’t want you to feel excluded, dear Microsoft users!

The second one is the support for classic MUC group chats. MUC Light was never intended as a complete replacement for original XEP-0045 solution. It means that numerous projects exists where mod_muc is still a better match than its younger sibling, and they may now benefit from inbox features as well!

Last but not least is the timestamp support. First of all, they are stored in DB and returned in Inbox query results. For those using mod_inbox from MIM 3.0: you’ll need to update your schemas but don’t worry - it isn’t very complicated. What’s more, a client may now request conversations from a certain time period and sort them by timestamp, both ascending and descending.

This is not our final word on this matter. You may expect further improvements to this extension in upcoming MongooseIM versions!

We’ve prepared a DEMO of the Inbox feature. It shows both the backed and the frontend side of it. The application used in the demo DEMO has been designed by Forward.

Lessons learnt regarding CI

OK, these are short and sweet but nevertheless important:

1. Avoid RPC in integration tests. They tend to time out in slow CI environments (such as Travis).
2. When test users exchange messages, always wait until they are received to ensure proper synchronisation.
3. On a slow machine, MSSQL SELECT query may return more than one row (even when retrieving by the exact primary key value) as a consequence of the transaction deadlock.
4. When you can’t use any other means of server state synchronisation, don’t use hardcoded sleep periods; replace them with an incremental backoff and verification in a loop. Sometimes you can’t predict whether a server state is updated properly in 500ms, 1000ms or 3000ms. Adding 5s waits everywhere may cause test suites to run veeery long.
5. Be careful about leaking presences between cases. This applies to XMPP testing. Best practice is to generate fresh user accounts for every scenario.
6. Some databases don’t support transactions so the new data may not be instantly available. For example, in the case of Riak (its Search function in particular) a delay between data insert and query is required.
7. Sometimes creating a schema in a DB may fail for the first time due to timing issues, so implement a retry loop in a DB provisioning scripts. This also applies to Riak.
8. Did I mention creating new user accounts for every test case? It actually applies not only to XMPP. With this practice, you won’t have to worry about possible leftovers of a user’s state.

Honorable mentions

ElasticSearch backend for Message Archive

Almost every MongooseIM module supports more than one type of backend. Usually it’s Mnesia, RDBMS and sometimes Riak. Message Archive Management is a noteworthy exception, as we’ve implemented RDBMS, Riak and Cassandra support for this module. Or “modules” actually, as it consists of over 30 Erlang files already.

It is a very special feature as it processes a vast amount of data and sometimes executes expensive queries. In order to ensure performance and match a project’s architecture, wide range of supported DB backends is essential.

It is our pleasure to announce that yet another backend has joined the family: ElasticSearch.

OTP 21 support

OTP 21.0 has been released ~1 month ago and we’ve added support for this version less than a week after! This is great news for all projects sticking to the most recent Erlang technology as pioneers in BEAM world. The new platform version brought not only improvements in regards to performance but also some incompatibilities that we’ve resolved, so MongooseIM still remains at a technological peak.

As a tradeoff, we’ve dropped official support for OTP 18.x. It should still be possible to compile 3.1 with this version with some minor code modifications, but we’re definitely moving forward. It has allowed us to get rid of non-typed maps specifications as an example. As a reminder, bear in mind, that MongooseIM always supports two most recent, stable OTP branches (currently these are 20.x and 19.x and one being under an active development, 21.x).

Jingle/SIP tutorial

SIP is a common choice for VoIP applications but certain XMPP features may be a very good match for such software. MongooseIM is able to liaise between these two worlds and now it’s easier than ever with significantly extended documentation (compared to the level in 3.0) and a tutorial on mod_jingle_sip usage.

Worker pool unification

Every developer writes a custom worker pool at some point of their career. Everyone. Certain MongooseIM components (the ones that use connection pools) were created with different preferred library in mind. As a result, we’ve ended up with many kinds of worker pools: cuesport, worker_pool and poolboy. It wasn’t only a matter of maintenance difficulty, but performance as well. As an example, cuesport supports only a simple round-robin job assignment algorithm, which is not optimal in every case. It also lacks inspection of any kind.

Given our experience gathered over the years, we’ve selected worker_pool as our primary library. It is very flexible and exposes a dedicated stats API. It was originally created by Inaka, however it is actively maintained by Erlang Solutions at this present time.

For now, the changes are purely internal. Some projects may observe better performance but the primary goal was to prepare for a second round of unification. Stay tuned for more details in near future.

Changelog

Please feel free to read the detailed changelog. Here, you can find a full list of source code changes and useful links.

Contributors Special thanks to our contributors: @SamuelNichols @Beisenbek @GalaxyGorilla!

Test our work on MongooseIM 3.1 and share your feedback

Help us improve the MongooseIM platform:

1. Star our repo: esl/MongooseIM
2. Report issues: esl/MongooseIM/issues
3. Share your thoughts via Twitter: @MongooseIM
4. Download Docker image with new release
5. Sign up to our dedicated mailing list to stay up to date about MongooseIM, messaging innovations and industry news.
6. Check out our MongooseIM product page for more information on the MongooseIM platform.

As previously mentioned, I have in mind to write a novel about the ancient Greek philosopher Pyrrho and his journey to central Asia and northwestern India with the army of Alexander the Great. As part of my research I'm reading a great deal of philosophy, history, and fiction. It's been fascinating to return to the reading of fiction after a lapse of many years - I feel like I have fresh eyes for everything. My readings in the genre of philosophical fiction have ranged all over the map - during a business trip last week I devoured The Name of the Rose by Umberto Eco, before that I re-read The Left Hand of Darkness by Ursula K. Le Guin, and I'm currently reading both The Picture of Dorian Gray by Oscar Wilde and The Awakening by Kate Chopin. Alongside these works of fiction I'm reading books of philosophical scholarship (the latest were a full-length study of Pyrrho and an exploration of epistemology after Protagoras) and I'm starting to explore the historical background to the massive cultural change from classical to hellenistic Greece. Umberto Eco's book in particular has given me a new ambition of combining the story of Pyrrho's intellectual development with a mystery novel about the still unexplained death of Alexander the Great! It turns out the Pyrrho was probably not quite the radical skeptic he's usually taken to be (although he did believe that the entities we encounter in reality are indefinite and thus hard to pin down in many ways). Combine this with political intrigue on an imperial scale and perhaps regicide (indeed, there's an ancient legend that Aristotle himself was somehow involved in the assassination of his former pupil Alexander, perhaps in revenge for Alexander's execution of Aristotle's nephew Kallisthenes), and we might have the makings of large-scale novel that hinges in part on the validity of the law of non-contradiction! This is all quite speculative at this point because I have over 200 more books to read during the research phase, and I would also need to intensively study how the better authors construct their novels (an art of which I am mostly ignorant). Although I'm not sure if I have a book this big in me, I'm enjoying the exploration of possibilities. And all this reading has kept me from blogging, thus my absence here. Hopefully the results will be worth waiting for......

I’m very happy and proud to announce the first alpha release of PGPainless!

PGPainless 0.0.1-alpha1 is the first non-snapshot release and is available from maven central. It was an interesting experience to go through the process of creating a release and I’m looking forward to have many more releases in the future

The current release contains a workaround for the bug I described in an earlier blog post. The issue was, that bouncycastle wouldn’t mark the public sub keys of a secret key ring as sub keys, which results in loss of keys if the user tries to create a public key ring from the exported public keys. My workaround fixes the issue by iterating through all sub keys of an existing key ring and converting the key packages of subkeys to subkey packages. The code is also available as a gist.

Ironically I had some issues related to OpenPGP during the release process. Releases to maven central have to be signed with an OpenPGP key, so I created a dedicated signing key pair using GnuPG, which I wanted to put into a separate GPG key box. After creating the key, I exported it using

gpg --armor --export-secret-keys [key-id] > pgpainless-singing-key.asc

imported it into a dedicated key box using

gpg --no-default-keyring --keyring pgpainless.gpg --import pgpainless-signing-key.asc

and deleted the key from my old key box, as well as the .asc-file. But when I tried to sign my release, I got the error, that a secret key would be needed. After checking the key box, I noticed, that only a public key was present.

Unfortunately that key had already been published to the key servers and I have no way to revoke it, due to lack of a secret key. I have no idea, what exactly happened or how it could happen, but its too late to recover the key. So in the end I had to create a new OpenPGP key, which I now carefully backed up on a freshly bought USB stick which will be locked away for the event that I lose the copy on my work station. Better safe than sorry.

Happy Hacking!

After a longer period of development, Babbler 0.8.0 has just been released today.

The new major features are certainly

• support for Java NIO (using the Netty framework), which is a corner stone for future development (like an XMPP server or Serverless Messaging),
• improvements to the JID implementation (e.g. it now uses the upgraded PRECIS framework as per RFC 8264)
• and support for XEP-0390: Entity Capabilities 2.0 and XEP-0392: Consistent Color Generation.

Further information about the release can be found in the release notes and in the changelog.

ProcessOne is excited today to announce the first release of RTB (Real-Time Benchmark), an open source benchmarking tool to stress-test real-time protocols. The aim of this project is to define an easy benchmark for XMPP and MQTT servers, with minimal configuration overhead.

Why

To compare server implementations and be sure they are ready for massive traffic, there is a need for a “golden standard” benchmarking tool that is easy to use and avoids ambiguity due to configuration or overhead – each test should have the same baseline and capabilities for the results to be comparable.

We believe RTB fits such a role, because it has sane defaults (gathered from statistics of real world servers) and is able to cover all the test features defined in the XMPP Compliance Suite 2018.

Status

RTB is in early stages of development: currently, only XMPP protocol is implemented, and support for Multi-User Chat (MUC) and Personal Eventing Protocol (PEP) is lacking.

However, “sane” defaults and what should be considered a “golden standard” for this benchmark is open for discussion with the XMPP community – we welcome issues and pull requests. The tool has already been battle-tested: we used RTB to stress test our ejabberd SaaS deployments with more than 2 000 000 concurrent connections.

RTB includes a script to populate your server’s database with user accounts and rosters, so it’s easy to create test pools. When you start a test with RTB, you can monitor a local website on port 8080 with live graphs like the one below:

Requirements

We want RTB to be straightforward and easy to deploy. Currently, only Unix-based operating systems are supported. To compile, you just need gcc, make, Erlang, expat, yaml, openssl, zlib and gnuplot. There’s a default YAML configuration file to get you started and only 5 mandatory parameters before you can lanuch your first benchmark.

RTB is available on GitHub under Apache License 2.0 and we are waiting for your feedback!

I passed the second evaluation phase Now begins the final spurt, as the last month of GSoC has begun. My main goal can be summarized as follows: Get everything merged!

To get that done, I have to polish up my smack-openpgp branch which has grown to a size of 7000 loc. There are still some minor quirks, but Florian recommended to focus on the big picture instead of spending too much time on small details and edge cases.

I also have to release pgpainless to maven central and establish some kind of release cycle. It will be a future challenge for me personally to synchronize the releases of smack-openpgp and pgpainless.

But now enough talking, I have to get to work

Happy Hacking!

Quite some time has passed since I bothered you with my last post A lot has happened since, I have been making steady process in both smack-openpgp, as well as pgpainless.

One big step that I took was to get rid of smack-openpgp-bouncycastle, which now has been merged into smack-openpgp. Having modular code may be worthwhile, however it poses some big challenges. The biggest problem with having smack-openpgp not depend on smack-openpgp-bouncycastle was, that I could not use classes that represent encryption keys directly in smack-openpgp. Instead I had to create interfaces that encapsule functionality and call those in order to get stuff done from inside smack-openpgp. Last week me and flow decided that it would make my job a lot easier if we just got rid of smack-openpgp-bouncycastle by merging the two modules. In case there will be another implementation at some point, the code would still be modular enough to allow extension by overriding classes and methods.

Now smack-openpgp depends on pgpainless directly, which means that I don’t have to create duplicate code to get bundled information from pgpainless to smack-openpgp for instance. This change gave me a huge performance boost in the development process, as it makes the next steps much more clear for me due to less abstraction.

I rewrote the whole storage backend of smack-openpgp, keeping everything as modular as possible. Now there are 3 different store types. One store is responsible for keys, another one for metadata and a third one for trust decisions. For all of those I created a file-based implementation which just writes information to files. An implementor can for example chose to write information to a database instead. For all those store classes I wrote a parametrized junit test, meaning new implementations can easily be tested by simply inserting an instance of the new store into an array.

Unfortunately I stumbled across yet another bug in bouncycastle, which makes it necessary to implement a workaround in my project until a patched version of bouncycastle is released.
The issue was, that a key ring which consists of a master key and some subkeys was not exported correctly. The subkeys would be exported as normal keys, which caused the constructor of the key ring to skip those, as it expected sub keys, not normal keys. That lead to the subkeys getting lost, which caused smack-openpgp to be unable to encrypt messages for contacts which use a master key and subkeys for OpenPGP.

This bug has been fixed pretty fast by the bouncycastle team and the minimal test I created to illustrate my problem has been incorporated into bouncycastle. Thank you

Currently I’m working on a workaround for the bug in smack-openpgp, but that work is already working. Next I will polish up my test client and do some more field testing to iron out all the edge cases I probably overlooked

Happy Hacking!

Today I added some checkstyle rules to PGPainless.Checkstyle is a gradle plugin, which checks the source code for style violations.

Some say, strict checkstyle rules are unnecessary and that it is annoying to be held back from pushing a commit to the master branch only to fix “style issues” for half an hour. I must say, in the beginning I thought the same way. I was annoyed thinking “why does it matter, if a line comment ends with a period or not?” But after being forced to give it a try when I first became a contributor to the Smack project, I became a fan of it. In the beginning I had to often recommit my changes because they broke the checkstyle rules. For example I often forgot to leave spaces between mathematical operators. I would write “i = 5+5;” instead of “i = 5 + 5;”. But after some amount of time, I got less and less warnings.

I adopted most of the (honestly *very* strict) rules in Smacks rule set to my own coding style. I like how it automatically leads to cleaner, more uniform code (not that it is impossible to write garbage with it of course). For that reason, I decided to put those rules into place in PGPainless today (I only left one rule out, because who the hell cares about the alphabetical sorting of imports???).

At some point, PGPainless will be released as a maven artifact. In preparation for this historical event, I bought the domain pgpainless.org. For now it is just a forwarding to the PGPainless git repository, but I will likely setup a small website with documentation etc. at some point.

During my testing of Smacks OX implementation, I came across an interesting problem. When a user queries a PubSub node in Smack, Smack first does a disco#info query on that node to determine, whether it is a LeafNode or a CollectionNode. This normally works fine. However, it becomes more and more popular to make use of the PubSub access model ‘open’. The open access model makes a PubSub node accessible to entities (like other users) which are not in the contact list of the user. This enables the use of OMEMO in group chats, where not every participant is in your contact list for example.

The problem is that a server which allows access to open PubSub nodes, does not necessarily allow the disco#info query. The question is: Should disco#info queries on open PubSub nodes be allowed or not? An argument against it is, that it might allow “jid-harvesting”. An attacker might use disco#info queries on open PubSub nodes in order to determine, whether the user exists or not. This is a bad thing, because it allows spammers to collect the Jabber IDs of potential victims. On the other hand however, the attacker could simply do a direct PubSub query on the open node and the result would be the same. The benefit of allowing disco#info queries would be, that you can in fact determine the node type.
For now my mail to the standards mailing list remained unanswered, but I think that there should be a well defined expected behavior for this edge case.

For now I worked around the issue by using Javas reflections to access the LeafNode constructor directly, avoiding the disco#info query.

Other than that, I didn’t get a whole lot done this week. Unlike the demotivating week though, this time the reason was primarily exciting new hardware

Happy Hacking!

Welcome to the XMPP newsletter.

If you have an article, tutorial or blog post you'd like us to include in the newsletter, please submit it on the XMPP wiki.

News

• Decentralized code forge, based on XMPP

Github has been acquired by Microsoft, prompting some FOSS developers with long memories to reconsider whether they still want to host their projects there.

The Salut à Toi project is working on a decentralized and federated solution for issues and merge requests, based on XMPP, and Jérôme Poisson explains in this article their motivation for doing this work.

• Project MAXS, which stands for Modular Android XMPP Suite, allows you to control your Android device and receive notifications over XMPP.

• XMPP vs Firebase: Choosing the right platform for real-time chat apps

Every now and then a blog post comparing XMPP and Firebase appears. Generally these posts seem to be poorly disguised attempts at SEO and to promote application development by a particular devshop. This submission seems to follow the trend and is also relatively superficial. It is however interesting to note that they mention multiple times that it's difficult to set up an XMPP server. Perhaps something to take note of and to consider how we could ease setup of an XMPP-based solution.

• Monal now has initial OMEMO support

Monal has received a lot of updates last months and now it also has initial OMEMO support. More work needs to be done, so don't expect to see this in the App Store before August.

• Paul Schaub continues writing about his Summer of Code adventures:

  • Command-line OX client
  • PGPainless 2.0
  • Evaluation and key lengths
  • The demotivating week
  • An almost three line fix to a three days problem

Software releases

Servers

• Jackal 0.2.0

The XMPP server written in Go has a new release with support for Websocket (RFC-6395), XEP-0191 (Blocking Command), XEP-0012 (Last Activity) and XEP-0237 (Roster Versioning).

• Ejabberd 18-06

The new release includes improved robustness of external authentication backends, support for REST API custom headers and more.

• MongooseIM 3.0.0

This release includes the so-called "Inbox" feature which was also discussed at the XSF summit in FOSDEM. It is a list of all ongoing conversations, with excerpts of last messages and unread messages count, to help you gain an overview of what happened while you were gone. There is no XEP for this yet, but Erlang Solutions have promised to submit one.

Clients

• Monal 3.0.2 has been released.

• JSXC 3.4.0 released including support for location sharing and improved roster management in our internal Nextcloud backend.

Libraries

• XMPPFramework 4.0

Chris Ballinger has released Version 4.0 of XMPPFramework, the XMPP library for iOS and macOS. It brings support for XEP-0048: Bookmarks and XEP-0359: Stanza Ids. It also has improved Swift support with optional Swift-only features.

• Smack 4.3.0-rc1 and 4.4.0-alpha1

This ejabberd 18.06 release includes, after two months of development, a lot of bug-fixes and many improvements. There are also a few changes, so please read carefully the release notes.

Noticeable changes

Stop ejabberd initialization on invalid/unknown options

Since some time now, we are warning ejabberd admins in log file about incorrect configuration options. We feel that we are ready to make these hints strong requirements about config file validity.

Starting with ejabberd 18.06, ejabberd will not ignore unknown options and doesn’t allow to have options with malformed values. The rationale for this is to avoid unexpected behaviour during runtime, i.e. to conform to “fail early” approach. We thus hope to be helpful to newcomers by making sure they properly format their config files to get the expected behaviour.

Note that it’s safe to reload a configuration with potentially invalid and/or unknown options:
this will not halt ejabberd, but will only prevent the configuration from loading.

NOTE FOR PACKAGE BUILDERS
This new behaviour should be documented in the upgrade notes.

Improve robustness of external authentication backends

Now all external ports are attached to supervising processes and requests are balanced in round-robin manner until the pool is exhausted.

The commit also deprecates extauth_instances option and introduces extauth_pool_size option instead, with the default value of a number of logical processors (i.e. CPU cores).

Add new options for OOM watchdog

• oom_watermark: 1..100
  Start OOM watchdog only when system memory usage exceeds this value in percents. When the usage drops below the value, OOM watchdog is stopped. The default is 80 (percents). Note that once OOM watchdog is started, it performs full garbage collection periodically: this can be seen as spikes in CPU utilization and drops in RAM usage. If your system is permanently above the watermark, it may cause significant CPU overhead.

• oom_queue: positive integer
  Only trigger OOM killer when total amount of messages in all queues of all Erlang processes is above this value. The default is 10000. Note that this value only takes effect when oom_killer is set to true (this is the default). Otherwise, only a warning will be logged.

Add support for REST API custom headers

ext_api_headers can be defined as a single string. Headers are separated by comma. Definition MUST NOT contain spaces. Example “X-MyHead:test,X-Token:082748”

Optimize HTTP requests memory usage

Due to historical reasons, ejabberd loads the whole file/data into the memory when serving an HTTP request. This is now improved:

1. For GET requests ejabberd uses sendfile(2) if the underlying connection is HTTP and falls back to read/write loop with 64kb buffer for HTTPS connections. This type of requests are handled by mod_http_fileserver, mod_http_upload, ejabberd_captcha, etc
2. POST requests are now limited to 20Mb and are fully downloaded into the memory for further processing (by ejabberd_web_admin, mod_bosh, etc)
3. PUT requests (e.g. for mod_http_upload) are handled by read/write loop with 64kb buffer

Support SASL PLAIN by xmpp_stream_out

Also, SASL mechanisms chaining is now supported: if several mechanisms are supported and authentication fails, next mechanism in the list is picked, until the list is exhausted. In the case of a failure, the latest SASL failure reason is returned within handle_auth_failure/3 callback.

Drop support of mod_irc

If you need IRC support, we suggest to use biboumi. We will not maintain mod_irc anymore and moved it to ejabberd-contrib repository.

SQL schema change

As mod_irc has been obsoleted, we removed the related SQL table from the schema: irc_custom

Changes

Admin

• Stop ejabberd initialization on invalid/unknown options
• Add new options for OOM watchdog: oom_watermark and oom_queue
• Add ability to modify version string
• Add option ext_api_headers to define REST API custom headers
• Fix Erlang limits in ejabberdctl.cfg.example to reflect current situation
• Make trusted_proxied ejabberd_http option accept ip masks
• Teach acl ip matching about ipv4 mapped ipv6 addresses
• Removed watchdog_admins option from config, as has no effect anymore
• Improve logging of external authentication failures
• ejabberd_auth: Don’t use cache if the option is disabled
• Make connected_users_info and user_sessions_info DB-agnostic

Core

• Support SASL PLAIN by xmpp_stream_out
• Add Resource Binding support to xmpp_stream_out
• Improve robustness of external authentication backends
• Don’t use ‘unsupported-version’ inside SM element
• Generate SASL failures on unencrypted connections only for s2s
• Fix reset_stream in websocket using pre-rfc protocol
• Don’t crash in bosh when we receive request with RID < prev_rid
• Get rid of all calls to jlib.erl module
• Support IPv6 connections for PostgreSQL, MySQL and LDAP
• Fix authentication for usernames containing uppercase characters
• Optimize HTTP requests memory usage
• PKIX: Just warn instead of ignore a certificate containing no domain names
• PKIX: Don't replace valid certificates with invalid ones

Modules

• Log modules startup
• mod_disco: Advertise disco#info and disco#items features
• mod_irc: is moved away from ejabberd repo to ejabberd-contrib
• mod_mam: Don't replace existing stanza ID
• HTTP upload: Generate HTTP Upload form using xdata codec
• HTTP upload: Improve error formatting
• HTTP upload: Return detailed error if HTTP upload is too large

MUC

• Always display room's xdata in disco#info
• Display muc#roomconfig_changesubject in room's disco#info
• Render roomname, allowinvites and allowpm in room disco#info
• Support for roomconfig_lang/roominfo_lang
• mod_muc_sql: Fix export to SQL

Push

• Omit summary for outgoing messages
• Further improve handling of carbons
• Also include sender/body for carbons
• Include a static body text by default
• keepalive: Increase default timeout to 3 days
• SQL: Check 'max_user_sessions' limit

Feedback

As usual, the release is tagged in the Git source code repository on Github.

The source package and binary installers are available at ProcessOne.

If you suspect that you’ve found a bug, please search or fill a bug report on Github.

OMEMO conversations work in Monal. There is still a lot work to be done here to properly handle keys and make UI elements in OSX and iOS for all the the new interactions.  There is a lot that also needs to be tested before this is in ready for the App Store. Realistically, expect to start seeing this stuff in the binaries I release some point in  August.

I have tested this with gajim on linux and chat secure on iOS. Will have dust off an android device somewhere to try it with Conversations.

diff --git a/src/main/java/de/vanitasvitae/crypto/pgpainless/decryption_verification/DecryptionStreamFactory.java b/src/main/java/de/vanitasvitae/crypto/pgpainless/decryption_verification/DecryptionStreamFactory.java
index d651b1b..bca7ba4 100644
--- a/src/main/java/de/vanitasvitae/crypto/pgpainless/decryption_verification/DecryptionStreamFactory.java
+++ b/src/main/java/de/vanitasvitae/crypto/pgpainless/decryption_verification/DecryptionStreamFactory.java
@@ -157,15 +157,13 @@ public class DecryptionStreamFactory {
         PGPPrivateKey decryptionKey = null;
         PGPPublicKeyEncryptedData encryptedSessionKey = null;
         while (iterator.hasNext()) {
-            encryptedSessionKey = (PGPPublicKeyEncryptedData) iterator.next();
-            long keyId = encryptedSessionKey.getKeyID();
+            PGPPublicKeyEncryptedData encryptedData = (PGPPublicKeyEncryptedData) iterator.next();
+            long keyId = encryptedData.getKeyID();
 
             resultBuilder.addRecipientKeyId(keyId);
             LOGGER.log(LEVEL, "PGPEncryptedData is encrypted for key " + Long.toHexString(keyId));
-            if (decryptionKey != null) {
-                continue;
-            }
 
             PGPSecretKey secretKey = decryptionKeys.getSecretKey(keyId);
             if (secretKey != null) {
                 LOGGER.log(LEVEL, "Found respective secret key " + Long.toHexString(keyId));
+                encryptedSessionKey = encryptedData;
                 decryptionKey = secretKey.extractPrivateKey(decryptionKeyDecryptor.getDecryptor(keyId));
                 resultBuilder.setDecryptionKeyId(keyId);
             }
         }
         PublicKeyDataDecryptorFactory keyDecryptor = new BcPublicKeyDataDecryptorFactory(decryptionKey);

The above 3 deletions and 1 addition are the fix for a bug in my decryption routines, which took me 3 days to find. Lets examine the bug in more detail in order to understand what the code does, what went wrong, and why it took me so long to find it.

As I described in an earlier blog post, an encrypted OpenPGP message basically consists of the symmetrically encrypted body and a bunch of public key encrypted session keys. In Bouncycastle, the encrypted session keys are called PGPPublicKeyEncryptedData objects. The are encrypted using the asymmetric public key of a recipient and they contain the symmetric session key which was used to encrypt the actual message. So if Alice writes a message to Bob and Carla, the message will contain at least two PGPPublicKeyEncryptedData elements. One containing the session key encrypted for Bob, the other one for Carla.

In PGPainless, I iterate through all PGPPublicKeyEncryptedData objects of an incoming message in order to get a list of all recipients of the message. I do that because I can imagine it would be cool if Bob can see, that the message was also encrypted for Carla. Also during every iteration I see, if the current PGPPublicKeyEncryptedData object is encrypted for a key of which I do have the secret key available. If so, I extract the private key for later use.

Lets have an example:

Alice encrypts a message to both herself and Bob. Then she sends the message to Bob. Bob tries to decrypt the message.
PGPainless saves the first PGPPublicKeyEncryptedData object in the variable encryptedSessionKey.
This first encrypted session key has Alice’s key ID, so PGPainless adds the id to the list of recipient key ids and goes to the next iteration. Again it stores the encrypted data object in encryptedSessionKey.
This second encrypted session key has Bob’s key ID. PGPainless finds the respective private key, extracts is and saves it to the variable decryptionKey.

Now the iteration ends, as all PGPPublicKeyEncryptedData objects have been processed. Now the decryption can begin. Using the extracted private key, Bob decrypts the encryptedSessionKey to retrieve his session key with which he decrypts the message. Easy Peasy Lemon Squeezy – at least that is what I thought.

I wrote some JUnit tests that encrypt a message for one recipient and those tests work just fine. I used them for example to determine the average message length using different algorithms.

For some strange reason however, during integration testing I noticed that every now and then decryption would fail. I thought it had to do with some race conditions at first. I blamed JUnits @Before and @After annotations which I used to delete the key rings from disk after the test was done to fire too early. Coincidentally after I removed the annotations the test would work flawlessly, which encouraged me and seemed to confirm my hypothesis.

However, the more tests I did, the more confused I became, as I could not find the cause of the failing. Bouncycastle has debugging disabled for their library, so I could not follow the calculations step by step. This is done for security reasons I guess. Fortunately there is a debug version which I discovered today. Using this library, I can see, which lines are responsible for throwing exceptions and step through the execution in great detail.

The final hint however came from a forum post. So lets see what exactly went wrong. For that purpose lets assume the same scenario as above, but with a slight twist.

Again, Alice encrypts a message to herself and Bob, but this time she prepends Bobs session key first.
Bob, when decrypting the message, stores the first PGPPublicKeyEncryptedData object in encryptedSessionKey, notices that this is his own session key and extracts his respective private key into the variable decryptionKey.
In the next iteration (remember, Bob wants to know all recipients), he stores the next PGPPublicKeyEncryptedData object in encryptedSessionKey. This is Alices session key.

Now he proceeds with decrypting the encryptedSessionKey with his decryptionKey – and hell breaks lose. Why? Because at this point in time encryptedSessionKey actually contains Alices session key, not Bobs.

The tricky part about this bug is, that it only happens by chance. I’m not sure in which order session keys are prepended to the message by Bouncycastle, but in certain cases this caused my code to fail – while in certain cases it did not. One benefit that OpenPGP has over OMEMO is, that you can write better tests. In OMEMO keys always change when the ratchet advances, so you cannot really test if certain messages decrypt. At least it is much easier using OpenPGP. But this bug told me a lesson, that you have to be very very careful with your JUnit tests. At some point I replaced randomly generated key pairs with some fixed keys to get more reliable test results and to be able to confirm the result using GnuPG. Ironically I was very lucky that as a result my test would reproduce the second scenario above. If instead it would have produced the first scenario, it would have taken me much much longer to discover the cause of the issue.

Fighting this bug took me 3 days. Luckily I didn’t spend 100% of my time bug hunting. I also wrote another integration test, which covers one very cool feature of OpenPGP for XMPP.

Secret Key Synchronization

In OX it is possible to store the secret OpenPGP key in a private PubSub node. That way the key can be easily transported to other devices, so you can read your encrypted messages in a web browser for example. This is one huge benefit over OMEMO, where you can only read messages received *after* you began using the new device.

During my testing, I also found out, that ejabberd despite announcing support for alternative PubSub access models, does not properly handle some requests.
For normal messaging, OX recommends to publish the public keys of users in a PubSub node configured with the ‘open’ access model. That allows users to access your public key, even if they are not in your roster.

Smack however does a disco info query on a PubSub node prior to fetching it in order to get some metadata about it. It turns out that ejabberd does return an error for such a request, stating that you have to be subscribed to the owner of the node in order to fetch information about it. I’m pretty sure though, that this bug will be fixed soon

Happy Hacking!

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning business aspects of real-time enterprise we found interesting in Issue #12. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

Cisco: 90% of internet traffic through Erlang-controlled nodes

A tweet from @guieevc and a subsequent discussion on HN gives us some insights into how internet traffic is routed and what powers all the machines doing the job.

Ruby vs Elixir vs Go: concurrency comparision

Siva Gollapalli writes: “As we all know, for comparison you need more than one language. So, I choose Golang and Elixir along with Ruby. When I search about building highly concurrent applications most of the results would involve either one of these languages and moreover I like these two languages. Golang for its simplicity and Elixir for its Ruby like features.”

Apple introduces the AI phone

This “new AI iPhone” — which, to be clear, is your same ol’ iPhone running a new mobile OS — will understand where you are, what you’re doing and what you need to know right then and there.

Your smartphone is listening and it’s not a paranoia

For your smartphone to actually pay attention and record you, there needs to be a trigger, like Hey Siri or Okay Google for example . Without these triggers, there’s no recording, with just some general metrics being sent to your service provider. This might not seem a cause for an alarm, but when it comes to apps like Facebook, no one knows what the triggers are. In fact, there could be thousands.

Use MQTT to stream real-time data

The ny-power project uses a set of microservices to consume open data about the performance of the NY State power grid, and computes an approximate level of carbon intensity. This computed data is served in a real-time MQTT stream, making it easy to consume in other applications. It also shows how to manage a set of data processing in Kubernetes with Helm.

Mining for Bitcoin vanity addresses with elixir

Let’s take another bite out of Mastering Bitcoin and implement the algorithm described for “mining for vanity addresses”. After we implement the basic algorithm, we’ll add our Elixir special sauce and turn it into a fully parallelized procedure.

Why you should never have a data room  for fund-raising

The data room is where your process goes to die. What happens is 18–20 firms access the data room and download all of your documents. You feel proud because data rooms have tracking on them so you know exactly who did and who did not access your data. So you sit around and wait for the next call. You convince yourself that it should take 1–2 weeks until they have gone through the data and you’ll get a call but it never comes.

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning tech aspects of real-time development we found interesting in Issue #12. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

Exception Management at the Heart of Artificial Intelligence Performance

Science fiction makes us dream about the promises of artificial intelligence, but it creates a mythology, which makes it difficult to understand the real issues at stake. Yet, there’s no magic ingredient in AI, but rather a lot of mathematics.

An AI Created New Doom Levels That Are as Fun as the Game’s Original Ones

The technical skills of programmer John Carmack helped create the 3D world of Doom, the first-person shooter that took over the world 25 years ago. But it was level designers like John Romero and American McGee that made the game fun to play.

Choosing the Right Platform for Real-time Chat Application Development: XMPP vs Firebase

With a variety of chat application frameworks available in the market, these are the two most advanced and high performing protocols: Firebase and XMPP. Here are their pros and cons.

How To use XMPP and I2P to Build Your Own Decentralized Messenger

Building decentralized messenger run by users instead of corporations is definitely not new but thanks to XMPP and I2P (Invisible Internet Protocol) it’s easier than ever before.

AI and the Augmentation Fallacy

The fundamental disruption introduced by AlphaZero’s hyperlearning in the chess world can teach business executives about AI. Many pundits, academics, and economists advise business executives on how artificial intelligence (AI) will augment human performance in the workplace.

XMPP Echo Bot

Do you know that situation, you really really need an XMPP echo bot, but you don’t have access to high-level tools like Python to write one? All you have is openssl, bash, dig, stdbuf and sed? Then this tool is for you. This is an XMPP echo bot written in (mostly) sed.

The Erlang Runtime System

The real goal with this book is to give you all the information, all the gory details, that you need in order to really understand the performance of your Erlang application.

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning business aspects of real-time enterprise we found interesting in Issue #11. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

France to build alternative to WhatsApp, Telegram due to foreign spying scare

According to a recent report, the French government is currently developing an end-to-end encrypted alternative to WhatsApp and Telegram that its officials could use without worrying about foreign spying.

BotChain wants to put bot-to-bot communication on the blockchain

Increasingly we are going to be having bots conducting business on a company’s behalf. As that happens, it is going to require a trust mechanism to ensure that bot-to-bot communication is legitimate.

How WhatsApp bot is helping redBus improve customer support

Case study: Many of users who booked a bus ticket on redBus in last few weeks would have received ticket details and lot more on WhatsApp. Some of them might have also conversed with redBus on WhatsApp.

Space communication is stuck in the dial-up age, which means it’s time for more lasers

In space, no one can hear you scream — because sound doesn’t travel in a vacuum, but also because you would need some sort of radio relay to carry the message, what with the distances being so extreme.

Smartphones are killing the planet faster than anyone expected

Before you upgrade your next iPhone, you may want to consider a $29 battery instead. Not only will the choice save you money, it could help save the planet.

​Red Hat and Microsoft bring OpenShift to Azure

At Red Hat Summit in San Francisco, Red Hat and Microsoft announced they were bringing Red Hat OpenShift, Red Hat’s Kubernetes container orchestration platform, to Microsoft’s Azure, Microsoft’s public cloud.

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning tech aspects of real-time development we found interesting in Issue #11. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

How to Build a Private One to One Chat App from Scratch?

In this article, the development of one to one chat app from scratch is discussed in detail. The technical stack involved in WhatsApp like chat app are Erlang (Language), Ejabberd (Framework), XMPP protocol, MySQL (Database). The following steps will help you to develop a one to one chat application in an hassle-free manner.

Deploy Bulletproof Embedded Software in Elixir with Nerves

Nerves defines an entirely new way to build embedded systems using Elixir. It is specifically designed for embedded systems, not desktop or server systems. It consists of a minimal Buildroot-derived Linux that boots directly to the BEAM VM, ready-to-go library of Elixir modules to get you up and running quickly and powerful command-line tools.

Connecting an Elixir Node to the Bitcoin Network

Pete Corey writes: “Since I first started diving into the world of Bitcoin development, I’ve wanted to build a simple node that connects to the network. The Elixir programming language gives us some fantastic tools to implement a server in the peer-to-peer network. Let’s see how far they can take us!”

Riot: A Distributed IRC and VOIP Client and Home Server

Riot is a free and open source decentralized instant messaging application that can be considered an alternative to Slack. This article takes a look at features of Riot, installation procedure and usage.

Elixir’s Phoenix Powering a Real-time Web UI

A problem WallarooLabs needed to solve early on was deciding on the tooling that would power our metrics monitoring system. They needed a monitoring solution to provide real-time updates on the several steps a data message may take along the way within a Wallaroo application. This post takes a deeper dive into the monitoring problem and how Phoenix and Elixir helped solve specific issues.

Fortnite: Postmortem of Service Outage at 3.4M CCU

Epic Games posted a postmortem on, among other things, their XMPP outage. They had a situation while mitigating a known instability problem that resulted in overloading a downstream system component and effectively paralyzing presence flow. Without presence, a user who is a friend cannot see that a player is online, breaking most of Fortnite social features including the ability to form parties.

I guess in anybodies project, there is one week that stands out from the others by being way less productive than the rest. I just had that week.

I had to take one day off on Friday due to circulation problems after a visit at the doctor (syringes suck!), so I had the joy of an extended weekend. On top that, I was not at home that time, so I didn’t write any code during these days.

At least I got some coding done last week. Yesterday I spent the whole day scratching my head about an error that I got when decrypting a message in Smack. Strangely that error did not happen in my pgpainless tests. Today I finally found the cause of the issue and a way to work around it. Turns out, somewhere between key generation and key loading from persistent storage, something goes wrong. If I run my test with fresh keys, everything works fine while if I run it after loading the keys from disk, I get an error. It will be fun working out what exactly is going wrong. My breakpoint-debugging skills are getting better, although I still often seem to skip over important code points during debugging.

My ongoing efforts of porting the Smack OX code over from using bouncy-gpg to pgpainless are still progressing slowly, but steady. Today I sent and received a message successfully, although the bug I mentioned earlier is still present. As I said, its just a matter of time until I find it.

Apart from that, I created another very small pull request against the Bouncycastle repository. The patch just fixes a log message which irritated me. The message stated, that some data could not be encrypted, while in fact date is being decrypted. Another patch I created earlier has been merged \o/.

There is some really good news:
Smack 4.4.0-alpha1 has been released! This version contains my updated OMEMO API, which I have been working on since at least half a year.

This week I will continue to integrate pgpainless into Smack. There is also still a significant lack of JUnit tests in both projects. One issue I have is, that during my project I often have to deal with objects, that bundle information together. Those data structures are needed in smack-openpgp, smack-openpgp-bouncycastle, as well as in pgpainless. Since smack-openpgp and pgpainless do not depend on one another, I need to write duplicate code to provide all modules with classes that offer the needed functionality. This is a real bummer and creates a lot of ugly boilerplate code.

I could theoretically create another module which bundles those structures together, but that is probably overkill.

On the bright side of things, I passed the first evaluation phase, so I got a ton of motivation for the coming days

Happy Hacking!

@Flow wrote:

The Smack developer community is proud to announce the availability of the first release candidate of Smack 4.3. Users of Smack are encouraged switch to the new 4.3 release family of Smack. The Smack 4.3 API is considered frozen and the API changes between 4.2 and 4.3 are not as significant compared to the changes between Smack 4.1 and 4.2. More information can be found in the Readme of Smack 4.3 (please note that the Readme is work in progress).

Together with the 4.3.0-rc1 release, we have also published the first alpha of Smack 4.4, which includes the updated and improved OMEMO API. Credits for this go to Paul.

As always, all the release artifacts are available on Maven Central.

Posts: 1

Participants: 1

Read full topic

I have released 3.0.2 to the iOS App Store.  So far I appear to have resolved the worst crashes.