Planet Jabber

Software development teams are beginning to realize the benefits of continuous, test-driven delivery of new releases.

Instead of a single, milestone release (waterfall development) or multiple, internal test releases before major external ones (agile development), continuous delivery focuses on constant releasing of features to market throughout the development process.

The goal of continuous delivery is that code is always ready to deploy and features are constantly rolled out independent of ‘releases’ – and doing so properly requires realtime data.

...

Newsflash: My Jingle File Transfer is compatible with Gajim!

The Gajim developers recently fixed a bug in their Jingle Socks5 transport implementation and now I can send and receive files without any problems between Gajim and my test client. Funny side note: The bug they fixed was very familiar to me *cough*

Happy Hacking!

Although I pay no attention to the news, I'm vaguely aware that perhaps the world is becoming more negative and irrational. Rather than getting lost in despair and confusion, I try to model a positive, reasonable alternative by living the best life I can and by writing philosophical books that are clear, concise, and constructive. I've written six such books, with a seventh half done and an eighth outlined. Even though each of these books takes 500-1000 hours to research and write, I make them all available for free on my website (however, I do charge for the paperback and ebook versions). To date I've also made no effort to market them, so I depend on word of mouth. If you have found value in any of these books, I'd greatly appreciate it if you could tell a friend, write a review at Amazon, or give a copy as a gift. Thanks. :-)...

Chat is one of the most popular uses of realtime data. In this article we’ll explain how to build a web chat app in Django, using Django EventStream and Fanout Cloud. The Django EventStream module makes it easy to push JSON events through Fanout Cloud to connected clients.

...

Happy summer with ejabberd 17.08 !
This release includes great improvements and new features. It also includes fixes and closes the biggest milestone about refactor we’ve made last couple of months.
If you have issues with 17.04 or troubles using PEP, upgrade to 17.08 will fix most known issues.

New features

Introduce ‘hosts’ option

The option can be used as a replacement of ‘host’ option when several (sub)domains are needed to be registered for the module.
Note that you cannot combine both ‘host’ and ‘hosts’ in the config because ‘host’ option is of a higher priority. Example:

mod_pubsub:
   ...
   hosts:
     - "pubsub1.@HOST@"
     - "pubsub2.@HOST@"

XEP-0357: Push Notifications

This module tries to keep pending stream management sessions of push clients alive (as long as the disconnected clients are reachable via push notifications).

Modular cluster

For configuring the cluster new global option ‘cluster_backend’ is introduced. The default and only available value at the moment is ‘mnesia’.

Use c2s ‘certfile’ option

Use the ‘certfile’ listener option rather than a ‘domain_certfile’ for ejabberd_c2s listeners that have “tls: true” configured. A ‘domain_certfile’ should only be preferred for STARTTLS connections.

New mod_muc hooks

There are four new events: create_room, destroy_room, join_room and leave_room.
Note: destroy_room was previously used already by mod_mam and was named remove_room. remove_room is not renamed back to destroy_room for consistency.

Changes

Core

• Erlang/OTP 17.5 or higher is required, and 20 is now supported
• Make ejabberd_cluster modular
• Replace gen_fsm with p1_fsm to avoid warnings in OTP20+
• Fix clustering table reg_users_counter
• ext_mod: Update spec from custom and allow modules dependencies
• extauth.py: Fix to support : in passwords
• Set high water mark in lager for all backends
• Fix old route record in mnesia’s route table haven’t been remove when restarting in some cases
• ejabberd_cluster*.erl: Add copyright and fix description
• Add support of rfc6120 section 4.9.3.16 on node shutdown

Configuration

• ejabberd_c2s: Fix priority of ‘certfile’ option
• Introduce ‘hosts’ modules option
• Fix ERLANG_OPTS, INET_DIST_INTERFACE and FIREWALL_WINDOW option
• Remove unused ‘managers’ option, related to the deferred XEP-0321

Commands

• Fix errors when running ejabberdctl as root
• Fix set_presence command to work in recent ejabberd
• Rename stop_all_connections to stop_s2s_connections for consistency
• Change policy of user_resources command, from user to admin
• Remove old command calling interface
• Describe more command arguments and results

Modules

• mod_http_api: Use hide_sensitive_log_data option when registering users
• mod_http_fileserver: Request basic auth dialog from browser
• mod_muc: Fix nick bug with MUC on riak
• mod_muc: new hooks
• mod_push: Support XEP-0357: Push Notifications
• mod_push_keepalive: New module

PubSub/PEP

• Keep disco#info on PEP compatible with XEP-0060
• Preliminary export PubSub data from Mnesia tables to SQL file
• Fix PubSub send last published items
• Fix PEP node removal
• Fix PEP node identity
• Fix disco#items on PEP service
• Fix getting cached last item
• Add import of PEP from prosody

Documentation

• Improved API documentation
  see https://docs.ejabberd.im/developer/ejabberd-api/admin-api/

Feedback

As usual, the release is tagged in the Git source code repository on Github.

The source package and binary installers are available at ProcessOne.

If you suspect that you’ve found a bug, please search or fill a bug report on Github.

You know what makes code of a software library even better? Client code that makes it do stuff! I present to you xmpp_sync!

Xmpp_sync is a small command line tool which allows you to sync file from one devices to one or more other devices via XMPP. It works a little bit like you might now it from eg. owncloud or nextcloud. Just drop the files into one folder and they automagically appear on your other devices. At the moment it works only unidirectional, so files get synchronized in one direction, but not in the other.

The program has to modes; master mode and slave mode. In general, a client started in master mode will send files to all clients started in slave mode. So lets say we want to mirror contents from one directory to another. We start the client on our master machine and give it a path to the directory we want to monitor. On the other machines we start the client in slave mode, and then add them to the master client. Whenever we now drop a file into the directory, it will automatically be sent to all registered slaves via Jingle File Transfer. Files do also get send when they get modified by the user. I registered a FileWatcher in order to get notified of such events. For this purpose I got in touch with java NIO again.

Currently the transmission is made unencrypted (as described in XEP-0234), but I plan to also utilize my Jingle Encrypted Transports (JET) code/spec to send the files OMEMO encrypted in the future. My plan for the long run is to further improve JET, so that it might get implemented by other clients.

Besides that I found the configuration error in my ejabberd configuration which prevented my Socks5 proxy from working. The server was listening at 127.0.0.1 by default, so the port was not reachable from the outside world. Now I can finally test on my own server

I also tested my code against Gajims implementation and found some more mistakes I made, which are now fixed. The Jingle InBandBytestream Transport is sort of working, but there are some more smaller things I need to change.

Thats all for the week.

Happy Hacking

Over the last years, the XMPP community has had a hard time competing with other Instant Messaging implementations, especially in the mobile / smartphone ecosystems. By focusing a small part of our resources on user experience (UX), we can gain significant improvements. This article is the first in a series of "Easy XMPP" posts: easy ways for application developers to make XMPP easy to use.

Complexity of Federation

As opposed to most other Instant Messaging solutions, XMPP is a federated protocol. That allows everyone to run their own servers, at the cost of additional complexity for users:

• the user identifier always consists of a user and a domain part,
• there is no central registry that will consume your phone book and tell you who else is using XMPP,
• some servers might be running an older stack not supporting modern features, etc.

This inherent complexity, together with many developers' lack of attention to good UX, have left us in a situation where on-boarding of new users and finding contacts is painfully hard, especially when compared to proprietary/centralized IM solutions.

It is not possible to remove the inherent complexity of federation without replacing XMPP with a completely different protocol. However, there is another federated system that is well established and used by people all over the world: email. XMPP and email share the same basic principles - they are federated, user identifiers are user@domain and it is possible (albeit less common) to run your own server (or to get your own domain hosted with a commercial provider).

By leveraging users' knowledge about how email works, learning from over forty years of email evolution, and applying ideas from modern UX design on top, it is possible to make XMPP easier to use today.

Our Challenges

There are several areas where our community needs to improve. This post provides an overview of the challenges we are currently facing in different areas. Subsequent posts will dive deeper into the individual topics and work out possible solutions.

Terminology

The XMPP terminology is driven by technical requirements and exposes complexities of the protocol. Normal IM users don't want to know about PubSub Publish Options, asymmetric presence subscription in their roster, or MUC-PM Carbons. All they care about is to see whether their friends are online and that they can exchange cat pictures.

It is our task as the developer community to create understandable abstractions of the technical complexities, and to set up a common glossary for the user-facing elements, including translations into common languages. We need to define what the difference between "Jabber" and "XMPP" is, whether the user identifier is a "JID", a "Jabber ID", an "XMPP address", or a "user identifier", and fix all the other terms that are exposed to users and make clients inconsistent today. And then we need to apply this glossary to our implementations.

First-time User Experience

The first-time user experience for IM users needs to be streamlined. The users who need the most assistance are newcomers to the ecosystem who got invited by a friend, and start out with nothing but their friend's Jabber ID. They need help to accomplish these tasks:

• install a modern client
• create an account
• add their friend
• automatically find all their other friends
• find other users and chat rooms

Projects like Easy XMPP Invitations, Pre-Authenticated Roster Subscription, and invite URLs are a first step in that direction, and getting them into your client today will make new users' lives easier. But there is also potential to further streamline the whole process.

Easy Group Chats

There are two typical IM use cases for group chats: chat with friends or family (private groups), or participate in a public chat room (typically for support purposes, pioneered by Internet Relay Chat).

The latter is well-covered by MUC, and the upcoming MIX should provide a solid technical basis for both use cases. The next logical step is to allow the easy creation and sharing of ad-hoc rooms between different clients, with a coherent user experience.

Fighting Spam

If the amount of spam is a measure of XMPP's popularity, the network is doing exceptionally well. Almost all of today's XMPP spam can be blocked with some easy pattern matching rules, and the inevitable arms race will move to the next round.

The community needs to prepare for that, by improving reporting between server operators, adding anti-spam features into clients and providing better whitelisting mechanisms to users.

Call for Action

You can help making XMPP easier to use. As a user, you can contribute ideas, add user stories, check your favorite client for confusing UI elements and non-helping wizards, and report those to the developers. As a developer, you can streamline the on-boarding process in your client and contribute to the common glossary.

Get yourself a wiki account and start hacking on Easy XMPP today!

Apropos of my latest journal entry "Politics is a Disease", here is a quote from Beyond Good and Evil (Section 251) by Friedrich Nietzsche:...

Finally!! I found a bug which I was after for the last week. Now I finally got that little bas****.

The bug happened in my code for the Jingle SOCKS5 Bytestream Transport (XEP-0260). SOCKS5 proxies are used whenever the two endpoints can’t reach one another directly due to firewalls etc. In such a case, another entity (eg. the XMPP server) can jump in to act as a proxy between both endpoints. For that reason, the initiator (Alice) first collects available proxies, and sends them over to the responder (Bob). The responder does the same and sends its candidates back to the initiator. Both then try to connect to the candidates (in this case proxies) they got sent from their peer. In order for the proxy to know, who wants to talk to whom, both include a destination address, which is calculated as SHA1(sid, providerJid, targetJid), where the provider is the party which sent the candidates to the target.

The alert reader will know, that there are two different destination addresses in the game by now. The first one being SHA1(sid, Alice, Bob) and the second one SHA1(sid, Bob, Alice). The issue is that somewhere in the logs I ended up with 3 different destination addresses. How the hell did that happen. For the answer lets look at an example stanza:

In the last 3 weeks I plan to stabilize the API some more. Currently you can only receive data into files, but I plan to add another method which gives back a bytestream instead.

Also I need more tests. Things like that nasty sid bug can be prevented and found using junit tests, so I’ll definitely stock up on that front.

Thats all for now Happy Hacking!

One of the first things that impressed me about Mac OS X when I first saw it was its screensaver. Instead of just showing a simple slideshow of your pictures, it actually used a ‘Ken Burns’ panning and zooming effect with a fancy fading transition to make the otherwise static pictures really come to life. It always sounded like a fun project to create a standalone tool to create slideshow movies that used this effect, with full control over where and how much pictures should be zoomed. It turns out you don’t really need a new tool: you can get the same result using just FFMpeg. In this post, I’ll walk through the steps of creating slideshows using the Ken Burns effect.

Oh, and cats. There will be cats!

Continue reading post

Today we’re pleased to introduce Django EventStream, a module that provides Server-Sent Events capability to your Django server application. It relies on Pushpin or Fanout Cloud to manage the client connections. Events can optionally be persisted to your database, for highly reliable delivery.

...

The 9th week of GSoC is there. I’m surprised of how fast time went by, but I guess that happens when you enjoy what you do

I’m happy to report that the third iteration of my Jingle code is working again. There are still many bugs and Socks5Transport is still missing, but all in all I’m really happy with how it turned out. Next I’ll make the implementation more solid and add more features like transport replacing etc.

Apart from normal Jingle File Transfer I also started working on my JET protocol. JET is short for Jingle Encrypted Transfers which is my approach to combining Jingle sessions with end-to-end encryption. My focus lays on modularity and easy extensibility. Roughly JET works as follows:

Lets assume, Alice wants to send an encrypted file to Bob. Luckily Alice and Bob already do have a secure OMEMO session. Alice now sends a JET File transfer request to Bob, which includes a security element which contains an OMEMO key transport message. Bob can decrypt the key transport message with his OMEMO session to retrieve an AES key. That key will be used to encrypt/decrypt the file Alice sends to Bob as soon as the jingle session negotiation is finished.

This protocol should in theory work with any end-to-end encryption method, for example also with OpenPGP. Also JET is in theory not limited to file transfer, but could also be used to secure other types of Jingle sessions, eg. Audio/Video calls. Since the development is in a very early state, it would be nice to get some feedback from more experienced developers and members of the XMPP community. A rendered version of the JET specification can be found here.

I’m very happy that encrypted File transfer already works in my implementation. I created an integration test for that, which transports the encryption key using OMEMO. Apropos tests: I created a basic JingleTransport test, which tests transport methods. Currently SOCKS5 is still failing, but I’m very close to a solution.

During the week I opened another PR against the XEPs repo, which adds a missing attribute to a XML schema in the Jingle File Transfer XEP.

It’s becoming the new normal that messaging and collaboration apps and platforms are available across multiple devices.

Business tools like Slack and JIRA offer feature-rich mobile apps, and users increasingly consume content from social networks like Facebook on their mobile devices instead of a desktop or laptop.

This isn’t a surprise – and we’re here to share our perspective on how developers can use realtime data to provide cross-platform users with the best notification experience.

...

The 8th. week of GSoC is there. The second evaluation phase has begun.

This week I was not as productive as the weeks before. This is due to me having my last (hopefully ) bachelors exam. But that is now done, so I can finally give 100% to coding .

I made some more progress on my Jingle code rework. Most of the transports code is now finished. I started to rework the descriptions code which includes the base classes as well as the JingleFileTransfer code. I’m very happy with the new design, since it is way more modular and less interlocked than the last iteration. Below you can find an UML-like diagram of the current structure.

During the rework I stumbled across a slight ambiguity in the Jingle XEP(s), which made me wonder: There are multiple jingle actions, which denote the purpose of a Jingle stanza (eg. transport-replace to replace the used transport message, session-initiate to initiate a session (duh) and so forth). Now there is the session-info Jingle action, which is used to announce session specific events. This is used for example in RTP sessions to let the peers client ring during a call, or to send the checksum of a file in a file transfer session. My problem with this is, that such use-cases are in my opinion highly description related and instead the description-info action should be used. The description part of a Jingle session is the part that represents the actual purpose of the session (eg. file transfer, video calls etc.).

The session itself is description agnostic, since it only bundles together a set of contents. One content is composed of one description, one transport and optionally one security element. In a content you should be able to combine different description, transport and security components in an arbitrary way. Thats the whole purpose of the Jingle protocol. In my opinion it does no make much sense to denote description related informational stanzas with the session-info action.

My proposal to make more use of the description-info element is also consistent with other uses of *-info actions. The transport-info action for example is used to denote transport related stanzas, while the security-info action is used for security related information.

But why do I even care?

Lets get back to my implementation for that . As you can see in the diagram above, I split the different Jingle components into different classes like JingleTransport, JingleSecurity, JingleDescription and so on. Now I’d like to pass component-related jingles down to the respective classes (a transport-info usually only contains information valuable to the transport-component). I’d like to do the same for the JingleDescription. At the moment I have no real “recipient” for the session-info action. It might contain session related infos, but might also be interesting for the description. As a consequence I have to make exceptions for those actions, which make the code more bloated and less logical.

Another point is, that such session-info elements (due to the fact that they target a single content in most cases) often contain a “name” attribute that matches the name of the targeted content. I’d propose to not only replace session-info with description-info, but also specify, that the description-info MUST have one or more content child elements that denote the targeted contents. That would make parsing much easier, since the parser always can expect content elements.

That’s all for now

Happy Hacking!

This is my update post for the 7th week of GSoC. The next evaluation phase is slowly approaching and it is still a lot of work to do.

This week I started to work on integrating encryption in my Jingle File Transfer code. I’m very pleased, that only very minor changes are required to my OMEMO code. The OmemoManager just has to implement a single interface with two methods and that’s it. The interface should be pretty forward to implement in other encryption methods as well.

Unfortunately the same is not true for the code I wrote during the GSoC. There are many things I haven’t thought about, which require major changes, so it looks like I’ll have to rethink the concept once again. My goal is to make the implementation so flexible, that description (eg. video chat, file transfer…), transport (eg. Socks5, IBB…) and security (XTLS, my Jet spec etc.) can be mixed in arbitrary ways without adding in glue code for new specifications. Flow told me, this is going to get complicated, but I want to try anyway . For “safety” reasons, I’ll keep a seperate working branch in case the next iteration does not turn out as I want.

Yesterday Flow found an error in smack-omemo, which caused bundles not getting fetched. The mistake I made was, that a synchronous CarbonListener was registered in the packet-reader thread. This caused the packet-reader thread to timeout on certain messages, even though the stanzas arrived. It is nice to have this out of the way and it was a good lesson about the pitfalls of parallel programming.

While reading the Jingle File Transfer XEP I also found some missing XML schemes and proposed to replace the usage of xs:nonNegativeInteger and xs:positiveNumber with xs:unsignedLong to simplify/unify the process of implementing the XEP.

Thats basically it for this week. Unfortunately I have an upcoming exam at university next week, which means even less free time for me, but I’ll manage that. In the worst case I always have a second try on the exam

Happy Hacking!