What do you mean, its on fire?

NITEC is the NCI Agency’s flagship annual industry conference, focusing on advancing technological solutions and business practices to strengthen NATO operations. This year’s NITEC was held in the Norwegian capital of Oslo between 20 and 22 May.

A regular visitor to NITEC, Isode spent three enjoyable days at this year’s show demonstrating our software for chat and email messaging to an audience of systems integrators, technical influencers and representatives of end-user organisations (the militaries of NATO nations).

Isode’s Louise Hill (Pre-Sales Engineer) and Jeff Tillotson (Business Development Manager: Defence) setting up the Isode stand.

Of particular interest to visitors was Isode’s ability to connect different services, employing different chat and messaging standards, so that they become seamless end-to-end systems. Our demonstration of connecting XMPP based chat systems to those employing the legacy IRC system (using Isode’s M-Link IRC Gateway) was particularly popular.

Both chat and messaging demonstrations also featured security label translation, mapping labelled messages from one security domain to another (for the show we used Norwegian and NATO labels on the different “sides” of the messaging/chat systems).

Evaluation software for both Email Messaging and XMPP Chat software is available from the Isode website.

ProcessOne curates two monthly newsletters – tech-focused Real-time Stack and business-focused Real-time Enterprise. Here are the articles concerning tech aspects of real-time development we found interesting in Issue #23. To receive this newsletter straight in your inbox on the day it’s published, subscribe here.

Mozilla relaunches open source IoT platform Project Things as WebThings

For the better part of two years, the folks at Mozilla have been diligently chipping away at Project Things, an open implementation of the World Wide Web Consortium’s Web of Things standard for monitoring and controlling connected devices, and today, they reached a major milestone.

Jaguar Land Rover to let drivers earn cryptocurrency by sharing data

Owners of cars made by Jaguar Land Rover will be able to earn cryptocurrency as they drive by enabling their vehicles to report “useful” data.

Easy start into the world of IoT with MQTT

The Internet of Things is getting everywhere – so more important it is to understand the principals behind. In this example we show how easy everybody can build its own ideas for the digital world based on open standards like Arduino, MQTT, Node-RED… in a simple and easy “visual” coding.

Mainflux LoRaWAN?tutorial

LoRa (Long Range) is a patented digital wireless data communication technology developed by Cycleo of Grenoble, France, and acquired by Semtech in 2012. LoRa uses license-free sub-gigahertz radio frequency bands like 169 MHz, 433 MHz, 868 MHz (Europe) and 915 MHz (North America).

Device originating communications, or how and why MQTT rocks

The story goes that a valve manufacturer wanted to have their networked smart valves certified for use in a nuclear reactor plant. The smart valve could report all sorts of critical data points to a database system and also be controlled by that SCADA system in the plant.

Public Jabber chat rooms

A listing of public XMPP/Jabber chat rooms/groups curated by Christopher Muclumbus.

I managed to get the basic in-band bytestream file transfers working. It still doesn’t have any user interface, that’s depending on a project by my mentor. For now, it’s simply triggered by a hardcoded target JID that gets a file transfer when it sends a presence.

After finishing the proof of concept, I added some error handling and did some code cleanup. With that done, I did some interoperability testing. With Gajim, the file transfers now worked well, so I moved on to test with Conversations. Here, something funny happened, my Dino code sent (abbreviated for brevity):

<iq type='set'>
  <jingle action='session-initiate'>
    <content>
      <description xmlns='urn:xmpp:jingle:apps:file-transfer:5'>
      	…
      </description>
      <transport xmlns='urn:xmpp:jingle:transports:ibb:1' block-size='4096' />
    </content>
  </jingle>
</iq>

And Conversations responded:

<iq type='set'>
  <jingle action='session-accept'>
    <content>
      <description xmlns='urn:xmpp:jingle:apps:file-transfer:5'>
      	…
      </description>
      <transport xmlns='urn:xmpp:jingle:transports:s5b:1'>
        <candidate host='146.255.57.229' type='proxy' jid='proxy.jabber.ccc.de' port='5224' />
      </transport>
    </content>
  </jingle>
</iq>

In essence, my Dino code asked to open a Jingle session for a file transfer, using in-band-bytestreams as the transport. Then, Conversations responded that it accepts my offer of Socks 5-based Jingle transport for the file transfer, accepting an offer I didn’t make. I think this is a violation of the Jingle XEP (XEP-0166): If the other party wants to negotiate a different transport, it should probably use the transport-replace Jingle action. I created an issue (Conversations#) which was quickly resolved. I have yet to test the fix, I don’t have a development setup for Conversations yet.

The in-band bytestream file transfers I tested were quite slow (<2KB/s), probably some rate-limiting of the servers, I didn’t investigate furtherly.

The XMPP Standards Foundation is happy to act as umbrella organization for Google Summer of Code (GSoC) 2019. This year we are glad to mentor and support the following three GSoC projects.

Prosody plugin installer

Student: João Duarte GSoC Blog: https://gsoc-prosody-2019.blogspot.com Project: Prosody - An XMPP server written in Lua Mentors: MattJ, Zash

João lives in Portugal and usually studies Aerospace Engineering. This summer he will be developing an integration between Prosody and the LuaRocks package manager and ecosystem,allowing installation and management of third-party Prosody modules via simple commands.

Poezio infinite scrolling using MAM

Student: Madhur Garg GSoC Blog: https://madhur96.github.io/gsoc19/ Project: Poezio - A free console XMPP client in Python Mentors: pep, Link Mauve

Madhur is a student of the International Institute of Information Technology in Hyderabad, India. He likes Python and C++. Outside of programming, he likes guitar and trekking. This summer he will be working on adding “Infinite scroll” support to poezio, the console client, using XEP-0313: Message Archive Management, as well as some other general improvements.

Jingle File Transfer for Dino

Student: hrxi GSoC Blog: https://hrxi.github.io/gsoc/ Project: Dino - A modern open-source chat client for the desktop in Vala Mentors: fixah, la|r|ma

hrxi will implement encrypted peer-to-peer Jingle file transfers in Dino. The encrypted connection will be set up by either one of the supported encryption protocols of Dino, OMEMO and OpenPGP. In order to achieve compatibility with web clients, it is planned to support WebRTC as a transport protocol. The implementation will be done in a way that allows building support for audio/video chat on top of it. hrxi is based in Germany, studies mathematics and computer science, and likes to write code in Rust.

Release 3.8 should make jumping on to XMPP easier for non technical users. The new registration screen will bring us closer to a chat platform that anyone can download and use.

@guus wrote:

Last weekend, I participated in an XSF-community organized sprint, in The Hague (conveniently located in my country of residence). Graciously hosted by IPerity, a group of people from various XMPP related projects got together.

Apart from it always being nice to meet up with fellow enthusiasts, we got quite some work done. In the sprint preparation, we had already established that one of the topics that we’d like to address, was the lack of top-notch (mobile) client support, in XMPP world.

the-hague-2019.jpg1920×1440 359 KB

Much of the work that we did revolved around adding better support for client notifications. This lead pep. and Laszlo to build a prototype for browser-based push notifications, which they got working in Converse (the web client that Openfire ships in the “inVerse” plugin) and the Prosody XMPP server. A goal for this effort is to evolve into a new XEP for web push.

Yours truly is happy to report that a new plugin was created for Openfire, that adds XEP-0357-defined push notifications to Openfire! The code is still somewhat rough around the edges, but I’d love for people to give this a test run, and how the prototype holds up!

Unrelated to the various notification-related efforts, Erik got quite a bit of work done on a refactoring project in the Gajim client, which over the past weekend is growing to be the basis for a refactoring of Gajim’s chat window UI.

I’m thrilled by the prospect of hosting a major, hopefully Openfire-based, hackathon in the near future, that is geared towards a particular industry in the Netherlands. That might be an exciting precursor for another increase in XMPP utilization! More on that later though!

I’ve had a blast! I loved meeting up with everyone, and enjoyed the discussions we had. Let’s do this again, soon!

For other Ignite Realtime release announcements and news follow us on Twitter

Posts: 3

Participants: 2

Read full topic

iOS 3.8 is ready to be tested and available on Testflight. The change list is:

1. In app registration on blabber.im
2. In app password changes
3. Addressed compatibility issue specifically with prosody that caused higher batter use.

All of these changes are important but we really need to test the registration flow. Please go to settings->quick setup and try registering a new account on blabber.im. This is the welcome screen new users see so this has to work perfectly. If you are having issues with Monal in general, having a blabber.im account is a good idea because I have tested with this server and can confirm that everything works as expected. This will let us determine if any bugs you experience are related to your specific server, Monal or your device.

As the first task, we planned to get a working, proof-of-concept Jingle file transfer over an “in-band bytestream” (IBB, XEP-0047, XEP-0261). To get to this point, I planned to write a module for basic Jingle interactions, supporting the necessities for file transfers, and a non-final API for transport methods on the other side.

What do in-band bytestreams look like? “In-band” means that the file data will go through the same connection as the rest of the XMPP stream that also contains your chat messages and metadata like read receipts, etc. It has to fit into the XML stream, an example with an 28-byte file below:

<iq xmlns="jabber:client" to="tish@example.com/gajim" from="blagovest@example.com/dino" type="set" id="e8656828-2da8-48a5-828d-804542945851">
	<data xmlns="http://jabber.org/protocol/ibb" seq="0" sid="514ef10b-22e7-4e58-bf46-cca01629e826">
		SGVsbG8sIGluLWJhbmQgYnl0ZXN0cmVhbXMhCg==
	</data>
</iq>

As you can see, this is quite inefficient. The <iq>/<data> tags amount to roughly 256 bytes. The actual data has to be base64-encoded because XML cannot contain arbitrary binary data, leading to an extra 33% overhead. With the default blocksize of 4096 bytes as specified in the IBB XEP, we get a total overhead of 40%. Nevertheless, this transport is simple to implement as it does not need additional network connections beyond the already existing.

While trying different approaches for the Jingle transport API, fiaxh (my mentor) helped me decide: I’m now using abstract classes for the different transports, and signals for the users of a Single session, a pretty Vala-esque approach. After this planning, I went ahead and tried to realize this in actual code. This resulted in basic session opening support (only initiating), support for in-band bytestreams and file transfers.

What’s still missing for merging this? I was pretty happy that the code almost worked after fixing syntax error and obvious oversights. It still has a pretty serious bug: Currently, when testing the file transfer from Dino to Gajim, Gajim drops the last chunk we send. Additionally, the code needs proper error handling, currently it just writes a message to the standard error output.

During the weekly organization chat, flow sent me an interesting document written by someone who implemented the Jingle protocol in multiple clients and wrote about the “enlightenment” he achieved in doing so. Thanks for that, I’ll check it out the next week.

README

Events:

Prosodyctl:

• Made a simple command function that prints out a welcoming message
• Made a command function that prints out a list of currently enabled modules
• Made a command function that prints out locally available modules

Research:

• Mercurial practice:
• evolve extension
• ssh connections
• branching
• resolving conflicts
• Prosody: Re-reading most of the documentation
• Luarocks: 
• Studying Luarocks-admin commands
• Looking for packages.path and packages.cpath related code

 Difficulties:

• Community Bounding Period mindset - I feel like I'm still biased to do research like if I was in the CBP in order to review and understand a growing number of concepts
• Few commits - A feeling that I'm still not totally aware of what is going on has led to few commits, in my opinion, this week
• Final weeks of the semester - Well, everyone should be having time issues related to this by now, but nevertheless it still has a considerable toll on project development

Possible/Planned future goals:

• Improve current prosodyctl commands to list all locally available modules and tell which ones are enabled and disabled
• Implement a command that enables or disables local modules (Probably similar with what happens with the telnet console commands)
• Decide how should the program behave with modules that require configuration
• Understand how Luarocks deals with package management and dependency resolution

Considerations:

Last week, Taiyo Watanabe from Nintendo shared some insights about Nintendo Switch Push Notification infrastructure (NPNS) implementation. Here are some details from his presentation at ElixirFest conference.

The Nintendo Switch NPNS is a large-scale realtime messaging service based on ejabberd. The game consoles are constantly connected to the infrastructure, listening for push notifications. So far, Nintendo shipped more than 34 million units, thus the infrastructure needs to handle millions of concurrent connections.

The notifications include user-based (directed at active players) and system-based (automatic updates, parental controls). They also include the PubSub topic-based ones like the update notifications and important news, and are available when connected and after a successful login.

In terms of technology, XMPP was chosen due to the large feature set of that protocol. ejabberd was chosen because of its scalability and robustness. It is built in Erlang and inherits many of Erlang strengths: scalability, robustness, native clustering support, and more.

To ease maintenance, Nintendo ejabberd cluster is split into two areas: the inner ejabberd (cpu-bound) and outer ejabberd (memory-bound). All the nodes are connected by the distributed Erlang communication.

The job of the inner ejabberd is to retrieve messages from Amazon Simple Queue Service (SQS), while the outer ejabberd handles the distribution to the Switch game consoles. Business requirements include 1 million concurrent connections, and the recovery for bursting logins in max 30 minutes.

The first performance results of the load test with a simple stock ejabberd service was not sufficient for the production requirements, with capacity at 300k connections and 150 connections per second. Therefore, performance tuning and system redesign was conducted.

The first improvement to reduce memory footprint was to remove the XML parser except from the session memory when it was not active. The second change was to tune OpenSSL parameters to reduce the buffer size. Consequently, the maximum capacity increased to around 750k concurrent connections.

Another issue was the congestion on sockets during burst logins and slow item distribution for some topics, due to the process bottleneck. To solve this, a dedicated distribution worker process was created for each socket. As a result, login throughput increased from 150 per second to 300.

However, the mass delivery of contents created another congestion between the databases and Erlang cluster. To mitigate this, the action was to move content distribution processes from the inner to the outer cluster. Additionally, selective relaying from the inner to outer cluster was set up using a process dictionary to speed up access.

Another interesting problem appeared from the fact that XMPP accepts multiple logins of the same ID (for example, multiple devices). This resulted in too many session processes being created and too many controls being performed to apply standard XMPP business rules. As those rules were not needed in Nintendo use case, the quick solution was to just do nothing when a session is disconnected, and to make the session process die in 1 minute instead of 30.

The above issue was found using Erlang tracing. However, log processing time became a pain and was increasing exponentially, because of reductions (erlang:bumpreductions/1). To be able to handle that, a “dam” process was created to perform rate limiting. Thanks to it, 500k log lines could be processed in 30 seconds instead of 2 hours!

After all the tuning, the infrastructure met business requirements and was ready for production deployment. But that wasn’t the end of tweaking.

Production service was up, but monitoring signalled Redis load was five times bigger than what was expected. A message loop was discovered under certain disconnection conditions. One possible cause might be unprocessed messages triggering an exchange loop. However, testing any hypothesis on the whole production system was too risky. Therefore, a manual hot code deployment was applied directly on only one of the nodes, handling just a small part of the traffic.

Luckily, the patch on that node was behaving properly, and it was extended to all other nodes. Another higher load was detected in relation to small messages, caused by “busy waiting” BEAM scheduler. The solution: switch +sbwt from medium to very_short resulting in CPU use dropping from 30% to 21%.

The key lesson from this project is to always perform load-testing and profiling. Searching for bottlenecks should be continuous, keeping a lookout for C extensions (NIFs). A good approach is to keep increasing parallelism, reduce per-process load and set rate-limiting. Scaling a service is hard and directly depends on your use case. As you scale, you may discover new bottlenecks and parameters to tune and adjust. Scaling a service is an ongoing effort.

Today, the Nintendo Switch NPNS handles 10 000 000 (10 million!) simultaneous connections, 2 000 000 000 (2 billion!!) messages per day with 100-200k connections per node and 600 messages per second!

The greatest advantage of ejabberd is its incredible scalability and throughput. Erlang VM was also very useful, allowing troubleshooting a live system using Erlang remote shell. In the end, this project was implemented in just 6 months, with 1 month of prototyping. It’s been up and running ever since without major issues.

Here is the link to the slides of the presentation (in Japanese).

Monal will support dark mode the day iOS 13 launches. There are a few things to fix here and there regarding images and contrast, but we are definitely ready for iOS 13. This may seem like an odd diversion but is very important. The first is this is someone’s first impression of XMPP and not supporting the latest UI is a great way to make them think XMPP is old. The second is the day iOS 13 comes out, people will seek out apps that support the latest features. These are people who will likely install Monal and experience XMPP for the first time. We need these new people.

There are a lot of new developments coming out to wwdc right now that will impact Monal and XMPP. I am still going over the technical documentation, but the most relevant items are :

1. I will switch to catalyst to bring iOS ans Mac apps to parity. Maintaining one ui will also speed up development
2. cryptokit will change the way we do omemo (for the better). I expect to be able to leverage this to release in France.

@wroot wrote:

The Ignite Realtime community is happy to announce the immediate release of version 1.2.1 of the User Status plugin for Openfire!

User Status plugin automatically saves the last status (presence, IP address, logon and logoff time) per user and resource to userStatus table in the Openfire database.

This update adds support for longer IPv6 addresses and fixes a few issues with plugin’s build.

Your instance of Openfire should automatically display the availability of the update in the next few hours. Alternatively, you can download the new release of the plugin at the User Status plugin archive page.

For other release announcements and news follow us on Twitter

Posts: 1

Participants: 1

Read full topic

The goal here is to get more users who aren’t power users. For these users there needs to be less exposure to the technical workings of XMPP and fewer options. The usual account screens are still there in settings but a new user will encounter something like this. Someone should be able to download Monal, hit register and talk to another XMPP user without any hand holding. Part of plan to get to that goal is the new registration screen. This is a work in progress and currently hard coded to blabber.im but could be any server if needed.

Lots of things in the pipeline and many changes are afoot. iOS 3.7 is available. 3.8 will come next week and there is a Mac refresh coming with the same fixes. The WWDC keynote is on Monday and we will see a lot of changes coming to Mac and iOS.

iOS is expected to have a dark mode and I plan on having Monal support on day of the iOS 13 release (likely Sept). I will likely start adding support for it and have it in betas as soon as I can. The process of adding dark mode in macOS was pretty straightforward and I have many of the art assets ready for it already.

Mac is going to see a huge change. At the moment I maintain two apps a Mac app and an iOS app. We expect to see marzipan based iOS apps running on Mac OS and depending on what the capabilities are, I hope to have it ready on day 1 as well. This will present an interesting scenario where I will have two Mac apps. As a long time Mac user, I know the thought of iOS apps seems dreadful at first glance, but the reality is this is likely the shot in the arm the OS needs for app support. The current one will continue to exist because there are still many Macs out there that will not run iOS apps. However the experience for the two apps will be very different, I am going to have to decide what I will name the two (the current one will likely be Monal Classic) .

The plan for both apps (but iOS first) is to have an app that requires no explanation. We should be able to tell people to grab the Monal app, register or login and then chat. There are a lot of messaging apps and either it works perfectly on the first try or it gets uninstalled.

The general roadmap before I take a chunk of the summer off:

iOS:

Registration, Password change and QR codes.
Dark Mode
Muc fixes /Mix
New Contact List
encrypted attachments
simple video/audio support

Mac:

Marzipan
Classic App iOS Parity

Late in testing there was a critical bug discovered that prevented openfire servers from working with Monal. This has been resolved in the new beta out today and I hope to have 3.7 released this week.

Testing on a local open fire server I was reminded of a few things, Openfire is super easy to use, its super popular and the fact that it doesn’t support push is maddening. Plugin development seems to be reasonably easy any Java developers out there interested in helping me implement push?

In the continuity with the previous post about jp, the following article present sat_pubsub, a XMPP Publish-Subscribe (Pubsub) Service Component, build for the need of the « Salut à Toi » project.

Salut à toi (SàT) is a unique XMPP client. As its official description says, it's a "multipurpose, multi front-end, free (libre) and decentralized communication tool". It has been actively developed by Jérôme Poisson (Goffi) and Adrien Cossa (Souliane) since 2008.

sat_pubsub allows us to use our own up-to-date persistent pubsub service.

This article is composed of several sections

• Why sat_pubsub ?
• Let's install it !
• Let's use it !
• A note about Agayon XMPP service
• Future of SàT
• Links

Why sat_pubsub ?

There are three main reasons to use sat_pubsub.

First, the XMPP servers come with variable pubsub support. Using an external component allows users to benefits constant features independent of the XMPP server or provider.

Secondly, sat_pubub is the only free (libre) implementation that support Pubsub MAM (Message Archive Management). It allows one to research a pubsub node. This feature is used to research blog post in the database.

Thirdly, sat_pubsub is used to test new functionalities like the ability to restrict blog post to specific group of contacts (think about circles on Google+). ¹.

Let's install it !

This section describes the installation process of sat_pubsub in a python virtualenv. Unfortunately, I have encountered difficulties to install twisted, a dependency of SàT, in a virtualenv with pip because my production machine has no compiler. I managed to avoid the problem by installing python2-twisted (the distribution package) then the virtualenv was created with the option --system-site-packages to give it access to the system packages. Thereafter, it is possible to install sat_tmp, a small python module that monkey patch wokkel, a dependency of sat_pubsub.

$ sudo apt-get install python2-twisted
[...]
$ hg clone http://repos.goffi.org/sat_tmp
$ cd sat_tmp/
$ virtualenv env -ppython2.7 --system-site-packages
$ source env/bin/activate
(env)$  python2 setup.py install
(env)$ cd ../
(env)$ hg clone https://repos.goffi.org/sat_pubsub
(env)$ cd sat_pubsub
(env)$ python2 setup.py install

Setup the database

According to the official documentation, we need to create the database and install the SQL schema.

$ sudo -u postgres createuser -d -P `whoami`
$ createdb pubsub
$ psql pubsub < sat_pubsub/db/pubsub.sql

Configure Prosody

In order to use sat_pubsub, we need to declare it in our prosody config file. Once again, the information is available in the documentation. Two files need to be modified:

• /etc/prosody/prosody.cfg.lua

[...]
modules_enabled = {
    [...]
    "delegation";
    "privilege";
}
[...]

• /etc/prosody/conf.avail/your_domain.cfg.lua

This file defines the domain configuration of your XMPP server.

This is a copy of the configuration for the agayon server. The pubsub component is available at the address pubsub.agayon.be. The following configuration:

• enables pubsub MAM,
• uses the component for microblogging activity,
• announces the pubsub nodes and items in disco requests,
• lets the component access the roster and presence informations (used for PEP).

VirtualHost "agayon.be"
    enabled = true
    [...]
    privileged_entities = {

            ["pubsub.agayon.be"] = {
            roster = "get";
            message = "outgoing";
            presence = "roster";
            },
       }
    delegations = {
                ["urn:xmpp:mam:1"] = {
                        filtering = {"node"};
                        jid = "pubsub.agayon.be";
                },
                 ["http://jabber.org/protocol/pubsub"] = {
                        jid = "pubsub.agayon.be";
                 },
                 ["http://jabber.org/protocol/pubsub#owner"] = {
                        jid = "pubsub.agayon.be";
                 },
                 ["https://salut-a-toi/protocol/schema:0"] = {
                        jid = "pubsub.agayon.be";
                 },
                 ["http://jabber.org/protocol/disco#items:*"] = {
                        jid = "pubsub.agayon.be";
                 },
            }


Component "pubsub.agayon.be"
        component_secret = "shared_secret"
        modules_enabled = {"delegation", "privilege"}

Launch

(env)$ twistd  sat_pubsub  --jid=pubsub.agayon.be --secret=shared_secret --rhost=127.0.0.1 \
--rport=5347 --backend=pgsql --dbuser=user --dbpass=pass_psql --dbname=pubsub --dbhost=localhost --dbport=5432

Let's use it !

Once prosody is restarted, the component is accessible through any XMPP client. See the following example with jp.

$ jp info disco pubsub.agayon.be -c
Features

http://jabber.org/protocol/disco#items
http://jabber.org/protocol/pubsub#auto-create
http://jabber.org/protocol/pubsub#config-node
http://jabber.org/protocol/pubsub#create-nodes
http://jabber.org/protocol/pubsub#delete-any
http://jabber.org/protocol/pubsub#delete-nodes
http://jabber.org/protocol/pubsub#groupblog
http://jabber.org/protocol/pubsub#instant-nodes
http://jabber.org/protocol/pubsub#item-ids
http://jabber.org/protocol/pubsub#meta-data
http://jabber.org/protocol/pubsub#outcast-affiliation
http://jabber.org/protocol/pubsub#persistent-items
http://jabber.org/protocol/pubsub#publish
http://jabber.org/protocol/pubsub#publish-options
http://jabber.org/protocol/pubsub#publisher-affiliation
http://jabber.org/protocol/pubsub#purge-nodes
http://jabber.org/protocol/pubsub#retract-items
http://jabber.org/protocol/pubsub#retrieve-affiliations
http://jabber.org/protocol/pubsub#retrieve-default
http://jabber.org/protocol/pubsub#retrieve-items
http://jabber.org/protocol/pubsub#retrieve-subscriptions
http://jabber.org/protocol/pubsub#subscribe
https://salut-a-toi.org/spec/pubsub_admin:0
https://salut-a-toi/protocol/schema:0
jabber:iq:version
urn:xmpp:mam:2
urn:xmpp:order-by:0

Identities

┌───────┬─────────┬───────────────────────────┐
│catego │ type    │ name                      │
├┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┤
│pubsub │ service │ Salut à Toi pubsub service│
└───────┴─────────┴───────────────────────────┘

Items

┌─────────────────┬───────────────────┬─┐
│entity           │ node              │ │
├┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┼┄┤
│pubsub.agayon.be │ be.agayon.errol:0 │ │
│pubsub.agayon.be │ tests             │ │
│pubsub.agayon.be │ Agayon            │ │
└─────────────────┴───────────────────┴─┘

The following commands are used to create a node, subscribe to it and edit an item (see the previous article).

$ jp pubsub node create -s pubsub.agayon.be -n node_name
$ jp pubsub subscribe -s pubsub.agayon.be -n node_name
$ jp pubsub edit -s pubsub.agayon.be -n node_name

The default text editor is then opened. It is possible to directly edit a XML file. This command is useful for debugging purpose.

About Agayon XMPP service

All the tests described in this blog are realized on the Agayon.be XMPP service. It is not opened for registration but having this field of experimentation is great to learn, practice and question the potential uses of XMPP. I use it with several accounts, depending on my use:

• obtain statistics about the server with chatty_server, a small python bot.
• write memos to save links in a dummy account,
• automatic file transfer,
• daily logs sender with logwatch
• ...

Most moderns XEPs are enabled in order to provide an up-to-date experience: PEP, Carbon, Stream Management, persistent pubsub, MAM, SRV records over TLS (useful to pass blocking WiFi accesses), etc.

Future of SàT

In a near future, the SàT project will make great announcements about new uses of pubsub. I hope this article will make you want to install sat_pubsub and experiment with nodes, data and notifications.

Stay tuned !

Links

• sat_pubsub repository
• Install notes (maybe outdated)
• The previous article about JP

1. See the following blog post for more information. ↩

Salut à toi is a unique XMPP client. As its official description says, it's a "multipurpose, multi front-end, free (libre) and decentralized communication tool". It has been actively developed by Jérôme Poisson (Goffi) and Adrien Cossa (Souliane) since 2008. Today, I will focus on the use of "JP", a command-line interface. It can be used to send or receive files directly from a shell, pipe commands to or from XMPP, use XMPP easily in a script and of course play with pubsub nodes.

The following article describes uses of JP. Several of them are availabled in the trunk version of Salut-à-Toi.

Introduction

JP can be used to launch complexes commands in script, for debugging purpose or to explore XMPP services. JP is the command line interface and it connect to the daemon Salut-à-Toi (SàT). You can share the session between the front-ends:

• JP of course.
• Primitivus as a console front-end.
• Livervia, a web based front-end.
• Sententia an Emacs front-end (WIP).
• Cagou an original mobile XMPP client based on SàT (WIP).

If you want to send a file easily, discover which services are available on a server, send messages in your scripts, manage your xmpp account, control video player, edit your blog post with your favorite editor, pipe streams, manage your pubsub nodes etc, JP is for you!

Usage

JP can be used as a command line tools or in a small shell environment.

First, we need to configure and launch the daemon sat.

$ sat

Your default profile will be connected. If you have no profile, JP can be used to create one.

$ jp profile create -j my_jid@example.org  -x mypassword profile_name
$ jp profile connect -p profile_name

Your password is saved in the sat settings. You can connect automatically with the option -c. It should be noted that SàT defines a default profile. It can be bypassed with the option -p. ¹

$ echo "test message" | jp message send recipient@example.org -p profile_name -c

You can obtain help about a command with the option -h.

$ jp pubsub -h
usage: jp pubsub [-h]
                 {get,delete,edit,subscribe,unsubscribe,subscriptions,node,affiliations,search,hook,uri}
                 ...

positional arguments:
  {get,delete,edit,subscribe,unsubscribe,subscriptions,node,affiliations,search,hook,uri}
    get                 get pubsub item(s)
    delete              delete an item
    edit                edit an existing or new pubsub item
    subscribe           subscribe to a node
    unsubscribe         unsubscribe from a node
    subscriptions       retrieve all subscriptions on a service
    node                node handling
    affiliations        retrieve all affiliations on a service
    search              search items corresponding to filters
    hook                trigger action on Pubsub notifications
    uri                 build URI

optional arguments:
  -h, --help            show this help message and exit

JP is a Swiss army knife. Let's discover its possibilities through a few examples.

Examples

$ jp -h
usage: jp [-h] [--version]
          {file,input,uri,message,event,info,account,param,debug,ad-hoc,ticket,invitation,profile,shell,avatar,pipe,pubsub,bookmarks,roster,identity,blog}
          ...

This software is a command line tool for XMPP.
Get the latest version at http://salut-a-toi.org

optional arguments:
  -h, --help            show this help message and exit
  --version             show programʼs version number and exit

Available commands:
  {file,input,uri,message,event,info,account,param,debug,ad-hoc,ticket,invitation,profile,shell,avatar,pipe,pubsub,bookmarks,roster,identity,blog}
    file                File sending/receiving
    input               launch command with external input
    uri                 XMPP URI parsing/generation
    message             messages handling
    event               event management
    info                Get various pieces of information on entities
    account             XMPP account management
    param               Save/load parameters template
    debug               debugging tools
    ad-hoc              Ad-hoc commands
    ticket              tickets handling
    invitation          invitation of user(s) without XMPP account
    profile             profile commands
    shell               launch jp in shell (REPL) mode
    avatar              avatar uploading/retrieving
    pipe                stream piping through XMPP
    pubsub              PubSub nodes/items management
    bookmarks           manage bookmarks
    roster              Manage an entityʼs roster
    identity            identity management
    blog                blog/microblog management

Send a message

$ echo "Hello World" > filetest
$ jp message send recipient@example.org < filetest

or

$ echo "test  jp" | jp message send recipient@example.org

Send files

The following command allows to send the file file.txt to recipient@example.org.

$ jp file send file.txt recipient@example.org

Receive files

The following command allows to receive a file in the /tmp directory.

$ jp file receive -p agayon --path /tmp

Get information about a server

$ jp info version agayon.be
Client name: Prosody
Client version: 0.10.0
Operating System: Linux

Get disco information

Query a Server

$ jp info disco agayon.be
Features

http://jabber.org/protocol/commands
http://jabber.org/protocol/disco#info
http://jabber.org/protocol/disco#items
http://jabber.org/protocol/pubsub#auto-create
http://jabber.org/protocol/pubsub#config-node
http://jabber.org/protocol/pubsub#create-nodes
http://jabber.org/protocol/pubsub#delete-any
http://jabber.org/protocol/pubsub#delete-nodes
http://jabber.org/protocol/pubsub#groupblog
http://jabber.org/protocol/pubsub#instant-nodes
http://jabber.org/protocol/pubsub#item-ids
http://jabber.org/protocol/pubsub#meta-data
http://jabber.org/protocol/pubsub#outcast-affiliation
http://jabber.org/protocol/pubsub#persistent-items
http://jabber.org/protocol/pubsub#publish
http://jabber.org/protocol/pubsub#publish-options
http://jabber.org/protocol/pubsub#publisher-affiliation
http://jabber.org/protocol/pubsub#purge-nodes
http://jabber.org/protocol/pubsub#retract-items
http://jabber.org/protocol/pubsub#retrieve-affiliations
http://jabber.org/protocol/pubsub#retrieve-default
http://jabber.org/protocol/pubsub#retrieve-items
http://jabber.org/protocol/pubsub#retrieve-subscriptions
http://jabber.org/protocol/pubsub#subscribe
https://salut-a-toi.org/spec/pubsub_admin:0
https://salut-a-toi/protocol/schema:0
jabber:iq:auth
jabber:iq:last
jabber:iq:private
jabber:iq:roster
jabber:iq:time
jabber:iq:version
msgoffline
urn:xmpp:blocking
urn:xmpp:carbons:1
urn:xmpp:carbons:2
urn:xmpp:http:upload
urn:xmpp:http:upload:0
urn:xmpp:mam:0
urn:xmpp:order-by:0
urn:xmpp:ping
urn:xmpp:time
vcard-temp

Identities

┌───────┬──────┬─────────────────┐
│catego │ type │ name            │
├┄┄┄┄┄┄┄┼┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┤
│store  │ file │ HTTP File Upload│
│pubsub │ pep  │ Prosody         │
│server │ im   │ Prosody         │
└───────┴──────┴─────────────────┘

Extensions

urn:xmpp:http:upload
        type: text-single
        var: max-file-size
        value: 1048576
urn:xmpp:http:upload:0
        type: text-single
        var: max-file-size
        value: 1048576

Items

┌──────────────────────┬──┬────────────────────┐
│entity                │  │ name               │
├┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┼┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┤
│test-pubsub.agayon.be │  │                    │
│upload.agayon.be      │  │                    │
│auth.agayon.be        │  │                    │
│proxy.agayon.be       │  │                    │
│pubsub.agayon.be      │  │                    │
│chat.agayon.be        │  │ Agayon.be chatrooms│
└──────────────────────┴──┴────────────────────┘

Query an account

$ jp info disco info@agayon.be
Items

┌───────────────┬───────────────────────────────────────────────────┬─┐
│entity         │ node                                              │ │
├┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┼┄┤
│info@agayon.be │ eu.siacs.conversations.axolotl.bundles:1948395845 │ │
│info@agayon.be │ http://jabber.org/protocol/activity               │ │
│info@agayon.be │ http://jabber.org/protocol/mood                   │ │
│info@agayon.be │ eu.siacs.conversations.axolotl.devicelist         │ │
│info@agayon.be │ storage:bookmarks                                 │ │
└───────────────┴───────────────────────────────────────────────────┴─┘

Manage pubsub nodes

The following examples show how to manage pubsub nodes on the service pubsub.agayon.be.

Create a node

$ jp pubsub node create -s pubsub.agayon.be -n node_name

Subscribe to a node

$ jp pubsub subscribe -s pubsub.agayon.be -n node_name

Edit

The edit command allows to edit an item under a node.

$ jp pubsub edit -s pubsub.agayon.be -n node_name

The default text editor is opened. It is possible to directly edit a XML file. This command is useful for debugging purpose.

If you want to edit a post without having to edit xml directly, use jp blog edit.

As an example, you can try to edit the following xml file.

<entry xmlns="http://www.w3.org/2005/Atom">
  <author>
    <name>author@example.com</name>
  </author>
  <generator>JP (SàT)</generator>
  <title>I am a pubsub post !</title>
  <content>This is the content of this great post.</content>
</entry>

Once the file is saved, a notification appears in Gajim. For now, the name of the post is a hash.

Manage your XMPP Blog

It is possible to manage a post with XMPP. It is based on PEP, a simplified version of pubsub. If your server support PEP, JP can help you to manage this blog easily.

Publish a post

First, you need to define your preferred syntax. In this example, I select markdown. This option can also be set in any other frontend (e.g. Primitivus). Whereafter you can edit a new post with the syntax jp blog edit.

$ jp param set Composition Syntax markdown -p agayon
$ jp blog edit

Your favorite editor open and you can edit your blog post with markdown syntax. When you save and close it, another file open. You can edit your settings:

{
    "allow_comments": true,
}

You can verify the accessibility of your post with the following command:

$ jp blog get 
content: 
Great test !
Make testing great again !
Yuuuuuuge publication.

It is also possible to modify your last blog post simply with jp blog edit --last-item.

Your blog post are also visible on other clients like Movim (see below) and interfaces like Livervia.

Use jp shell:

In order to ease debugging of services, JP comes with a shell interface. You only need to launch jp shell. You can obtain help by typing ?.

$ jp shell
cmd pubsub
pubsub> ?
Shell commands:
Documented commands (type help <topic>):
========================================
cmd    do    help  shell  use_clear  version
debug  exit  quit  use    verbose    whoami 

Action commands:
positional arguments:
  {get,delete,edit,subscribe,unsubscribe,subscriptions,node,affiliations,search,hook,uri}
    get                 get pubsub item(s)
    delete              delete an item
    edit                edit an existing or new pubsub item
    subscribe           subscribe to a node
    unsubscribe         unsubscribe from a node
    subscriptions       retrieve all subscriptions on a service
    node                node handling
    affiliations        retrieve all affiliations on a service
    search              search items corresponding to filters
    hook                trigger action on Pubsub notifications
    uri                 build URI

Select a command

> cmd pubsub
pubsub> use node urn:xmpp:microblog:0

Navigate into commands

> cmd bookmarks/list
bookmarks/list> -c
... CMD result
bookmarks/list> cmd ..
bookmarks> cmd
>

Example: List bookmarks

$  jp shell
> cmd bookmarks
bookmarks> cmd list
bookmarks/list> ?
Shell commands:
Documented commands (type help <topic>):
========================================
cmd    do    help  shell  use_clear  version
debug  exit  quit  use    verbose    whoami 

Action commands:
optional arguments:
  -h, --help            show this help message and exit
  -p PROFILE, --profile PROFILE
                        Use PROFILE profile key (default: @DEFAULT@)
  --pwd PASSWORD        Password used to connect profile, if necessary
  -c, --connect         Connect the profile before doing anything else
  -l {all,local,private,pubsub}, --location {all,local,private,pubsub}
                        storage location (default: all)
  -t {muc,url}, --type {muc,url}
                        bookmarks type (default: muc)
bookmarks/list> -t muc
private:
    Movim [movim@conference.movim.eu] (*)
    Archlinux - blah blah [archlinux-fr@chat.jabberfr.org]
    bot [bot@chat.agayon.be] (*)

Example: get disco information

> cmd info
info> use jid pubsub.agayon.be 
info> disco
Features

http://jabber.org/protocol/disco#info
http://jabber.org/protocol/disco#items
http://jabber.org/protocol/pubsub
http://jabber.org/protocol/pubsub#create-nodes
http://jabber.org/protocol/pubsub#delete-items
http://jabber.org/protocol/pubsub#delete-nodes
http://jabber.org/protocol/pubsub#instant-nodes
http://jabber.org/protocol/pubsub#item-ids
http://jabber.org/protocol/pubsub#publish
http://jabber.org/protocol/pubsub#publisher-affiliation
http://jabber.org/protocol/pubsub#purge-nodes
http://jabber.org/protocol/pubsub#retract-items
http://jabber.org/protocol/pubsub#retrieve-items
http://jabber.org/protocol/pubsub#retrieve-subscriptions
http://jabber.org/protocol/pubsub#subscribe

Identities

┌───────┬─────────┬───────────────────────┐
│catego │ type    │ name                  │
├┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┼┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┤
│pubsub │ service │ Prosody PubSub Service│
└───────┴─────────┴───────────────────────┘

More

• Export command (input/output) to a contact
• Create a universal remote for VLC (media player)

Some alternatives

SleekXMPP

SleekXMPP is a xmpp python library. It provide two examples of pubsub clients. The first, pubsub_client.py can manage nodes and the second one, pubsub_events.py, can manage events.

./pubsub_client.py -j jid --password=passwd pubsub.agayon.be create node_name

Gajim

Gajim is a great desktop client. It can be used to post on pubsub nodes too.

Movim

Movim is a social network platform based on XMPP. It massively use pubsub to allow users to interact.

Conclusions

Salut-à-Toi is a great XMPP client. JP, its command-line interface, is a powerful tool. It can be useful to debug services, pubsub nodes and play with 'social' capacities of XMPP. It is actively developed and new features are coming. We will be soon able to make todolist or fill a bug ticket on a XMPP based system.

Links

• Salut à toi
• SàT room: sat@chat.jabberfr.org
• JP
• SàT repository
• Archlinux package
• Gajim
• Movim

1. It is possible to use jp profile modify to set the default profile. ↩