Vaxbot, a service built by the Monal team to vaccinate people in the US and Canada was featured on the Canadian Broadcast Corporation (CBC) The National and was broadcast nationwide.
This was great exposure for the project and XMPP in general, introducing it to a whole new population that likely skews older and less tech savvy. They don’t show Monal but you can see one of the users in the clip using Converse.js to access the bot. There has been a massive spike in new users since it aired on TV. At its peak in the US, we were doing about 1 million messages a day. We are approaching 800k again in Canada now.
The Ignite Realtime Community is pleased to announce the release of Openfire version 4.6.4. This release contains a number of bug fixes and denotes our desire to provide stable 4.6.x series releases while development work continues on a 4.7.0 release.
You can find download artifacts available having the following sha1sum values:
Please report any issues you have with this release in our Community Forums.
We are always desperate for folks interested in pitching in with development, testing, and code reviews. Please consider dropping by our webchat forum if you are interested in helping.
In recent posts we've looked into holding opinions about fewer topics, holding multiple opinions about the same topic, and changing our opinions about the opinions that other people hold. But what exactly is an opinion? Let's take a closer look....
we have implemented an attention flag in profanity.
The attention flag can be used to mark chats and groupchats where you would like to
pay particular attention.
This is only available on master, but will be in the next release (0.11.0).
How it works
Open the chat or groupchat window and press shortcut ALT+F. Profanity will
display a line to inform you when the attention flag has been activated and
deactivated.
05/06/21 15:25:49 - Staff restaurant: Lunch recommendations:
05/06/21 15:27:04 ! Attention flag has been activated
05/06/21 15:27:04 ! Attention flag has been deactivated
You can use the shortcut ALT+F to toggle the flag.
The /wins attention command can be used to display all windows with you pay
attention.
Welcome to the XMPP Newsletter covering the month of May 2021.
Many projects and their efforts in the XMPP community are a result of people’s voluntary work. If you are happy with the services and software you may be using, especially throughout the current situation, please consider to say thanks or help these projects!
Many thanks to the translators and their work! This is a great help to spread the news! Please join them in their work or start over with another language!
XSF Announcement
Voting the membership applications can be done via xmpp:memberbot@xmpp.org (by XSF members only). We will hold a member meeting on June 10th to formally approve the voting results. The meeting particulars are:
Date: June 10th, 2021
Time: 19:00 UTC
Location: xmpp:xsf@muc.xmpp.org
Further information: https://wiki.xmpp.org/web/Membership_Applications_Q2_2021
Ingo Jürgensmann wrote the article The Fediverse – What About Resources? on the resources of different messaging technologies. He states that XMPP consumes a lot less hardware resources and thereby energy than comparing services.
Vaxbot US has been shutdown after change of the situation in the USA. But the service has been deployed for Canada instead. All in all, that was an interesting approach and use of XMPP technology!
Software news
Clients and applications
Video calls in Dino are slowly coming together. Dino's developers are already making
successful OMEMO encrypted video calls. The feature is included in their nightly builds now, but there's still further work to be done.
Gajim development News: This month brought improved Ad-Hoc commands, fixes for Gajim Portable,
and new image preview capabilities. Meanwhile, work on Gajim’s next version made some progress: better code block styling, chat filters, note to myself, and much more. Also in Gajim news: Gajim celebrated its 17th birthday this month. Philipp Hörist (lovetox), maintainer of Gajim, gave a preview of the new Gajim 1.4 User Interface. Gajim is an XMPP client written in Python. It currently receives a big UI overhaul, of which the first results were presented at the XMPP Office Hours.
Kaidan 0.8 has been released with noteworthy new features, including typing notifications (Chat State Notifications) and message history synchronization (using MAM)!
Prosody 0.11.9 has been released: This release addresses a number of important security issues that
affect most deployments of Prosody. Full details are available in a separate security advisory. We recommend that all deployments upgrade or apply the mitigations described in the advisory.
Snikket just published their May update to the Snikket server software. This includes a few security fixes from Prosody, so upgrade soon! It also now allows you to manage user roles and access levels.
Libraries
No updates on XMPP libraries reach our attention :-(
Extensions and specifications
Developers and other standards experts from around the world collaborate on these extensions, developing new specifications for emerging practices, and refining existing ways of doing things. Proposed by anybody, the particularly successful ones end up as Final or Active - depending on their type - while others are carefully archived as Deferred. This life cycle is described in XEP-0001, which contains the formal and canonical definitions for the types, states, and processes. Read more about the standards process. Communication around Standards and Extensions happens in the Standards Mailing List (online archive).
Proposed
The XEP development process starts by writing up an idea and submitting it to the XMPP Editor. Within two weeks, the Council decides whether to accept this proposal as an Experimental XEP.
This document defines XMPP application categories for different use cases (Core, Web, IM, and Mobile), and specifies the required XEPs that client and server software needs to implement for compliance with the use cases.
New
No new XEP this month.
Deferred
If an experimental XEP is not updated for more than twelve months, it will be moved off Experimental to Deferred. If there is another update, it will put the XEP back onto Experimental.
Recommend PubSub access model 'open' for public key data node and metadata node. (ps)
Version 1.3 of XEP-0013 (Flexible Offline Message Retrieval)
Deprecate after council vote of 2021-03-31 (XEP Editor (jsc))
Last Call
Last calls are issued once everyone seems satisfied with the current XEP status. After the Council decides whether the XEP seems ready, the XMPP Editor issues a Last Call for comments. The feedback gathered during the Last Call help improving the XEP before returning it to the Council for advancement to Draft.
No Last Call this month.
Draft
No Draft this month.
Call for Experience
A Call For Experience - like a Last Call, is an explicit call for comments, but in this case it's mostly directed at people who've implemented, and ideally deployed, the specification. The Council then votes to move it to Final.
No Call for Experience this month.
Thanks all!
This XMPP Newsletter is produced collaboratively by the community.
Therefore many thanks to emus, Florent Zara, Goffi, jeybe, Licaon_Kter, mdosch, nicola, pmaziere, snark, wurstsalat and Ysabeau for their support and help in creation, review and translation!
We started drafting in this simple pad in parallel to our efforts in the XSF Github repository. We are always happy to welcome contributors. Do not hesitate to join the discussion in our Comm-Team group chat (MUC) and thereby help us sustain this as a community effort. We really need more support!
You have a project and write about it? Please consider sharing your news or events here, and promote it to a large audience!
And even if you can only spend a few minutes of support, these would already be helpful!
Tasks which need to be done on a regular basis are for example:
Aggregation of news in the XMPP universe
Short formulation of news and events
Summary of the monthly communication on extensions (XEP)
Sometimes it's difficult to hold fewer opinions in your own mind or to engage in cognitive empathy toward others; that's when it can help to hold multiple opinions at the same time. This might sound like the mental equivalent of juggling plates, but it's a skill worth cultivating (and one with an ancient pedigree, as evidenced for example by the Letter to Pythocles by the ancient Greek philosopher Epicurus)....
We are happy publish a new release of Kaidan that brings it closer fullfilling the daily messaging needs.
As promised, this release includes some major new features:
The chat history is now being synchronized across devices, to allow finding information in old messages.
It is now indicated when the chat partner is typing, which makes conversations easier.
We have also polished the general usability, so that now the size of the windows is preserved across restarts.
Some of the strings in the user interface have received some rewording, to make them easier to understand.
The registration now integrates as good as possible with servers which have registration through chat apps disabled, by showing the link to register through the server’s website.
On the cross-platform side, we have decided to use the breeze theme also on macOS, as the macOS style does not support a few features we want to use in the user interface.
Changelog
This release adds the following features:
Add typing notifications (XEP-0085: Chat State Notifications)
Add message history syncing (XEP-0313: Message Archive Management)
Window size is restored
The server’s website link is displayed if account creation is disabled
This month brings improved Ad-Hoc commands, fixes for Gajim Portable, and new image preview capabilities. Meanwhile, work on Gajim’s next version made some progress: better code block styling, chat filters, note to myself, and much more.
Changes in Gajim
Through Gajim’s Ad-Hoc commands window, you are able to configure service settings, gather infos, or trigger various other actions offered by your provider. The Ad-Hoc commands window has now been ported to the new Assistant, which you already know from Gajim’s Account Creation for example. This lowers maintenance effort and allows all tasks with a ‘wizard’ workflow to have consistent design and behaviour.
On Windows, you can choose whether you want to install Gajim on your system or if you want to have a portable version of Gajim inside a single folder. For distribution of Gajim Portable, we use an installer which extracts all relevant files into a directory of your choice. However, this process sometimes fails if you upgrade an existing Gajim Portable folder. This issue has now been fixed by applying a cleaning mechanism before extracting the new version. Your user data won’t be touched, of course.
Meanwhile, work on the next Gajim version made some progress:
‘Note to myself’ feature: write messages to your own contact (e.g. to another device), now improved
Code block styling is now aligned to your theme (light/dark)
Chats can be filtered by ‘chats’ and ‘group chats’ in the Start Chat window and in the Chat List
Status message styling has been improved
File transfers (uploading) are now displayed inline
Gajim update notifications will now be displayed in a separate App Page, so there won’t be disrupting update notifications anymore
Status Icon has been fixed, and the App Indicator plugin has been integrated
Status message window has been removed: it’s now a simple input field
If you haven’t seen it yet: lovetox (Philipp Hörist), current maintainer of Gajim, gave a first introduction to what’s coming with the next version:
This is only part of what we’re planning to do for the next release of Gajim. Stay tuned!
What else happened
Profile window: Fixed initial state of privacy switches
Profile window: Added Note element
#10559: Fixed issue with Server Info window when using PLAIN connection
Plugin updates
Gajim’s URL Image Preview plugin is now able to generate previews for WEBP and JXL files. MIME type guessing has been improved as well.
Changes in python-nbxmpp
This month, Ad-Hoc Commands (XEP-0050) compliance has been improved.
By default, ejabberd uses the Mnesia internal database. It is great for home and small office environments, but in larger companies, as the amount of chat logs and users grows, we need more scalability. Today, I will show you how to install MariaDB, a MySQL-compatible database, migrate your data and configure ejabberd to use MariaDB instead of Mnesia.
» Don’t want to migrate data yourself?
ProcessOne experts will make your communication scalable. Contact us »
Installing MariaDB
We assume the usual Debian configuration as in my previous tutorials. I have updated my ejabberd to version 21.01 (the update process is the same as the initial ejabberd installation, so check my first tutorial).
To install MariaDB simply use:
apt-get install mariadb-server
Then run the installation wizard and follow the instructions:
mysql_secure_installation
Preparing MariaDB for ejabberd
To get MariaDB ready for ejabberd, we need to create a new database, its user, and then populate the database with the ejabberd SQL schema.
First, let’s create the database using your MariaDB root user:
Next, let’s create a dedicated ejabberd user authenticated with a password, and assign it to this database. The Enter password prompt is again asking about the root MariaDB user:
echo "GRANT ALL ON ejabberd.* TO 'ejabberd'@'localhost' IDENTIFIED BY 'password';" | mysql -h localhost -u root -p
Finally, let’s download the latest ejabberd SQL schema and load it into our database. This time, we are switching to using the ejabberd MariaDB user, and the Enter password prompt is asking for the password we just specified in the GRANT command above:
To verify that everything is correct, run a command to display all the database tables, again using the ejabberd MariaDB user, and the output should look something like that:
Now that our MariaDB tables are ready, we need to configure ejabberd to use this MySQL-compatible database. Edit your ejabberd.yml config and add the following settings, where password refers to the ejabberd MariaDB user:
sql_type: mysql
sql_server: "localhost"
sql_database: "ejabberd"
sql_username: "ejabberd"
sql_password: "password"
Migrating Mnesia data to MariaDB database
At this point, if you restart your ejabberd, it won’t be using MariaDB just yet. Let’s first migrate Mnesia data into our new SQL database using ejabberdctl – we first export the data into a mnesia.sql file, and then we import it into the MariaDB database:
cd /opt/ejabberd-21.01/bin/
./ejabberdctl export2sql marekfoss.org /tmp/mnesia.sql
mysql -h localhost -D ejabberd -u ejabberd -p < /tmp/mnesia.sql
rm /tmp/mnesia.sql
It’s a good practice to remove the /tmp/mnesia.sql after we are done with it. Now, add default_db: sql and auth_method: sql to your ejabberd.yml configuration file:
Then, restart your ejabberd instance – it will now use the MariaDB database! Please note that the Mnesia database will still be started up and used for non-persistent data and clustering.
Today is Gajim’s birthday! 🎉 Seventeen years of Instant Messaging with open standards and open source software.
About Gajim
On Mai 21th 2004, Asterix (Yann Leboulanger) released Gajim 0.1. Gajim (is) aJabber Instant Messenger. Development started back when XEPs (XMPP Extension Protocols) were named JEPs (Jabber Enhancement Proposals) and the XSF (XMPP Standards Foundation) was called JSF (Jabber Software Foundation). Right from the first year, Gajim was available for both Linux and Windows. Supported by many contributors, there have been more than 70 releases until today. The code repository counts around 18,000 commits, and the issue tracker (once powered by Trac, now Gitlab) lists a history of over 10,500 opened issues in total.
Today, Gajim supports around 85 XEPs. Driven by the community, Gajim has been translated into 29 languages. A calculated effort of around 31 person-years went into developing Gajim.
With the release of Gajim 1.0 in March 2018, a big switch from Python 2 and GTK 2 to Python 3 and GTK 3 had been completed. For the following releases, Gajim’s core had been gradually rewritten. Since February 2021, we’ve been working on the next version. It will include a complete rework of Gajim’s main component: the message window. lovetox (Philipp Hörist), current maintainer of Gajim, gave a first introduction to what’s coming with the next version:
This is only part of what we’re planning to do for the next release of Gajim. Stay tuned!
We’re pleased to introduce a new release of the Snikket server. The Snikket
server is an easy-to-install server package that allows you to run your own
private messaging service for family, friends and other small groups.
As well as some new features, this release has some important security
fixes for the built-in Prosody component. We advise all
administrators to update as soon as possible.
For information on how to upgrade, see the (very short) upgrade guide.
Web interface
User and role management
This release brings a new interface for viewing and editing user accounts on
the server. Among the changes is the ability to select the “access level” of
an account via the web interface. In particular this allows you to add/remove
other administrators of your server.
In the future we will also be adding an additional ‘limited’ access level that
can be used to restrict access to features such as invites and federation for
certain user accounts (such as guests and minors).
Invitations
Invitation pages now include a link to download the Snikket app from
F-Droid, as well as Google Play. Although F-Droid doesn’t yet
support the seamless registration flow, it’s important that
we help people discover free (as in freedom) alternatives whenever possible!
Translations
Translation improvements have been made for Polish, German, Danish, Spanish
(Mexican), Indonesian and Swedish.
Certificate renewal
A bug has been fixed
that eventually caused Snikket to present an expired certificate for web links
(the web interface and also shared files). Restarting the service is a
temporary fix, but this release will prevent it happening again in the future.
Technical improvements
Here’s a bunch of lower-level changes for advanced users that are included in
this release:
You can now configure what address Snikket’s built-in HTTP server will
listen for connections on (useful for certain advanced setups behind a
reverse proxy)
Add docker health checks, allowing docker to inform you about the health of
the Snikket services
Switch to a more robust DNS resolver (used for federation when connecting to
other servers)
Allow configuration of an external TURN server (replacing the built-in one)
Fix support for BOSH and websockets (allowing third-party web clients)
If you have any questions or feedback about this release, let us know!
In the second half of this blog series, Erik Schön looks next at the ‘right tool for the jobs’ that are required for FinTech 3.0. The Erlang/Elixir/OTP open-source ecosystem for software development has its roots in telecom – an industry that has already resolved many of the new challenges confronting FinTech. Revisit Part One.
The Right Tool for the Job
One of the most important tools in the telecoms toolbox for real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost, is the open-source programming language and associated frameworks, tools and libraries called Erlang/OTP (Open Telecom Platform) originally developed by the telecom giant Ericsson.
The Need: Quick Development of the Right Thing
Mike Williams, co-creator of Erlang/OTP, wrote down his credo in 1985 which was then used to guide the development of Erlang/OTP during the 80s and 90s (Däcker, 2009):
“Find the right methods – design by prototyping.”
“Make mistakes on a small scale – not in a production project.”
“It’s not good enough to have ideas – you must also be able to implement them to know that they work.”
These insights contributed to making Erlang/OTP suitable for iterative, incremental development with quick feedback from real customers. This also ensured a smooth developer experience making it easier to build the right thing that:
customers find valuable, usable and sometimes even desirable;
is feasible from a technical perspective;
is viable from a commercial perspective;
is reasonable from a societal perspective, e.g. sustainable and ethical.
The Need: Real-Time, Secure, Reliable and Scalable Solutions
Bjarne Däcker, head of the Ericsson computer science lab and Mike Williams’s manager at the time, formulated the requirements on the new programming language system as follows (Däcker, 2000):
Real-time: “Actions to be performed at a certain point in time or within a certain time.”
Security: “Stringent quality”
Reliability: “Very large software systems … Interaction with hardware … Complex functionality such as feature interaction … Continuous operation for many years … Software maintenance (reconfiguration, etc.) without stopping the system … Fault tolerance both to hardware failures and software errors”
Scalability: “Systems distributed over several computers … handling of a very large number of concurrent activities.”
Joe Armstrong, co-creator of Erlang/OTP summarised it as “making reliable distributed systems in the presence of software errors”. (Armstrong, 2003).
Business Outcomes: Faster, Safer, Better, and, More for Less
Over the past 20+ years Erlang/OTP has provided the following business outcomes (Cesarini, 2019):
2x FASTER development of new services thanks to the language’s design, the OTP middleware, the set of frameworks, principles, and design patterns that guide and support the structure, design, implementation, and deployment where e.g. Motorola saw 4-20 times less code compared to traditional languages.
10x BETTER services that are down less than 5 minutes per year thanks to built-in fault tolerance and resilience mechanisms built into the language and the OTP middleware, e.g. software upgrades and generic error handling, as seen by e.g. Ericsson in their mobile network packet routers.
10x SAFER services that are hard to hack and crash through denial of service attacks thanks to the language construction with lack of pointers, use of messages instead of shared memory and immutable state rather than variable assignments, as seen by the number of vulnerabilities compared to other languages (CVE, 2021).
10x MORE users (billions), transactions per second (millions) within milliseconds thanks to a battle-tested virtual machine as seen by e.g. WhatsApp with more than 2 billion users.
10x LESS costs and energy consumption thanks to fewer servers needed where e.g. Bleacher Report were able to reduce their hardware requirements from 150 servers to 5.
Robert Virding, co-creator of Erlang/OTP formulated the unique value proposition like this: “Any sufficiently complicated concurrent program in another language [for this job to be done] contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.” (Virding, 2008).
What about Elixir?
Elixir has all the benefits of Erlang with a lower threshold for developers used to traditional programming languages like Ruby since Elixir’s syntax looks much more familiar to them. Additionally, Elixir gives a very smooth developer experience including state-of-the-art libraries e.g. for graphical user interfaces with Phoenix LiveView.
Both Erlang and Elixir are easy to learn within a couple of weeks for people with practical experiences and skills equivalent to a computer science degree as well as a curiosity to learn. And, experience shows that they get up to speed within a couple of months which is what it normally takes to understand a new product codebase or business domain (Däcker, 2000).
Engineers and developers love the experience of using Elixir and Erlang, and as of today, there are over 50,000 members in over 140 Meetup groups in all continents of the world except Antarctica (Schön, 2021).
The Road Ahead
The Erlang/Elixir open-source ecosystem is thriving like never before.
During 2020-2021 we have seen companies like WhatsApp and Klarna working together on improving the developer experience further and companies like Ericsson evolving the OTP middleware where the next OTP release in May, 2021 is expected to improve the out-of-the-box performance of Erlang and Elixir applications by 30-130% (Larsson, 2020) and reduce the energy consumption by 25% (Cathcart, 2021).
And, we haven’t even mentioned the recent and exciting announcement of Elixir optimised for quick development of safe machine learning solutions for new innovative, automated services (Valim, 2021).
Conclusion
We hope that you now see what FinTech. can learn from telecom and how FinTech. companies can use the Erlang/Elixir/OTP open-source ecosystem to go 2x FASTER, 10x SAFER, 10x BETTER with 10x MORE for 10x LESS – resulting in happy and loyal customers as well as engaged developers.
Kudos to Michael Jaiyeola, for the original idea, helpful pointers, examples and feedback; Noman Azim, for valuable input and concrete examples; Steve Roberts for helpful feedback, insights and examples from a telecom perspective; Phil Harrison for insights and feedback from a FinTech. perspective; Francesco Cesarini for co-creating a generous and welcoming community, for spreading the word and feedback; Joe Armstrong, Robert Virding, Mike Williams and Bjarne Däcker for perseverance, professionalism and respect in co-creating and managing Erlang/OTP.
Writer
Erik Schön, Managing Director, Erlang Solutions Nordic, is an executive with 20+ years of telecoms experience from global standardisation, system design and managing R&D organisations of up to 400 people at the global telecom giant Ericsson. Erik is a big fan of Erlang since the 90s and his latest book is The Art of Strategy.
Vaccines are readily available in the US now, you can walk in to a pharmacy anywhere and get it. There is no need for the Vaxbot service (http://vaccine.monal.im) here anymore. After 11 million alerts sent out, we have shutdown US operations last week on May 8th and start focusing on Canada (where we just expanded) and likely add the EU (or anywhere else in the world that needs help) next.
Thanks to everyone on the Monal and the server administrators and developers from Yaxim and Prosody teams for making this project a success! We’ve made a difference in the lives of thousands of people and there is still much more to do.
If you are still interested in the open and free technology behind Vaxbot, please take a look at Monal IM which is the main project which enabled the simple setup of Vaxbot.
RabbitMQ, the message broker, is not the first thing that comes to mind when we talk about integrating legacy services. We use high level frameworks to hide the complexity of the underlying plumbing. In this post we will show how the RabbitMQ, the most widely deployed open source message broker, can help with reliable and resilient messaging.
What is RabbitMQ?
RabbitMQ is a free, open-source and extensible message queuing solution. It is a message broker that understands AMQP (Advanced Message Queuing Protocol), but can also be used with other popular messaging solutions like MQTT. It is highly available, fault tolerant and scalable. It is implemented in Erlang OTP, a technology tailored for building stable, reliable, fault tolerant and highly scalable systems which possess native capabilities of handling very large numbers of concurrent operations. The benefits of Erlang OTP can easily be seen in RabbitMQ and other systems like WhatsApp, MongooseIM, to mention a few.
At a very high level, it is a middleware layer that enables different services in your application to communicate with each other without worrying about message loss while providing different quality of service (QoS) requirements. It also enables fine-grained and efficient message routing enabling extensive decoupling of applications.
RabbitMQ Connecting Legacy Services
When designing system interactions we usually start with a top-to-bottom approach. We identify the components and use the technology available to connect them. If we need to integrate existing systems with newer ones, then it is often infeasible to change the older system to use modern protocols. In cases like this, connector applications can be developed acting as a bridge between the old system and the new.
Once we know the components and we have all the translation layers, we can deal with the interconnection. There are two major ways how parts of the system can communicate:
Directly (synchronously). When both components are online, they can use an active network connection and a preferred protocol like HTTP, protobuf or even FTP. Using these technologies results in a simple solution with its trade-offs, such as unavailability of multiple services, or having to implement client side buffering, when things go wrong.
Indirectly (asynchronously). To mitigate the trade-offs from the direct communication we can introduce a messaging layer into the solution that can retain the messages even if the receiving application is not online. Although this adds complexity to the architecture it removes a lot of edge cases from the applications. Scaling the messaging layer is often easier than introducing scale to the legacy applications.
RabbitMQ is a well-suited message broker because it scales well not only with the message load requirements, but also with the business requirements. Finding message brokers that can handle hundreds or thousands or even tens of thousands of messages is easy. But finding a message bus that can cater for several messaging patterns, can handle different protocols (AMQP, MQTT, STOMP, JMS and many more) and can be modified via plugins is much more difficult.
Features come with trade-offs, but when RabbitMQ is put into a heterogeneous environment, it provides an easy way to move the messages between virtually any systems. All major frameworks support an AMQP adapter, but even if the component uses a proprietary protocol then it is possible to develop a plugin and extend RabbitMQ’s functionality.
RabbitMQ provides a high degree of availability and consistent messaging if the following three features are used. First we see why it is recommended to run RabbitMQ as a cluster, then how to guarantee that messages are received and kept safe by RabbitMQ and how quorum queues improve on classic mirroring.
Clustering
Although a single node RabbitMQ provides the highest consistency guarantees, it is not resilient to hardware errors or planned downtimes due to upgrades. If anything happens to the single RabbitMQ instance it means that the service is no longer available system-wide. If any of the message store files gets corrupted on the file system, then messages can be lost.
To avoid these problems RabbitMQ can run as a cluster, each node providing the full RabbitMQ functionality. It does not matter for the clients which RabbitMQ node it connects, RabbitMQ hides the complexities of distributing and replicating messages, queues and exchanges.
The big question is how many nodes we should use in a cluster. Using two nodes might be tempting, but it only works with systems running in an active-passive nature. RabbitMQ, as we discussed above, runs in an active-active fashion where each node in a cluster provides identical functionality. This means that in case of any problems, RabbitMQ needs a consensus of the majority of the nodes to function. By restricting the number of nodes in a cluster to an odd number, we can guarantee that the cluster will consistently survive the loss of the minority of the nodes (in case of 5 nodes, RabbitMQ can tolerate the loss of 2 nodes.)
To enforce that the minority nodes don’t commit changes, pausing them when detecting a network issue can be automated by setting the cluster partition handling mechanism to pause_minority. This is a built-in feature and it not only detects that a network partition occurred but also detects when the network is working again and can restart the nodes paused automatically. RabbitMQ does not require manual intervention to heal.
Publisher confirm and acknowledgements
RabbitMQ can automagically cluster and heal itself, but due to the distributed nature of things, it requires some cooperation from the clients to guarantee no message loss. When publishing a message to RabbitMQ the clients can only “forget” about the message when it was acknowledged by RabbitMQ. This acknowledgement on the publishing side is called the Publisher Confirm feature. It is more lightweight than the transaction mechanism in AMQP and performs much better under load. This RabbitMQ specific feature is supported by all major client libraries.
On the consuming side, the clients should acknowledge the message only when they finished processing the message. This guarantees that once a message is fully processed it is removed from RabbitMQ, but if any problem happens during message processing, the same message can be redelivered to the same or a different client. This gives a good opportunity to introduce horizontal scaling for parts of the system.
Quorum Queues
The last bit of the puzzle is to make sure that the messages that are in RabbitMQ can survive network errors, hardware failures or planned maintenance. With the introduction of clustering it is possible to distribute the workload among many nodes, but it is also important for introducing resiliency. Quorum queues are the new addition to the RabbitMQ feature family providing an improved solution to queue mirroring. It is based on the state of the art Raft algorithm to manage the integrity of the queue leader and the contents of the queues. Using it together with the publisher confirms, it is guaranteed that messages acknowledged by RabbitMQ will be safe until they are delivered to consumers.
Quorum queues improve on one of the major issues of classic queue mirroring: queue synchronisation after a network partition. With the Raft protocol, it is possible to only copy the messages that are missing from the mirror. This reduces the synchronisation network pressure which was a cause for some hard-to-debug RabbitMQ issues in the past.
Summary
In this blog post, we discussed the major challenges of integrating legacy and non-legacy systems. Introducing a versatile messaging layer like RabbitMQ can provide solutions to lots of problems and applying the three major features highlighted in this blogpost will enable a successful project.
Here at Erlang Solutions, we have world leading experts in RabbitMQ and can help with finding solutions to the unique problems of distributed systems. These challenges range from auditing the planned or deployed RabbitMQ solution or designing bespoke solutions with RabbitMQ. Get in touch to find out more about how we can help with your integration project or get your ticket to RabbitMQ Summit 2021 to learn more about why RabbitMQ is trusted by many of the world’s biggest companies.
We are pleased to announce a new minor release from our stable branch.
This release addresses a number of important security issues that affect most
deployments of Prosody. Full details are available in a separate security
advisory. We recommend that
all deployments upgrade or apply the mitigations described in the advisory.
Note: We have updated the default config file. Your package manager may
warn you about this, and ask if you want to use the new file or keep your
existing one. You should usually keep your existing one, but make sure you
update it to enable mod_limits after the upgrade.
A summary of changes in this release:
Security
mod_limits, prosody.cfg.lua: Enable rate limits by default
certmanager: Disable renegotiation by default
mod_proxy65: Restrict access to local c2s connections by default
util.startup: Set more aggressive defaults for GC
mod_c2s, mod_s2s, mod_component, mod_bosh, mod_websockets: Set default stanza size limits
mod_authinternal{plain,hashed}: Use constant-time string comparison for secrets
again, lot of things have happened since last progress note, so I'll only talk about major changes here.
"Salut à Toi" is now "Libervia"
The project has been renamed to "Libervia". Even if I personally loved the former name (which was a reference to a French punk band song, an which could be translated to "hi to you", a nice fit for a communication tool), it proved to be hard to pronounce and remember for non French speakers, and the many names of frontends and project components were confusing. The name change has been discussed for long in the association, but the new ActivityPub/Pubsub end-to-end encryption project accelerated things: after a talk with NLnet, we decided to move forward on this so project name would not change in the middle of its development.
After doing a quick poll, we confirmed that "Libervia" (which was formerly the name of the web frontend only) would be the new name.
All parts are now named in straightforward way: "Libervia Backend", "Libervia Web", "Libervia Desktop/Mobile" (currently the same Kivy frontend for both), "Libervia TUI" and "Libervia CLI", with matching executable names (libervia-backend, libervia-web, libervia-desktop, libervia-tui, libervia-cli also aliased as li). The former names are kept internally and as aliases.
The non-profit (French "loi 1901") association behind it stays with the name "Salut à Toi".
This renaming has involved a lot for work, it took weeks to update code, web sites, doc, etc. and according to our statuses, we had to make a general assembly to discuss this decision. It's still not fully finished (notably the official web site URL is still https://salut-a-toi.org, while https://www.libervia.org is currently used for the demo server), and source code repositories have not been modified for the moment, but most of the renaming is done, and you can now reference the whole project as "Libervia"
Official Website Update
Following the changes in Libervia Web themes, the official website one has also been updated and is now based on Libervia's Web Bulma theme. The news now links to my personal blog as it is where you'll have most up-to-date informations about Libervia development (and the former page was broken). Tickets/Bug tracker is now directly accessible from the official site, as it makes more sense to have it there. It's still accessible from goffi.org, and thanks to its decentralised nature, it's usable transparently on both locations.
I've also temporarily disabled account registration on the bug-tracker due a wave of spammy accounts. I will have to put in place a protection for that, but I'm reluctant to use popular non-libre options.
Flatpak and Docker
While working on the renaming and website, I've updated the Flatpaks (they were really outdated), and Docker images. Flatpaks is for now using a specific dev repos, but I hope to see Libervia on Flathub after the release.
I've created Docker images and Docker Compose file to run quickly a local demo of Libervia Web, you can see the instruction on the Official Website.
Ideally, I would like to also create Snaps, Appimages, Nix packages, etc. But I'm lacking time (Flatpak and Docker are already too much time consuming) and prefer to focus on the code rather than on the packaging, help is more than welcome though.
User Friendly URLs
As you may have noticed on the last blog posts, URLs are now more user friendly:
A blog post is referenced using its item ID, and previously a unique ID was used for that, which is relatively long and doesn't give any information about the content, but is necessary to avoid conflict (writing a blog post with an existing ID will overwrite the previous one).
The new behaviour directly use URL friendly item IDs, and to avoid conflict, a short random suffix is appended (on the example above, QGqK is the suffix). After some tests, the collision risk for a short suffix like that is not that high (I've tested millions of IDs without collision), and it may anyway happen only if 2 blog posts have the exact same title, so the risk is very low. The resulting URL is more pleasant.
This URL friendly ID is used by default when a blog post is created, but it can be deactivated if user_friendly_id is set to false in blog post metadata, or by specifying manually an item id.
To accompany this change, a new Libervia CLI rename subcommand has been added to li blog and li pubsub, which will change the ID of an item. As there is no standard rename operation in XMPP Pubsub, this is done by copying the item to the new ID, then delete the former one in case of success.
Navigation Helpers in Libervia Web
It was not really easy so far to know where we were in Libervia Web. To help with this, the selected menu is now shown activated, and a breadcrumb has been added.
The breadcrumb is only shown when there are at least 2 elements to show (i.e. not on root pages). It is generated automatically by default, but can be customised with specific label, sub-elements, or even icons, like in the file sharing screenshot below:
Blog Editor
As it was not possible anymore to write a new blog item with Libervia Web, I've made a blog item editor, which is relatively basic for now, but working. If JavaScript is activated, you'll get a tags editor, preview, and autosaving:
File Sharing Quotas
One last missing piece I was needing before release was to put in place quotas on the file sharing component, this is now done.
Indeed, this component doesn't work with a per-file limit like most others do, but with a per-user quota, and you can upload any file size you want at long as you're not over quota.
Release to come
It's more than time to think about the release. I wanted to improve the chat notably in Libervia Web where it's still really basic since we moved out from the former frontend, but finally I've decided to report this to next release, as I plan to refactor messages handling, and for now I need to concentrate on the ActivityPub gateway.
So I'll soon prepare a beta version, and plan to do the release in a couple of weeks. I'll do bugfix on the 0.8 version during this time, but will avoid any important new development.
ActivityPub gateway project
With all the work done above (and other things, I've not mentioned everything), I've been late to start working on the ActivityPub project, but now I can focus on it. The first task is about developing a Pubsub cache as Libervia is currently getting its data for Pubsub related feature directly from the services.
Beside the obvious speed improvement, having a local cache will give the possibility to do data search/manipulation (such as doing Full-Text Search when the Pubsub service doesn't implement it, or doing feature-specific data analysis), handle message received unordered, allow to keep decrypted data when received from e2ee items, etc.
So far, SQLite was used for data storage in Libervia, by using Twisted's adbapi and custom semi-automatic schema update/data migration. It has been working relatively well so far, but it's no pleasant to maintain.
Fortunately, SQLAlchemy has recently added support for AsyncIO, thus it can now be used in Libervia. This is great, as SQLAlchemy is popular and rock solid, so I've decided to go with it. This will open the possibility to use other backends (like PostgreSQL), and refactor Libervia to use SQLAlchemy's ORM. Logically, Alembic will be used for data migration, which should make database modifications easier.
Such a cache will make possible to implements things like items discovery based on categories (or search by "hashtags" as it named in other software).
This two-part blog series looks at the jobs to be done in fintech and argues that the right tool for the job, the Erlang/Elixir/OTP open-source ecosystem for software development, has its roots in telecom – an industry that has already resolved many of the new challenges confronting fintech currently.
The world of financial services using technology to make individuals’ lives easier and more prosperous is becoming increasingly similar to the world of telecommunications.
The simple reason is that since the introduction of the iPhone (a smart telecommunications device also known as a “smartphone”) in 2007, most of our interactions with banks and other financial institutions are through smartphones.
This means that we expect the banking experience to be as good as all our other smartphone experiences, be it entertainment or work related, including around the clock availability and responses within fractions of a second. This is backed up by research showing that response times above half a second makes people wonder what is going on and get stressed.
Since it is about money and other financial instruments, we also expect all interactions and personal data to be highly secure from e.g. intrusions and theft. The essence of all financial services is trust. You trust banks to not lose your money. If you lose the trust in your bank, you will switch to another bank or at least move parts of your business elsewhere.
In more technical terms, we want our financial interactions to be done in real-time and be extremely secure and reliable.
The Job to Be Done
From a customer perspective, we have just seen how our financial services needs are for real-time, secure and reliable products and solutions. Also, we expect new innovative services that simplify our daily lives, e.g. saving us time and money while growing our financial wealth. One example is instant payments using your smartphone as a digital wallet connected to your bank account where work is ongoing to enable instant person-to-person payments also across national borders.
The traditional banks have a track record of security and reliability leading to trust, loyalty and long-standing personal relationships with their customers with e.g. personal banking experts available for financial advice. In the past that trust was established with large imposing buildings signifying they were here to stay. Over time banks have eliminated buildings – particularly in the branch network in order to reduce costs and as a consequence “trust” has become more intangible.
As a traditional bank, you would like to extend the experience to the digital world and at the same time reduce your large cost base which is due to very old hardware and software. One example is a European bank that developed a new, well-designed and innovative smartphone application that unfortunately did not work since it took up to 30 seconds to fetch customer data from the legacy software system. They realised that their legacy software system was not fit for the future and have since then started to replace the system with future-proof technology, one “elephant carpaccio slice” at a time.
If you are a startup providing financial services, your needs are your customers’ needs, so real-time, security and reliability in order to secure their trust and loyalty are still valid. Since you are competing with the incumbent giants, you need a unique, focused offering that can be developed quickly and operated at low cost. Additionally, you want to provide a smooth onboarding experience to acquire new customers quickly. Backbase and Tide offer one minute customer onboarding time, a process that usually requires at least 48 hours to fully vet and comply with regulations.
If you are one of the big US technology giants, you may want to consider your particular superpower, e.g. social networking, superb user experience or superior customer understanding, to expand your offering to embrace selected financial services as an integral part of your brand. One example is Apple Pay which extends the real time facility without having a current account or complying with onboarding requirements. Another example is Uber Money with digital wallets and credit and debit cards.
If you are one of the clearing houses, banking centrals, credit card companies or central banks, the huge number of financial transactions due to the real-time interactions lead to massive requirements on scalability and resilience of your technical infrastructure since the systems you operate are critical, not only for your specific customers but for society at large. We are seeing experiments with digital currencies run by several central banks in the world, including Sweden’s “Riksbank”, founded in 1668.
If you are one of the providers of digital currencies based on blockchain technologies you also face the challenge of minimising the energy consumption for all the computations needed.
Considering the engineers and developers who design, deliver and operate the financial systems, they want an environment where you can develop and deploy easily – including smooth programming languages requiring few lines of code supported by libraries, frameworks and tooling – and get quick feedback from both development, test and commercial deployments. Motivated and engaged employees lead to better services and happier and more loyal customers.
To summarise: we need real-time, secure, reliable and scalable systems that can be developed quickly and smoothly, operating at low cost with interoperability across borders and currencies.
Fortunately, the needs of real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost have already been fulfilled in the telecoms world starting in the 60s and 70s when the telecom operators started deploying the first digital software switches for local, regional and international phone calls. This infrastructure critical for society was designed, developed and tested for global, real-time around the clock operation with billions of users from the very beginning and global standards for interoperability were key enablers.
The technology evolution grew very rapidly in the 80s and 90s due to the increased demands for communication on the move using handsets connected to mobile networks, first using national network standards (the first generation, 1G, for voice calls only), then regional network standards (the second generation, 2G, for voice, text messages and limited data transmission), and, finally global network standards (the 3rd, 4th and 5th generations, 3G/4G/5G, optimised for higher and higher rates of data transmission).
This evolution happened in parallel with the evolution of systems for secure and reliable financial transactions. The biggest difference was that the telecom systems had far more transactions to handle due to the real-time nature of phone calls, mobile phone calls and browsing the web whereas the financial world handled this particular challenge through daily batches around midnight where all the financial transactions during the past 24 hours were processed.
Finally, telecoms have been commoditised by regulation, standards and global competition and is now a low margin business. There is a similar threat to financial services. In telecom, threats such as Google and others were well known to telecoms yet the telecom operators failed to act and now the value has been taken by the Big US Tech players with a global mindset.
To summarise, telecoms have already mastered the challenge of real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost. At Erlang Solutions we believe that a lot of our learnings from this industry can be applied to exciting new projects in the fintech space.
Writer
Erik Schön, Managing Director, Erlang Solutions Nordic, is an executive with 20+ years of telecoms experience from global standardisation, system design and managing R&D organisations of up to 400 people at the global telecom giant Ericsson. Erik is a big fan of Erlang since the 90s and his latest book is The Art of Strategy.
This blog post looks at the jobs to be done in fintech and argues that the right tool for the job, the Erlang/Elixir/OTP open-source ecosystem for software development, has its roots in telecom – an industry that has already resolved many of the new challenges confronting fintech currently.
The Brave New World of Fintech
The world of financial services using technology to make individuals’ lives easier and more prosperous is becoming increasingly similar to the world of telecommunications.
The simple reason is that since the introduction of the iPhone (a smart telecommunications device also known as a “smartphone”) in 2007, most of our interactions with banks and other financial institutions are through smartphones.
This means that we expect the banking experience to be as good as all our other smartphone experiences, be it entertainment or work related, including around the clock availability and responses within fractions of a second. This is backed up by research showing that response times above half a second makes people wonder what is going on and get stressed.
Since it is about money and other financial instruments, we also expect all interactions and personal data to be highly secure from e.g. intrusions and theft. The essence of all financial services is trust. You trust banks to not lose your money. If you lose the trust in your bank, you will switch to another bank or at least move parts of your business elsewhere.
In more technical terms, we want our financial interactions to be done in real-time and be extremely secure and reliable.
The Job to Be Done
From a customer perspective, we have just seen how our financial services needs are for real-time, secure and reliable products and solutions. Also, we expect new innovative services that simplify our daily lives, e.g. saving us time and money while growing our financial wealth. One example is instant payments using your smartphone as a digital wallet connected to your bank account where work is ongoing to enable instant person-to-person payments also across national borders.
The traditional banks have a track record of security and reliability leading to trust, loyalty and long-standing personal relationships with their customers with e.g. personal banking experts available for financial advice. In the past that trust was established with large imposing buildings signifying they were here to stay. Over time banks have eliminated buildings – particularly in the branch network in order to reduce costs and as a consequence “trust” has become more intangible.
As a traditional bank, you would like to extend the experience to the digital world and at the same time reduce your large cost base which is due to very old hardware and software. One example is a European bank that developed a new, well-designed and innovative smartphone application that unfortunately did not work since it took up to 30 seconds to fetch customer data from the legacy software system. They realised that their legacy software system was not fit for the future and have since then started to replace the system with future-proof technology, one “elephant carpaccio slice” at a time.
If you are a startup providing financial services, your needs are your customers’ needs, so real-time, security and reliability in order to secure their trust and loyalty are still valid. Since you are competing with the incumbent giants, you need a unique, focused offering that can be developed quickly and operated at low cost. Additionally, you want to provide a smooth onboarding experience to acquire new customers quickly. Backbase and Tide offer one minute customer onboarding time, a process that usually requires at least 48 hours to fully vet and comply with regulations.
If you are one of the big US technology giants, you may want to consider your particular superpower, e.g. social networking, superb user experience or superior customer understanding, to expand your offering to embrace selected financial services as an integral part of your brand. One example is Apple Pay which extends the real time facility without having a current account or complying with onboarding requirements. Another example is Uber Money with digital wallets and credit and debit cards.
If you are one of the clearing houses, banking centrals, credit card companies or central banks, the huge number of financial transactions due to the real-time interactions lead to massive requirements on scalability and resilience of your technical infrastructure since the systems you operate are critical, not only for your specific customers but for society at large. We are seeing experiments with digital currencies run by several central banks in the world, including Sweden’s “Riksbank”, founded in 1668.
If you are one of the providers of digital currencies based on blockchain technologies you also face the challenge of minimising the energy consumption for all the computations needed.
Considering the engineers and developers who design, deliver and operate the financial systems, they want an environment where you can develop and deploy easily – including smooth programming languages requiring few lines of code supported by libraries, frameworks and tooling – and get quick feedback from both development, test and commercial deployments. Motivated and engaged employees lead to better services and happier and more loyal customers.
To summarise: we need real-time, secure, reliable and scalable systems that can be developed quickly and smoothly, operating at low cost with interoperability across borders and currencies.
The Telecoms World
Fortunately, the needs of real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost have already been fulfilled in the telecoms world starting in the 60s and 70s when the telecom operators started deploying the first digital software switches for local, regional and international phone calls. This infrastructure critical for society was designed, developed and tested for global, real-time around the clock operation with billions of users from the very beginning and global standards for interoperability were key enablers.
The technology evolution grew very rapidly in the 80s and 90s due to the increased demands for communication on the move using handsets connected to mobile networks, first using national network standards (the first generation, 1G, for voice calls only), then regional network standards (the second generation, 2G, for voice, text messages and limited data transmission), and, finally global network standards (the 3rd, 4th and 5th generations, 3G/4G/5G, optimised for higher and higher rates of data transmission).
This evolution happened in parallel with the evolution of systems for secure and reliable financial transactions. The biggest difference was that the telecom systems had far more transactions to handle due to the real-time nature of phone calls, mobile phone calls and browsing the web whereas the financial world handled this particular challenge through daily batches around midnight where all the financial transactions during the past 24 hours were processed.
Finally, telecoms have been commoditised by regulation, standards and global competition and is now a low margin business. There is a similar threat to financial services. In telecom, threats such as 9Google and others were well known to telecoms yet the telecom operators failed to act and now the value has been taken by the Big US Tech players with a global mindset.
To summarise, telecoms have already mastered the challenge of real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost. At Erlang Solutions we believe that a lot of our learnings from this industry can be applied to exciting new projects in the fintech space.
The Right Tool for the Job
One of the most important tools in the telecoms toolbox for real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost, is the open-source programming language and associated frameworks, tools and libraries called Erlang/OTP (Open Telecom Platform) originally developed by the telecom giant Ericsson.
The Need: Quick Development of the Right Thing
Mike Williams, co-creator of Erlang/OTP, wrote down his credo in 1985 which was then used to guide the development of Erlang/OTP during the 80s and 90s (Däcker, 2009):
“Find the right methods – design by prototyping.”
“Make mistakes on a small scale – not in a production project.”
“It’s not good enough to have ideas – you must also be able to implement them to know that they work.”
These insights contributed to making Erlang/OTP suitable for iterative, incremental development with quick feedback from real customers. This also ensured a smooth developer experience making it easier to build the right thing that:
customers find valuable, usable and sometimes even desirable;
is feasible from a technical perspective;
is viable from a commercial perspective;
is reasonable from a societal perspective, e.g. sustainable and ethical.
The Need: Real-Time, Secure, Reliable and Scalable Solutions
Bjarne Däcker, head of the Ericsson computer science lab and Mike Williams’s manager at the time, formulated the requirements on the new programming language system as follows (Däcker, 2000):
Real-time: “Actions to be performed at a certain point in time or within a certain time.”
Security: “Stringent quality”
Reliability: “Very large software systems … Interaction with hardware … Complex functionality such as feature interaction … Continuous operation for many years … Software maintenance (reconfiguration, etc.) without stopping the system … Fault tolerance both to hardware failures and software errors”
Scalability: “Systems distributed over several computers … handling of a very large number of concurrent activities.”
Joe Armstrong, co-creator of Erlang/OTP summarised it as “making reliable distributed systems in the presence of software errors”. (Armstrong, 2003).
Business Outcomes: Faster, Safer, Better, and, More for Less
Over the past 20+ years Erlang/OTP has provided the following business outcomes (Cesarini, 2019):
2x FASTER development of new services thanks to the language’s design, the OTP middleware, the set of frameworks, principles, and design patterns that guide and support the structure, design, implementation, and deployment where e.g. Motorola saw 4-20 times less code compared to traditional languages.
10x BETTER services that are down less than 5 minutes per year thanks to built-in fault tolerance and resilience mechanisms built into the language and the OTP middleware, e.g. software upgrades and generic error handling, as seen by e.g. Ericsson in their mobile network packet routers.
10x SAFER services that are hard to hack and crash through denial of service attacks thanks to the language construction with lack of pointers, use of messages instead of shared memory and immutable state rather than variable assignments, as seen by the number of vulnerabilities compared to other languages (CVE, 2021).
10x MORE users (billions), transactions per second (millions) within milliseconds thanks to a battle-tested virtual machine as seen by e.g. WhatsApp with more than 2 billion users.
10x LESS costs and energy consumption thanks to fewer servers needed where e.g. Bleacher Report were able to reduce their hardware requirements from 150 servers to 5.
Robert Virding, co-creator of Erlang/OTP formulated the unique value proposition like this: “Any sufficiently complicated concurrent program in another language [for this job to be done] contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.” (Virding, 2008).
What about Elixir?
Elixir has all the benefits of Erlang with a lower threshold for developers used to traditional programming languages like Ruby since Elixir’s syntax looks much more familiar to them. Additionally, Elixir gives a very smooth developer experience including state-of-the-art libraries e.g. for graphical user interfaces with Phoenix LiveView.
The Developer Experience
Both Erlang and Elixir are easy to learn within a couple of weeks for people with practical experiences and skills equivalent to a computer science degree as well as a curiosity to learn. And, experience shows that they get up to speed within a couple of months which is what it normally takes to understand a new product codebase or business domain (Däcker, 2000).
Engineers and developers love the experience of using Elixir and Erlang, and as of today, there are over 50,000 members in over 140 Meetup groups in all continents of the world except Antarctica (Schön, 2021).
The Road Ahead
The Erlang/Elixir open-source ecosystem is thriving like never before.
During 2020-2021 we have seen companies like WhatsApp and Klarna working together on improving the developer experience further and companies like Ericsson evolving the OTP middleware where the next OTP release in May, 2021 is expected to improve the out-of-the-box performance of Erlang and Elixir applications by 30-130% (Larsson, 2020) and reduce the energy consumption by 25% (Cathcart, 2021).
And, we haven’t even mentioned the recent and exciting announcement of Elixir optimised for quick development of safe machine learning solutions for new innovative, automated services (Valim, 2021).
Conclusion
We hope that you now see what fintech can learn from telecom and how fintech companies can use the Erlang/Elixir/OTP open-source ecosystem to go 2x FASTER, 10x SAFER, 10x BETTER with 10x MORE for 10x LESS – resulting in happy and loyal customers as well as engaged developers.
What are your jobs to be done? What tools are you using? How can we help?
Acknowledgements
Kudos to Michael Jaiyeola, for the original idea, helpful pointers, examples and feedback; Noman Azim, for valuable input and concrete examples; Steve Roberts for helpful feedback, insights and examples from a telecom perspective; Phil Harrison for insights and feedback from a fintech perspective; Francesco Cesarini for co-creating a generous and welcoming community, for spreading the word and feedback; Joe Armstrong, Robert Virding, Mike Williams and Bjarne Däcker for perseverance, professionalism and respect in co-creating and managing Erlang/OTP.
Writer
Erik Schön, Managing Director, Erlang Solutions Nordic, is an executive with 20+ years of telecoms experience from global standardisation, system design and managing R&D organisations of up to 400 people at the global telecom giant Ericsson. Erik is a big fan of Erlang since the 90s and his latest book is The Art of Strategy.
And now for something completely different: a rollicking discussion on the Big Yellow Praxis podcast about some underappreciated musical recordings, in which Jacob and I explore the age-old question: what exactly makes a concept album? Check it out on YouTube!...
Welcome to the XMPP Newsletter covering the month of April 2021.
Many projects and their efforts in the XMPP community are a result of people’s voluntary work. If you are happy with the services and software you may be using, especially throughout the current situation, please consider to say thanks or help these projects!
The Italian translation of the last newsletter appeared at NicFab.it
XSF Announcement
The XMPP Standards Foundation is now on YouTube! You can find recordings of conferences, the XMPP Office Hours, and other fun stuff over on the channel.
Have a talk, demo, or roundtable discussion about XMPP or related technologies? XMPP Office Hours are weekly meetings of XMPP enthusiasts for discussion and presentations. For more information, or to sign up, have a look at the wiki page.
Berlin XMPP Meetup (remote): Monthly Meeting of XMPP Enthusiasts in Berlin - always 2nd Wednesday of every month.
Articles
From May 2021, there have been major content updates for freie-messenger.de as well as additions. From now on, the content will be internationalised. As a first step, there is the new Quick Overview of Messengers in both German and English. Call for supporters!
Gajim Development News: April brought an all new message styling parser, making Gajim fully compliant with XEP-0393. This post will also give you a sneak peek on some features we’ve been developing for the past months: the new Chat view and Contact Info window.
Libervia (formerly Salut à Toi) has been selected for a grant by NLNet/NGI0 Discovery Fund (with financial support from European Commission's Next Generation Internet programme) for an gateway between XMPP and ActivityPub doubled with Pubsub end-to-end encryption project. The gateway will join two major open and decentralised protocols. In practice it will be a XMPP server component (usable with any server), and implement the ActivityPub server to server protocol (or "Federation Protocol"). On XMPP side, it will be mostly a Pubsub service (with some extra, like private messages converted to XMPP message stanza).
Vaxbot continues to grow: VaxBot is a free service that helps you find vaccine appointments in your state or country if available. During the last weeks, Vaxbot continued to grow. The service just hit 8 million messages sent and it sends around 800,000 messages a day. XMPP and Vaxbot have been on TV news in Las Vegas and South Carolina, the radio in Washington as well newspapers all over the country.
Refering the Vaxbot news: Georg Lukas, who runs the server behind it, wrote an article regarding the increased subscriptions and how the services have to adapt the infrastructure to keep up with this new "VaxBot performance challenge".
Snikket announces that XMPP Account Portability was funded under the NGI DAPSI initiative. It will cover the needed standards (beyond older XEP-0227 and XEP-0238), open-source tools and integrating said functionality into Snikket.
Since there appeared a lot of sendxmpp alternatives over the last years there is now an work-inprogress overview in the XSF wiki.
UWPX is replacing the SQLite-net DB backend with a new Entity Framework Core DB. This comes with a fix and update of the OMEMO implementation to the latest version of the draft (OMEMO 0.7.0 (2020-09-05)). This version is incompatible with all other previous versions of OMEMO and UWPX allows you now to only exchange OMEMO encrypted messages with contacts that support at least OMEMO 0.7.0 (2020-09-05) draft. Now UWPX can act as a client with a proper one to one OMEMO implementation!
Spark 3.0.0 Beta has been released bringing fixes as usual. Pade Meetings is now integrated (it enables audio and video chat) and all these over a complete UI refresh.
Servers
ejabberd 21.04 has been released: The new ejabberd 21.04 release includes many bugfixes and a few improvements. This release includes minor improvements to fully support
Erlang/OTP 24 and Rebar3. At the same time, it maintains support back to the old Erlang/OTP 19.3 and Rebar2.
Openfire 4.6.3 has been released: This release contains a number of bug fixes and denotes the desire to provide stable 4.6.x series releases while development work continues on a 4.7.0 release.
Libraries
slixmpp 1.7.1 has been released, which is a bugfix release for a rarely used code path tied to OMEMO.
Extensions and specifications
Developers and other standards experts from around the world collaborate on these extensions, developing new specifications for emerging practices, and refining existing ways of doing things. Proposed by anybody, the particularly successful ones end up as Final or Active - depending on their type - while others are carefully archived as Deferred. This life cycle is described in XEP-0001, which contains the formal and canonical definitions for the types, states, and processes. Read more about the standards process. Communication around Standards and Extensions happens in the Standards Mailing List (online archive).
Proposed
The XEP development process starts by writing up an idea and submitting it to the XMPP Editor. Within two weeks, the Council decides whether to accept this proposal as an Experimental XEP.
This specification defines a Unicode-formatted fancy text syntax for use in instant messages.
Initial published version. (XEP Editor (jsc))
Deferred
If an experimental XEP is not updated for more than twelve months, it will be moved off Experimental to Deferred. If there is another update, it will put the XEP back onto Experimental.
No XEPs deferred this month.
Updated
Version 0.4.0 of XEP-0420 (Stanza Content Encryption)
Use 'envelope' and 'content' consistently by renaming elements Update namespace to (melvo)
Versions 0.4.0 and 0.5.0 of XEP-0434 (Trust Messages (TM))
Add new section, use more precise sentences, apply consistent formatting:
Add new section for Trust Message URIs
Use 'that' instead of 'which' for restrictive clauses
Apply consistent formatting for paragraphs and revision history
Update to XEP-0420 version 0.4.0 and adapt namespace to shortname
Replace SCE's old 'content' element by its new 'envelope' element
Replace SCE's old 'payload' element by its new 'content' element
Update SCE's namespace to 'urn:xmpp:sce:1'
Change namespace to 'urn:xmpp:tm:0' (melvo)
Versions 0.2.0 and 0.3.0 of XEP-0450 (Automatic Trust Management (ATM)
Add usage of Trust Message URIs, use more precise sentences, apply consistent formatting
Add usage of Trust Message URIs for initial authentications
Use 'that' instead of 'which' for restrictive clauses
Apply consistent formatting for paragraphs and revision history
Update to XEP-0420 version 0.4.0 and XEP-0434 version 0.5.0
Replace SCE's old 'content' element by its new 'envelope' element
Replace SCE's old 'payload' element by its new 'content' element
Update SCE's namespace to 'urn:xmpp:sce:1'
Update TM's namespace to 'urn:xmpp:tm:0'
Update namespace to 'urn:xmpp:atm:1' (melvo)
Last Call
Last calls are issued once everyone seems satisfied with the current XEP status. After the Council decides whether the XEP seems ready, the XMPP Editor issues a Last Call for comments. The feedback gathered during the Last Call help improving the XEP before returning it to the Council for advancement to Draft.
A Call For Experience - like a Last Call, is an explicit call for comments, but in this case it's mostly directed at people who've implemented, and ideally deployed, the specification. The Council then votes to move it to Final.
No Call for Experience this month.
Thanks all!
This XMPP Newsletter is produced collaboratively by the community.
Thanks to alkino, emus, Licaon_Kter, IM, Martin, mathieui, MattJ, nicola, peetah, Sam, seveso, therealjeybe, wurstsalat and Ysabeau for their help in creating it!
We started drafting in this simple pad in parallel to our efforts in the XSF Github repository. We are always happy to welcome contributors. Do not hesitate to join the discussion in our Comm-Team group chat (MUC) and thereby help us sustain this as a community effort.
You have a project and write about it? Please consider sharing your news or events here, and promote it to a large audience!
And even if you can only spend a few minutes of support, these would already be helpful!
Tasks which need to be done on a regular basis are for example:
Aggregation of news in the XMPP universe
Short formulation of news and events
Summary of the monthly communication on extensions (XEP)
My friend Paul sent me a few thoughts about my recent post on holding fewer opinions. He's formulated an approach that involves holding fewer opinions about other people's opinions. This seems valuable, and related to a post I wrote four years ago entitled "Why Do You Think What You Think?" My introspective conclusion then was that I can't always assign praise or blame for the contents of my thoughts; extending this to other people is a good example of what Arnold Kling calls cognitive empathy: instead of demonizing people who disagree with you, suspend judgment or at least try to understand where they might be coming from. (It's probably best to build up such a practice first in matters that aren't very consequential - and we all have our limits regarding the opinions we find acceptable!) The flip side is not deifying your own thoughts, which we could express in the following aphorism: "if you think well of your own opinions just because they're yours, then you're not thinking well."...
We have some exciting news to share! An important piece of the
Snikket roadmap has been selected for funding by NGI DAPSI, an
EU-funded project focused on data portability and services.
What is DAPSI?
The Data Portability and Services Incubator (DAPSI) is a EU funded
project, under the European Commission’s Next Generation Internet (NGI)
initiative. In their own words, DAPSI was established to:
[…] empower top internet innovators to develop human-centric solutions,
addressing the challenge of personal data portability on the internet, as
foreseen under the GDPR and make it significantly easier for citizens to
have any data which is stored with one service provider transmitted directly
to another provider.
Over the years we have seen many XMPP providers come and go, and
when a provider decides to shut down, it’s too often not easy for people to
obtain their data and move it elsewhere. This contributes to
user churn on the XMPP network - individuals are likely to leave
XMPP rather than figure out the necessary steps to migrate to a new XMPP
service.
There are other reasons for wanting to move your data, such as seeking
providers with better privacy or reliability. You may also want to relocate
from a provider to a self-hosted solution, or vice-versa.
As part of Snikket’s mission to improve all aspects of XMPP usability, clear
data ownership and portability options have been an important goal since the
project’s beginning.
In particular we believe:
People should not be locked into the service by the first provider they sign
up with.
People should be able to export their full data at any time, in a standard
format.
People should be able to easily migrate their account data to a new provider
without losing important contact relationships.
The XMPP account portability project
The need for account and data portability goes beyond Snikket, we want to see
improved portability and data interoperability across the whole XMPP
ecosystem. DAPSI have funded an extensive project that over the next
nine months will cover:
Standardizing the necessary protocols and formats for account data import
and export
Developing open-source easy-to-use tools that allows people to export,
import and migrate their account between XMPP services
Building this functionality into Snikket
The standards will be submitted through the usual XMPP standards process
and the implementations will be open-source.
XMPP already has some existing standards that overlap with this project, in
particular XEP-0227 and XEP-0283. Both specifications are outdated
and incomplete (XEP-0227 doesn’t support many modern features, and assumes
your password will be exported in plain text!). We will update and/or
complement these documents as needed.
The final stage of work will be to integrate the migration mechanism into
Snikket. This will allow people to move their accounts between Snikket
servers, including to or from our hosted service as well as
other XMPP servers.
Our not-for-profit organization is committed to sustaining the
Snikket project through ethical means and without the influence of private
investment. We are very grateful for initiatives such as NGI, allowing
projects like ours to fulfil our ambitious goals with open and transparent
funding. Every project funded by them is helping to rebalance the internet.
We look forward to sharing further updates on this project in the coming
months, so stay tuned! You can follow us on Mastodon and Twitter, or
subscribe to this blog’s RSS feed.
