Planet Jabber

June 21, 2021

Monal IM

Vaxbot on Canadian National News

Vaxbot, a service built by the Monal team to vaccinate people in the US and Canada was featured on the Canadian Broadcast Corporation (CBC) The National and was broadcast nationwide.

This was great exposure for the project and XMPP in general, introducing it to a whole new population that likely skews older and less tech savvy. They don’t show Monal but you can see one of the users in the clip using Converse.js to access the bot. There has been a massive spike in new users since it aired on TV. At its peak in the US, we were doing about 1 million messages a day. We are approaching 800k again in Canada now.

by Anu at June 21, 2021 12:54

June 20, 2021

Monal IM

Monal 5.0 RC2 ready

We fixed some bugs in this second release candidate:

  • fixed background image bug
  • fixed wrong message direction displayed for some incoming messages in non-anon mucs
  • fixed reconnect-all button in logs settings
  • fixed changing of roster names of contact via contact details

by Thilo at June 20, 2021 23:35

June 18, 2021

Ignite Realtime Blog

Openfire 4.6.4 is released

The Ignite Realtime Community is pleased to announce the release of Openfire version 4.6.4. This release contains a number of bug fixes and denotes our desire to provide stable 4.6.x series releases while development work continues on a 4.7.0 release.

You can find download artifacts available having the following sha1sum values:

42bff5adc51a2347952f436fed25b5013ee0a146  openfire_4_6_4.dmg
d1f85a2854e42ae48e4000480fbbb55e39f84e56  openfire-4.6.4-1.i686.rpm
1a5c43e904726192e2abc36858eb2df0b17244a2  openfire-4.6.4-1.noarch.rpm
689c046afdbd63e22615a599f076851a49af50cd  openfire-4.6.4-1.x86_64.rpm
18117c3fee17d1b6d704b85c42d7d3386e520983  openfire_4.6.4_all.deb
36b1953db86406a1eb2344d1b0ffada0f57b8b32  openfire_4_6_4_bundledJRE.exe
72de355806bb2ce414655cf0da2801ce47032f03  openfire_4_6_4_bundledJRE_x64.exe
3086b70e04077b56dbb59931a87937e619995f0b  openfire_4_6_4.exe
7dab9c9a61456508793ede0ab13c7fe1f18e12d1  openfire_4_6_4.tar.gz
4f05ea94729800bbc6272b8ff98f3c8aa81d9207  openfire_4_6_4_x64.exe
792f68efc3943658e71586dbe305420499aab6b2  openfire_4_6_4.zip
9b3e031e4b064b5e4c618476b5429e077d4c1abb  openfire_src_4_6_3.tar.gz
bd79eae0079cba4a84cc9d1e249a62c3f45d6539  openfire_src_4_6_3.zip

Please report any issues you have with this release in our Community Forums.

We are always desperate for folks interested in pitching in with development, testing, and code reviews. Please consider dropping by our webchat forum if you are interested in helping.

Thanks again for your interest in Openfire!

For other release announcements and news follow us on Twitter

1 post - 1 participant

Read full topic

by guus at June 18, 2021 20:20

Peter Saint-Andre

Opinions vs. Truths

In recent posts we've looked into holding opinions about fewer topics, holding multiple opinions about the same topic, and changing our opinions about the opinions that other people hold. But what exactly is an opinion? Let's take a closer look....

June 18, 2021 00:00

June 17, 2021

Monal IM

Monal 5.0 RC1 ready

We are pleased to announce that we just published a new beta build (746) of Monal 5.0 (iOS and macOS)

If nothing big happens, this will be the last beta build of Monal 5.0 and we’ll finally see the release of Monal 5.0 stable next week!

by Thilo at June 17, 2021 05:14

June 06, 2021

Peter Saint-Andre

The Number Six

Everyone has their idiosyncrasies. One of mine is a particular fascination with the number six....

June 06, 2021 00:00

June 05, 2021

Profanity

Attention, attention!

Hello folks,

we have implemented an attention flag in profanity.

The attention flag can be used to mark chats and groupchats where you would like to pay particular attention.

This is only available on master, but will be in the next release (0.11.0).

How it works

Open the chat or groupchat window and press shortcut ALT+F. Profanity will display a line to inform you when the attention flag has been activated and deactivated.

05/06/21 15:25:49 - Staff restaurant: Lunch recommendations:
05/06/21 15:27:04 ! Attention flag has been activated
05/06/21 15:27:04 ! Attention flag has been deactivated

You can use the shortcut ALT+F to toggle the flag.

The /wins attention command can be used to display all windows with you pay attention.

15:38:19 - 3: Room roomA@conference.domain.tld
15:38:19 - 15: Room roomB@conference.server.tld, 1 unread
15:38:19 - 28: Room roomC@chat.server.tld
15:38:19 - 29: Room roomD@chat.server.tld, 3 unread

You can just circle around the marked windows with shortcut ALT+M.

June 05, 2021 13:07

June 04, 2021

The XMPP Standards Foundation

The XMPP Newsletter May 2021

Welcome to the XMPP Newsletter covering the month of May 2021.

Many projects and their efforts in the XMPP community are a result of people’s voluntary work. If you are happy with the services and software you may be using, especially throughout the current situation, please consider to say thanks or help these projects!

Read this Newsletter via our RSS Feed!

Interested in supporting the Newsletter team? Read more at the bottom.

Other than that - enjoy reading!

Newsletter translations

Translations of the XMPP Newsletter will be released here (with some delay):

Many thanks to the translators and their work! This is a great help to spread the news! Please join them in their work or start over with another language!

XSF Announcement

Voting the membership applications can be done via xmpp:memberbot@xmpp.org (by XSF members only). We will hold a member meeting on June 10th to formally approve the voting results. The meeting particulars are:

  • Date: June 10th, 2021
  • Time: 19:00 UTC
  • Location: xmpp:xsf@muc.xmpp.org
  • Further information: https://wiki.xmpp.org/web/Membership_Applications_Q2_2021

Events

XMPP Office Hours each week - Also, checkout our new YouTube channel!

Berlin XMPP Meetup (remote): Monthly Meeting of XMPP Enthusiasts in Berlin - always 2nd Wednesday of the month. Next topic will be the Curated XMPP provider list on Wednesday, 2021-06-09 18:00 CEST.

Videos

Gajim 1.4 UI/UX Preview presented by Philipp Hörist.

Gajim 1.4 UI/UX Preview

Articles

JC Brand, the developer behind Converse.js, the web client, blogs about the current development towards version 8.0.0 in Mergebounce: Increasing performance by batching IndexedDB writes.

Ingo Jürgensmann wrote the article The Fediverse – What About Resources? on the resources of different messaging technologies. He states that XMPP consumes a lot less hardware resources and thereby energy than comparing services.

Sumit Khanna wrote an article about moving their phone numbers from Google Hangouts/Voice to a SIP/XMPP Service by using XMPP and jmp.chat.

jmp.chat is also used by craftyguy to send test MMS to himself during mmsd development.

Vaxbot US has been shutdown after change of the situation in the USA. But the service has been deployed for Canada instead. All in all, that was an interesting approach and use of XMPP technology!

Software news

Clients and applications

Video calls in Dino are slowly coming together. Dino's developers are already making successful OMEMO encrypted video calls. The feature is included in their nightly builds now, but there's still further work to be done.

Dino AV calls

Gajim development News: This month brought improved Ad-Hoc commands, fixes for Gajim Portable, and new image preview capabilities. Meanwhile, work on Gajim’s next version made some progress: better code block styling, chat filters, note to myself, and much more. Also in Gajim news: Gajim celebrated its 17th birthday this month. Philipp Hörist (lovetox), maintainer of Gajim, gave a preview of the new Gajim 1.4 User Interface. Gajim is an XMPP client written in Python. It currently receives a big UI overhaul, of which the first results were presented at the XMPP Office Hours.

Kaidan 0.8 has been released with noteworthy new features, including typing notifications (Chat State Notifications) and message history synchronization (using MAM)!

"Salut à Toi" is now "Libervia". Read more about the changes behind the curtain.

UWXP, a Microsoft Windows (UWP) client, has been released in version 0.32.0.0 with focus on MUC and MAM bugfixes.

Servers

ProcessOne published a tutorial on how to install and configure MariaDB with ejabberd.

Prosody 0.11.9 has been released: This release addresses a number of important security issues that affect most deployments of Prosody. Full details are available in a separate security advisory. We recommend that all deployments upgrade or apply the mitigations described in the advisory.

Snikket just published their May update to the Snikket server software. This includes a few security fixes from Prosody, so upgrade soon! It also now allows you to manage user roles and access levels.

Libraries

No updates on XMPP libraries reach our attention :-(

Extensions and specifications

Developers and other standards experts from around the world collaborate on these extensions, developing new specifications for emerging practices, and refining existing ways of doing things. Proposed by anybody, the particularly successful ones end up as Final or Active - depending on their type - while others are carefully archived as Deferred. This life cycle is described in XEP-0001, which contains the formal and canonical definitions for the types, states, and processes. Read more about the standards process. Communication around Standards and Extensions happens in the Standards Mailing List (online archive).

Proposed

The XEP development process starts by writing up an idea and submitting it to the XMPP Editor. Within two weeks, the Council decides whether to accept this proposal as an Experimental XEP.

  • XMPP Compliance Suites 2022
    • This document defines XMPP application categories for different use cases (Core, Web, IM, and Mobile), and specifies the required XEPs that client and server software needs to implement for compliance with the use cases.

New

  • No new XEP this month.

Deferred

If an experimental XEP is not updated for more than twelve months, it will be moved off Experimental to Deferred. If there is another update, it will put the XEP back onto Experimental.

  • No XEP deferred this month.

Updated

  • Version 0.7.0 of XEP-0373 (OpenPGP for XMPP)

    • Recommend PubSub access model 'open' for public key data node and metadata node. (ps)
  • Version 1.3 of XEP-0013 (Flexible Offline Message Retrieval)

    • Deprecate after council vote of 2021-03-31 (XEP Editor (jsc))

Last Call

Last calls are issued once everyone seems satisfied with the current XEP status. After the Council decides whether the XEP seems ready, the XMPP Editor issues a Last Call for comments. The feedback gathered during the Last Call help improving the XEP before returning it to the Council for advancement to Draft.

  • No Last Call this month.

Draft

  • No Draft this month.

Call for Experience

A Call For Experience - like a Last Call, is an explicit call for comments, but in this case it's mostly directed at people who've implemented, and ideally deployed, the specification. The Council then votes to move it to Final.

  • No Call for Experience this month.

Thanks all!

This XMPP Newsletter is produced collaboratively by the community.

Therefore many thanks to emus, Florent Zara, Goffi, jeybe, Licaon_Kter, mdosch, nicola, pmaziere, snark, wurstsalat and Ysabeau for their support and help in creation, review and translation!

Spread the news!

Please share the news on "social networks":

Find and place job offers in the XMPP job board.

Also check out our RSS Feed!

Help us to build the newsletter

We started drafting in this simple pad in parallel to our efforts in the XSF Github repository. We are always happy to welcome contributors. Do not hesitate to join the discussion in our Comm-Team group chat (MUC) and thereby help us sustain this as a community effort. We really need more support!

You have a project and write about it? Please consider sharing your news or events here, and promote it to a large audience! And even if you can only spend a few minutes of support, these would already be helpful!

Tasks which need to be done on a regular basis are for example:

  • Aggregation of news in the XMPP universe
  • Short formulation of news and events
  • Summary of the monthly communication on extensions (XEP)
  • Review of the newsletter draft
  • Preparation for media images
  • Translations: especially German and Spanish

License

This newsletter is published under CC BY-SA license.

by emus at June 04, 2021 22:00

May 31, 2021

Peter Saint-Andre

Holding Multiple Opinions

Sometimes it's difficult to hold fewer opinions in your own mind or to engage in cognitive empathy toward others; that's when it can help to hold multiple opinions at the same time. This might sound like the mental equivalent of juggling plates, but it's a skill worth cultivating (and one with an ancient pedigree, as evidenced for example by the Letter to Pythocles by the ancient Greek philosopher Epicurus)....

May 31, 2021 00:00

May 28, 2021

Kaidan

Kaidan 0.8 released: Typing notifications & message synchronization

We are happy publish a new release of Kaidan that brings it closer fullfilling the daily messaging needs.

As promised, this release includes some major new features: The chat history is now being synchronized across devices, to allow finding information in old messages. It is now indicated when the chat partner is typing, which makes conversations easier.

Typing notifications in action

We have also polished the general usability, so that now the size of the windows is preserved across restarts. Some of the strings in the user interface have received some rewording, to make them easier to understand.

The registration now integrates as good as possible with servers which have registration through chat apps disabled, by showing the link to register through the server’s website.

On the cross-platform side, we have decided to use the breeze theme also on macOS, as the macOS style does not support a few features we want to use in the user interface.

Changelog

This release adds the following features:

  • Add typing notifications (XEP-0085: Chat State Notifications)
  • Add message history syncing (XEP-0313: Message Archive Management)
  • Window size is restored
  • The server’s website link is displayed if account creation is disabled
  • Use breeze theme on macOS
  • Improved user strings & descriptions

Download

Or install Kaidan from your distribution:

Packaging status

May 28, 2021 21:01

Gajim

Development News May 2021

This month brings improved Ad-Hoc commands, fixes for Gajim Portable, and new image preview capabilities. Meanwhile, work on Gajim’s next version made some progress: better code block styling, chat filters, note to myself, and much more.

Changes in Gajim

Through Gajim’s Ad-Hoc commands window, you are able to configure service settings, gather infos, or trigger various other actions offered by your provider. The Ad-Hoc commands window has now been ported to the new Assistant, which you already know from Gajim’s Account Creation for example. This lowers maintenance effort and allows all tasks with a ‘wizard’ workflow to have consistent design and behaviour.

On Windows, you can choose whether you want to install Gajim on your system or if you want to have a portable version of Gajim inside a single folder. For distribution of Gajim Portable, we use an installer which extracts all relevant files into a directory of your choice. However, this process sometimes fails if you upgrade an existing Gajim Portable folder. This issue has now been fixed by applying a cleaning mechanism before extracting the new version. Your user data won’t be touched, of course.

Meanwhile, work on the next Gajim version made some progress:

  • ‘Note to myself’ feature: write messages to your own contact (e.g. to another device), now improved
  • Code block styling is now aligned to your theme (light/dark)
  • Chats can be filtered by ‘chats’ and ‘group chats’ in the Start Chat window and in the Chat List
  • Status message styling has been improved
  • File transfers (uploading) are now displayed inline
  • Gajim update notifications will now be displayed in a separate App Page, so there won’t be disrupting update notifications anymore
  • Status Icon has been fixed, and the App Indicator plugin has been integrated
  • Status message window has been removed: it’s now a simple input field

If you haven’t seen it yet: lovetox (Philipp Hörist), current maintainer of Gajim, gave a first introduction to what’s coming with the next version:

This is only part of what we’re planning to do for the next release of Gajim. Stay tuned!

What else happened

  • Profile window: Fixed initial state of privacy switches
  • Profile window: Added Note element
  • #10559: Fixed issue with Server Info window when using PLAIN connection

Plugin updates

Gajim’s URL Image Preview plugin is now able to generate previews for WEBP and JXL files. MIME type guessing has been improved as well.

Changes in python-nbxmpp

This month, Ad-Hoc Commands (XEP-0050) compliance has been improved.

As always, feel free to join gajim@conference.gajim.org to discuss with us.

Gajim

May 28, 2021 00:00

May 27, 2021

ProcessOne

Install and configure MariaDB with ejabberd

By default, ejabberd uses the Mnesia internal database. It is great for home and small office environments, but in larger companies, as the amount of chat logs and users grows, we need more scalability. Today, I will show you how to install MariaDB, a MySQL-compatible database, migrate your data and configure ejabberd to use MariaDB instead of Mnesia.

» Don’t want to migrate data yourself?
ProcessOne experts will make your communication scalable. Contact us »

MariaDB with ejabberd

Installing MariaDB

We assume the usual Debian configuration as in my previous tutorials. I have updated my ejabberd to version 21.01 (the update process is the same as the initial ejabberd installation, so check my first tutorial).

To install MariaDB simply use:

apt-get install mariadb-server

Then run the installation wizard and follow the instructions:

mysql_secure_installation

Preparing MariaDB for ejabberd

To get MariaDB ready for ejabberd, we need to create a new database, its user, and then populate the database with the ejabberd SQL schema.

First, let’s create the database using your MariaDB root user:

echo "CREATE DATABASE ejabberd;" | mysql -h localhost -u root -p

Next, let’s create a dedicated ejabberd user authenticated with a password, and assign it to this database. The Enter password prompt is again asking about the root MariaDB user:

echo "GRANT ALL ON ejabberd.* TO 'ejabberd'@'localhost' IDENTIFIED BY 'password';" | mysql -h localhost -u root -p

Finally, let’s download the latest ejabberd SQL schema and load it into our database. This time, we are switching to using the ejabberd MariaDB user, and the Enter password prompt is asking for the password we just specified in the GRANT command above:

wget https://raw.githubusercontent.com/processone/ejabberd/master/sql/mysql.sql
mysql -h localhost -D ejabberd -u ejabberd -p < mysql.sql

To verify that everything is correct, run a command to display all the database tables, again using the ejabberd MariaDB user, and the output should look something like that:

echo "SHOW TABLES;" | mysql -h localhost -D ejabberd -u ejabberd -p --table
Enter password: 
+-------------------------+
| Tables_in_ejabberd      |
+-------------------------+
| archive                 |
| archive_prefs           |
| bosh                    |
| caps_features           |
| last                    |
| mix_channel             |
| mix_pam                 |
| mix_participant         |
| mix_subscription        |
| motd                    |
| mqtt_pub                |
...

Configuring ejabberd for MariaDB

Now that our MariaDB tables are ready, we need to configure ejabberd to use this MySQL-compatible database. Edit your ejabberd.yml config and add the following settings, where password refers to the ejabberd MariaDB user:

sql_type: mysql
sql_server: "localhost"
sql_database: "ejabberd"
sql_username: "ejabberd"
sql_password: "password"

Migrating Mnesia data to MariaDB database

At this point, if you restart your ejabberd, it won’t be using MariaDB just yet. Let’s first migrate Mnesia data into our new SQL database using ejabberdctl – we first export the data into a mnesia.sql file, and then we import it into the MariaDB database:

cd /opt/ejabberd-21.01/bin/
./ejabberdctl export2sql marekfoss.org /tmp/mnesia.sql
mysql -h localhost -D ejabberd -u ejabberd -p < /tmp/mnesia.sql
rm /tmp/mnesia.sql

It’s a good practice to remove the /tmp/mnesia.sql after we are done with it. Now, add default_db: sql and auth_method: sql to your ejabberd.yml configuration file:

default_db: sql
auth_method: sql

sql_type: mysql
sql_server: "localhost"
sql_database: "ejabberd"
sql_username: "ejabberd"
sql_password: "password"

Then, restart your ejabberd instance – it will now use the MariaDB database! Please note that the Mnesia database will still be started up and used for non-persistent data and clustering.

In this ejabberd tutorial series:

Photo by Amy Asher on Unsplash

The post Install and configure MariaDB with ejabberd first appeared on ProcessOne.

by Marek Foss at May 27, 2021 13:33

May 21, 2021

Gajim

Gajim’s Birthday

Today is Gajim’s birthday! 🎉 Seventeen years of Instant Messaging with open standards and open source software.

About Gajim

On Mai 21th 2004, Asterix (Yann Leboulanger) released Gajim 0.1. Gajim (is) a Jabber Instant Messenger. Development started back when XEPs (XMPP Extension Protocols) were named JEPs (Jabber Enhancement Proposals) and the XSF (XMPP Standards Foundation) was called JSF (Jabber Software Foundation). Right from the first year, Gajim was available for both Linux and Windows. Supported by many contributors, there have been more than 70 releases until today. The code repository counts around 18,000 commits, and the issue tracker (once powered by Trac, now Gitlab) lists a history of over 10,500 opened issues in total.

Today, Gajim supports around 85 XEPs. Driven by the community, Gajim has been translated into 29 languages. A calculated effort of around 31 person-years went into developing Gajim.

With the release of Gajim 1.0 in March 2018, a big switch from Python 2 and GTK 2 to Python 3 and GTK 3 had been completed. For the following releases, Gajim’s core had been gradually rewritten. Since February 2021, we’ve been working on the next version. It will include a complete rework of Gajim’s main component: the message window. lovetox (Philipp Hörist), current maintainer of Gajim, gave a first introduction to what’s coming with the next version:

This is only part of what we’re planning to do for the next release of Gajim. Stay tuned!

As always, feel free to join gajim@conference.gajim.org to discuss with us.

Gajim

May 21, 2021 00:00

May 19, 2021

Snikket

May 2021 server release

We’re pleased to introduce a new release of the Snikket server. The Snikket server is an easy-to-install server package that allows you to run your own private messaging service for family, friends and other small groups.

As well as some new features, this release has some important security fixes for the built-in Prosody component. We advise all administrators to update as soon as possible.

For information on how to upgrade, see the (very short) upgrade guide.

Web interface

User and role management

This release brings a new interface for viewing and editing user accounts on the server. Among the changes is the ability to select the “access level” of an account via the web interface. In particular this allows you to add/remove other administrators of your server.

In the future we will also be adding an additional ‘limited’ access level that can be used to restrict access to features such as invites and federation for certain user accounts (such as guests and minors).

Invitations

Invitation pages now include a link to download the Snikket app from F-Droid, as well as Google Play. Although F-Droid doesn’t yet support the seamless registration flow, it’s important that we help people discover free (as in freedom) alternatives whenever possible!

Translations

Translation improvements have been made for Polish, German, Danish, Spanish (Mexican), Indonesian and Swedish.

Certificate renewal

A bug has been fixed that eventually caused Snikket to present an expired certificate for web links (the web interface and also shared files). Restarting the service is a temporary fix, but this release will prevent it happening again in the future.

Technical improvements

Here’s a bunch of lower-level changes for advanced users that are included in this release:

  • You can now configure what address Snikket’s built-in HTTP server will listen for connections on (useful for certain advanced setups behind a reverse proxy)
  • Add docker health checks, allowing docker to inform you about the health of the Snikket services
  • Switch to a more robust DNS resolver (used for federation when connecting to other servers)
  • Allow configuration of an external TURN server (replacing the built-in one)
  • Fix support for BOSH and websockets (allowing third-party web clients)

If you have any questions or feedback about this release, let us know!

by Snikket Team (team@snikket.org) at May 19, 2021 12:45

Erlang Solutions

Lessons FinTech Can Learn From Telecom – Part 2/2

In the second half of this blog series, Erik Schön looks next at the ‘right tool for the jobs’ that are required for FinTech 3.0. The Erlang/Elixir/OTP open-source ecosystem for software development has its roots in telecom – an industry that has already resolved many of the new challenges confronting FinTech. Revisit Part One.

The Right Tool for the Job

One of the most important tools in the telecoms toolbox for real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost, is the open-source programming language and associated frameworks, tools and libraries called Erlang/OTP (Open Telecom Platform) originally developed by the telecom giant Ericsson.

erlang and elixir stickers on laptop

The Need: Quick Development of the Right Thing

Mike Williams, co-creator of Erlang/OTP, wrote down his credo in 1985 which was then used to guide the development of Erlang/OTP during the 80s and 90s (Däcker, 2009):

  • “Find the right methods – design by prototyping.”
  • “Make mistakes on a small scale – not in a production project.”
  • “It’s not good enough to have ideas – you must also be able to implement them to know that they work.”

These insights contributed to making Erlang/OTP suitable for iterative, incremental development with quick feedback from real customers. This also ensured a smooth developer experience making it easier to build the right thing that: 

  • customers find valuable, usable and sometimes even desirable
  • is feasible from a technical perspective; 
  • is viable from a commercial perspective;
  • is reasonable from a societal perspective, e.g. sustainable and ethical.

The Need: Real-Time, Secure, Reliable and Scalable Solutions 

Bjarne Däcker, head of the Ericsson computer science lab and Mike Williams’s manager at the time, formulated the requirements on the new programming language system as follows (Däcker, 2000):

  • Real-time:  “Actions to be performed at a certain point in time or within a certain time.”
  • Security: “Stringent quality”
  • Reliability: “Very large software systems … Interaction with hardware … Complex functionality such as feature interaction … Continuous operation for many years … Software maintenance (reconfiguration, etc.) without stopping the system … Fault tolerance both to hardware failures and software errors”
  • Scalability: “Systems distributed over several computers … handling of a very large number of concurrent activities.”

Joe Armstrong, co-creator of Erlang/OTP summarised it as “making reliable distributed systems in the presence of software errors”. (Armstrong, 2003).

Business Outcomes: Faster, Safer, Better, and, More for Less

Over the past 20+ years Erlang/OTP has provided the following business outcomes (Cesarini, 2019):

  • 2x FASTER development of new services thanks to the language’s design, the OTP middleware, the set of frameworks, principles, and design patterns that guide and support the structure, design, implementation, and deployment where e.g. Motorola saw 4-20 times less code compared to traditional languages. 
  • 10x BETTER services that are down less than 5 minutes per year thanks to built-in fault tolerance and resilience mechanisms built into the language and the OTP middleware, e.g. software upgrades and generic error handling, as seen by e.g. Ericsson in their mobile network packet routers.
  • 10x SAFER services that are hard to hack and crash through denial of service attacks thanks to the language construction with lack of pointers, use of messages instead of shared memory and immutable state rather than variable assignments, as seen by the number of vulnerabilities compared to other languages (CVE, 2021).
  • 10x MORE users (billions), transactions per second (millions) within milliseconds thanks to a battle-tested virtual machine as seen by e.g. WhatsApp with more than 2 billion users.
  • 10x LESS costs and energy consumption thanks to fewer servers needed where e.g. Bleacher Report were able to reduce their hardware requirements from 150 servers to 5.

It has been used in telecom since the 90s by e.g.

  • Vendors: Cisco, Ericsson, Nokia, Motorola, 2600Hz
  • Operators: T-Mobile, Telia
  • Social: WhatsApp

and in FinTech. since the mid-00s by e.g.

Robert Virding, co-creator of Erlang/OTP formulated the unique value proposition like this:
Any sufficiently complicated concurrent program in another language [for this job to be done] contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.” (Virding, 2008).

What about Elixir? 

Elixir has all the benefits of Erlang with a lower threshold for developers used to traditional programming languages like Ruby since Elixir’s syntax looks much more familiar to them. Additionally, Elixir gives a very smooth developer experience including state-of-the-art libraries e.g. for graphical user interfaces with Phoenix LiveView.

The Developer Experience

Both Erlang and Elixir are easy to learn within a couple of weeks for people with practical experiences and skills equivalent to a computer science degree as well as a curiosity to learn. And, experience shows that they get up to speed within a couple of months which is what it normally takes to understand a new product codebase or business domain (Däcker, 2000).

Engineers and developers love the experience of using Elixir and Erlang, and as of today, there are over 50,000 members in over 140 Meetup groups in all continents of the world except Antarctica (Schön, 2021). 

The Road Ahead

The Erlang/Elixir open-source ecosystem is thriving like never before.

During 2020-2021 we have seen companies like WhatsApp and Klarna working together on improving the developer experience further and companies like Ericsson evolving the OTP middleware where the next OTP release in May, 2021 is expected to improve the out-of-the-box performance of Erlang and Elixir applications by 30-130% (Larsson, 2020) and reduce the energy consumption by 25% (Cathcart, 2021).

And, we haven’t even mentioned the recent and exciting announcement of Elixir optimised for quick development of safe machine learning solutions for new innovative, automated services (Valim, 2021).

Conclusion

We hope that you now see what FinTech. can learn from telecom and how FinTech. companies can use the Erlang/Elixir/OTP open-source ecosystem to go 2x FASTER, 10x SAFER, 10x BETTER with 10x MORE for 10x LESS – resulting in happy and loyal customers as well as engaged developers.

What are your jobs to be done? What tools are you using? How can we help? Visit our FinTech Hub Page or, if you’re ready to find out how we can work together, tell us about your development project or idea.

Acknowledgements

Kudos to Michael Jaiyeola, for the original idea, helpful pointers, examples and feedback; Noman Azim, for valuable input and concrete examples; Steve Roberts for helpful feedback, insights and examples from a telecom perspective; Phil Harrison for insights and feedback from a FinTech. perspective; Francesco Cesarini for co-creating a generous and welcoming community, for spreading the word and feedback; Joe Armstrong, Robert Virding, Mike Williams and Bjarne Däcker for perseverance, professionalism and respect in co-creating and managing Erlang/OTP.

Writer

Erik Schön, Managing Director, Erlang Solutions Nordic, is an executive with 20+ years of telecoms experience from global standardisation, system design and managing R&D organisations of up to 400 people at the global telecom giant Ericsson. Erik is a big fan of Erlang since the 90s and his latest book is The Art of Strategy.

References

Armstrong, Joe (2003). Making reliable distributed systems in the presence of software errors

Cathcart, Will (2021), Improving WhatsApp’s server efficiency by 25%

Cesarini, Francesco (2019). Which companies are using Erlang, and why? 

CVE (2021). CVE Security Vulnerability Database

Däcker, Bjarne (2000). Concurrent Functional Programming for Telecommunications: A Case Study of Technology Introduction

Däcker, Bjarne (2009). CS Lab and all that … 

Larsson, Lukas (2020). Implement BeamAsm – a JIT for Erlang/OTP.

Rubio, Manuel (2019). Which companies are using Elixir, and why?

Schön, Erik (2021). Elixir & Erlang Developers

Valim, José (2021). Nx (Numerical Elixir) is now publicly available.

Virding, Robert (2008). Virding’s First Rule of Programming

The post Lessons FinTech Can Learn From Telecom – Part 2/2 appeared first on Erlang Solutions.

by Erik Schön at May 19, 2021 12:40

May 13, 2021

Monal IM

Vaxbot US shutting down

Vaccines are readily available in the US now, you can walk in to a pharmacy anywhere and get it. There is no need for the Vaxbot service (http://vaccine.monal.im) here anymore. After 11 million alerts sent out, we have shutdown US operations last week on May 8th and start focusing on Canada (where we just expanded) and likely add the EU (or anywhere else in the world that needs help) next.

Thanks to everyone on the Monal and the server administrators and developers from Yaxim and Prosody teams for making this project a success! We’ve made a difference in the lives of thousands of people and there is still much more to do.

If you are still interested in the open and free technology behind Vaxbot, please take a look at Monal IM which is the main project which enabled the simple setup of Vaxbot.

by Anu at May 13, 2021 15:00

Erlang Solutions

How to use RabbitMQ in service integration

RabbitMQ, the message broker, is not the first thing that comes to mind when we talk about integrating legacy services. We use high level frameworks to hide the complexity of the underlying plumbing. In this post we will show how the RabbitMQ, the most widely deployed open source message broker, can help with reliable and resilient messaging.

What is RabbitMQ?

RabbitMQ is a free, open-source and extensible message queuing solution. It is a message broker that understands AMQP (Advanced Message Queuing Protocol), but can also be used with other popular messaging solutions like MQTT. It is highly available, fault tolerant and scalable. It is implemented in Erlang OTP, a technology tailored for building stable, reliable, fault tolerant and highly scalable systems which possess native capabilities of handling very large numbers of concurrent operations. The benefits of Erlang OTP can easily be seen in RabbitMQ and other systems like WhatsApp, MongooseIM, to mention a few.

At a very high level, it is a middleware layer that enables different services in your application to communicate with each other without worrying about message loss while providing different quality of service (QoS) requirements. It also enables fine-grained and efficient message routing enabling extensive decoupling of applications.

RabbitMQ Connecting Legacy Services

When designing system interactions we usually start with a top-to-bottom approach. We identify the components and use the technology available to connect them. If we need to integrate existing systems with newer ones, then it is often infeasible to change the older system to use modern protocols. In cases like this, connector applications can be developed acting as a bridge between the old system and the new.

Once we know the components and we have all the translation layers, we can deal with the interconnection. There are two major ways how parts of the system can communicate:

  1. Directly (synchronously). When both components are online, they can use an active network connection and a preferred protocol like HTTP, protobuf or even FTP. Using these technologies results in a simple solution with its trade-offs, such as unavailability of multiple services, or having to implement client side buffering, when things go wrong.
  2. Indirectly (asynchronously). To mitigate the trade-offs from the direct communication we can introduce a messaging layer into the solution that can retain the messages even if the receiving application is not online. Although this adds complexity to the architecture it removes a lot of edge cases from the applications. Scaling the messaging layer is often easier than introducing scale to the legacy applications.

RabbitMQ is a well-suited message broker because it scales well not only with the message load requirements, but also with the business requirements. Finding message brokers that can handle hundreds or thousands or even tens of thousands of messages is easy. But finding a message bus that can cater for several messaging patterns, can handle different protocols (AMQP, MQTT, STOMP, JMS and many more) and can be modified via plugins is much more difficult.

Features come with trade-offs, but when RabbitMQ is put into a heterogeneous environment, it provides an easy way to move the messages between virtually any systems. All major frameworks support an AMQP adapter, but even if the component uses a proprietary protocol then it is possible to develop a plugin and extend RabbitMQ’s functionality.

Resilient and Reliable Messaging

RabbitMQ provides a high degree of availability and consistent messaging if the following three features are used. First we see why it is recommended to run RabbitMQ as a cluster, then how to guarantee that messages are received and kept safe by RabbitMQ and how quorum queues improve on classic mirroring.

Clustering

Although a single node RabbitMQ provides the highest consistency guarantees, it is not resilient to hardware errors or planned downtimes due to upgrades. If anything happens to the single RabbitMQ instance it means that the service is no longer available system-wide. If any of the message store files gets corrupted on the file system, then messages can be lost.

To avoid these problems RabbitMQ can run as a cluster, each node providing the full RabbitMQ functionality. It does not matter for the clients which RabbitMQ node it connects, RabbitMQ hides the complexities of distributing and replicating messages, queues and exchanges.

The big question is how many nodes we should use in a cluster. Using two nodes might be tempting, but it only works with systems running in an active-passive nature. RabbitMQ, as we discussed above, runs in an active-active fashion where each node in a cluster provides identical functionality. This means that in case of any problems, RabbitMQ needs a consensus of the majority of the nodes to function. By restricting the number of nodes in a cluster to an odd number, we can guarantee that the cluster will consistently survive the loss of the minority of the nodes (in case of 5 nodes, RabbitMQ can tolerate the loss of 2 nodes.)

To enforce that the minority nodes don’t commit changes, pausing them when detecting a network issue can be automated by setting the cluster partition handling mechanism to pause_minority. This is a built-in feature and it not only detects that a network partition occurred but also detects when the network is working again and can restart the nodes paused automatically. RabbitMQ does not require manual intervention to heal.

Publisher confirm and acknowledgements

RabbitMQ can automagically cluster and heal itself, but due to the distributed nature of things, it requires some cooperation from the clients to guarantee no message loss. When publishing a message to RabbitMQ the clients can only “forget” about the message when it was acknowledged by RabbitMQ. This acknowledgement on the publishing side is called the Publisher Confirm feature. It is more lightweight than the transaction mechanism in AMQP and performs much better under load. This RabbitMQ specific feature is supported by all major client libraries.

On the consuming side, the clients should acknowledge the message only when they finished processing the message. This guarantees that once a message is fully processed it is removed from RabbitMQ, but if any problem happens during message processing, the same message can be redelivered to the same or a different client. This gives a good opportunity to introduce horizontal scaling for parts of the system.

Quorum Queues

The last bit of the puzzle is to make sure that the messages that are in RabbitMQ can survive network errors, hardware failures or planned maintenance. With the introduction of clustering it is possible to distribute the workload among many nodes, but it is also important for introducing resiliency. Quorum queues are the new addition to the RabbitMQ feature family providing an improved solution to queue mirroring. It is based on the state of the art Raft algorithm to manage the integrity of the queue leader and the contents of the queues. Using it together with the publisher confirms, it is guaranteed that messages acknowledged by RabbitMQ will be safe until they are delivered to consumers.

Quorum queues improve on one of the major issues of classic queue mirroring: queue synchronisation after a network partition. With the Raft protocol, it is possible to only copy the messages that are missing from the mirror. This reduces the synchronisation network pressure which was a cause for some hard-to-debug RabbitMQ issues in the past.

Summary

In this blog post, we discussed the major challenges of integrating legacy and non-legacy systems. Introducing a versatile messaging layer like RabbitMQ can provide solutions to lots of problems and applying the three major features highlighted in this blogpost will enable a successful project.

Here at Erlang Solutions, we have world leading experts in RabbitMQ and can help with finding solutions to the unique problems of distributed systems. These challenges range from auditing the planned or deployed RabbitMQ solution or designing bespoke solutions with RabbitMQ. Get in touch to find out more about how we can help with your integration project or get your ticket to RabbitMQ Summit 2021 to learn more about why RabbitMQ is trusted by many of the world’s biggest companies.

The post How to use RabbitMQ in service integration appeared first on Erlang Solutions.

by Gabor Olah at May 13, 2021 13:10

May 12, 2021

Prosodical Thoughts

Prosody 0.11.9 released

We are pleased to announce a new minor release from our stable branch.

This release addresses a number of important security issues that affect most deployments of Prosody. Full details are available in a separate security advisory. We recommend that all deployments upgrade or apply the mitigations described in the advisory.

Note: We have updated the default config file. Your package manager may warn you about this, and ask if you want to use the new file or keep your existing one. You should usually keep your existing one, but make sure you update it to enable mod_limits after the upgrade.

A summary of changes in this release:

Security

  • mod_limits, prosody.cfg.lua: Enable rate limits by default
  • certmanager: Disable renegotiation by default
  • mod_proxy65: Restrict access to local c2s connections by default
  • util.startup: Set more aggressive defaults for GC
  • mod_c2s, mod_s2s, mod_component, mod_bosh, mod_websockets: Set default stanza size limits
  • mod_authinternal{plain,hashed}: Use constant-time string comparison for secrets
  • mod_dialback: Remove dialback-without-dialback feature
  • mod_dialback: Use constant-time comparison with hmac

Minor changes

  • util.hashes: Add constant-time string comparison (binding to CRYPTO_memcmp)
  • mod_c2s: Don’t throw errors in async code when connections are gone
  • mod_c2s: Fix traceback in session close when conn is nil
  • core.certmanager: Improve detection of LuaSec/OpenSSL capabilities
  • mod_saslauth: Use a defined SASL error
  • MUC: Add support for advertising muc#roomconfig_allowinvites in room disco#info
  • mod_saslauth: Don’t throw errors in async code when connections are gone
  • mod_pep: Advertise base pubsub feature (fixes #1632: mod_pep missing pubsub feature in disco)
  • prosodyctl check config: Add ‘gc’ to list of global options
  • prosodyctl about: Report libexpat version if known
  • util.xmppstream: Add API to dynamically configure the stanza size limit for a stream
  • util.set: Add is_set() to test if an object is a set
  • mod_http: Skip IP resolution in non-proxied case
  • mod_c2s: Log about missing conn on async state changes
  • util.xmppstream: Reduce internal default xmppstream limit to 1MB

Download

As usual, download instructions for many platforms can be found on our download page

If you have any questions, comments or other issues with this release, let us know!

by The Prosody Team at May 12, 2021 18:32

May 07, 2021

Jérôme Poisson

Libervia progress note 2021-W18

Hi,

again, lot of things have happened since last progress note, so I'll only talk about major changes here.

"Salut à Toi" is now "Libervia"

The project has been renamed to "Libervia". Even if I personally loved the former name (which was a reference to a French punk band song, an which could be translated to "hi to you", a nice fit for a communication tool), it proved to be hard to pronounce and remember for non French speakers, and the many names of frontends and project components were confusing. The name change has been discussed for long in the association, but the new ActivityPub/Pubsub end-to-end encryption project accelerated things: after a talk with NLnet, we decided to move forward on this so project name would not change in the middle of its development.

After doing a quick poll, we confirmed that "Libervia" (which was formerly the name of the web frontend only) would be the new name.

All parts are now named in straightforward way: "Libervia Backend", "Libervia Web", "Libervia Desktop/Mobile" (currently the same Kivy frontend for both), "Libervia TUI" and "Libervia CLI", with matching executable names (libervia-backend, libervia-web, libervia-desktop, libervia-tui, libervia-cli also aliased as li). The former names are kept internally and as aliases.

The non-profit (French "loi 1901") association behind it stays with the name "Salut à Toi".

This renaming has involved a lot for work, it took weeks to update code, web sites, doc, etc. and according to our statuses, we had to make a general assembly to discuss this decision. It's still not fully finished (notably the official web site URL is still https://salut-a-toi.org, while https://www.libervia.org is currently used for the demo server), and source code repositories have not been modified for the moment, but most of the renaming is done, and you can now reference the whole project as "Libervia"

Official Website Update

Following the changes in Libervia Web themes, the official website one has also been updated and is now based on Libervia's Web Bulma theme. The news now links to my personal blog as it is where you'll have most up-to-date informations about Libervia development (and the former page was broken). Tickets/Bug tracker is now directly accessible from the official site, as it makes more sense to have it there. It's still accessible from goffi.org, and thanks to its decentralised nature, it's usable transparently on both locations.

I've also temporarily disabled account registration on the bug-tracker due a wave of spammy accounts. I will have to put in place a protection for that, but I'm reluctant to use popular non-libre options.

Flatpak and Docker

While working on the renaming and website, I've updated the Flatpaks (they were really outdated), and Docker images. Flatpaks is for now using a specific dev repos, but I hope to see Libervia on Flathub after the release.

I've created Docker images and Docker Compose file to run quickly a local demo of Libervia Web, you can see the instruction on the Official Website.

Ideally, I would like to also create Snaps, Appimages, Nix packages, etc. But I'm lacking time (Flatpak and Docker are already too much time consuming) and prefer to focus on the code rather than on the packaging, help is more than welcome though.

User Friendly URLs

As you may have noticed on the last blog posts, URLs are now more user friendly:

A blog post is referenced using its item ID, and previously a unique ID was used for that, which is relatively long and doesn't give any information about the content, but is necessary to avoid conflict (writing a blog post with an existing ID will overwrite the previous one).

To make it more pleasant, a URL friendly extension was then added, and not used to retrieve the item, so in the example above, www.goffi.org/b/LFMqr7xC2aNf4MDgkbamBY links to the same blog post as www.goffi.org/b/LFMqr7xC2aNf4MDgkbamBY/sat-progress-note-2020-w53. The resulting URL is long and not easy to read, but the item is unique.

The new behaviour directly use URL friendly item IDs, and to avoid conflict, a short random suffix is appended (on the example above, QGqK is the suffix). After some tests, the collision risk for a short suffix like that is not that high (I've tested millions of IDs without collision), and it may anyway happen only if 2 blog posts have the exact same title, so the risk is very low. The resulting URL is more pleasant.

This URL friendly ID is used by default when a blog post is created, but it can be deactivated if user_friendly_id is set to false in blog post metadata, or by specifying manually an item id.

To accompany this change, a new Libervia CLI rename subcommand has been added to li blog and li pubsub, which will change the ID of an item. As there is no standard rename operation in XMPP Pubsub, this is done by copying the item to the new ID, then delete the former one in case of success.

Navigation Helpers in Libervia Web

It was not really easy so far to know where we were in Libervia Web. To help with this, the selected menu is now shown activated, and a breadcrumb has been added.

The breadcrumb is only shown when there are at least 2 elements to show (i.e. not on root pages). It is generated automatically by default, but can be customised with specific label, sub-elements, or even icons, like in the file sharing screenshot below:

Libervia Web 0.8 Breadcrumbs Screenshot

Blog Editor

As it was not possible anymore to write a new blog item with Libervia Web, I've made a blog item editor, which is relatively basic for now, but working. If JavaScript is activated, you'll get a tags editor, preview, and autosaving:

Libervia Web 0.8 Blog Editor Screenshot

File Sharing Quotas

One last missing piece I was needing before release was to put in place quotas on the file sharing component, this is now done.

Indeed, this component doesn't work with a per-file limit like most others do, but with a per-user quota, and you can upload any file size you want at long as you're not over quota.

Release to come

It's more than time to think about the release. I wanted to improve the chat notably in Libervia Web where it's still really basic since we moved out from the former frontend, but finally I've decided to report this to next release, as I plan to refactor messages handling, and for now I need to concentrate on the ActivityPub gateway.

So I'll soon prepare a beta version, and plan to do the release in a couple of weeks. I'll do bugfix on the 0.8 version during this time, but will avoid any important new development.

ActivityPub gateway project

With all the work done above (and other things, I've not mentioned everything), I've been late to start working on the ActivityPub project, but now I can focus on it. The first task is about developing a Pubsub cache as Libervia is currently getting its data for Pubsub related feature directly from the services.

Beside the obvious speed improvement, having a local cache will give the possibility to do data search/manipulation (such as doing Full-Text Search when the Pubsub service doesn't implement it, or doing feature-specific data analysis), handle message received unordered, allow to keep decrypted data when received from e2ee items, etc.

So far, SQLite was used for data storage in Libervia, by using Twisted's adbapi and custom semi-automatic schema update/data migration. It has been working relatively well so far, but it's no pleasant to maintain.

Fortunately, SQLAlchemy has recently added support for AsyncIO, thus it can now be used in Libervia. This is great, as SQLAlchemy is popular and rock solid, so I've decided to go with it. This will open the possibility to use other backends (like PostgreSQL), and refactor Libervia to use SQLAlchemy's ORM. Logically, Alembic will be used for data migration, which should make database modifications easier.

Such a cache will make possible to implements things like items discovery based on categories (or search by "hashtags" as it named in other software).

That's all for this note, see you soon.

by goffi at May 07, 2021 09:13

May 06, 2021

Erlang Solutions

Lessons FinTech Can Learn From Telecom – Part 1/2

This two-part blog series looks at the jobs to be done in fintech and argues that the right tool for the job, the Erlang/Elixir/OTP open-source ecosystem for software development, has its roots in telecom – an industry that has already resolved many of the new challenges confronting fintech currently.

To ensure you don’t miss part two of this blog and other high-quality content from us you can subscribe to our fintech newsletter here.

The Brave New World of Fintech

The world of financial services using technology to make individuals’ lives easier and more prosperous is becoming increasingly similar to the world of telecommunications.

The simple reason is that since the introduction of the iPhone (a smart telecommunications device also known as a “smartphone”) in 2007, most of our interactions with banks and other financial institutions are through smartphones. 

This means that we expect the banking experience to be as good as all our other smartphone experiences, be it entertainment or work related, including around the clock availability and responses within fractions of a second. This is backed up by research showing that response times above half a second makes people wonder what is going on and get stressed.

Since it is about money and other financial instruments, we also expect all interactions and personal data to be highly secure from e.g. intrusions and theft. The essence of all financial services is trust. You trust banks to not lose your money. If you lose the trust in your bank, you will switch to another bank or at least move parts of your business elsewhere. 

In more technical terms, we want our financial interactions to be done in real-time and be extremely secure and reliable. 

developers collaborating

The Job to Be Done

From a customer perspective, we have just seen how our financial services needs are for real-time, secure and reliable products and solutions. Also, we expect new innovative services that simplify our daily lives, e.g. saving us time and money while growing our financial wealth. One example is instant payments using your smartphone as a digital wallet connected to your bank account where work is ongoing to enable instant person-to-person payments also across national borders.

The traditional banks have a track record of security and reliability leading to trust, loyalty and long-standing personal relationships with their customers with e.g. personal banking experts available for financial advice. In the past that trust was established with large imposing buildings signifying they were here to stay.  Over time banks have eliminated buildings – particularly in the branch network in order to reduce costs and as a consequence “trust” has become more intangible. 

As a traditional bank, you would like to extend the experience to the digital world and at the same time reduce your large cost base which is due to very old hardware and software.  One example is a European bank that developed a new, well-designed and innovative smartphone application that unfortunately did not work since it took up to 30 seconds to fetch customer data from the legacy software system. They realised that their legacy software system was not fit for the future and have since then started to replace the system with future-proof technology, one “elephant carpaccio slice” at a time.

If you are a startup providing financial services, your needs are your customers’ needs, so real-time, security and reliability in order to secure their trust and loyalty are still valid. Since you are competing with the incumbent giants, you need a unique, focused offering that can be developed quickly and operated at low cost. Additionally, you want to provide a smooth onboarding experience to acquire new customers quickly. Backbase and Tide offer one minute customer onboarding time, a process that usually requires at least 48 hours to fully vet and comply with regulations. 

If you are one of the big US technology giants, you may want to consider your particular superpower, e.g. social networking, superb user experience or superior customer understanding, to expand your offering to embrace selected financial services as an integral part of your brand. One example is Apple Pay which extends the real time facility without having a current account or complying with onboarding requirements. Another example is Uber Money with digital wallets and credit and debit cards.

If you are one of the clearing houses, banking centrals, credit card companies or central banks, the huge number of financial transactions due to the real-time interactions lead to massive requirements on scalability and resilience of your technical infrastructure since the systems you operate are critical, not only for your specific customers but for society at large. We are seeing experiments with digital currencies run by several central banks in the world, including Sweden’s “Riksbank”, founded in 1668.

If you are one of the providers of digital currencies based on blockchain technologies you also face the challenge of minimising the energy consumption for all the computations needed.

Considering the engineers and developers who design, deliver and operate the financial systems, they want an environment where you can develop and deploy easily – including smooth programming languages requiring few lines of code supported by libraries, frameworks and tooling – and get quick feedback from both development, test and commercial deployments. Motivated and engaged employees lead to better services and happier and more loyal customers.

To summarise: we need real-time, secure, reliable and scalable systems that can be developed quickly and smoothly, operating at low cost with interoperability across borders and currencies.

The Telecoms World

Fortunately, the needs of real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost have already been fulfilled in the telecoms world starting in the 60s and 70s when the telecom operators started deploying the first digital software switches for local, regional and international phone calls. This infrastructure critical for society was designed, developed and tested for global, real-time around the clock operation with billions of users from the very beginning and global standards for interoperability were key enablers. 

The technology evolution grew very rapidly in the 80s and 90s due to the increased demands for communication on the move using handsets connected to mobile networks, first using national network standards  (the first generation, 1G, for voice calls only), then regional network standards (the second generation, 2G, for voice, text messages and limited data transmission), and, finally global network standards (the 3rd, 4th and 5th generations, 3G/4G/5G, optimised for higher and higher rates of data transmission).

This evolution happened in parallel with the evolution of systems for secure and reliable financial transactions. The biggest difference was that the telecom systems had far more transactions to handle due to the real-time nature of phone calls, mobile phone calls and browsing the web whereas the financial world handled this particular challenge through daily batches around midnight where all the financial transactions during the past 24 hours were processed.

Finally, telecoms have been commoditised by regulation, standards and global competition and is now a low margin business. There is a similar threat to financial services. In telecom, threats such as Google and others were well known to telecoms yet the telecom operators failed to act and now the value has been taken by the Big US Tech players with a global mindset. 

To summarise, telecoms have already mastered the challenge of real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost. At Erlang Solutions we believe that a lot of our learnings from this industry can be applied to exciting new projects in the fintech space. 

Writer

Erik Schön, Managing Director, Erlang Solutions Nordic, is an executive with 20+ years of telecoms experience from global standardisation, system design and managing R&D organisations of up to 400 people at the global telecom giant Ericsson. Erik is a big fan of Erlang since the 90s and his latest book is The Art of Strategy.

For part two of this blog series and other great fintech content you can subscribe to our newsletter here.

The post Lessons FinTech Can Learn From Telecom – Part 1/2 appeared first on Erlang Solutions.

by Erik Schön at May 06, 2021 09:56

Lessons FinTech Can Learn From Telecom

This blog post looks at the jobs to be done in fintech and argues that the right tool for the job, the Erlang/Elixir/OTP open-source ecosystem for software development, has its roots in telecom – an industry that has already resolved many of the new challenges confronting fintech currently.

The Brave New World of Fintech

The world of financial services using technology to make individuals’ lives easier and more prosperous is becoming increasingly similar to the world of telecommunications.

The simple reason is that since the introduction of the iPhone (a smart telecommunications device also known as a “smartphone”) in 2007, most of our interactions with banks and other financial institutions are through smartphones. 

This means that we expect the banking experience to be as good as all our other smartphone experiences, be it entertainment or work related, including around the clock availability and responses within fractions of a second. This is backed up by research showing that response times above half a second makes people wonder what is going on and get stressed.

Since it is about money and other financial instruments, we also expect all interactions and personal data to be highly secure from e.g. intrusions and theft. The essence of all financial services is trust. You trust banks to not lose your money. If you lose the trust in your bank, you will switch to another bank or at least move parts of your business elsewhere. 

In more technical terms, we want our financial interactions to be done in real-time and be extremely secure and reliable. 

developers collaborating

The Job to Be Done

From a customer perspective, we have just seen how our financial services needs are for real-time, secure and reliable products and solutions. Also, we expect new innovative services that simplify our daily lives, e.g. saving us time and money while growing our financial wealth. One example is instant payments using your smartphone as a digital wallet connected to your bank account where work is ongoing to enable instant person-to-person payments also across national borders.

The traditional banks have a track record of security and reliability leading to trust, loyalty and long-standing personal relationships with their customers with e.g. personal banking experts available for financial advice. In the past that trust was established with large imposing buildings signifying they were here to stay.  Over time banks have eliminated buildings – particularly in the branch network in order to reduce costs and as a consequence “trust” has become more intangible. 

As a traditional bank, you would like to extend the experience to the digital world and at the same time reduce your large cost base which is due to very old hardware and software.  One example is a European bank that developed a new, well-designed and innovative smartphone application that unfortunately did not work since it took up to 30 seconds to fetch customer data from the legacy software system. They realised that their legacy software system was not fit for the future and have since then started to replace the system with future-proof technology, one “elephant carpaccio slice” at a time.

If you are a startup providing financial services, your needs are your customers’ needs, so real-time, security and reliability in order to secure their trust and loyalty are still valid. Since you are competing with the incumbent giants, you need a unique, focused offering that can be developed quickly and operated at low cost. Additionally, you want to provide a smooth onboarding experience to acquire new customers quickly. Backbase and Tide offer one minute customer onboarding time, a process that usually requires at least 48 hours to fully vet and comply with regulations. 

If you are one of the big US technology giants, you may want to consider your particular superpower, e.g. social networking, superb user experience or superior customer understanding, to expand your offering to embrace selected financial services as an integral part of your brand. One example is Apple Pay which extends the real time facility without having a current account or complying with onboarding requirements. Another example is Uber Money with digital wallets and credit and debit cards.

If you are one of the clearing houses, banking centrals, credit card companies or central banks, the huge number of financial transactions due to the real-time interactions lead to massive requirements on scalability and resilience of your technical infrastructure since the systems you operate are critical, not only for your specific customers but for society at large. We are seeing experiments with digital currencies run by several central banks in the world, including Sweden’s “Riksbank”, founded in 1668.

If you are one of the providers of digital currencies based on blockchain technologies you also face the challenge of minimising the energy consumption for all the computations needed.

Considering the engineers and developers who design, deliver and operate the financial systems, they want an environment where you can develop and deploy easily – including smooth programming languages requiring few lines of code supported by libraries, frameworks and tooling – and get quick feedback from both development, test and commercial deployments. Motivated and engaged employees lead to better services and happier and more loyal customers.

To summarise: we need real-time, secure, reliable and scalable systems that can be developed quickly and smoothly, operating at low cost with interoperability across borders and currencies.

The Telecoms World

Fortunately, the needs of real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost have already been fulfilled in the telecoms world starting in the 60s and 70s when the telecom operators started deploying the first digital software switches for local, regional and international phone calls. This infrastructure critical for society was designed, developed and tested for global, real-time around the clock operation with billions of users from the very beginning and global standards for interoperability were key enablers. 

The technology evolution grew very rapidly in the 80s and 90s due to the increased demands for communication on the move using handsets connected to mobile networks, first using national network standards  (the first generation, 1G, for voice calls only), then regional network standards (the second generation, 2G, for voice, text messages and limited data transmission), and, finally global network standards (the 3rd, 4th and 5th generations, 3G/4G/5G, optimised for higher and higher rates of data transmission).

This evolution happened in parallel with the evolution of systems for secure and reliable financial transactions. The biggest difference was that the telecom systems had far more transactions to handle due to the real-time nature of phone calls, mobile phone calls and browsing the web whereas the financial world handled this particular challenge through daily batches around midnight where all the financial transactions during the past 24 hours were processed.

Finally, telecoms have been commoditised by regulation, standards and global competition and is now a low margin business. There is a similar threat to financial services. In telecom, threats such as 9Google and others were well known to telecoms yet the telecom operators failed to act and now the value has been taken by the Big US Tech players with a global mindset. 

To summarise, telecoms have already mastered the challenge of real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost. At Erlang Solutions we believe that a lot of our learnings from this industry can be applied to exciting new projects in the fintech space. 

The Right Tool for the Job

One of the most important tools in the telecoms toolbox for real-time, secure, reliable, scalable and interoperable systems that can be developed quickly and operated at low cost, is the open-source programming language and associated frameworks, tools and libraries called Erlang/OTP (Open Telecom Platform) originally developed by the telecom giant Ericsson.

erlang and elixir stickers on laptop

The Need: Quick Development of the Right Thing

Mike Williams, co-creator of Erlang/OTP, wrote down his credo in 1985 which was then used to guide the development of Erlang/OTP during the 80s and 90s (Däcker, 2009):

  • “Find the right methods – design by prototyping.”
  • “Make mistakes on a small scale – not in a production project.”
  • “It’s not good enough to have ideas – you must also be able to implement them to know that they work.”

These insights contributed to making Erlang/OTP suitable for iterative, incremental development with quick feedback from real customers. This also ensured a smooth developer experience making it easier to build the right thing that: 

  • customers find valuable, usable and sometimes even desirable
  • is feasible from a technical perspective; 
  • is viable from a commercial perspective;
  • is reasonable from a societal perspective, e.g. sustainable and ethical.

The Need: Real-Time, Secure, Reliable and Scalable Solutions 

Bjarne Däcker, head of the Ericsson computer science lab and Mike Williams’s manager at the time, formulated the requirements on the new programming language system as follows (Däcker, 2000):

  • Real-time:  “Actions to be performed at a certain point in time or within a certain time.”
  • Security: “Stringent quality”
  • Reliability: “Very large software systems … Interaction with hardware … Complex functionality such as feature interaction … Continuous operation for many years … Software maintenance (reconfiguration, etc.) without stopping the system … Fault tolerance both to hardware failures and software errors”
  • Scalability: “Systems distributed over several computers … handling of a very large number of concurrent activities.”

Joe Armstrong, co-creator of Erlang/OTP summarised it as “making reliable distributed systems in the presence of software errors”. (Armstrong, 2003).

Business Outcomes: Faster, Safer, Better, and, More for Less

Over the past 20+ years Erlang/OTP has provided the following business outcomes (Cesarini, 2019):

  • 2x FASTER development of new services thanks to the language’s design, the OTP middleware, the set of frameworks, principles, and design patterns that guide and support the structure, design, implementation, and deployment where e.g. Motorola saw 4-20 times less code compared to traditional languages. 
  • 10x BETTER services that are down less than 5 minutes per year thanks to built-in fault tolerance and resilience mechanisms built into the language and the OTP middleware, e.g. software upgrades and generic error handling, as seen by e.g. Ericsson in their mobile network packet routers.
  • 10x SAFER services that are hard to hack and crash through denial of service attacks thanks to the language construction with lack of pointers, use of messages instead of shared memory and immutable state rather than variable assignments, as seen by the number of vulnerabilities compared to other languages (CVE, 2021).
  • 10x MORE users (billions), transactions per second (millions) within milliseconds thanks to a battle-tested virtual machine as seen by e.g. WhatsApp with more than 2 billion users.
  • 10x LESS costs and energy consumption thanks to fewer servers needed where e.g. Bleacher Report were able to reduce their hardware requirements from 150 servers to 5.

It has been used in telecom since the 90s by e.g.

  • Vendors: Cisco, Ericsson, Nokia, Motorola, 2600Hz
  • Operators: T-Mobile, Telia
  • Social: WhatsApp

and in fintech since the mid-00s by e.g.

Robert Virding, co-creator of Erlang/OTP formulated the unique value proposition like this:
Any sufficiently complicated concurrent program in another language [for this job to be done] contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.” (Virding, 2008).

What about Elixir? 

Elixir has all the benefits of Erlang with a lower threshold for developers used to traditional programming languages like Ruby since Elixir’s syntax looks much more familiar to them. Additionally, Elixir gives a very smooth developer experience including state-of-the-art libraries e.g. for graphical user interfaces with Phoenix LiveView.

The Developer Experience

Both Erlang and Elixir are easy to learn within a couple of weeks for people with practical experiences and skills equivalent to a computer science degree as well as a curiosity to learn. And, experience shows that they get up to speed within a couple of months which is what it normally takes to understand a new product codebase or business domain (Däcker, 2000).

Engineers and developers love the experience of using Elixir and Erlang, and as of today, there are over 50,000 members in over 140 Meetup groups in all continents of the world except Antarctica (Schön, 2021). 

The Road Ahead

The Erlang/Elixir open-source ecosystem is thriving like never before.

During 2020-2021 we have seen companies like WhatsApp and Klarna working together on improving the developer experience further and companies like Ericsson evolving the OTP middleware where the next OTP release in May, 2021 is expected to improve the out-of-the-box performance of Erlang and Elixir applications by 30-130% (Larsson, 2020) and reduce the energy consumption by 25% (Cathcart, 2021).

And, we haven’t even mentioned the recent and exciting announcement of Elixir optimised for quick development of safe machine learning solutions for new innovative, automated services (Valim, 2021).

Conclusion

We hope that you now see what fintech can learn from telecom and how fintech companies can use the Erlang/Elixir/OTP open-source ecosystem to go 2x FASTER, 10x SAFER, 10x BETTER with 10x MORE for 10x LESS – resulting in happy and loyal customers as well as engaged developers.

What are your jobs to be done? What tools are you using? How can we help?

Acknowledgements

Kudos to Michael Jaiyeola, for the original idea, helpful pointers, examples and feedback; Noman Azim, for valuable input and concrete examples; Steve Roberts for helpful feedback, insights and examples from a telecom perspective; Phil Harrison for insights and feedback from a fintech perspective; Francesco Cesarini for co-creating a generous and welcoming community, for spreading the word and feedback; Joe Armstrong, Robert Virding, Mike Williams and Bjarne Däcker for perseverance, professionalism and respect in co-creating and managing Erlang/OTP.

Writer

Erik Schön, Managing Director, Erlang Solutions Nordic, is an executive with 20+ years of telecoms experience from global standardisation, system design and managing R&D organisations of up to 400 people at the global telecom giant Ericsson. Erik is a big fan of Erlang since the 90s and his latest book is The Art of Strategy.

References

Armstrong, Joe (2003). Making reliable distributed systems in the presence of software errors

Cathcart, Will (2021), Improving WhatsApp’s server efficiency by 25%

Cesarini, Francesco (2019). Which companies are using Erlang, and why? 

CVE (2021). CVE Security Vulnerability Database

Däcker, Bjarne (2000). Concurrent Functional Programming for Telecommunications: A Case Study of Technology Introduction

Däcker, Bjarne (2009). CS Lab and all that … 

Larsson, Lukas (2020). Implement BeamAsm – a JIT for Erlang/OTP.

Rubio, Manuel (2019). Which companies are using Elixir, and why?

Schön, Erik (2021). Elixir & Erlang Developers

Valim, José (2021). Nx (Numerical Elixir) is now publicly available.Virding, Robert (2008). Virding’s First Rule of Programming

The post Lessons FinTech Can Learn From Telecom appeared first on Erlang Solutions.

by Erik Schön at May 06, 2021 09:30

Peter Saint-Andre

What Is a Concept Album?

And now for something completely different: a rollicking discussion on the Big Yellow Praxis podcast about some underappreciated musical recordings, in which Jacob and I explore the age-old question: what exactly makes a concept album? Check it out on YouTube!...

May 06, 2021 00:00

May 05, 2021

The XMPP Standards Foundation

The XMPP Newsletter April 2021

Welcome to the XMPP Newsletter covering the month of April 2021.

Many projects and their efforts in the XMPP community are a result of people’s voluntary work. If you are happy with the services and software you may be using, especially throughout the current situation, please consider to say thanks or help these projects!

Read this Newsletter via our RSS Feed!

Interested in supporting the Newsletter team? Read more at the bottom! Other than that - enjoy reading!

Newsletter translations

Translations of the XMPP Newsletter will be released here (with some delay):

XSF Announcement

The XMPP Standards Foundation is now on YouTube! You can find recordings of conferences, the XMPP Office Hours, and other fun stuff over on the channel.

Have a talk, demo, or roundtable discussion about XMPP or related technologies? XMPP Office Hours are weekly meetings of XMPP enthusiasts for discussion and presentations. For more information, or to sign up, have a look at the wiki page.

Events

XMPP Office Hours each week - Also, checkout our new YouTube channel!

Berlin XMPP Meetup (remote): Monthly Meeting of XMPP Enthusiasts in Berlin - always 2nd Wednesday of every month.

Articles

From May 2021, there have been major content updates for freie-messenger.de as well as additions. From now on, the content will be internationalised. As a first step, there is the new Quick Overview of Messengers in both German and English. Call for supporters!

Isode presents their military capabilities at DSEI 2021, offering Military Instant Messaging and more.

Gajim Development News: April brought an all new message styling parser, making Gajim fully compliant with XEP-0393. This post will also give you a sneak peek on some features we’ve been developing for the past months: the new Chat view and Contact Info window.

Gajim chat view

Libervia (formerly Salut à Toi) has been selected for a grant by NLNet/NGI0 Discovery Fund (with financial support from European Commission's Next Generation Internet programme) for an gateway between XMPP and ActivityPub doubled with Pubsub end-to-end encryption project. The gateway will join two major open and decentralised protocols. In practice it will be a XMPP server component (usable with any server), and implement the ActivityPub server to server protocol (or "Federation Protocol"). On XMPP side, it will be mostly a Pubsub service (with some extra, like private messages converted to XMPP message stanza).

Vaxbot continues to grow: VaxBot is a free service that helps you find vaccine appointments in your state or country if available. During the last weeks, Vaxbot continued to grow. The service just hit 8 million messages sent and it sends around 800,000 messages a day. XMPP and Vaxbot have been on TV news in Las Vegas and South Carolina, the radio in Washington as well newspapers all over the country.

Refering the Vaxbot news: Georg Lukas, who runs the server behind it, wrote an article regarding the increased subscriptions and how the services have to adapt the infrastructure to keep up with this new "VaxBot performance challenge".

Yaxim server registrations

Snikket announces that XMPP Account Portability was funded under the NGI DAPSI initiative. It will cover the needed standards (beyond older XEP-0227 and XEP-0238), open-source tools and integrating said functionality into Snikket.

Nicola Fabiano published an article titled: Whatsapp? No thanks, I prefer to have control over my data

Software news

Clients and applications

Conversation 2.9.11 and 2.9.12 have been released, fixing 'No Connectivity' issues on Android 7.1 and adding support for roster pre-authentication.

Gajim 1.3.2 has been released: This release brings back translations for Windows users. It also adds some small fixes and improvements.

Monal 5 Beta 4 has been released with lots of updates.

Since there appeared a lot of sendxmpp alternatives over the last years there is now an work-inprogress overview in the XSF wiki.

UWPX is replacing the SQLite-net DB backend with a new Entity Framework Core DB. This comes with a fix and update of the OMEMO implementation to the latest version of the draft (OMEMO 0.7.0 (2020-09-05)). This version is incompatible with all other previous versions of OMEMO and UWPX allows you now to only exchange OMEMO encrypted messages with contacts that support at least OMEMO 0.7.0 (2020-09-05) draft. Now UWPX can act as a client with a proper one to one OMEMO implementation!

Spark 3.0.0 Beta has been released bringing fixes as usual. Pade Meetings is now integrated (it enables audio and video chat) and all these over a complete UI refresh.

Servers

ejabberd 21.04 has been released: The new ejabberd 21.04 release includes many bugfixes and a few improvements. This release includes minor improvements to fully support Erlang/OTP 24 and Rebar3. At the same time, it maintains support back to the old Erlang/OTP 19.3 and Rebar2.

Openfire 4.6.3 has been released: This release contains a number of bug fixes and denotes the desire to provide stable 4.6.x series releases while development work continues on a 4.7.0 release.

Libraries

slixmpp 1.7.1 has been released, which is a bugfix release for a rarely used code path tied to OMEMO.

Extensions and specifications

Developers and other standards experts from around the world collaborate on these extensions, developing new specifications for emerging practices, and refining existing ways of doing things. Proposed by anybody, the particularly successful ones end up as Final or Active - depending on their type - while others are carefully archived as Deferred. This life cycle is described in XEP-0001, which contains the formal and canonical definitions for the types, states, and processes. Read more about the standards process. Communication around Standards and Extensions happens in the Standards Mailing List (online archive).

Proposed

The XEP development process starts by writing up an idea and submitting it to the XMPP Editor. Within two weeks, the Council decides whether to accept this proposal as an Experimental XEP.

  • No XEPs have been proposed this month.

New

  • Version 1.0.0 of XEP-0457 (Message Fancying)
    • This specification defines a Unicode-formatted fancy text syntax for use in instant messages.
    • Initial published version. (XEP Editor (jsc))

Deferred

If an experimental XEP is not updated for more than twelve months, it will be moved off Experimental to Deferred. If there is another update, it will put the XEP back onto Experimental.

  • No XEPs deferred this month.

Updated

  • Version 0.4.0 of XEP-0420 (Stanza Content Encryption)

    • Use 'envelope' and 'content' consistently by renaming elements Update namespace to (melvo)
  • Versions 0.4.0 and 0.5.0 of XEP-0434 (Trust Messages (TM))

    • Add new section, use more precise sentences, apply consistent formatting:
    • Add new section for Trust Message URIs
    • Use 'that' instead of 'which' for restrictive clauses
    • Apply consistent formatting for paragraphs and revision history
    • Update to XEP-0420 version 0.4.0 and adapt namespace to shortname
    • Replace SCE's old 'content' element by its new 'envelope' element
    • Replace SCE's old 'payload' element by its new 'content' element
    • Update SCE's namespace to 'urn:xmpp:sce:1'
    • Change namespace to 'urn:xmpp:tm:0' (melvo)
  • Versions 0.2.0 and 0.3.0 of XEP-0450 (Automatic Trust Management (ATM)

    • Add usage of Trust Message URIs, use more precise sentences, apply consistent formatting
    • Add usage of Trust Message URIs for initial authentications
    • Use 'that' instead of 'which' for restrictive clauses
    • Apply consistent formatting for paragraphs and revision history
    • Update to XEP-0420 version 0.4.0 and XEP-0434 version 0.5.0
    • Replace SCE's old 'content' element by its new 'envelope' element
    • Replace SCE's old 'payload' element by its new 'content' element
    • Update SCE's namespace to 'urn:xmpp:sce:1'
    • Update TM's namespace to 'urn:xmpp:tm:0'
    • Update namespace to 'urn:xmpp:atm:1' (melvo)

Last Call

Last calls are issued once everyone seems satisfied with the current XEP status. After the Council decides whether the XEP seems ready, the XMPP Editor issues a Last Call for comments. The feedback gathered during the Last Call help improving the XEP before returning it to the Council for advancement to Draft.

Draft

  • No Drafts this month.

Call for Experience

A Call For Experience - like a Last Call, is an explicit call for comments, but in this case it's mostly directed at people who've implemented, and ideally deployed, the specification. The Council then votes to move it to Final.

  • No Call for Experience this month.

Thanks all!

This XMPP Newsletter is produced collaboratively by the community.

Thanks to alkino, emus, Licaon_Kter, IM, Martin, mathieui, MattJ, nicola, peetah, Sam, seveso, therealjeybe, wurstsalat and Ysabeau for their help in creating it!

Spread the news!

Please share the news on "social networks":

Find and place job offers in the XMPP job board.

Also check out our RSS Feed!

Help us to build the newsletter

We started drafting in this simple pad in parallel to our efforts in the XSF Github repository. We are always happy to welcome contributors. Do not hesitate to join the discussion in our Comm-Team group chat (MUC) and thereby help us sustain this as a community effort.

You have a project and write about it? Please consider sharing your news or events here, and promote it to a large audience! And even if you can only spend a few minutes of support, these would already be helpful!

Tasks which need to be done on a regular basis are for example:

  • Aggregation of news in the XMPP universe
  • Short formulation of news and events
  • Summary of the monthly communication on extensions (XEP)
  • Review of the newsletter draft
  • Preparation for media images
  • Translations: especially German and Spanish

License

This newsletter is published under CC BY-SA license.

by emus at May 05, 2021 18:00

May 04, 2021

Peter Saint-Andre

Opinions about Opinions

My friend Paul sent me a few thoughts about my recent post on holding fewer opinions. He's formulated an approach that involves holding fewer opinions about other people's opinions. This seems valuable, and related to a post I wrote four years ago entitled "Why Do You Think What You Think?" My introspective conclusion then was that I can't always assign praise or blame for the contents of my thoughts; extending this to other people is a good example of what Arnold Kling calls cognitive empathy: instead of demonizing people who disagree with you, suspend judgment or at least try to understand where they might be coming from. (It's probably best to build up such a practice first in matters that aren't very consequential - and we all have our limits regarding the opinions we find acceptable!) The flip side is not deifying your own thoughts, which we could express in the following aphorism: "if you think well of your own opinions just because they're yours, then you're not thinking well."...

May 04, 2021 00:00

April 30, 2021

Snikket

XMPP Account Portability funded by NGI DAPSI

We have some exciting news to share! An important piece of the Snikket roadmap has been selected for funding by NGI DAPSI, an EU-funded project focused on data portability and services.

What is DAPSI?

The Data Portability and Services Incubator (DAPSI) is a EU funded project, under the European Commission’s Next Generation Internet (NGI) initiative. In their own words, DAPSI was established to:

[…] empower top internet innovators to develop human-centric solutions, addressing the challenge of personal data portability on the internet, as foreseen under the GDPR and make it significantly easier for citizens to have any data which is stored with one service provider transmitted directly to another provider.

You can learn more about the initiative on the NGI DAPSI website.

Data portability in Snikket and XMPP

Over the years we have seen many XMPP providers come and go, and when a provider decides to shut down, it’s too often not easy for people to obtain their data and move it elsewhere. This contributes to user churn on the XMPP network - individuals are likely to leave XMPP rather than figure out the necessary steps to migrate to a new XMPP service.

There are other reasons for wanting to move your data, such as seeking providers with better privacy or reliability. You may also want to relocate from a provider to a self-hosted solution, or vice-versa.

As part of Snikket’s mission to improve all aspects of XMPP usability, clear data ownership and portability options have been an important goal since the project’s beginning.

In particular we believe:

  • People should not be locked into the service by the first provider they sign up with.
  • People should be able to export their full data at any time, in a standard format.
  • People should be able to easily migrate their account data to a new provider without losing important contact relationships.

The XMPP account portability project

The need for account and data portability goes beyond Snikket, we want to see improved portability and data interoperability across the whole XMPP ecosystem. DAPSI have funded an extensive project that over the next nine months will cover:

  • Standardizing the necessary protocols and formats for account data import and export
  • Developing open-source easy-to-use tools that allows people to export, import and migrate their account between XMPP services
  • Building this functionality into Snikket

The standards will be submitted through the usual XMPP standards process and the implementations will be open-source.

XMPP already has some existing standards that overlap with this project, in particular XEP-0227 and XEP-0283. Both specifications are outdated and incomplete (XEP-0227 doesn’t support many modern features, and assumes your password will be exported in plain text!). We will update and/or complement these documents as needed.

The final stage of work will be to integrate the migration mechanism into Snikket. This will allow people to move their accounts between Snikket servers, including to or from our hosted service as well as other XMPP servers.

Our not-for-profit organization is committed to sustaining the Snikket project through ethical means and without the influence of private investment. We are very grateful for initiatives such as NGI, allowing projects like ours to fulfil our ambitious goals with open and transparent funding. Every project funded by them is helping to rebalance the internet.

We look forward to sharing further updates on this project in the coming months, so stay tuned! You can follow us on Mastodon and Twitter, or subscribe to this blog’s RSS feed.

by Snikket Team (team@snikket.org) at April 30, 2021 13:14