Nobody here knew quite what a year 2020 was going to be! However despite
pandemics and lockdowns, we have continued to work on Prosody. This post
is a summary of how the project is doing, and what we’ve been up to in
the past year.
One quick note before we begin… Prosody is an independent open-source
project and exists only because the developers have been fortunate enough
to be in a position to work on it. A couple of core team members are
currently looking for freelance work. If you have projects in need of a
Prosody expert, check the bottom of this post for more details!
More users than ever before
Prosody does not “phone home” in any way, which means we do not have a
lot of insight into how many people are using Prosody. But there are
some indicators that we can use to see at least the growth of the project.
Many years ago, circa 2014, I was filling out a form that asked how
many users the project had. I thought long and hard, but with no idea how
to measure, I wrote down an estimate of “500” based on nothing but a gut
feeling. Only a few weeks later I learned that the XMPP Observatory
had already seen over 2200 domains submitted that were running Prosody.
As most deployments were unlikely to have been submitted to xmpp.net, my
estimate was clearly far out. These days I jump at any chance for even a
vague estimate of our userbase. It helps us to know that people are out
One useful tool is Shodan. This project scans the
entire internet, just to see what it can find, and it records the results.
Often used by academics and security researchers, a free account can also
be used by anyone to run simple queries over the data they collect.
A Shodan search in 2017 turned up nearly 7000 Prosody deployments. The
same search 8 months later returned over 16000. Today we’re at over 52000
Prosody servers! And this only counts instances using port 5269 and
accessible to the internet. There are also many private/internal deployments
of Prosody that are not included in these numbers. Unfortunately we
didn’t run a report in 2019, but here’s a graph of the previous
reports we have run:
Shodan reports over 85000 federating XMPP servers on port 5269 in total.
Based on this, Prosody makes up 44% of the public XMPP network. That’s
quite an achievement!
Another handy insight into one sector of Prosody deployments is via Debian’s
“popularity contest” service. This is an
automated survey that administrators of Debian servers can choose to opt
into. It reports anonymously to Debian what software packages are installed
and in use. Although it reflects only a small slice of even Debian
installations, it is useful to see trends.
March 2020 marked an unprecedented spike in Prosody installations!
Although we don’t know for certain, we suspect this was caused by an surge
of interest in the self-hosted video conferencing software, Jitsi Meet. Jitsi Meet integrates with Prosody, which is used to power the
authentication, signaling and chat of the video conferences. Jitsi’s Videobridge component handles the media routing. Together they make
a very powerful and flexible communication system, and it is hardly surprising
that interest has spiked this year when there has been a massive shift to
remote work, and online meetings have replaced physical ones.
The code counts
Now let’s look at some stats about the Prosody codebase itself.
|Other (build tools, etc.)
Changes in 2020
In 2020 (looking at the development branch) we added 597 commits, changing
191 files. The changes added 9872 lines of code, and 3637 lines of code
A significant portion of the new lines were in our unit and integration
tests (2200 lines, about 22%) which we have been working hard to expand
over the past couple of years.
If you use Prosody, you know we have an emphasis on modularity and
extensibility. We like to make developing plugins for Prosody as easy as possible, whether it’s for integration with other
systems or crazy experiments.
Here’s a random selection of the 363 modules currently in the repository:
- Send debug logs to RAM unless they are needed.
- Allow a Prosody server to run as a (XEP-0114) external component of
another (Prosody or not) XMPP server.
- Powerful rules-based scripts for filtering and redirecting XMPP traffic.
- An experiment in making MUC joins persistent (like MIX).
During 2020 we saw 37 new modules published in the community repository,
and 499 commits from 19 contributors. Together they added over 10,000
lines of code (and removed 728 lines). This makes it our most active
year apart from 2018!
Outside of the Prosody dev team, the most active contributor was
Seve, who also made their first contribution this year and added a total of four new modules.
But the most important part… what features have we been working on? All
these things are scheduled for the 0.12 release (more on that in a bit).
We have been applying further polish to, and setting up the infrastructure
for the plugin installer. This was a Google Summer of Code project by
JoĂŁo Duarte. It utilizes the
Lua package manager, LuaRocks, to download, install and manage community
Although the the installer was completed in 2019, to make it generally
usable we also had to ensure every module in the community repository
could be packaged and served in an automated way by our server. We now
have this working.
Bye bye telnet, hello prosodyctl shell!
The telnet console is one of the best things about Prosody, and we’ve
been working on its successor. An early version of
prosodyctl shell is
already available to try out in trunk nightly builds.
Using prosodyctl allows us to more easily support advanced features such
as line editing and history (previously attainable using a third-party
utility, rlwrap). It also allows for some richer UIs and is more secure
on shared servers (it uses a unix socket instead of TCP).
Since Prosody needs to resolve special DNS record types (such as SRV
records) and in an asynchronous manner, the built-in operating system APIs
are generally inadequate.
For a long time we’ve been using an adopted library simply known as
‘dns.lua’ combined with our own asynchronous wrapper around it. Although
it hasn’t been terrible, it has a few issues, especially in some uncommon
environments. It also doesn’t support many advanced features such as DNSSEC.
Now we are migrating to libunbound, part of the unbound
project. This is one of the leading DNS implementations, and will be a
big improvement over our current DNS library. To try it out, you can
simply install luaunbound (already
available in luarocks, Debian testing, AUR and others - poke your distro
maintainers if you don’t have it yet!).
HTTP server upload performance
We didn’t set out to write a HTTP server, but we ended up with one anyway!
Originally added so that we could natively support BOSH (XMPP over HTTP)
clients, it grew to support websockets, and various modules now provide
HTTP APIs for integration between Prosody and other systems.
One big problem is that the original implementation was designed for only
small amounts of data. Since the widespread of adoption of
XEP-0363 people now want to
be able to upload files, pictures and videos using Prosody’s internal
HTTP server. We have limits in place to protect against denial of service
attacks, but those same limits prevent large uploads from trusted users.
We’ve put some work into supporting “streaming uploads”, where incoming
data can be saved directly to disk instead of RAM. This means it will be
safe to increase file upload limits without opening up your server to
increase RAM usage and denial of service attacks.
In general though, we do recommend using a real external HTTP server in
a production or high traffic deployment (using mod_http_upload_external).
Passwords are the fundamental means of authenticating to your server in
XMPP today. XMPP is quite good at this, adopting strong standard authentication
mechanisms such as SCRAM far earlier than the rest of the industry. But
the rest of the industry is also moving away from passwords in many places.
We’re aiming to follow this movement also. Not that we are scrapping passwords
entirely, but making it easier to offer alternatives.
Prosody actually has a number of non-password authentication modules already,
such as mod_auth_oauthbearer (OAuth2 tokens),
mod_auth_ccert (client certificates) and
But most of the modules have limitations and are not well integrated
(e.g. you can set up Prosody to accept passwords, or set it up to accept
tokens, but you can’t offer both methods at the same time).
An important related aspect is authorization. In most systems authentication
via a token also provides limited access to the account (e.g. if a password
is associated with an account, a session that logged in using a token
should not be allowed to reset the password).
We’ve been working on two things. Firstly, a built-in authorization system
(more flexible than the current
admins configuration option) where users
and sessions can be associated with specific permissions and roles.
Secondly we’re using this authorization layer to add built-in support
for OAuth2-style authentication and authorization.
This is exciting for a number of reasons. It will allow, for example,
specialized clients to request and receive (when granted by the user)
limited access to an account without giving it your password. Some example
scenarios could be:
- Allow a third-party service to send messages via your account, but
not access your contacts or message history
- Allow an account backup/migration tool read-only access to your account
without giving it your password
- A specialized client could be granted access to only specific types of
incoming messages. For example a chess game that used your XMPP account
to communicate with a remote player, but wouldn’t be able to read your
We’re still working out the details right now, but the expectation is
that some of these features may require or be enhanced by extensions to
the XMPP protocol. Once we have some implementation experience, we will
standardize such extensions as XEPs. That way we can open the door to a
world where every XMPP client doesn’t require a password, and doesn’t
automatically get full access to your account.
The boring branches
All the above work has been happening in our trunk branch (nightly
builds are available, by the way!
). But we are also maintaining our 0.11 stable branch with bug fixes.
In 2020 we made 98 commits to the 0.11 branch, and made 4 releases. The
latest version is 0.11.7, and 0.11.8 will be coming soon.
We also have two other branches open - 0.9 (old old stable) and 0.10 (old
stable). We have a soft policy to keep supporting our branches while they
are in an active Debian release. Debian stable is on 0.11.x, and the previous
Debian release (stretch) was on 0.9.x and stopped receiving security
updates from the core Debian team in July. As such, we will be
formally deprecating these branches shortly.
In summary: if you are not on 0.11.x already, get moving :)
Not to be missed…
Through 2020 we put a lot of attention into helping increase adoption of
XMPP and helping it reach new users. This includes the launch of our
sister project, Snikket
at FOSDEM, and the backporting of Snikket’s invite-based registration to generic Prosody
Up next: Prosody 0.12
We’re currently working towards the next major release of Prosody. It’s
a lot of work to polish off all the new features and get them
sufficiently tested and documented. We are likely to start with a beta
or release candidate in the coming months.
Many thanks to all the testers of the Prosody trunk nightly builds, who
provide valuable feedback and deployment experience.
Available for hire
Prosody has been in active development for over a decade. Over this time
we’ve deliberately strived to keep the core project free from any commercial
influence or activities. We believe that this helps keep the project
focused on what the community wants, instead of simply what people with
money want :)
That said, we also need to pay our bills so we can continue to work on
Prosody. Two members of the dev team are currently available for
freelancing on a full or part-time basis for any size project. We do
strongly prefer projects related to Prosody/XMPP, and open-source if
Whether you want to hire us to integrate Prosody with your application,
review your architecture, or develop a new feature in Prosody itself, get
in touch. You can reach us at firstname.lastname@example.org.
Finally, we hope everyone is having a great start to the new year. That’s
all for now. Stay healthy!