Welcome to the XMPP Newsletter, great to have you here again! This issue covers the month of November 2022. This is the final release for this year and we will have a well-deserved winter break until the 5th of February 2023! Many thanks to all readers and all contributors!
Like this newsletter, many projects and their efforts in the XMPP community are a result of people’s voluntary work. If you are happy with the services and software you may be using, especially throughout the current situation, please consider saying thanks or help these projects! Interested in supporting the Newsletter team? Read more at the bottom.
Newsletter Contributors & Translations
This is a community effort, and we would like to thank translators for their contributions. Volunteers are welcome! Translations of the XMPP Newsletter will be released here (with some delay):
The XSF membership application period for the fourth quarter 2022 has ended. Voting results will be announced on Dec 13th.
Google Summer of Code 2022
XSF and Google Summer of Code 2022
The Google Summer of Code 2022 has finished! The two new contributors Patiga and PawBud worked on open-source software projects in the XMPP environment. Read the final collection blog posts:
ejabberd turns 20! Congratulations and many thanks to all who contributed to this great project!
ejabberd turns 20
IT-Business writes about the most important internet protocols - XMPP is part of it [DE]!
Read about how Libervia bridges XMPP and ActivityPub as the gateway developed by Goffi (thanks to a NLnet/NGI0 grant) hit alpha state. Also announced is OMEMO:2 support, for messages, files, file transfers via Jingle and all Pubsub related features.
Gajim 1.5.4 has been released, and it comes with a reworked file transfer interface, better URL detection, message selection improvements, and many fixes under the hood.
New Gajim file transfer interface
Spark 3.0.0 has been released with a new look and includes the Pade Meetings plugin for audio and video calls via Jitsi Meet.
Servers
Openfire 4.7.4 has been released, and it includes enhancements to cluster-specific
implementation of Multi-User Chat functionality, improved websocket handling and improved statistics. Furthermore from the ignite realtime community:
Developers and other standards experts from around the world collaborate on these extensions, developing new specifications for emerging practices, and refining existing ways of doing things. Proposed by anybody, the particularly successful ones end up as Final or Active - depending on their type - while others are carefully archived as Deferred. This life cycle is described in XEP-0001, which contains the formal and canonical definitions for the types, states, and processes. Read more about the standards process. Communication around Standards and Extensions happens in the Standards Mailing List (online archive).
The XEP development process starts by writing up an idea and submitting it to the XMPP Editor. Within two weeks, the Council decides whether to
accept this proposal as an Experimental XEP.
Specifies an OpenPGP for XMPP (XEP-0373) profile for the pubsub use case.
New
No new XEP this month.
Deferred
If an experimental XEP is not updated for more than twelve months, it will be moved off Experimental to Deferred. If there is another update, it
will put the XEP back onto Experimental.
No XEPs deferred this month.
Updated
No XEPs updated this month.
Last Call
Last calls are issued once everyone seems satisfied with the current XEP status. After the Council decides whether the XEP seems ready, the XMPP Editor issues a Last Call for comments. The feedback gathered during the Last Call help improving the XEP before returning it to the Council for advancement to Stable.
No Last Call this month.
Stable
No XEP moved to stable this month.
Deprecated
No XEP deprecated this month.
Call for Experience
A Call For Experience - like a Last Call, is an explicit call for comments, but in this case it’s mostly directed at people who’ve implemented, and ideally deployed, the specification. The Council then votes to move it to Final.
Looking for job offers or want to hire a professional consultant for your XMPP project? Visit our XMPP job board.
Help us to build the newsletter
This XMPP Newsletter is produced collaboratively by the XMPP community. Each month’s newsletter issue is drafted in this simple pad. At the end of each month, the pad’s content is merged into the XSF Github repository. We are always happy to welcome contributors. Do not hesitate to join the discussion in our Comm-Team group chat (MUC) and thereby help us sustain this as a community effort. You have a project and want to spread the news? Please consider sharing your news or events here, and promote it to a large audience.
Tasks we do on a regular basis:
gathering news in the XMPP universe
short summaries of news and events
summary of the monthly communication on extensions (XEPs)
The fantastic folks behind Jitsi have discovered a Denial of Service (DoS) vulnerability in Smack (JSA-2022-0002, JSA-2022-0003), which is possible if a combination of Smack components is used. The root of the vulnerability is interesting because it is due to a countermeasure against DoS attacks, namely FEATURE_SECURE_PROCESSING of the Java API for XML Processing (JAXP).
The DoS is possible because the older XMPPTCPConnection implementation of Smack parses the XMPP stream as one large XML document. Suppose the connection instance uses a parser where FEATURE_SECURE_PROCESSING is enabled. In that case, it is easy for an attacker to craft a stanza that triggers one of the various limits imposed by FEATURE_SECURE_PROCESSING, causing an exception, leaving the parser in an unrecoverable state, and closing the connection.
This vulnerability was relatively recently introduced in Smack with the addition of the support for JAXP’s Streaming API for XML (StaX) parser. Historically, Smack only used XPP3 as XML pull parser. The default implementation of XPP3 is a fast, lightweight, and, to the best of our knowledge, secure parser. XPP3 is used, for example, by Android. However, with version 4.4.0 (SMACK-591), Smack gained support for using Java’s Streaming API for XML (StAX) in addition to XPP3, to facilitate code-reuse on Java SE platforms and avoiding the XPP3 dependency.
So this DoS is possible if the XMPP connection is of type XMPPTCPConnection and if the Smack connection instance uses a StAX parser for XMPP parsing.
On a related note, Smack’s newer modular connection architecture is not affected by this, because it splits the individual top-level XMPP stream elements and parses them as standalone document. The splitting is done very early in the input processing step by XmlSplitter (of jxmpp), which also enforces size limits for the XML elements. Therefore, the DoS is not possible over connections that are established via Smack’s modern ModularXmppClientToServerConnection.
If you are affected, then the following countermeasures are possible:
Relax the FEATURE_SECURE_PROCESSING_LIMITS
Switch to XPP3 (smack-xmlparser-xpp3)
Use ModularXmppClientToServerConnection
Option A has the drawback that it is only possible to relax the limits globally. That is, it will affect XML processing regardless if Smack or some other component performs it. If you still want to go down that route, then
Gajim 1.5.4 comes with a reworked file transfer interface, better URL detection, message selection improvements, and many fixes under the hood. Thank you for all your contributions!
What’s New
Gajim’s interface for sending files has been reworked, and should be much easier to use now. For each file you’re about to send, Gajim will generate a preview. This way, you can avoid sending the wrong file to somebody. Regardless of how you start a file transfer, be it drag and drop, pasting a screen shot, or simply clicking the share button, you’ll always be able to check what you’re about to send.
Gajim’s new file transfer interface
More Changes
Performance: Chat history is now displayed quicker
Support for Jingle XTLS has been dropped, since it hasn’t been standardized
Christmas is getting closer and with that, the annual Advent of Code begins. For those who do not know, Advent of Code is a fun and inclusive event which provides a new programming puzzle every day. The fun is that these puzzles can be solved in any programming language and are accessible for varying levels of coding experience and skills. The real test is in your problem-solving. This year, we’ll be solving each of the problems in Erlang and publishing the results. We hope you enjoy it – if you’ve come up with a different solution and want to discuss it with us, we encourage you to comment on Twitter.
We are given a file with a list of values for calories in snacks carried by elves. The Elves take turns writing down the number of Calories contained in the various snacks that they’ve brought with them, one item per line. Each Elf separates its own inventory from the previous Elf’s inventory (if any) by a blank line. So the first task is to read and parse the data and decide how to represent it.
I decided to represent the input as a list of two-element tuples, where each tuple stores the elf’s number and a list of snacks. Here is an example data representation for the example file:
Now we just need to define a max function, which compares elements based on the sum of elements in a list of the calories in the snacks. I assumed that the list of elves is not empty, so I start with its first element as the current max and then started comparisons with other elves. Every time I find an elf with a bigger calorie sum I replace my current elf with the new one so that I will end up with the elf with the highest calorie total. Once the elf with the most calories is found, we can return the sum of calories. See the code: https://github.com/aleklisi/AdventOfCode2022/blob/main/day1_puzzle1/src/day1_puzzle1.erl#L48-L56 .
Day 1- Puzzle 2
The only difference in puzzle 2 compared to puzzle 1 is that now we need to find 3 elves with the most calories total instead of just 1 elf and sum their calories altogether.
We can heavily rely on solutions from puzzle 1.
To find the top 3 elves I will just:
Find the elf with the highest calories and save this elf as the first one.
Remove the first elf from the list of elves.
Find the next top elf with the highest calories from the list of elves (without the first elf) and save this elf as the second one.
Remove the second elf from the list of elves.
Find the next top elf with the highest calories from the list of elves (without the first and second elf) and save this elf as the third one.
Return the sum of calories of the first, second and third elf.
Voila!
Day 2
Day 2 of Advent of Code sees us helping the Elves to score a game of Rock, Paper, Scissors. The Elves have provided us with a strategy guide and it’s our job to help them score their game.
Day 2- Puzzle 1
To complete the task we need to calculate the results of the above games. Since the games are unrelated (the result of previous games does not impact the next games, the best way to approach the problem is to start by implementing a single game’s score count function and then map a list of games with this function, to get the scores for each of the games. To get the final score (which is a sum of the games’ scores) we then sum all of the elements of the list.
The data structure I decided to use to represent a single game is a two element tuple, where the first element is the opponent’s move and the second element is my move.
The list of games is parsed into something like this:
Looking back (after solving the problem) I could have used maps with a structure like the one below:
#{my_move = > rock, opponent_move => paper}
It might have helped me debug and avoid errors, which I did when first approaching the problem. That error was to confuse my move with my opponent’s move. In other words, I decoded A, B and C to be my moves and X, Y, and Z to be my opponent’s moves. It is an obvious mistake when you spot it, but easy oversight when reading the puzzle’s description fast. I obviously had to read the description carefully 2 more times to spot my mistake, so as we say in Polish: “the cunning one loses twice”.
Both parsing and solving today’s puzzle heavily rely on pattern matching, so let’s see that in action.
Firstly, let’s take a look at how the data is decoded using pattern matching:
% "The first column is what your opponent is going to play:
% A for Rock,
% B for Paper,
% C for Scissors.
translate_opponent_move("A") -> rock;
translate_opponent_move("B") -> paper;
translate_opponent_move("C") -> scissors.
% The second column, you reason, must be what you should play in response:
% X for Rock,
% Y for Paper,
% Z for Scissors.
translate_my_move("X") -> rock;
translate_my_move("Y") -> paper;
translate_my_move("Z") -> scissors.
A smart observer might notice that I could have used a single function to handle that translation, but I find dividing the decoding into two separate functions much more readable.
Now let’s take a look at how scoring can be conveniently calculated:
count_games_score({OpponentMove, MyMove}) ->
count_shape_score(MyMove) + count_result_score(OpponentMove, MyMove).
% The score for a single round is the score for the shape you selected (
% 1 for Rock,
% 2 for Paper,
% 3 for Scissors
count_shape_score(rock) -> 1;
count_shape_score(paper) -> 2;
count_shape_score(scissors) -> 3.
% ) plus the score for the outcome of the round (
% 0 if you lost,
% 3 if the round was a draw,
% 6 if you won
% ).
count_result_score(rock, scissors) -> 0;
count_result_score(paper, rock) -> 0;
count_result_score(scissors, paper) -> 0;
count_result_score(OpponentMove, MyMove) when MyMove == OpponentMove -> 3;
count_result_score(scissors, rock) -> 6;
count_result_score(rock, paper) -> 6;
count_result_score(paper, scissors) -> 6.
Again a keen observer might notice that it could be done in a single function, but I think most people will agree that translating the specifications one-to-one is way more convenient and much easier to understand and possibly debug if the need arises.
Puzzle two introduces a plot twist. It turns out that the second part of the input for each of the games is not what we are supposed to play, but the expected game result. We need to figure out what to play, based on what our opponent’s move is and the expected result. Notice that the game score count does not change, so if we determine what we have played based on the new understanding of input and provide parsing output to follow the same rules as we did in today’s puzzle 1 when doing the parsing, the rest of the code should work correctly without any change.
Let’s now see how to achieve that in practice.
In the `read_and_parse_data/1` function I modified the anonymous function inside a map function to translate the predicted result into my move:
And this is the implementation of the translating functions:
% "The first column is what your opponent is going to play:
% A for Rock,
% B for Paper,
% C for Scissors.
translate_opponent_move("A") -> rock;
translate_opponent_move("B") -> paper;
translate_opponent_move("C") -> scissors.
% The second column says how the round needs to end:
% X means you need to lose,
% Y means you need to end the round in a draw,
% Z means you need to win.
translate_result("X") -> lose;
translate_result("Y") -> draw;
translate_result("Z") -> win.
find_my_move(OpponentMove, draw) -> OpponentMove;
find_my_move(rock, lose) -> scissors;
find_my_move(paper, lose) -> rock;
find_my_move(scissors, lose) -> paper;
find_my_move(rock, win) -> paper;
find_my_move(paper, win) -> scissors;
find_my_move(scissors, win) -> rock.
Firstly, ALWAYS carefully read the requirements, and do not skip any part, because you find it “obvious”.
Secondly, pattern matching is a great tool to have, it allows us to easily implement readable code.
And last but not least, if you struggle or get stuck with something, it helps to add readable printing/logging to your code. When my implementation of `find_my_move/2` function (when solving puzzle 2) did not work. I added the following printing debug to the parsing data function:
OpponentMoveRaw: "A" OpponentMove: rock GameResultRaw: "X" MyMove: scissors GameResult: lose Result Score: 3
OpponentMoveRaw: "A" OpponentMove: rock GameResultRaw: "Y" MyMove: rock GameResult: draw Result Score: 4
OpponentMoveRaw: "A" OpponentMove: rock GameResultRaw: "Z" MyMove: paper GameResult: win Result Score: 8
OpponentMoveRaw: "B" OpponentMove: paper GameResultRaw: "X" MyMove: rock GameResult: lose Result Score: 1
OpponentMoveRaw: "B" OpponentMove: paper GameResultRaw: "Y" MyMove: paper GameResult: draw Result Score: 5
OpponentMoveRaw: "B" OpponentMove: paper GameResultRaw: "Z" MyMove: scissors GameResult: win Result Score: 9
OpponentMoveRaw: "C" OpponentMove: scissors GameResultRaw: "X" MyMove: paper GameResult: lose Result Score: 2
OpponentMoveRaw: "C" OpponentMove: scissors GameResultRaw: "Y" MyMove: scissors GameResult: draw Result Score: 6
OpponentMoveRaw: "C" OpponentMove: scissors GameResultRaw: "Z" MyMove: rock GameResult: win Result Score: 7
Which I found extremely helpful when finding the mistake. It turned out that instead of:
In the event that I was unable to locate my mistake, I would recommend implementing unit tests, hoping not to duplicate the mistake there.
That’s it for day 2. Come back on Monday for the solutions to the weekend’s puzzles as well as Monday’s solution.
Day 3
In today’s puzzles we are tasked with finding intersections of sets. I know that for 99% of my audience sets intersection is an obvious term, but for the 1% that do not know the intersection of sets is a fancy name for a collection of common elements of those sets.
An important note is that this definition works not only for two sets but for any number of sets.
In the diagram above, the intersection for the Sets’ is marked in green.
Day 3- Puzzle 1
In this puzzle, we need to divide each input line in half and then find the intersection (a single element) of the first and second half of each line. This can be done very simply using functions from Erlang’s sets (https://www.erlang.org/doc/man/sets.html) module:
Once we have the found intersections, we need to map them to priorities. This is one of the rare moments when using a dollar sign followed by character syntax makes the code much more readable. The one thing that we need to remember is that a letter is stored as a number, it’s ASCII code. So $a is stored internally in Erlang as number 97 etc. Having those 2 facts established we can implement mapping items (chars) to priorities:
For letters other than a, z, A and Z the code is not as obvious, but fortunately, we are provided with a description that can be translated directly into unit tests:
% Lowercase item types a through z have priorities 1 through 26.
% Uppercase item types A through Z have priorities 27 through 52.
priority_conversions_are_correct_for_edge_cases_test() ->
?assertEqual(1, item_to_priority($a)),
?assertEqual(26, item_to_priority($z)),
?assertEqual(27, item_to_priority($A)),
?assertEqual(52, item_to_priority($Z)).
% the priority of the item type that appears in both compartments of each rucksack is
% 16 (p),
% 38 (L),
% 42 (P),
% 22 (v),
% 20 (t),
% 19 (s)
priority_conversions_are_correct_test() ->
?assertEqual(16, item_to_priority($p)),
?assertEqual(38, item_to_priority($L)),
?assertEqual(42, item_to_priority($P)),
?assertEqual(22, item_to_priority($v)),
?assertEqual(20, item_to_priority($t)),
?assertEqual(19, item_to_priority($s)).
To run the tests execute: `rebar3 eunit` which should result in a message like this one:
Having the item-to-priority conversion implemented and tested is enough to get the intersections for the first and second compartments of each rucksack, map intersections to priorities and then sum priorities.
Today’s second puzzle seems to actually be simpler than the first one. The task requires us to divide the input into groups of 3 lines (Elves group), where each line represents the Elf’s rucksack content. This can be achieved by a custom recursive function:
The next step is to find the intersection of those 3 collections (which is the Elve’s group badge, a single character) and reuse the item for the priority conversion function from puzzle 1. This will reveal the priorities of the badges. Last but not least, just sum up the priorities to get the answer.
Today’s puzzles are about cleaning up the camp. From a mathematical/algorithmic perspective, it is all about numerical ranges (discrete intervals, aka with integers only) and finding common elements between them.
For the input, we are given a list of paired ranges.
Day 4- Puzzle 1
The first task is to find how many assignment pairs in one range, fully contains the other. This is a simple enough task. After parsing the data we just need to apply a filter with a custom predicate of one range containing the other.
Take a second to notice that the first range contains the second one when both of the following conditions are met:
the smallest number in the second range is bigger or equal to the smallest number in the first range
the biggest number in the second range is smaller or equal to the biggest number in the first range
Also spot that the second range might be the one containing the first one, so we have to reverse the condition (that is the part after `orelse`).
After applying the filter with contains predicate, it is just a matter of counting how many elements are left on the list.
Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day4_puzzle1
Day 4 – Puzzle 2
The second puzzle today differs from the first one only in the nature of the predicate, because the second task is to find pairs which have any common part (overlap).
The only change that needs to be made is the predicate, this time instead of `contains/1` I defined a new predicate `overlap/1`. Notice that there can be a few approaches to this problem:
Two ranges overlap when one end of one of the ranges is in the other range or one range contains the other.
The size of the intersection of the two sets of integers that those ranges contain is bigger than zero (0), in other words, the two rangers overlap if they have at least 1 common element.
Let’s see how to implement those two approaches.
Firstly, the conditions of one range containing one of the ends of the other range. Notice that instead of checking if the `contains/` predicate we could alternatively reverse the roles and check if any of the ends of the first range is in the second range.
The second approach is leveraging the erlang sets module:
Playing around with multiple implementations and checking if they all work correctly I added some unit tests at the end of the module. If you want to play around with the conditions change the code and run `rebar3 eunit`.
If you are wondering how the test cases were constructed to cover all cases (I hope I did not miss anyone). I fixed one range (5 to 7) and placed the other range in different configurations, starting from both ends of the second range being smaller and ending on both ends of the second range being bigger (the only 2 cases when the two ranges do not overlap). This is how the code doing that looks:
When there are many cases to be considered I think that the saying “a picture is worth a thousand words” applies. So I’ve provided the image above.
Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day4_puzzle2
Day 5
Today’s puzzle is about implementing an interpreter for manipulating crates between stacks. The most difficult part of today’s puzzle turned out to be parsing and loading initial stacks into a reasonable data structure.
Here is an example input file:
First, I divided the input into initial stacks of cranes and a list of commands. Since parsing the list of commands is pretty straightforward and can be done in many ways I will skip describing how to parse commands. If you are interested in looking more into how I did that, take a look at the code here.
To parse initial stacks, I read the rows, divided them into crates and if there is an empty space (for lower crates stack), replace it with an atom `empty`. Now the partially parsed crates look like this:
Next for a list of parsed rows, I picked the heads of those lists to a separate list (my first stack) and then reapplied the same function recursively for the tails of those rows (to deliver the following stacks).
Let’s see a simplified version of getting a single stack:
There were a few minor operations to be made to finally get the following data structure to represent my stacks:
This creates a map where the key is the stack’s number, and the value is a list of crates from the top crate at the beginning of a list to the bottom, the last element at this list.
Day 5 – Puzzle 1
After creating the initial stacks of crates and a list of commands parsed, we can execute the operations. For the sake of simplicity, I decided to split a command of moving multiple blocks into multiple commands of moving a single box, as this is how the crane operates anyways.
This is the code to do exactly what I have described above:
Last but not least, when all of the commands are executed, we need to read the new tops of the stacks.
This is done by changing the map of stacks back into a list of tuples with stack numbers as the first element and stack as the second one. Then, a relatively rarely used function `lists:keysort/1` comes in handy, as we need to sort those tuples by stack number.
After sorting it remains to get only the top element for each stack.
Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day5_puzzle1
Day 5 – Puzzle 2
The only modification for this puzzle is how we move the crates. Previously they were moved one by one, now all of the crates are moved at once. So, the only change that needs to be made to the code is how the command is executed. Now, I split the stack into parts using `lists:split/2` function, instead of just getting the top one element from the source stack, like it was done in today’s puzzle 1. Pay attention to the fact that now we need to concatenate a list of moving elements, instead of adding the moving elements as a single head of the destination stack like it was done in the previous puzzle.
As always if you want to discuss, or compare your solutions with us you are welcome to do so on Twitter.
Day 6
Today, both puzzles are about finding the first subsequence of N (4 in the first puzzle and 14 in the second puzzle) characters that are all different in a given input sequence.
Unfortunately using mapping does not work, because we need to process more than one element at a time. Folding does not work either (obviously I bet someone can make it work, but it is way too complicated to be interested in doing) as it always needs to process the whole list. But we want to be able to stop processing as soon as we find the expected subsequence.
Day 6- Puzzle 1
For the first puzzle today, we need to find where in the given sequence of characters ends the first subsequence of 4 unique (all are different) characters. To find the first such sequence, I recursively check if the current 4 characters are all different. If they are then I return the result position, otherwise I remove the first character from the analysed sequence, increment the position counter by one and run the search again and so on until eventually the searched section is found.
In the snippet above, a very powerful concept of using recursion with an accumulator is shown. The idea of recursion with an accumulator is basically to add another parameter to the recursive function and to store partial results in this parameter. To see the difference, let’s take a look at 2 approaches to implementing finding the Nth element of the Fibonacci sequence. Firstly the classical implementation:
I cannot disagree that at first glance the implementation with an accumulator is more complex, but much faster code. Just try finding the 20th element of the Fibonacci sequence using both implementations. And they give the same results (at least for the first few elements of the sequence) which I checked using this assertion:
Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day6_puzzle1
Day 6-Puzzle 2
The second puzzle only differs in the length of the subsequence where all elements must be different. In the first puzzle, I defined checking if all four elements are different “by hand”:
first_4_chars_are_all_different([A, B, C, D | _]) ->
A /= B andalso A /= C andalso A /= D
andalso B /= C andalso B /= D
andalso C /= D.
This would be ridiculously long and easy to make mistake code, therefore I decided that it makes much more sense to use a property of set (sets module). That is when an element is added to a set multiple times it is stored only once. In other words, sets do not keep the duplicates of elements. Knowing that, if I decide to build a set from a list of a given number of elements, the size of this set will only be equal to the length of the list it was constructed from if all of the elements on the source list were different from each other. I know it sounds complicated, but let’s see the code and everything should be clear.
Then it would work for both the first and second puzzles by just changing parameter N, setting it to 4 or 14.
Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day6_puzzle2
Day 7
Today, both puzzles are about finding directories and the sizes of files and subdirectories they contain. The biggest challenge was how to change input into data representation. The issue is mostly about using `cd ..` as when creating any recursive data structure, the inner part often does not know about the outer part. I tried 2 approaches, which failed.
I found that I was not able to traverse the tree conveniently, therefore I gave it up.
For the second approach, I was to create actual files and directories to represent the data given in the input. But then I came to the conclusion that it would be cheating, so I also gave up this idea quickly. I ended up parsing the given shell history into a list of files and directories, where each file and directory was represented by a tuple, like those:
Let’s consider input like the above. We can see that the “/” directory should consider all of the files in its size. The [“/”, “A”] directory should only consider “d.txt” file’s size to know its own size, as it never matches its path’s second element with any of the other files. And directory [“/”, “B”] matches none of the files on its second element at all.
Such comparisons can be done by a surprisingly simple function:
We already know which files are to be counted when a directory size is calculated. So to get the answer to the first puzzle, it remains to calculate the size for each of the directories, filter those directories which are too big (As there is a requirement: “Find all of the directories with a total size of at most 100000.”) and then sum the sizes of the remaining once.
Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day7_puzzle1
Day 7 – Puzzle 2
The second part of the puzzle requires finding the smallest directory to be deleted to have enough free memory to install an update of the given size.
It is needed to know how much memory is taken. Fortunately, the amount of memory taken is the size of the “/” directory. To get how much memory is already available,we subtract taken memory from the total space available. To find the smallest folder that will release enough memory, it is necessary to calculate how much memory needs to be freed. To calculate that, subtract the free memory from how much memory needs to be freed.
Then last but not least, filter directories which are too small (deleting them will not free enough memory) and from big enough directories we pick the one with the smallest size (the directory’s name is dropped as the puzzle asks only about this directory’s size).
Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day7_puzzle2
In cloud architectures (or microservices), applications are broken down into smaller independent blocks that can be quickly developed, deployed, and maintained. Imagine you have a cloud architecture that has many services and many requests per second, you have to make sure that no requests are lost and your web service is always ready to receive new requests instead of locked by processing the previous request and must ensure that the services communicate with each other smoothly and efficiently.
So how do you? How can different applications communicate with each other? The answer is Message Broker!
Nowadays, there are many Message Broker software that can be listed as AWS Simple Queue Service (SQS), Apache Kafka, Apache ActiveMQ,… But the most popular among the names listed above are RabbitMQ!
So, What is a message broker? What is RabbitMQ? Why use RabbitMQ? Your concerns will be answered by the RabbitMQ team right in the article!
What is a Message Broker?
A message broker (an integration broker or interface engine) is an intermediary module that transfers messages from sender to receiver. It is an architectural pattern for inspecting, relaying, and navigating messages; mediating between applications, simplifying communication between them, and maximizing the efficiency of splitting into smaller blocks. The main task of a Message broker is to receive messages from applications and perform some action. Let’s take a look at the sequence diagram below:
How does Message Broker work?
During the Initialize process, Service1 and Sevice2 load the proxy and register to the Broker. From there, the Broker will forward the messages to the pre-registered proxy. The advantages can see here are:
Service1 and Service2 don’t need to know each other and don’t need to speak in the same language. They just need to send the message to the proxy, and from here proxy will forward the message to the Broker. The Broker will take care of validating, transforming, and routing messages between Service1 and Service2.
With this design pattern, we can set up \an asynchronous mechanism. Service1 doesn’t need to care when a message is delivered to Service2 and when Service2 finishes handling the message, everything Sevice1 should do is send the message to the Broker, Service2 will pick up the message whenever it wants.
To put it simply, a message broker is an intermediary program developed to serve the needs of easy communication between different applications. You can also understand message broker as a message broker software program.
Some highlights about RabbitMQ
RabbitMQ is an open-source message broker. In the beginning, it was used for Advanced Message Queueing Protocol (AMQP), and after that growing up to support the Streaming Text Oriented Messaging Protocol (STOMP), Message Queuing Telemetry Transport (MQTT), and other protocols.
In a word, RabbitMQ is like an intermediary message carrier or queue manager. RabbitMQ presented in Erlang language gives programmers an intermediary means to exchange data between members of the system and securely stores the data before it is pushed to another location.
It can be understood simply, as a large-scale system, the exchange of messages between components is more and more complex. Therefore, RabbitMQ was born as an effective solution in the system structure. Not only that, but the capacity of RabbitMQ is also quite light, programmers can deploy it on both fixed and cloud environments.
RabbitMQ also supports multiple Messaging protocols, develops in a distributed, federated environment, and meets the need for scalability. RabbitMQ also provides a wide range of tools for the most popular languages such as C++, C#, Java, PHP, Python,…
The most outstanding features of RabbitMQ
As the software of choice is used so much, it must contain many outstanding features:
Interface:
RabbitMQ has a fairly simple interface, easy to use. Users can easily manage, monitor and control all problems in the programs.
Reliability:
RabbitMQ offers a variety of features to let you trade off performance with reliability, including persistence, delivery acknowledgments, publisher confirms, and high availability.
When a connection fails, messages may be in transit between the client and server. They may be stuck in the middle of being decoded or encoded on either side, stuck in the TCP buffer, … In such events, messages in transit will not be delivered, they will need to be retransmitted. The RabbitMQ Acknowledgements and Confirms feature will let the server and clients know when to do this.
RabbitMQ can detect dead TCP connections with the Heartbeats feature.
In a RabbitMQ cluster, all definitions (of exchanges, bindings, users, etc) are replicated across the entire cluster. Queues may be located on a single node, or replicate their content for higher availability.
Quorum queues is a modern replicated queue type that focuses on data safety. They provide a more understandable, in some cases less resource intensive, new implementation for achieving replicated queues and high availability.
Streams is another replicated messaging data type that allows for repeatable consumption.
Flexibility:
The message is routed through the Exchange before reaching the Queue. RabbitMQ provides some commonly used Exchange types, we can also define our own Exchange as a plugin. For more complex routing you can bind exchanges together.
RabbitMQ enables users to control the trade-offs between messages, throughput, and performance. All the messages in the queues can specify where they should be saved to a disc before their delivery. Queues in a cluster can span multiple servers while ensuring that no messages are lost in the case of a server failure.
Multi-protocol, the multi-language feature creates a diversity of users.
Lightweight:
RabbitMQ is lightweight and requires less than 40MB of RAM to run the application core and plugins like the Management UI.
High availability of queues:
With queues in RabbitMQ, users can replicate it on several different machines in the same cluster. This will help to keep the messages safe even if the hardware fails.
Traceability:
If messaging is not working properly, RabbitMQ will step in and take action. Thanks to its traceability, users can track the system’s operating status or tell if the system has any problems.
Plugins system:
RabbitMQ also supports plugin extension through many forms. If you have the ability, you can also create these utilities yourself. For more information on how to develop a RabbitMQ Plugin, please refer to the Plugin Development Basics page.
RabbitMQ can be used when the web server needs to quickly respond to requests. This eliminates the need to perform resource-intensive operations while the user waits for results. RabbitMQ is also used to transport messages to many different recipients for processing or to share the load among highly loaded workers (20K+ messages/sec).
RabbitMQ can be used for:
Applications need to support legacy protocols, such as STOMP, MQTT, AMQP, 0-9-1.
Fine-grained control over consistency/set of guarantees on a per-message basis
Complex routing to consumers
Applications need multiple to publish/subscribe, point-to-point request/reply messaging capabilities.
Who is using RabbitMQ?
RabbitMQ is an open-source tool with 10.1K GitHub stars and 3.7K GitHub forks. Here’s a link to RabbitMQ’s open-source repository on GitHub.
There are a lot of things that go into building a chat system, such as client, server, and protocol. Even for only making a client there are lots of areas of focus, such as user experience, features, and performance. To keep this post a manageable size, we will just be building a client and will use an existing server and protocol (accessing Jabber network services using the XMPP protocol). We’ll make a practical GUI so we can test things, but not spend too much time on polish, and look at getting to a useful baseline of features.
You can find all the code for this post in git. All code licensed AGPL3+.
Use a Library
As with most large programming tasks, if we wanted to do every single thing ourselves we would spend a lot more time, so we should find some good libraries. There is another reason to use a library: any improvements we make to the library benefits others. While releasing our code might help someone else if they choose to read it, a library improvement can be picked up by users of that library right away.
We need to speak the XMPP protocol so let’s choose Blather. We need a GUI so we can see this working, but don’t really want to futz with it much so let’s choose Glimmer. The code here will use these libraries and be written in the Ruby programming language, but these ideas are general purpose to the task and hopefully we won’t get too bogged down in syntax specifics.
One little language-specific thing you will need to create is a description of which ruby packages are being used, so let’s make that file (named Gemfile):
Let’s get the bare minimum: a connection to a Jabber service and a window.
client.rb
require "glimmer-dsl-libui"
require "blather/client"
BLATHER = self
include Glimmer
Thread.new do
window("Contacts") {
on_destroy {
BLATHER.shutdown
}
}
end
When required in this way, Blather will automatically set up a connection with event processing on the main thread, and will process command line arguments to get connection details. So we put the GUI on a second thread to not have them block each other. When the window is closed (on_destroy), be sure to disconnect from the server too. You can run this barely-a-client like this:
The arguments are a Jabber ID (which you can get from manyexistingservices), and the associated password.
You should get a blank window and no errors in your terminal. If you wanted to you could even look in another client and confirm that it is connected to the account by seeing it come online.
Show a Contact List
Let’s fetch the user’s contacts from the server and show them in the window (if you use this with a new, blank test account there won’t be any contacts yet of course, but still).
$roster = [["", ""]]
Thread.new do
window("Contacts") {
vertical_box {
table {
button_column("Contact") {
}
editable false
cell_rows $roster
}
}
on_destroy {
BLATHER.shutdown
}
}.show
end
after(:roster) do
LibUI.queue_main do
$roster.clear
my_roster.each do |item|
$roster << [item.name || item.jid, item.jid]
end
end
end
In a real app you would probably want some kind of singleton object to represent the contacts window and the contact list (“roster”) etc. For simplicity here we just use a global variable for the roster, starting with some dummy data so that the GUI framework knows what it will look like, etc.
We fill out the window from before a little bit to have a table with a column of buttons, one for each contact. The button_column is the first (and in this case, only) column definition so it will source data from the first element of each item in cell_rows. It’s not an editable table, and it gets its data from the global variable.
We then add an event handler to our XMPP connection to say that once the roster has been loaded from the server, we hand control over to the GUI thread and there we clear out the global variable and fill it up with the roster as we now see it. The first item in each row is the name that will be shown on the button (either item.name or item.jid if there is no name set), the second item is the Jabber ID which won’t be shown because we didn’t define that column when we made the window. Any updates to the global variable will be automatically painted into the GUI so we’re done.
One Window Per Conversation
For simplicity, let’s say we want to show one window per conversation, like so:
$conversations = {}
class Conversation
include Glimmer
def self.open(jid, m=nil)
return if $conversations[jid]
($conversations[jid] = new(jid, m)).launch
end
def initialize(jid, m=nil)
@jid = jid
@messages = [["", ""]]
new_message(m) if m
end
def launch
window("Conversation With #{@jid}") {
vertical_box {
table {
text_column("Sender")
text_column("Message")
editable false
cell_rows @messages
@messages.clear
}
horizontal_box {
stretchy false
@message_entry = entry
button("Send") {
stretchy false
on_clicked do
BLATHER.say(@jid, @message_entry.text)
@messages << [ARGV[0], @message_entry.text]
@message_entry.text = ""
end
}
}
}
on_closing do
$conversations.delete(@jid)
end
}.show
end
def format_sender(jid)
BLATHER.my_roster[jid]&.name || jid
end
def message_row(m)
[
format_sender(m.from&.stripped || BLATHER.jid.stripped),
m.body
]
end
def new_message(m)
@messages << message_row(m)
end
end
message :body do |m|
LibUI.queue_main do
conversation = $conversations[m.from.stripped.to_s]
if conversation
conversation.new_message(m)
else
Conversation.open(m.from.stripped.to_s, m)
end
end
end
Most of this is the window definition again, with a table of the messages in this conversation sourced from an instance variable @messages. At the bottom of the window is an entry box to type in text and a button to trigger sending it as a message. When the button is clicked, send that message to the contact this conversation is with, add it to the list of messages so that it shows up in the GUI, and make the entry box empty again. When the window closes (on_closing this time because it’s not the “main” window) delete the object from the global set of open conversations.
This object also has a helper to open a conversation window if there isn’t already one with a given Jabber ID (jid), some helpers to format message objects into table rows by extracting the sender and body (including format_sender which gets the roster item if there is one, uses &.name to get the name if there was a roster item or else nil, and if there was no roster item or no name just show jid) and a helper that adds new messages into the GUI.
Finally we add a new XMPP event handler for incoming messages that have a body. Any such incoming message we look up in the global if there is a conversation open already, if so we pass the new message there to have it appended to the GUI table, otherwise we open the conversation with this message as the first thing it will show.
Getting from the Contact List to a Conversation
Now we wire up the contact list to the conversation view:
button_column("Contact") {
on_clicked do |row|
Conversation.open($roster[row][1].to_s)
end
}
When a contact button is clicked, grab the Jabber ID from the hidden end of the table row that we had stashed there, and open the conversation.
And let’s provide a way to start a new conversation with an address that isn’t a contact too. An entry to type in a Jabber ID and a button that opens the conversation.
Adding a Contact
Might as well add a button to the main window that re-uses that entry box to allow adding a contact as well:
button("Add Contact") {
stretchy false
on_clicked do
BLATHER.my_roster << jid_entry.text
end
}
Handling Multiple Devices
In many chat protocols, it is common to have multiple devices or apps connected simultaneously. It is often desirable to show messages sent to or from one device on all the others as well. So let’s implement that. First, a helper for creating XML structures we may need:
We need to tell the server that we support this feature:
when_ready do
self << Blather::Stanza::Iq.new(:set).tap { |iq|
xml_child(iq, :enable, "urn:xmpp:carbons:2")
}
end
We will be handling live messages from multiple event handlers so let’s pull the live message handling out into a helper:
def handle_live_message(m, counterpart: m.from.stripped.to_s)
LibUI.queue_main do
conversation = $conversations[counterpart]
if conversation
conversation.new_message(m)
else
Conversation.open(counterpart, m)
end
end
end
And the helper that will handle messages from other devices of ours:
def handle_carbons(fwd, counterpart:)
fwd = fwd.first if fwd.is_a?(Nokogiri::XML::NodeSet)
return unless fwd
m = Blather::XMPPNode.import(fwd)
return unless m.is_a?(Blather::Stanza::Message) && m.body.present?
handle_live_message(m, counterpart: counterpart.call(m))
end
This takes in the forwarded XML object (allowing for it to be a set of which we take the first one) and imports it with Blather’s logic to become hopefully a Message object. If it’s not a Message or has no body, we don’t really care so we stop there. Otherwise we can handle this extracted message as though we had received it ourselves.
And then wire up the event handlers:
message(
"./carbon:received/fwd:forwarded/*[1]",
carbon: "urn:xmpp:carbons:2",
fwd: "urn:xmpp:forward:0"
) do |_, fwd|
handle_carbons(fwd, counterpart: ->(m) { m.from.stripped.to_s })
end
Because XMPP is just XML, we can use regular XPath stuff to extract from incoming messages. Here we say that if the message contains a forwarded element inside a carbons received element, then we should handle this with the carbons handler instead of just the live messages handler. The XML that matches our XPath comes in as the second argument and that is what we pass to the handler to get converted into a Message object.
message(
"./carbon:sent/fwd:forwarded/*[1]",
carbon: "urn:xmpp:carbons:2",
fwd: "urn:xmpp:forward:0"
) do |_, fwd|
handle_carbons(fwd, counterpart: ->(m) { m.to.stripped.to_s })
end
This handler is for messages sent by other devices instead of received by other devices. It is pretty much the same, except that we know the “other side of the conversation” (here called counterpart) is in the to not the from.
message :body do |m|
handle_live_message(m)
end
And our old message-with-body handler now just needs to call the helper.
History
So far our client only processes and displays live messages. If you close the app, or even close a conversation window, the history is gone. If you chat with another client or device, you can’t see that when you re-open this one. To fix that we’ll need to store messages persistently, and also fetch any history from while we were disconnected from the server. We will need a few more lines in our Gemfile first:
gem "sqlite3"
gem "xdg"
And then to set up a basic database schema:
require "securerandom"
require "sqlite3"
require "xdg"
DATA_DIR = XDG::Data.new.home + "jabber-client-demo"
DATA_DIR.mkpath
DB = SQLite3::Database.new(DATA_DIR + "db.sqlite3")
if DB.user_version < 1
DB.execute(<<~SQL)
CREATE TABLE messages (
mam_id TEXT PRIMARY KEY,
stanza_id TEXT NOT NULL,
conversation TEXT NOT NULL,
created_at INTEGER NOT NULL,
stanza TEXT NOT NULL
)
SQL
DB.execute("CREATE TABLE data (key TEXT PRIMARY KEY, value TEXT)")
DB.user_version = 1
end
user_version is a SQLite feature that allows storing a simple integer alongside the database. It starts at 0 if never set, and so here we use it to check if our schema has been created or not. We store the database in a new directory created according to the XDG Base Directory specification. There are two relevant IDs for most XMPP operations: the MAM ID (the ID in the server’s archive) and the Stanza ID (which was usually selected by the original sender). We also create a data table for storing basic key-value stuff, which we’ll use in a minute to remember where we have sync’d up to so far. Let’s edit the Conversation object to store messages as we send them, updating the send button on_clicked handler:
on_clicked do
m = message
EM.defer do
BLATHER << m
DB.execute(<<~SQL, [nil, m.id, @jid, m.to_s])
INSERT INTO messages
(mam_id, stanza_id, conversation, created_at, stanza)
VALUES (?,?,?,unixepoch(),?)
SQL
end
@messages << message_row(m)
@message_entry.text = ""
end
When we send a message we don’t yet know the server’s archive ID, so we set that to nil for now. We set mam_id to be the primary key, but SQLite allows multiple rows to have NULL in there so this will work. We don’t want to block the GUI thread while doing database work so we use EM.defer to move this to a worker pool. We also want to store messages when we receive them live, so add this to the start of handle_live_message:
Here we extract the server archive’s ID for the message (added by the server in a stanza-id with by="Your Jabber ID") and figure out what time the message was originally sent (usually this is just right now for a live message, but if it is coming from offline storage because every client was offline or similar, then there can be a “delay” set on it which we can use). Now that we have stored the history of message we received we need to load them into the GUI when we start up a Conversation so add this at the end of initialize:
EM.defer do
mam_messages = []
query = <<~SQL
SELECT stanza
FROM messages
WHERE conversation=?
ORDER BY created_at
SQL
DB.execute(query, [@jid]) do |row|
m = Blather::XMPPNode.import(
Nokogiri::XML.parse(row[0]).root
)
mam_messages << m
end
LibUI.queue_main do
mam_messages.map! { |m| message_row(m) }
@messages.replace(mam_messages + @messages)
end
end
In the worker pool we load up all the stored messages for the current conversation in order, then we take the XML stored as a string in the database and parse it into a Blather Message object. Once we’ve done as much of the work as we can in we worker pool we use queue_main to switch back to the GUI thread and actually build the rows for the table and replace them into the GUI.
With these changes, we are now storing all messages we see while connected and displaying them in the conversation. But what about messages sent or received by other devices or clients while we were not connected? For that we need to sync with the server’s archive, fetching messages at a reasonable page size from whatever we already have until the end.
def sync_mam(last_id)
start_mam = Blather::Stanza::Iq.new(:set).tap { |iq|
xml_child(iq, :query, "urn:xmpp:mam:2").tap do |query|
xml_child(query, :set, "http://jabber.org/protocol/rsm").tap do |rsm|
xml_child(rsm, :max, "http://jabber.org/protocol/rsm").tap do |max|
max.content = (EM.threadpool_size * 5).to_s
end
next unless last_id
xml_child(rsm, :after, "http://jabber.org/protocol/rsm").tap do |after|
after.content = last_id
end
end
end
}
client.write_with_handler(start_mam) do |reply|
next if reply.error?
fin = reply.find_first("./ns:fin", ns: "urn:xmpp:mam:2")
next unless fin
handle_rsm_reply_when_idle(fin)
end
end
The first half of this creates the XML stanza to request a page from the server’s archive. We create a query with a max page size based on the size of our worker threadpool, and ask for messages only after the last known id (if we have one, which we won’t on first run). Then we use write_with_handler to send this request to the server and wait for a reply. The reply is sent after all messages have been sent down (sent seperately, not returned in this reply, see below), but we may still be processing some of them in the worker pool so we next create a helper to wait for the worker pool to be done:
def handle_rsm_reply_when_idle(fin)
unless EM.defers_finished?
EM.add_timer(0.1) { handle_rsm_reply_when_idle(fin) }
return
end
last = fin.find_first(
"./ns:set/ns:last",
ns: "http://jabber.org/protocol/rsm"
)&.content
if last
DB.execute(<<~SQL, [last, last])
INSERT INTO data VALUES ('last_mam_id', ?)
ON CONFLICT(key) DO UPDATE SET value=? WHERE key='last_mam_id'
SQL
end
return if fin["complete"].to_s == "true"
sync_mam(last)
end
Poll with a timer until the worker pool is all done so that we aren’t fetching new pages before we have handled the last one. Get the value of the last archive ID that was part of the page just processed and store it in the database for next time we start up. If this was the last page (that is, complete="true") then we’re all done, otherwise get the next page. We need to make sure we actually start this sync process inside the when_ready handler:
last_mam_id = DB.execute(<<~SQL)[0]&.first
SELECT value FROM data WHERE key='last_mam_id' LIMIT 1
SQL
sync_mam(last_mam_id)
And also, we need to actually handle the messages as they come down from the server archive:
message "./ns:result", ns: "urn:xmpp:mam:2" do |_, result|
fwd = result.xpath("./ns:forwarded", ns: "urn:xmpp:forward:0").first
fwd = fwd.find_first("./ns:message", ns: "jabber:client")
m = Blather::XMPPNode.import(fwd)
next unless m.is_a?(Blather::Stanza::Message) && m.body.present?
mam_id = result.first["id"]&.to_s
# Can't really race because we're checking for something from the past
# Any new message inserted isn't the one we're looking for here anyway
sent = DB.execute(<<~SQL, [m.id])[0][0]
SELECT count(*) FROM messages WHERE stanza_id=? AND mam_id IS NULL
SQL
if sent < 1
counterpart = if m.from.stripped.to_s == jid.stripped.to_s
m.to.stripped.to_s
else
m.from.stripped.to_s
end
delay =
fwd.find_first("./ns:delay", ns: "urn:xmpp:delay")
&.[]("stamp")&.then(Time.method(:parse))
delay = delay&.to_i || m.delay&.stamp&.to_i || Time.now.to_i
DB.execute(<<~SQL, [mam_id, m.id, counterpart, delay, m.to_s])
INSERT OR IGNORE INTO messages
(mam_id, stanza_id, conversation, created_at, stanza)
VALUES (?,?,?,?,?)
SQL
else
DB.execute(<<~SQL, [mam_id, m.id])
UPDATE messages SET mam_id=? WHERE stanza_id=?
SQL
end
end
Any message which contains a MAM (server archive) result will get handled here. Just like with carbons we extract the forwarded message and import, making sure it ends up as a Blather Message object with a body.
Remember how when we stored a sent message we didn’t know the archive ID yet? Here we check if there is anything in our database already with this stanza ID and no archive ID, if no we will insert it as a new message, but otherwise we can update the row we already have to store the server archive ID on it, which we now know.
And with that, our client now stores and syncs all history with the server, to give the user a full view of their conversation no matter where or when it happened.
Display Names
If a user is added to the contact list with a name, we already show that name instead of their address in conversations. What if a user is not a contact yet, or we haven’t set a name for them? It might be useful to be able to fetch any display name they advertise for themselves and show that. First we add a simple helper to expose write_with_handler outside of the main object:
public def write_with_handler(stanza, &block)
client.write_with_handler(stanza, &block)
end
We need an attribute on the Conversation to hold the nickname:
attr_accessor :nickname
And then we can use this in Conversation#initialize to fetch the other side’s nickname if they advertise one and we don’t have one for them yet:
Inside the window declaration we can use this as the window title:
title <=> [self, :nickname]
and in format_sender we can use this as well:
return nickname if jid.to_s == @jid.to_s
Avatars
Names are nice, but what about pictures? Can we have nice avatar images that go with each user? What should we display if they don’t have an avatar set? Well not only is there a protocol to get an avatar, but a specification that allows all clients to use the same colours to represent things, so we can use a block of that if there is no avatar set. Let’s generate the colour blocks first. Add this to Gemfile:
This takes the SHA-1 of a string, unpacks the first two bytes as a 16-bit little-endian integer, converts the range from 0 to MAX_SHORT into the range from 0 to 360 for hue degrees, then passes to the library we added to convert from HSV to RGB colour formats. The GUI library expects images as a byte string where every 4 bytes are 0 to 255 for red, then green, then blue, then transparency. Because we want a square of all one colour, we can create the byte string for one pixel and then multiply the string by the width and height (multiplying a string by a number in Ruby make a new string with that many copies repeated) to get the whole image.
In Conversation#initialize we can use this to make a default avatar on the dummy message row then the window first opens:
@messages = [[default_avatar(""), "", ""]]
And we will need to add a new column definition to be beginning of the table { block:
image_column("Avatar")
And actually add the image to message_row:
def message_row(m)
from = m.from&.stripped || BLATHER.jid.stripped
[
$avatars[from.to_s] || default_avatar(from.to_s),
format_sender(from),
m.body
]
end
If you run this you should now see a coloured square next to each message. We would now like to get actual avatars, so add this somewhere at the top level to advertise support for this:
Then in the when_ready block make sure to send it to the server:
send_caps
And handle the avatars as they come in:
pubsub_event(
"//ns:items[@node='urn:xmpp:avatar:metadata']",
ns: "http://jabber.org/protocol/pubsub#event"
) do |m|
id = m.items.first&.payload_node&.children&.first&.[]("id")
next $avatars.delete(m.from.stripped.to_s) unless id
path = DATA_DIR + id.to_s
key = m.from.stripped.to_s
if path.exist?
LibUI.queue_main { $avatars[key] = image(path.to_s, 32, 32) rescue nil }
else
write_with_handler(
Blather::Stanza::PubSub::Items.new(:get).tap { |iq|
iq.node = "urn:xmpp:avatar:data"
iq.to = m.from
}
) do |reply|
next if reply.error?
data = Base64.decode64(reply.items.first&.payload_node&.text.to_s)
path.write(data)
LibUI.queue_main { $avatars[key] = image(path.to_s, 32, 32) rescue nil }
end
end
end
When an avatar metadata event comes in, we check what it is advertising as the ID of the avatar for this user. If there is none, that means they don’t have an avatar anymore so delete anything we may have in the global cache for them, otherwise create a file path in the same folder as the database based on this ID. If that file exists already, then no need to fetch it again, create the image from that path on the GUI thread and set it into our global in-memory cache. If the file does not exist, then use write_with_handler to request their avatar data. It comes back Base64 encoded, so decode it and then write it to the file.
If you run this you should now see avatars next to messages for anyone who has one set.
Delivery Receipts
The Internet is a wild place, and sometimes things don’t work out how you’d hope. Sometimes something goes wrong, or perhaps just all of a user’s devices are turned off. Whatever the reason, it can be useful to see if a message has been delivered to at least one of the intended user’s devices yet or not. We’ll need a new database column to store that status, add after the end of the DB.user_version < 1 if block:
if DB.user_version < 2
DB.execute(<<~SQL)
ALTER TABLE messages ADD COLUMN delivered INTEGER NOT NULL DEFAULT 0
SQL
DB.user_version = 2
end
And update the row builder to include this new data:
def message_row(m, delivered=false)
from = m.from&.stripped || BLATHER.jid.stripped
[
$avatars[from.to_s] || default_avatar(from.to_s),
format_sender(from),
m.body,
delivered,
m.id
]
end
Inbound messages are always considered delivered, since we have them:
def new_message(m)
@messages << message_row(m, true)
end
And a method to allow signalling that a delivery receipt should be displayed, using the fact that we now hide the stanza id off the end of the rows in the table to find the relevant message to update:
In the Send button’s on_clicked handler we need to actually request that others send us receipts:
m = message
xml_child(m, :request, "urn:xmpp:receipts")
And we need to handle the receipts when they arrive:
message "./ns:received", ns: "urn:xmpp:receipts" do |m, received|
DB.execute(<<~SQL, [received.first["id"].to_s])
UPDATE messages SET delivered=1 WHERE stanza_id=?
SQL
conversation = $conversations[m.from.stripped.to_s]
return unless conversation
LibUI.queue_main do
conversation.delivered_message(received.first["id"].to_s)
end
end
When we get a received receipt, we get the id attribute off of it, which represents a stanza ID that this receipt is for. We update the database, and inform any open conversation window so the GUI can be updated.
Finally, if someone requests a receipt from us we should send it to them:
message :body do |m|
handle_live_message(m)
if m.id && m.at("./ns:request", ns: "urn:xmpp:receipts")
self << m.reply(remove_children: true).tap { |receipt|
xml_child(receipt, :received, "urn:xmpp:receipts").tap { |received|
received["id"] = m.id
}
}
end
end
If the stanza has an id and a receipt request, we construct a reply that contains just the received receipt and send it.
Message Correction
Sometimes people send a message with a mistake in it and want to send another to fix it. It is convenvient for the GUI to support this and render only the new version of the message. So let’s implement that. First we add it to the list of things we advertise support for:
Then we need a method on Conversation to process incoming corrections and update the GUI:
def new_correction(replace_id, m)
row = @messages.find_index { |r| r.last == replace_id }
return unless row
@messages[row] = message_row(m, true)
end
We look up the message row on the stanza id, just as we did for delivery receipts, and just completely replace it with a row based on the new incoming message. That’s it for the GUI. Corrections may come from live messages, from carbons, or even from the server archive if they happened while we were disconnected, so we create a new insert_message helper to handle any case we previously did the SQL INSERT for an incoming message:
def insert_message(
m,
mam_id:,
counterpart: m.from.stripped.to_s,
delay: m.delay&.stamp&.to_i
)
if (replace = m.at("./ns:replace", ns: "urn:xmpp:message-correct:0"))
DB.execute(<<~SQL, [m.to_s, counterpart, replace["id"].to_s])
UPDATE messages SET stanza=? WHERE conversation=? AND stanza_id=?
SQL
else
delay ||= Time.now.to_i
DB.execute(<<~SQL, [mam_id, m.id, counterpart, delay, m.to_s])
INSERT OR IGNORE INTO messages
(mam_id, stanza_id, conversation, created_at, stanza, delivered)
VALUES (?,?,?,?,?,1)
SQL
end
end
The else case here is the same as the INSERTs we’ve been using up to this point, but we also check first for an element that signals this as a replacement and if that is the case we issue an UPDATE instead to correct our internal archive to the new version.
Then in handle_live_message we also signal the possibly-open GUI:
if (replace = m.at("./ns:replace", ns: "urn:xmpp:message-correct:0"))
conversation.new_correction(replace["id"].to_s, m)
else
conversation.new_message(m)
end
We can now display incoming corrections, but it would also be nice to be able to send them. Add a second button after the Send button in Conversation that can re-use the @message_entry box to correct the most recently sent message:
button("Correct") {
stretchy false
on_clicked do
replace_row = @messages.rindex { |message|
message[1] == format_sender(BLATHER.jid.stripped)
}
next unless replace_row
m = message
m << xml_child(m, :replace, "urn:xmpp:message-correct:0").tap { |replace|
replace["id"] = @messages[replace_row].last
}
EM.defer do
BLATHER << m
DB.execute(<<~SQL, [m.to_s, @jid, @messages[replace_row].last])
UPDATE messages SET stanza=? WHERE conversation=? AND stanza_id=?
SQL
end
@messages[replace_row] = message_row(m, @messages[replace_row][-2])
@message_entry.text = ""
end
}
When the button is clicked we find the row for the most recently sent message, construct a message to send just as in the Send case but add the message correction replace child with the id matching the stanza id of the most recently sent message. We send that message and also update our own local copy of the stanza both in the database and in the memory model rendered in the GUI.
Conclusion
There are a lot more features that a chat system can implement, but hopefully this gives you a useful taste of how each one can be incrementally layered in, and what the considerations might be for a wide variety of different kinds of features. All the code for the working application developed in this article is available in git under AGPLv3+, with commits that corrospond to the path we took here.
This release primarily enhances functionality when running in an Openfire cluster. All changes can be reviewed in the changelog for this release of the plugin.
As always, your instance of Openfire should automatically display the availability of the update. Alternatively, you can download the new release of the plugin at the HTTP File Upload plugin’s archive page.
As our work continues on modernizing XMPP authentication,
we have some more new milestones to share with you. Until now our work has
mostly been focused on internal Prosody improvements, such as the new roles
and permissions framework. Now we are starting to extend our
work to the actual client-to-server protocol in XMPP.
Prosody and Snikket are both
regularly used from mobile devices, which have intermittent connectivity. Even
if it’s only a change between networks, or when driving through a tunnel for
a few minutes, these things can temporarily break your connection - requiring
a new one to be established.
We’ve had solutions and optimizations in the XMPP protocol for this situation
for years (really… the first version of XEP-0198 was published in 2004!).
XEP-0198 allows a client to reconnect to the server as soon as the network
comes back, easily discover if anything failed to be sent/received due to the
network interruption, and then resync any lost packets in either direction.
This effectively allows resuming and repairing the session as if no disconnect
occurred, while skipping a bunch of traffic that would usually be exchanged
when establishing a new session (instead, everything is simply cached from the
old session).
However, there is one important thing we don’t allow the client to skip. To
keep this resumption step secure, we require authentication. It’s a new
connection, and we need to prove it’s from who it claims to be from.
Authentication in XMPP today
The most common authentication method for XMPP connections today is SCRAM.
This is a neat password-based authentication mechanism that has many nice
properties, such as allowing both the client and the server to store only a
hash of the password. It also allows the client to determine that the server
really knows the user’s password, and supports channel binding. These
features allow the client to detect various kinds of attack.
Even though we have been using SCRAM in XMPP for many years now, it still
offers more protective features today than the vast majority of online
services you use - which generally all send your password to the server in
plain text, albeit within TLS or HTTPS.
A new SCRAM alternative is currently being developed, known as OPAQUE, which
adds even more nice properties. But that’s for future blog post… :)
However, there are some drawbacks of SCRAM (and similar mechanisms, including
OPAQUE) that can’t realistically be solved. To adequately protect your
password, it requires some back-and-forth negotiation with the server. In
protocol speak, we refer to such situations as “round trips” - every time the
client sends something to the server and has to wait for a response before it
can proceed. On a very slow network, round trips can add a bunch of latency,
and as anyone who has used audio/video calls or gaming online knows, latency
can be frustrating and hard to eliminate from a connection.
Simpler authentication methods just have the client say “here are my
credentials”, and the server say “your credentials look great, you’re
authenticated!“. That’s how HTTP and most websites work today. Such approaches
are quick and easy, but they don’t protect your credentials as well as SCRAM
does.
Passwords are the problem
SCRAM’s protections are important for passwords. Passwords are (unfortunately)
often chosen by users to be the same or similar across multiple services, and
even if they are strong and unique they can be vulnerable to phishing. If
leaked, many memorable passwords contain private information about the user.
We don’t want to drop any of our important password security features just to
improve connection speed. So instead we found a better solution: drop
passwords!
Our new solution allows the client to log in initially using a
password (or any other method the XMPP server supports). After that, it can
upgrade to a strong unique authentication token provided by the server, which it
can use to quickly re-authenticate on future connections.
Tokens are the answer
Tokens have many advantages compared to passwords:
They are unique to the service that generated them, so cross-service attacks
like credential stuffing
are useless against tokens.
Tokens don’t need to be memorable, so they can be very long and random (both
desirable properties for increasing account security!).
As they are not memorized by the user, they can be rotated frequently
without any inconvenience.
Different tokens can be generated for each of a user’s devices, instead of
sharing the user’s password across all of them. This also allows selectively
removing a device’s access from the user’s account, e.g. if it gets lost or
stolen.
With these security advantages, we suddenly unlock the ability to use simpler
authentication mechanisms without risking the security of the user’s account
or password.
Still, we can do a bit better than just sending the token to the server as
plain text. Fortunately, just the kind of modern token authentication
method we need has already been in development by Florian Schmaus: the
SASL HT mechanism family.
HT mechanisms have the following properties:
The actual token itself is not exchanged over the connection during authentication.
And yet, the server receives proof that the client has the full correct token.
The client also receives proof that the server has the full correct token (and isn’t just impersonating the real server).
Finally, if channel binding is used, both sides receive proof that no MITM or relay attack being performed.
And… all this can be completed within a single round trip!
The protocol to achieve this has been submitted to the XSF as “Fast
Authentication Streamlining Tokens”.
It is in the acceptance queue, so doesn’t have a XEP number assigned yet.
Updating and integrating with SASL2
If FAST authentication was the only thing we had been working on recently, we
would be happy enough. But there’s more…
In collaboration with Thilo Molitor from the Monal project,
a new version of XEP-0388 (SASL 2) has been submitted.
SASL 2 was originally proposed back in 2017, and it defines a new
authentication protocol for XMPP (still based on SASL, so we can reuse all the
existing mechanisms we already have in place).
Several features of SASL 2 are very relevant to our work. For example, it
allows negotiation of session features in parallel with the authentication
process. The old way required the client to authenticate, and then proceed to
negotiate whatever features and parameters it wanted for the new session. With
SASL2 the client can provide this information at the same time it provides its
credentials. This saves yet more round trips.
As well as SASL 2, we’ve also updated a related proposal from around the same
time, XEP-0386 (Bind 2). This is also a critical piece of session
establishment that integrates with SASL 2.
With the work we’ve done across these three specifications - XEP-0388,
XEP-0386 and FAST - we’ve essentially overhauled the entire authentication and
session establishment protocol of XMPP. Even with all our additional authentication
security features, it’s now possible for a client to connect, authenticate, and resume
or create a session in a single request and response.
This post shouldn’t be taken as being entirely about performance improvements.
It’s nice to be able to (re)connect to the server in the blink of an eye. But
there are other reasons to be working on this.
As anyone who used XMPP in 2012 and 2022 knows, XMPP has been continuously
evolving as both the internet and the way people use it has changed. Over time
we have “bolted on” various features to the connection process to achieve this
evolution.
Now, with these new changes, we are bringing all these enhancements together
into a single framework that was designed for them to fit neatly into. Not
only are we reducing round trips, we are also simplifying connection
establishment for the next generation of XMPP developers.
When can I use all this?
Even though this is all cutting edge stuff, you’ll be able to use it much
sooner than you might think!
Prosody has support for the new SASL 2, Bind 2 and FAST protocols. They are
all available as community modules right now, though we intend for them to
become part of the main Prosody distribution eventually.
To get started, you’ll need a Prosody trunk nightly build,
and simply enable the following community modules:
To take advantage of the new features, you’ll need a compatible client. FAST
is already implemented in multiple clients, and will be available from
Conversations 2.11
for Android, as well as the next major versions of Monal,
Siskin and Beagle for iOS and MacOS.
Gajim already has SASL 2 implemented, and other client
developers have also already expressed an interest in support.
If you’re a client or library developer interested in supporting any of this,
we have a test server available that you are welcome to use. Just let us know!
Do remember that all this is still very new and experimental. The relevant
protocol specifications are still working their way through the XSF standards
process and there may be changes to come in the future. There may also be
undiscovered bugs. We encourage brave souls to help test it all in real world
deployments, but if your priority is keeping a stable setup, you should probably
wait a little longer before deploying any of this.
TCP Fast Open
While this post is not just about performance improvements, we’ve talked a
lot about performance improvements. Therefore it’s worth noting an extra
little side feature at this point.
Prosody trunk builds, when used with the new LuaSocket 3.1.0, support
something known as
TCP Fast Open. This is a
low-level TCP extension that allows new connections to skip a round trip, by
exchanging initial data packets while the connection is being established.
It’s disabled for servers by default on Linux, but you can enable it on most
modern systems by creating the file /etc/sysctl.d/tcp-fastopen.conf with
the contents:
net.ipv4.tcp_fastopen=3
Run systemctl restart systemd-sysctl.service to apply the changes. More
information on the sysctl configuration can be found in the Linux kernel
documentation.
In Prosody’s config, add the following in the global section:
network_settings = {
tcp_fastopen = 256;
}
Restart Prosody to apply these changes. Be aware that some networks and routers
have been reported to be incompatible with TCP Fast Open (support was removed
from Firefox for this reason). Although Linux has built-in recovery mechanisms
that should work around such issues, if you experience trouble connecting to
your server from certain networks, you may want to try turning this off again.
We’re also looking at support for TLS 1.3’s 0-RTT mode, which can be combined
with FAST authentication and TCP Fast Open to achieve full connection
establishment within a single round-trip. Pretty impressive!
Next steps
These protocol changes are yet another step on our XMPP authentication
modernization journey. With the new protocols now written and implemented, we
can start looking forward to the next milestones for the project.
In the coming months, we’ll be working on the ability to sign in to your
XMPP account from third-party clients and services without sharing your
password with them. Subscribe to our blog or Mastodon account
and keep an eye out for that future post!
Hello, it's time for a long overdue progress note.
I'll talk here about the work made on ActivityPub (AP) gateway and on end-to-end encryption around pubsub.
Oh, and if everything goes well, this blog post should be accessible from XMPP and ActivityPub (and HTTP and ATOM feed), using the same identifier goffi@goffi.org.
Forewords
The work made on the AP gateway has been possible thanks to a NLnet/NGI0 grant (with financial support from the European Commission's Next Generation Internet programme).
I especially appreciated that the team was really there to help bring the ideas to life, and not once did they get in the way: little paperwork, no unnecessary pressure, caring, contacts when help was needed, etc.
I wish there were more organizations like this one that really help develop libre projects for the common good.
So once again I want to thank them for all that.
XMPP ⬌ ActivityPub Gateway
There is probably no need to explain here what is ActivityPub, we can simply write that it is an open protocol that allows to do things that XMPP also allows doing, and that until now these 2 protocols could not communicate together. The work on the ActivityPub gateway aims to allow software implementing one of these 2 protocols to communicate as easily as possible. I firmly believe that all open protocols should be able to communicate which each other, to avoid creating more silos, proprietary software is already good enough at that.
To be useful, a gateway must use the full potential of both protocols. A simple bot transcribing messages as we see too often, using unsuitable features (such as instant messaging for blog posts), or using a very limited set of features to ensure compatibility are flaws that I have tried to avoid. Building a good gateway is a difficult and time-consuming task. If done right, the gateway should be as invisible as possible to the end user.
XMPP is featuring blogging since long before AP, however the set of features is not exactly the same. Current use of AP is clearly inspired from commercial "social" networks, and metadata such as subscribers/subscribed nodes (or followers/following in AP terms) are highlighted, feature such as like/favourite were missing in XMPP, and some implementation such as Pleroma do implement reactions. To integrate that in the gateway, I've been working on new specifications:
Pubsub Public Subscriptions: a way to publicly announce subscriptions, in an opt-in way. With this it's possible to implement followers/following features in a way respectful of privacy.
Pubsub Attachments: a generic way to attach any kind of data to a pubsub item. It's notably used to implements noticed/favourite button (see here and reactions.
Calendar Events: handling of events and all the RSVP mechanism. Libervia was handling events for years, but it was an experimental implementation, this specification is a next step in the effort to make it a standard.
Note that this XEP and the others linked below have been accepted but are not yet visible in official list.
You may wonder why there is a specification for Calendar Events… It's because the AP gateway also handles them, making it compatible with Mobilizon. The gateway may evolve in the future to support other non (micro)blogging use cases.
The gateway is now finished in terms of functionalities, however the code is clearly of an alpha quality for the moment. Now the goal in the coming months will be to stabilize and possibly implement other features if there is a demand for it.
Early adopters are encouraged to try and test it as long as they keep in mind that it's not stable. So if you do try it, I recommend keeping a separate ActivityPub account in whatever stable implementation you use at the moment, this way you can check if messages or media are missing, if there is any inconsistency or other bugs, and report them to me. If you test it, please join the XMPP room libervia@chat.jabberfr.org (click here to connect from your browser) for help and feedback. Stabilization will probably take weeks, but I hope to have it done by early 2023.
Installation instructions and details on how the conversion between protocols is done is available in the documentation and notably here
A question I've been asked a lot: yes, you can use the same identifier for XMPP (JID) and AP (WebFinger actor handle) as long as you use "simple" characters (i.e. alphanumeric ASCII chars, _, . and -). If you use something more complicated, you'll have to use the escaping mechanism explained in the doc (this is due to constraints with some AP implementations).
As for blogs on pubsub nodes (what Movim calls "communities"), I made it simple: you can use directly the name of the node that holds the blog in the local part (i.e. before the "@") of your actor handle: a blog named community_bog at the XMPP pubsub service pubsub.example.org can thus be addressed with the AP actor community_blog@pubsub.example.org. This way you can use a rather user-friendly identifier to share your blog with people who are only on ActivityPub.
This gateway should work with any XMPP server, and any client that implement blogging features (Only Libervia itself and Movim implement it for now, but I have heard that other clients are planning support for it). To enjoy the whole feature set of the gateway, the new specifications need to be implemented by the clients, so you can start to fill feature requests…
With this gateway, the door is open to have a client able to talk to the ActivityPub network, while having the feature of XMPP, including e2e encrypted private messages (e2e encrypted only if you communicate with an XMPP account, not with an AP one).
Oh, and please update your graphics, drawing and other texts to include XMPP in the fediverse ;)
End-to-End Encryption
Much effort has also gone into end-to-end encryption.
OMEMO implementation has been updated (OMEMO:2 is now used), including Stanza Content Encryption which allows encrypting arbitrary elements instead of only the \<body/> of the message, I believe that Libervia is the first XMPP client to implement it. OpenPGP for XMPP (or "OX") has also been implemented, all that thanks to the work of Tim Henkes "Syndace", the author of python-omemo.
Beside instant messaging, end-to-end encryption has also been introduced to pubsub. I've made specifications for two methods:
An OpenPGP profile for pubsub which is thought to encrypt a whole node, with a system of secret sharing/rotation/revocation. With it, it is easy to give access to new entities after publication, and to retrieve old items for newcomers. This specification can be used to encrypt any pubsub based features: (micro)blogging, calendar events, lists, etc.
Pubsub Targeted Encryption which is a way to apply the same cryptographic system used in instant messaging to pubsub. This way, OMEMO can be used with its forward secrecy property. It is not a good option to use this specification to encrypt a whole node, as archive is then not accessible to newcomers, and to add access to a new entity you have to re-encrypt all items, but it's an interesting option to encrypt an element occasionally, for instance to restrict access of a specific post in an otherwise public blog.
Specifications have also been written to sign a pubsub item in a backward compatible way (client which don't implement those specifications can still work normally):
All those specifications are already implemented in Libervia, but they are only usable from CLI frontend at the moment. All you have to do is to use the --encrypt and/or --sign options from pubsub or blog commands (check documentation for details).
Uploaded files were already encrypted with OMEMO Media Sharing which is what is commonly used these days, but this method has not been accepted as a standard as it was a workaround for limitation of legacy OMEMO implementation. The proper way is now specified with Stateless File Sharing and is encrypted with Encryption For Stateless File Sharing. Those methods are currently only usable when OMEMO:2 is implemented in the peer client, and with them metadata on the shared file can be attached, including thumbnails.
Encryption has also been implemented for Jingle (XEP-0391 and XEP-0396), which is notably used for Jingle File Transfer (specially useful for large files transfers).
So to summarize, nearly everything (instant messaging, files uploaded, large file transfers, all pubsub related features) can now be e2e encrypted with Libervia.
Possible Future
With the AP gateway permitting to reach the whole AP network, all the new features implemented, and the work done on e2e encryption, Libervia has everything to be a solid option for communication. After the recent events regarding a famous commercial network, we see a breakthrough of ActivityPub that will hopefully last over time. We can now access AP from XMPP, while having the possibility to have e2e encrypted private conversations or even blogs or calendar events.
As far as I know this is, so far, something unique for a Libre decentralized software. However, there is still work to do on stabilization on UI/UX update before this is really usable.
Those feature were planned for very long (years), but the lack of resources made them slow to come. The grant has made it possible to greatly accelerate the pace of development, and I doubt that it would have been possible to have all that without it.
Regarding how large the project is, and my family life, it's not possible any more to develop seriously this project on my free time alone (and I would like to do other things, sometimes, of my free time).
In other words, I need to find a way to sustain the development of Libervia for the years to come, so I can work full-time on it, and with some luck, build a team. I'm thinking very seriously about it these days, I'll probably write on this topic in a little while. If you are willing to help in any way, please contact me (on the Libervia room linked above for instance).
That's all for this progress note. I'm now working on stabilization and UI/UX update on the web frontend.
Earlier today, we have released version 2.4.0 of the Openfire Monitoring Service plugin. This plugin adds both statistics, as well as message archiving functionality to Openfire.
In this release, compatibility with future versions of Openfire is added. A bug that affects MSSQL users has been fixed, and the dreaded “Unable to save XML properties” error message has been resolved. A few other minor tweaks have been added.
As always, your instance of Openfire should automatically display the availability of the update. Alternatively, you can download the new release of the plugin at the Monitoring plugin’s archive page.
ejabberd is a piece of software that was born 20 years ago. This is a long time, even at the scale of Internet. And yet, what ejabberd represents has not always been obvious. It took us a long time to realize what was so important about ejabberd. Why have we been developing it for 20 years? Why are we pushing it further even today? What makes it so special?
ejabberd is a scalable messaging server. That sums it all and that does not do justice to this critical piece of the Internet infrastructure. Sure, it is known to be the most scalable XMPP server, so scalable that it was used as a building brick to build Whatsapp messaging service. This is something that we have always been proud of, something you can easily brag about when meeting your friends.
But is that just it? Of course not. Today, with the troubles at Twitter, something appeared clearly.
ejabberd is important because it helped build much more than Whatsapp or any other big name high-profile projects we have built. It is important because it makes people communicate, in a federated way. It is important because it implements open protocols, and now several of them: XMPP, MQTT, SIP and now Matrix.
It’s about federation
ejabberd is about federation. It is helping people on different servers, domains, companies, communities or even countries chatting together. And today even more than 20 years ago, it really matters. We have built ejabberd for 20 years, because it is a critical building brick of what makes the Internet exists. Openness, interoperability, federation. It is one of the few software that prosper outside of the spotlights and make the Internet what it is, along for example with web and mail servers.
This is something we are pondering as we are thinking about the next steps, the next 20 years. But deep down, we know for sure, what we are about. ejabberd is about federation. You will read more from us here soon. It is a tradition. No birthday celebration speech is complete without looking back at the past.
It is hard to track all ejabberd usage, but we know that ejabberd empowers more than a billion users. Not bad for a piece of code we wrote. Trillions of messages went through our lines of code.
Closed protocols come and go – ejabberd and XMPP remains
Happy 20th birthday, ejabberd!
Brief timeline
The very first public commit in ejabberd’s source code was done by Alexey Shchepin the 16th November of 2002. That was in the Jabber.ru CVS server. Later when that machine had technical problems, the development code moved to JabberStudio CVS.
The first official ejabberd release was ejabberd 0.5 in November 2003. The ejabberd home page at that time was a simple HTML. It’s also worth checking the early stage of the Ejabberd Installation and Operation Guide. Notice this first ejabberd logo represented a frog-like animal sitting on a “Jabber globe bulb,” with bat wings, dangerous-looking cogs, and an Erlang suit.
After the 0.7.5 release in October 2004, ejabberd home page moved from JabberStudio to ejabberd.jabber.ru, and the bug tracker to Jabber.ru’s Bugzilla. For this Drupal site, the logo changed to a hedgehog, and that would remain ten years until the final website and logo update in 2015.
At the beginning of 2005, JabberStudio CVS had technical difficulties and the development code moved to ProcessOne SVN. Notice that ProcessOne contracted Alexey to work in J-EAI, a project based in ejabberd specially designed for some business usages, and later extended that relationship to ejabberd.
In February 2005 the source code repository moved from SVN to Git, and the bug tracker to JIRA. Around October 2010 the source code repository and the bug tracker were finally moved to GitHub.
The ejabberd code base got a relevant massive change with the data binarization (use Erlang binaries instead of Erlang strings for data representation) in March 2013, which jumped ejabberd version from 2.1.12 to 13.03.
The next years followed another major source code change: the movement of many C/C++ code to independent external libraries.
ejabberd keeps on improving at a steady pace and is happy to open to other protocols and communities.
Some source code statistics
The oldest unchanged function in ejabberd is probably one of the least used: stop/0. And the oldest functional line is the SETS macro definition.
The ejabberd repository got 1,070,325 line insertions and 901,287 line deletions. When counting both ejabberd and the dependency libraries, they got 1.693.180 line insertions and 1.108.873 line deletions. With all this, the ejabberd source code went from 13 files to 868, from 1,448 lines of code to 480,961.
Looking at the programming languages of ejabberd and its libraries, Erlang is obviously the major one, and C comes as a relevant second:
The Ignite Realtime community is happy to announce the release of Spark 3.0.0 version.
We decided to increase major version to 3.x to coincide with a complete UI refresh of Spark which was contributed by Amos . Now Spark uses only FlatLaf Look and Feel. We are very much grateful for his incredible work. Along that Pade Meetings plugin was added by Dele. Spark now uses the latest version of the Smack 4.4 library, thanks Flow. Now Spark works on any Java 8 and more. Improved file transfer compared to Spark 2.9.4 and also updated the HttpFileUpload plugin for file transfer.
Pade Meetings (formerly SparkMeet) plugin is included in this release. You can find it in a chat window, a button with P letter. This plugin works with Openfire Meetings and enables audio and video chat. It was modified by Dele Olajide to use Electron technology. First time a user logs in using this version Spark will download required libraries into user’s profile (it is around 150 MB on Windows, so can take a minute to download). You can also manage all plugins in Spark via Client Control Pluigin and disable them if you want.
Installers:
For Windows has been added 64bit installer and MSI installer, which will be useful for use in a corporate environment.
For MacOS has been replaced the installer because modern versions require a digital signature.
The release also contains bug fixes and many improvements.
Full list of changes can be found in the changelog .
We encourage users and developers to get involved with Spark project by providing feedback in the forums or submitting pull requests on our GitHub page.
You can download Spark from the Downloads page. Below are the sha256 checksums:
Stateless file sharing (sfs) is a generic file sharing message which, alongside metadata, sends a list of sources where the file can be retrieved from.
It is generic in the sense, that sources can be from different kinds of file transfer methods.
HTTP, Jingle and any other file transfers can be encapsulated with it.
The big idea is that functionality can be implemented for all file transfer methods at once, thanks to this wrapper.
Source Attaching
The idea is simple: Once somebody started sfs, anybody can contribute more sources to the list of sources.
A sfs-attachment consists of the message id of the original sfs, as well as a list of sources that should be attached.
You might ask: “But why? I have used many messengers and never heard of such a feature, why should somebody else in the chat be able to do that?”
I was surprised as well, but there actually are some reasonable use cases.
Peer-to-Peer file sharing in groups
Reviving dead download links
Announcing file uploads before the upload is finished
Security Considerations
We obviously have to prevent the possibility of other members attaching wrong files to other people’s sfs.
To prevent this, sources can only be attached to sfs, which have checksums in their metadata.
Clients need to dismiss source attachments,
if the sfs didn’t provide a checksum
if the checksum algorithm is not trusted or unknown
if the checksum of the downloaded file doesn’t match
Google Summer of Code Retrospect
This was the first time with Google Summer of Code and honestly, it’s been a nice ride.
I’ve come to enjoy my stay and take pride in finishing the different components.
When I started earlier this year, I had developed a bit of an Imposter syndrome.
For one I wasn’t sure if I understood the full scope of the project, but I was also uncertain if I had the skill to work on it.
Confidence came over time, when I managed to get an overview of everything that needed to be done.
Everything suddenly looks much more manageable, when you understand the purpose and inner workings of each subcomponent that will be required over the way.
Of course, it’s not completely straight forward.
I often had to realize that I will need another component somewhere, but one thing at a time is manageable.
Since I had my mentor to back me up whenever I got stuck, working on the project didn’t get frustrating.
Blog posts
Parallel to working on the project, we were expected to publish blog posts.
Originally I was going for a weekly interval, but yea…
I don’t think writing the blog posts helped me in any meaningful way.
However, I do think writing blogs is a good habit and hope that
I improved my blog writing skills
The blogs will someday help someone
Progress
The GSoC period has already finished.
See my pull request.
The Ignite Realtime Community is happy to announce the 4.7.4 release of Openfire. This release fixes a number of bugs and represents our effort to provide a stable 4.7.x series while work continues on the next feature release of Openfire.
Notable fixes include enhancements to cluster-specific implementation of Multi-User Chat functionality, improved websocket handling and improved statistics. Additionally, the dependency on Apache Commons Text has been upgraded, to prevent statistic analyzers raising unneeded alarms with regards to its recently published CVE.
You can find download artifacts available here with the following sha256sum values
If you have any questions, please stop by our community forum or our live groupchat. We are always looking for volunteers interested in helping out with Openfire development!
The Ignite Realtime community is happy to announce the immediate availability of version 2.6.1 of the Hazelcast plugin for Openfire! The Hazelcast plugin is what allows you to deploy Openfire as a clustered solution.
This release includes only one improvement, but that one can bring a significant performance improvement as compared to older versions of the plugin.
The updated plugin should become available for download in your Openfire admin console in the course of the next few hours. Alternatively, you can download the plugin directly, from the plugin’s archive page.
We are happy to announce the immediate availability of version 1.10.1 of the REST API plugin for Openfire!
This is a bugfix release, that will improve the endpoints that make modifications to MUC rooms.
The updated plugin should become available for download in your Openfire admin console in the course of the next few hours. Alternatively, you can download the plugin directly, from the plugin’s archive page.
ejabberd is already the most versatile and scalable messaging server. In this post, we are giving a sneak peak at what is coming next.
ejabberd just get new ace in it sleeve – you can now use ejabberd to talk with other Matrix servers, a protocol sometimes used for small corporate server messaging.
Of course, you all know ejabberd supports the XMPP instant messaging protocol with hundreds of XMPP extensions, this is what it is famous for.
The second major protocol in ejabberd is MQTT. ejabberd support MQTT 5 with clustering, and is massively scalable. ejabberd can be used to implement Internet of Things projects, using either XMPP or MQTT and it also supports hybrid workflow, where you can mix humans and machines exchanging messages on the same platform.
It also supports SIP, as you can connect to ejabberd with a SIP client, so that you can use a softphone directly with ejabberd for internal calls.
So far, so good, ejabberd leading both in terms of performance and number of messaging protocol it supports.
We always keep an eye on new messaging protocol. Recently, the Matrix protocol emerged as a new way to implement messaging for the small corporate servers.
Of course, by design, the Matrix protocol cannot scale as well as XMPP or MQTT protocols. At the heart of Matrix protocol, you have a kind of merging algorithm that reminds a bit of Google Wave. It means that a conversation is conceptually represented as a sort document you constantly merge on the server. This is a consuming process that is happening on the server for each message received in all conversations. That’s why Matrix has the reputation to be so difficult to scale.
Even if it is not as scalable as XMPP, we believe that we can make Matrix much more scalable than what it is now. That’s what we are doing right now.
As a first step, we have been working on implementing a large subset of the Matrix protocol as a bridge in ejabberd.
It means that an ejabberd server will be able to act as a Matrix server in the Matrix ecosystem. XMPP users will be able to exchange messages with Matrix users, transparently.
To do that, we implemented the Matrix protocol for conversations and the server-to-server protocol to allow interop between XMPP and Matrix protocol.
This feature coming first for our customers, in the coming weeks, whether they are using ejabberd Business Edition internally or on Fluux ejabberd SaaS platform. It will come later to ejabberd Community Edition.
Welcome to the XMPP Newsletter, great to have you here again! This issue covers the month of October 2022.
Like this newsletter, many projects and their efforts in the XMPP community are a result of people’s voluntary work. If you are happy with the services and software you may be using, especially throughout the current situation, please consider saying thanks or help these projects! Interested in supporting the Newsletter team? Read more at the bottom.
Newsletter Contributors & Translations
This is a community effort, and we would like to thank translators for their contributions. Volunteers are welcome! Translations of the XMPP Newsletter will be released here (with some delay):
The XSF membership application period for the fourth quarter 2022 is currently open. If you are interested in becoming a XSF member then you can apply for membership. Please submit by November 27, 2022.
Google Summer of Code 2022
XSF and Google Summer of Code 2022
The Google Summer of Code 2022 has finished! The two new contributors Patiga and PawBud worked on open-source software projects in the XMPP environment. Read the final collection blog posts:
Berlin XMPP Meetup (remote): monthly meeting of XMPP enthusiasts in Berlin, every 2nd Wednesday of the month
Capitole du Libre: Members of the “JabberFR” and “Libre en Communs” associations will be present in Toulouse on the 19-20th of November
Articles
ejabberd has added Matrix protocol support bringing their experience of scaling XMPP and MQTT to this new messaging avenue. At first a subset of the protocol will be implemented, acting like a bridge between protocols. Bussiness customers will get a taste of it first and later it will come to the Community Edition.
The JMP Newsletter details a big new release of the Cheogram Android client with features including XHTML-IM, Bits of Binary, a form of message retraction, contact tagging, and more. They also talk about an XMPP channel integration they have built for the Chatwoot support software, and offer to let community projects use their instance to try it out. This month they also wrote an article about SMS Account Verification and how it affects the ecosystem.
A new automated testing technique was recently adopted to help the Prosody developers during their daily development work: Mutation Testing in Prosody
Modernization of the authentication protocol is ongoing around the community, keyword “SASL”. Multiple hints of the work have been seen in the last two newsletters, in libs, XEPs updates and funding news. It spans multiple clients, multiple platforms and at least one server. Thilo Molitor, the Monal developer, blogged about the current broken state, the ideal properties of the system and the solutions being worked on. It’s a 30 minutes long read but will sure come handy as more news of the SASL implementations will start to appear from the other involved projects.
Software news
Clients and applications
Converse has released version 10.0.0 with a lot of polishing and an updated feature set.
Dino 0.3.1 has been released, which is a maintenance release for libsoup3 support and bug fixes.
Gajim 1.5.2 and 1.5.3 have been released. These releases bring another performance boost, better emojis, selection of multiple messages, and many bug fixes.
Monal 5.3.3 has been live in the AppStore for a while now, and even if it does not yet bring the SASL improvements mentioned above or the much awaited audio and video calls capabilities funded last month, it does feature several fixes for registrations, screenshots sharing and read markers.
aTalk has been released with an almost monthly cadence. Each new version brings more polish, like location/map views, calls, libraries updates and keeping up to date on the newer Android requirements. It’s biggest adoption impediment unfortunately still stays and stems from an outdated UI that precedes Android 5 Material UI.
Servers
ejabberd 22.10 release includes six months of work, over 140 commits, including relevant improvements in MIX, MUC, SQL, and installers, and bug fixes as usual.
Jackal 0.62.3 is out bringing 3 fixes for storage/archive and s2s stanzas.
Developers and other standards experts from around the world collaborate on these extensions, developing new specifications for emerging practices, and refining existing ways of doing things. Proposed by anybody, the particularly successful ones end up as Final or Active - depending on their type - while others are carefully archived as Deferred. This life cycle is described in XEP-0001, which contains the formal and canonical definitions for the types, states, and processes. Read more about the standards process. Communication around Standards and Extensions happens in the Standards Mailing List (online archive).
The XEP development process starts by writing up an idea and submitting it to the XMPP Editor. Within two weeks, the Council decides whether to
accept this proposal as an Experimental XEP.
Specifies an OpenPGP for XMPP (XEP-0373) profile for the pubsub use case.
New
No new XEP this month.
Deferred
If an experimental XEP is not updated for more than twelve months, it will be moved off Experimental to Deferred. If there is another update, it
will put the XEP back onto Experimental.
No XEPs deferred this month.
Updated
No XEPs updated this month.
Last Call
Last calls are issued once everyone seems satisfied with the current XEP status. After the Council decides whether the XEP seems ready, the XMPP Editor issues a Last Call for comments. The feedback gathered during the Last Call help improving the XEP before returning it to the Council for advancement to Stable.
No Last Call this month.
Stable
No XEP moved to stable this month.
Deprecated
No XEP deprecated this month.
Call for Experience
A Call For Experience - like a Last Call, is an explicit call for comments, but in this case it’s mostly directed at people who’ve implemented, and ideally deployed, the specification. The Council then votes to move it to Final.
Looking for job offers or want to hire a professional consultant for your XMPP project? Visit our XMPP job board.
Help us to build the newsletter
This XMPP Newsletter is produced collaboratively by the XMPP community. Therefore, we would like to thank Adrien Bourmault (neox), anubis, Anoxinon e.V., Benoît Sibaud, cpm, daimonduff, emus, Gooya, Holger, IM, Ludovic Bocquet, martin, MattJ, MSavoritias (fae,ve), nicfab, Pierre Jarillon, Sam Whited, TheCoffeMaker, wh0nix, vanitasvitae, wurstsalat, Zash for their support and help in creation, review, translation and deployment. Many thanks to all contributors and their continuous support!
Each month’s newsletter issue is drafted in this simple pad. At the end of each month, the pad’s content is merged into the XSF Github repository. We are always happy to welcome contributors. Do not hesitate to join the discussion in our Comm-Team group chat (MUC) and thereby help us sustain this as a community effort. You have a project and want to spread the news? Please consider sharing your news or events here, and promote it to a large audience.
Tasks we do on a regular basis:
gathering news in the XMPP universe
short summaries of news and events
summary of the monthly communication on extensions (XEPs)
Welcome to the latest edition of your pseudo-monthly JMP update!
In case it’s been a while since you checked out JMP, here’s a refresher: JMP lets you send and receive text and picture messages (and calls) through a real phone number right from your computer, tablet, phone, or anything else that has a Jabber client. Among other things, JMP has these features: Your phone number on every device; Multiple phone numbers, one app; Free as in Freedom; Share one number with multiple people.
October saw the release of Cheogram Android 2.10.10-3, the largest release in awhile. The app now stores data de-duplicated, so if you send or receive the same file multiple times only one copy will be stored. This also lays the groundwork for some new file transfer improvements that will be coming in the future. The app also now supports displaying rich text messages sent by clients which support that (such as Gajim), including the image protocol needed to display stickers sent by Movim users. A form of message retraction is also supported, and should work with most Jabber clients out there. A reminder that message retraction is a social convention and not a security feature – the target still has a full copy of your un-retracted message if they want it.
We know lots of you have big contact lists, across multiple accounts, and that’s why this release introduces the ability to edit tags on your contacts and a tag navigation widget integrated into contact search: to make finding the right conversation just a little bit easier. We would love to hear feedback about this UI and how well it works for you.
For those of you who start a lot of group texts, there is an easy way to do that built into the app now as well. When you start a “private group chat” and select only @cheogram.com contacts, you will be prompted to ask if you meant to start a group text instead. This flow seemed necessary, as many have accidentally created private channels with their SMS contacts instead of the intended group text, so checking at this point was likely to be necessary anyway.
There are also some smaller quality of life improvements in this release, including the ability to copy any link in a message to the clipboard (not just the first one), dumping app logs to a special directory on your device after every call in order to make debugging issues easier, asking if you want to keep app data on uninstall (to make switching back and forth to custom builds possible without always needing export/re-import), a new first-start welcome screen, performance improvements, and more.
As JMP grows so does our support load. Up until this month we have been managing all our support requests using normal Jabber clients (mostly Gajim and Dino), which worked well enough but less and less well as we grew. It would be hard to know if someone else was handling a request, who had previously handled a request, or even what the status of some requests were (if they had been resolved elsewhere in the public channel or otherwise). We’re a small enough team that we can just talk to each other to solve these things, but that does take time, and more time as there are more things to talk out. So this month we built an XMPP channel integration for Chatwoot and have migrated our main support infrastructure to a Cheogram-hosted instance. So far we like this a lot, and so much we’ve decided to share. If you have a project that handles support using Jabber (or SMS with JMP!) you can use it on the Cheogram Chatwoot instance. Just come by the chatroom and let us know you’re interested. Only the XMPP channel works on our instance for now, but we’d be happy to enable other channels as well if that would be useful.
And finally, we know many of you are excited about the JMP Data Plan. Roll out to the waiting list has gone a bit slower than we hoped, but many SIMs did go out in October. There have been some bumps as you might expect with any test phase, but overall things are looking good and we hope to speed up the roll out and even move on to the next phase soon.
To learn what’s happening with JMP between newsletters, here are some ways you can find out:
Gajim 1.5.3 brings back a feature many of you missed: selecting and copying multiple messages. Emoji shortcodes have been improved and cover even more emojis now. Gajim also lets you mark workspaces as read, so you don’t have to go through all conversations. Thank you for all your contributions!
What’s New
Since we changed the way Gajim displays messages in Gajim 1.4, selecting multiple messages to copy them was not possible anymore. With Gajim 1.5.3 you can now select multiple messages via message menu, followed by marking each message you would like to copy.
Thanks to @mjk, Gajim now offers even more emojis when using shortcodes. You will now get suggestions for categories as well, which makes finding the right emoji easier 🎉
Emoji shortcodes
Gajim now fully supports message drafts. If you type a message and switch to another chat, you will now see your draft in the chat list as well.
More Changes
You can now select a default workspace for new chats (for each account)
Some apps and services (but not JMP!) require an SMS verification code in order to create a new account. (Note that this is different from using SMS for authentication; which is a bad idea since SMS can be easily intercepted, are not encrypted in transit, and are vulnerable to simple swap scams, etc.; but has different incentives and issues.) Why do they do this, and how can it affect you as a user?
Tarpit
In the fight against service abuse and SPAM, there are no sure-fire one-size-fits-all solutions. Often preventing abusive accounts and spammers entirely is not possible, so targets turn to other strategies, such as tarpits. This is anything that slows down the abusive activity, thus resulting in less of it. This is the best way to think about most account-creation verification measures. Receiving an SMS to a unique phone number is something that is not hard for most customers creating an account. Even a customer who does not wish to give out their phone number or does not have a phone number can (in many countries, with enough money) get a new cell phone and cell phone number fairly quickly and use that to create the account.
If a customer is expected to be able to pass this check easily, and an abuser is indistiguishable from a customer, then how can any SMS verification possibly help prevent abuse? Well, if the abuser needs to create only one account, it cannot. However, in many cases an abuser is trying to create tens of thousands of accounts. Now imagine trying to buy ten thousand new cell phones at your local store every day. It is not going to be easy.
“VoIP Numbers”
Now, JMP can easily get ten thousand new SMS-enabled numbers in a day. So can almost any other carrier or reseller. If there is no physical device that needs to be handed over (such as with VoIP, eSIM, and similar services), the natural tarpit is gone and all that is left is the prices and policies of the provider. JMP has many times received requests to help with getting “10,000 numbers, only need them for one day”. Of course, we do not serve such customers. JMP is not here to facilitate abuse, but to help create a gateway to the phone network for human beings whose contacts are still only found there. That doesn’t mean there are no resellers who will work with such a customer, however.
So now the targets are in a pickle if they want to keep using this strategy. If the abuser can get ten thousand SMS-enabled numbers a day, and if it doesn’t cost too much, then it won’t work as a tarpit at all! So many of them have chosen a sort of scorched-earth policy. They buy and create heuristics to guess if a phone number was “too easy” to get, blocking entire resellers, entire carriers, entire countries. These rules change daily, are different for every target, and can be quite unpredictable. This may help when it comes to foiling the abusers, but is bad if you are a customer who just wants to create an account. Some targets, especially “big” ones, have made the decision to lose some customers (or make their lives much more difficult) in order to slow the abusers down.
De-anonymization
Many apps and services also make money by selling your viewing time to advertisers (e.g. ads interspersed in a social media feed, as pre-/mid-roll in a video, etc.) based on your demographics and behaviour. To do this, they need to know who you are and what your habits are so they can target the ads you see for the advertisers’ benefit. As a result, they have an incentive to associate your activity with just one identity, and to make it difficult for you to separate your behaviour in ways that reduce their ability to get a complete picture of who you are. Some companies might choose to use SMS verification as one of the ways they try to ensure a given person can’t get more than one account, or for associating the account (via the provided phone number) with information they can acquire from other sources, such as where you are at any given time.
Can I make a new account with JMP numbers?
The honest answer is, we cannot say. While JMP would never work with abusers, and has pricing and incentives set up to cater to long-term users rather than those looking for something “disposable”, communicating that to every app and service out there is a big job. Many of our customers try to help us with this job by contacting the services they are also customers of; after all, a company is more likely to listen to their own customers than a cold-call from some other company. The Soprani.ca project has a wiki page where users keep track of what has worked for them, and what hasn’t, so everyone can remain informed of the current state (since a service may work today, but not tomorrow, then work again next week, it is important to track success over time).
Part of why we can’t say whether you can make a new account with JMP numbers is because the reasons some companies choose to use SMS verification are opaque, so we may not know all of their criteria for sure.
Many customers use JMP as their only phone number, often ported in from their previous carrier and already associated with many online accounts. This often works very well, but everyone’s needs are different. Especially those creating new personas which start with a JMP number find that creating new accounts at some services for the persona can be frustrating to impossible. It is an active area of work for us and all other small, easy-access phone network resellers.
This ejabberd 22.10 release includes six months of work, over 140 commits, including relevant improvements in MIX, MUC, SQL, and installers, and bug fixes as usual.
This version brings support for latest MIX protocol version, and significantly improves detection and recovery of SQL connection issues.
There are no breaking changes in SQL schemas, configuration, or commands API. If you develop an ejabberd module, notice two hooks have changed: muc_subscribed and muc_unsubscribed.
A more detailed explanation of those topics and other features:
Erlang/OTP 19.3
You may remember than in the previous ejabberd release, ejabberd 22.05, support for Erlang/OTP 25 was introduced, even if 24.3 is still recommended for stable deployments.
It is expected that around April 2023, GitHub Actions will remove Ubuntu 18 and it will not be possible to run automatic tests for ejabberd using Erlang 19.3, the lowest possible will be Erlang 20.0.
For that reason, the planned schedule is:
ejabberd 22.10
Usage of Erlang 19.3 is discouraged
Anybody still using Erlang 19.3 is encouraged to upgrade to 24.3, or at least 20.0
ejabberd 23.05 (or later)
Support for Erlang 19.3 is deprecated
Erlang requirement softly increased in `configure.ac`
Announce: no warranty ejabberd can compile, start or pass the Common Tests suite using Erlang 19.3
Provide instructions for anybody to manually re-enable it and run the tests
ejabberd 23.xx+1 (or later)
Support for Erlang 19.3 is removed completely in the source code
New log_burst_limit_* options
Two options were added in #3865 to configure logging limits in case of high traffic:
log_burst_limit_window_time defines the time period to rate-limit log messages by.
log_burst_limit_count defines the number of messages to accept in that time period before starting to drop them.
Support ERL_DIST_PORT option to work without epmd
The option ERL_DIST_PORT is added to ejabberdctl.cfg, disabled by default.
When this option is set to a port number, the Erlang node will not start epmd and will not listen in a range of ports for erlang connections (typically used for ejabberdctl and for clustering). Instead, the erlang node will simply listen in the configured port number.
Please note:
Erlang/OTP 23.1 or higher is required to use ERL_DIST_PORT
make relive doesn’t support ERL_DIST_PORT, neither rebar3 nor elixir
To start several ejabberd nodes in the same machine, configure a different port in each node
Support version macros in captcha_cmd option
Support for the @VERSION@ and @SEMVER@ macros was added to the captcha_cmd option in #3835.
Those macros are useful because the example captcha scripts are copied in a path like ejabberd-VERSION/priv/bin that depends on the ejabberd version number and changes for each release. Also, depending on the install method (rebar3 or Elixir’s mix), that VERSION may be in XX.YY or in SEMVER format (respectively).
Two hooks have changed: muc_subscribed and muc_unsubscribed. Now they get the packet and room state, and can modify the sent packets. If you write source code that adds functions to those hooks, please notice that previously they were ran like:
being Packet1b a copy of Packet1a without the jid attribute in the muc_subscribe element.
Translations Updates
Several translations were improved: Ukrainian, Chinese (Simplified), French, German, Russian, Portuguese (Brazil), Spanish and Catalan. Thanks to all this people that contribute in ejabberd at Weblate!
WebAdmin page for external modules
A new page is added in ejabberd’s WebAdmin to view available external modules, update their source code, install, upgrade and remove them. All this is equivalent to what was already available using API commands from the modules tag.
Many modules in the ejabberd-contrib git repository have been improved, and their documentation updated. Additionally, those modules are now automatically tested, at least compilation, installation and static code analysis.
Documentation Improvements
In addition to the normal improvements and fixes, two sections in the ejabberd Documentation are greatly improved:
This is part 2 of a small series on verifying the validity of packet sequences using tools from theoretical computer science. Read part 1 here.
In the previous blog post I discussed how a formal grammar can be transformed into a pushdown automaton in order to check if a sequence of packets or tokens is part of the language described by the grammar. In this post I will discuss how I implemented said automaton in Java in order to validate OpenPGP messages in PGPainless.
In the meantime, I made some slight changes to the automaton and removed some superfluous states. My current design of the automaton looks as follows:
If you compare this diagram to the previous iteration, you can see that I got rid of the states “Signed Message”, “One-Pass-Signed Message” and “Corresponding Signature”. Those were states which had ε-transitions to another state, so they were not really useful.
For example, the state “One-Pass-Signed Message” would only be entered when the input “OPS” was read and ‘m’ could be popped from the stack. After that, there would only be a single applicable rule which would read no input, pop nothing from the stack and instead push back ‘m’. Therefore, these two rule could be combined into a single rule which reads input “OPS”, pops ‘m’ from the stack and immediately pushes it back onto it. This rule would leave the automaton in state “OpenPGP Message”. Both automata are equivalent.
One more minor detail: Since I am using Bouncy Castle, I have to deal with some of its quirks. One of those being that BC bundles together encrypted session keys (PKESKs/SKESKs) with the actual encrypted data packets (SEIPD/SED). Therefore when implementing, we can further simplify the diagram by removing the SKESK|PKESK parts:
Now, in order to implement this automaton in Java, I decided to define enums for the input and stack alphabets, as well as the states:
public enum InputAlphabet {
LiteralData,
Signature, // Sig
OnePassSignature, // OPS
CompressedData,
EncryptedData, // SEIPD|SED
EndOfSequence // End of message/nested data
}
public enum StackAlphabet {
msg, // m
ops, // o
terminus // #
}
public enum State {
OpenPgpMessage,
LiteralMessage,
CompressedMessage,
EncryptedMessage,
Valid
}
Note, that there is no “Start” state, since we will simply initialize the automaton in state OpenPgpMessage, with ‘m#’ already put on the stack.
We also need an exception class that we can throw when OpenPGP packet is read when its not allowed. Therefore I created a MalformedOpenPgpMessageException class.
Now the first design of our automaton itself is pretty straight forward:
public class PDA {
private State state;
private final Stack<StackAlphabet> stack = new Stack<>();
public PDA() {
state = State.OpenPgpMessage; // initial state
stack.push(terminus); // push '#'
stack.push(msg); // push 'm'
}
public void next(InputAlphabet input)
throws MalformedOpenPgpMessageException {
// TODO: handle the next input packet
}
StackAlphabet popStack() {
if (stack.isEmpty()) {
return null;
}
return stack.pop();
}
void pushStack(StackAlphabet item) {
stack.push(item);
}
boolean isEmptyStack() {
return stack.isEmpty();
}
public boolean isValid() {
return state == State.Valid && isEmptyStack();
}
}
As you can see, we initialize the automaton with a pre-populated stack and an initial state. The automatons isValid() method only returns true, if the automaton ended up in state “Valid” and the stack is empty.
Whats missing is an implementation of the transition rules. I found it most straight forward to implement those inside the State enum itself by defining a transition() method:
public enum State {
OpenPgpMessage {
@Overrides
public State transition(InputAlphabet input, PDA automaton)
throws MalformedOpenPgpMessageException {
StackAlphabet stackItem = automaton.popStack();
if (stackItem != OpenPgpMessage) {
throw new MalformedOpenPgpMessageException();
}
swith(input) {
case LiteralData:
// Literal Packet,m/ε
return LiteralMessage;
case Signature:
// Sig,m/m
automaton.pushStack(msg);
return OpenPgpMessage;
case OnePassSignature:
// OPS,m/mo
automaton.push(ops);
automaton.push(msg);
return OpenPgpMessage;
case CompressedData:
// Compressed Data,m/ε
return CompressedMessage;
case EncryptedData:
// SEIPD|SED,m/ε
return EncryptedMessage;
case EndOfSequence:
default:
// No transition
throw new MalformedOpenPgpMessageException();
}
}
},
LiteralMessage {
@Overrides
public State transition(InputAlphabet input, PDA automaton)
throws MalformedOpenPgpMessageException {
StackAlphabet stackItem = automaton.popStack();
switch(input) {
case Signature:
if (stackItem == ops) {
// Sig,o/ε
return LiteralMessage;
} else {
throw new MalformedOpenPgpMessageException();
}
case EndOfSequence:
if (stackItem == terminus && automaton.isEmptyStack()) {
// ε,#/ε
return valid;
} else {
throw new MalformedOpenPgpMessageException();
}
default:
throw new MalformedOpenPgpMessageException();
}
}
},
CompressedMessage {
@Overrides
public State transition(InputAlphabet input, PDA automaton)
throws MalformedOpenPgpMessageException {
StackAlphabet stackItem = automaton.popStack();
switch(input) {
case Signature:
if (stackItem == ops) {
// Sig,o/ε
return CompressedMessage;
} else {
throw new MalformedOpenPgpMessageException();
}
case EndOfSequence:
if (stackItem == terminus && automaton.isEmptyStack()) {
// ε,#/ε
return valid;
} else {
throw new MalformedOpenPgpMessageException();
}
default:
throw new MalformedOpenPgpMessageException();
}
}
},
EncryptedMessage {
@Overrides
public State transition(InputAlphabet input, PDA automaton)
throws MalformedOpenPgpMessageException {
StackAlphabet stackItem = automaton.popStack();
switch(input) {
case Signature:
if (stackItem == ops) {
// Sig,o/ε
return EncryptedMessage;
} else {
throw new MalformedOpenPgpMessageException();
}
case EndOfSequence:
if (stackItem == terminus && automaton.isEmptyStack()) {
// ε,#/ε
return valid;
} else {
throw new MalformedOpenPgpMessageException();
}
default:
throw new MalformedOpenPgpMessageException();
}
}
},
Valid {
@Overrides
public State transition(InputAlphabet input, PDA automaton)
throws MalformedOpenPgpMessageException {
// Cannot transition out of Valid state
throw new MalformedOpenPgpMessageException();
}
}
;
abstract State transition(InputAlphabet input, PDA automaton)
throws MalformedOpenPgpMessageException;
}
It might make sense to define the transitions in an external class to allow for different grammars and to remove the dependency on the PDA class, but I do not care about this for now, so I’m fine with it.
Now every State has a transition() method, which takes an input symbol and the automaton itself (for access to the stack) and either returns the new state, or throws an exception in case of an illegal token.
Next, we need to modify our PDA class, so that the new state is saved:
public class PDA {
[...]
public void next(InputAlphabet input)
throws MalformedOpenPgpMessageException {
state = state.transition(input, this);
}
}
Now we are able to verify simple packet sequences by feeding them one-by-one to the automaton:
// LIT EOS
PDA pda = new PDA();
pda.next(LiteralData);
pda.next(EndOfSequence);
assertTrue(pda.isValid());
// OPS LIT SIG EOS
pda = new PDA();
pda.next(OnePassSignature);
pda.next(LiteralData);
pda.next(Signature);
pda.next(EndOfSequence);
assertTrue(pda.isValid());
// COMP EOS
PDA pda = new PDA();
pda.next(CompressedData);
pda.next(EndOfSequence);
assertTrue(pda.isValid());
You might say “Hold up! The last example is a clear violation of the syntax! A compressed data packet alone does not make a valid OpenPGP message!”.
And you are right. A compressed data packet is only a valid OpenPGP message, if its decompressed contents also represent a valid OpenPGP message. Therefore, when using our PDA class, we need to take care of packets with nested streams separately. In my implementation, I created an OpenPgpMessageInputStream, which among consuming the packet stream, handling the actual decryption, decompression etc. also takes care for handling nested PDAs. I will not go into too much details, but the following code should give a good idea of the architecture:
public class OpenPgpMessageInputStream {
private final PDA pda = new PDA();
private BCPGInputStream pgpIn = ...; // stream of OpenPGP packets
private OpenPgpMessageInputStream nestedStream;
public OpenPgpMessageInputStream(BCPGInputStream pgpIn) {
this.pgpIn = pgpIn;
switch(pgpIn.nextPacketTag()) {
case LIT:
pda.next(LiteralData);
...
break;
case COMP:
pda.next(CompressedData);
nestedStream = new OpenPgpMessageInputStream(decompress());
...
break;
case OPS:
pda.next(OnePassSignature);
...
break;
case SIG:
pda.next(Signature);
...
break;
case SEIPD:
case SED:
pda.next(EncryptedData);
nestedStream = new OpenPgpMessageInputStream(decrypt());
...
break;
default:
// Unknown / irrelevant packet
throw new MalformedOpenPgpMessageException();
}
boolean isValid() {
return pda.isValid() &&
(nestedStream == null || nestedStream.isValid());
@Override
close() {
if (!isValid()) {
throw new MalformedOpenPgpMessageException();
}
...
}
}
The key thing to take away here is, that when we encounter a nesting packet (EncryptedData, CompressedData), we create a nested OpenPgpMessageInputStream on the decrypted / decompressed contents of this packet. Once we are ready to close the stream (because we reached the end), we not only check if our own PDA is in a valid state, but also whether the nestedStream (if there is one) is valid too.
This code is of course only a rough sketch and the actual implementation is far more complex to cover many possible edge cases. Yet, it still should give a good idea of how to use pushdown automata to verify packet sequences Feel free to check out my real-world implementation here and here.
If you are early in the process of learning Elixir or considering learning it in the future, you may have wondered a few things. What is the experience like? How easy is it to pick up functional and concurrent programming concepts when coming from a background in languages which lack those features? Which aspects of the language are the most challenging for newcomers to learn?
In this article, I will relate my experience as a new Elixir developer, working to implement the dice game Yatzy as my first significant project with the language.
So far in my education and career, I have worked primarily with Java.
This project was my first extensive exposure to concepts such as recursive functions, concurrent processes, supervision trees, and finite state machines, all of which will be covered in more depth throughout this article.
The rules of Yatzy
Yatzy is a variation of Yahtzee, with slight but notable differences to the rules and scoring. Players take turns rolling a set of five dice. They have the option to choose any number of their dice to re-roll up to two times each turn. After this, they must choose one of fifteen categories to score in. The “upper half” of the scorecard consists of six categories- “ones” through “sixes”. The score for each simply is the sum of all dice with the specified number. The “lower half”, consisting of the remaining nine categories, has more specific requirements, such as “two pairs”, “three of a kind”, “full house”, etc..
If a player’s total score in the upper half is equal to or greater than 63, they receive a 50-point bonus. The player with the highest total across the whole scorecard once all categories have been filled wins the game.
Requirements of the project
Given this ruleset, a functioning implementation of Yatzy would need to do the following:
Simulate dice rolls, including those where certain dice are kept for subsequent rolls
Calculate the score a roll would result in for each category
Save each player’s scorecard throughout the entire game
Determine the winner at the end of the game,
Allow the players to take these actions via a simple UI.
Due to my object-oriented background, my approach to this project in prior years would be to define classes to represent relevant concepts, such as the player, the scorecard, and the roll, and maintain the state via instances of these objects.
Additionally, I would make sure of iteration via loops to traverse data structures. Working with Elixir requires these problems to be tackled in different ways. The concepts are instead represented by processes that can be run concurrently, and data structures are traversed with recursive functions.
Adapting to this different structure and way of thinking was the most challenging and rewarding part of this project.
Score calculations and pattern matching
My first step in writing the project was to implement functions for rolling a set of five dice and calculating the potential scores of those dice rolls in each available category. The dice roll itself was fairly simple, but makes use of a notable feature of Elixir that I had not previously encountered: setting a default argument for a function.
In this instance, the roll function takes a single argument, ‘keep’, representing the dice from a previous roll that the player has chosen to keep.
def roll(keep \\ []) do
dice = 1..6
number_of_dice = 5 - Enum.count(keep)
func = fn -> Enum.random(dice) end
roll = Stream.repeatedly(func) |> Enum.take(number_of_dice)
keep ++ roll
end
Here ‘keep’ has a default value of an empty list that will be used if ‘roll’ is called with no arguments, as it would be for the first roll in any turn. If a list is passed to ‘roll’, the function will only generate enough new numbers to fill out the rest of the roll, and then combine this list with ‘keep’ for its final output. This allowed my code to be simpler, defining one function head that could be used in multiple different scenarios.
The score calculations themselves were far more complex and required making use of Elixir’s pattern-matching capabilities.
In this case, testing for a valid score in each category required accounting for every possible configuration the dice could appear in when passed into the function. I was able to greatly reduce the number of cases by ensuring the dice were sorted descending when passed, but this still left a lot to account for. However, Elixir’s pattern matching makes this process easier than it would be otherwise: the cases can be handled entirely in the function heads, and each function can be written in a single line:
def two_pairs([x, x, y, y, _]) when x != y, do: x * 2 + y * 2
def two_pairs([x, x, _, y, y]) when x != y, do: x * 2 + y * 2
def two_pairs([_, x, x, y, y]) when x != y, do: x * 2 + y * 2
def two_pairs(_roll), do: 0
Processes and GenServers
The next step of building the game was to implement processes, starting with those for the player and the scorecard. Processes in Elixir are vital for maintaining state and allowing concurrency – as many of them can be run simultaneously. I was able to set up a process for each player in the game -one for the scorecard belonging to each of those players, as well as one more to handle the score calculations.
As processes are dissimilar to the object-oriented model, they were the aspect of Elixir that took me the longest time to adjust to. I became comfortable with them by first learning how to work with raw processes, in order to better understand the theory behind them. After this, I converted these processes into GenServers, which contain improved functionality and handle most of the client/server interactions automatically.
The supervision tree
Another benefit of GenServers over raw processes is that they can be used as part of a supervision tree. In Elixir, a supervisor is a process that monitors other processes and restarts them if they crash. A supervision tree is a branching structure consisting of multiple supervisors and their child processes. In my Yatzy application, the supervision tree consists of a head supervisor with the scoring process as a child, along with another child supervisor for each player in the game. Each of these player supervisors has two children: a player and a scorecard.
Due to supervisors being syntactically similar to GenServers, the majority of this step of the process was simple. I had already learned how to implement the relevant API and callback functions, however, one mistake that took some time to notice was accidentally using GenServer.start_link instead of Supervisor.start_link in the API for the player supervisor. This problem was particularly hard to diagnose as it resulted in no compile or runtime errors in the application but did result in the supervisor’s child processes not starting and the game not functioning.
Finite state machine
After setting up the supervision tree, I still needed to define one more process to handle the functions for running through a single player’s turn. This process was implemented as another child of the head supervisor. As this process needed to handle multiple different states representing different stages of the turn, I constructed it as a finite state machine using the GenStateMachine module.
In this case, I defined four states, representing how many rolls are remaining in the turn: three, two, one, and none. It contains functions handling calls that represent a roll of the dice, which will set the machine to its next state, and functions that will reset it to its initial state for the end of the turn, including if the player decides not to use all their rolls.
Below is an example of one of the calls, representing a player making their second roll in a turn.
def handle_event({:call, from}, {:roll, keep}, :two_rolls, data) do
data = data ++ keep
{:next_state, :one_roll, data, [{:reply, from, data}]}
end
Compared to learning how to work with GenServers and Supervisors, this functionality was actually rather simple to pick up. I had never worked with finite state machines in other languages, but the examples of GenStateMachine in the Elixir documentation were easy to understand and contained all the information I needed in order to implement this process.
User interface and recursion
Once the required processes were in place in a supervision tree, it was time to implement a simple text-based interface allowing a full game of Yatzy to be played all the way through.
This would require each player in turn to receive the results of a dice roll, be prompted to choose which, if any, dice to keep for their subsequent rolls, and then again prompt them to choose which category to score in for that turn. It should loop through the players in this way until the game is complete, at which point it should declare the winner and prompt the user to reset the scorecards and play again.
Implementing the interface was the most complex and time-consuming part of the project. This required a significant amount of trial-and-error and researching through the Elixir docs, in order to get something functioning. However, one aspect that was easier than expected was working with recursive functions. I had rarely used recursion while working in Java due to the language’s focus on iterative loops, and as such never became fully comfortable with the technique. Implementing the interface required me to use recursion in several different places, and I was surprised at how easy it was to pick up in this language, with the pattern matching on function parameters making it simple to account for the end of the loop. The following is one of the recursive functions I implemented, which maps the results of a dice roll to the letters a, b, c, d, and e, allowing the player to pick which of the five they want to keep in the text-based interface.
def map_dice([head | tail], indexes) do
index = String.to_atom(head)
key_in_indexes = Map.has_key?(indexes, index)
case index do
index when key_in_indexes ->
value = Map.get(indexes, index)
[value | map_dice(tail, indexes)]
_index ->
[map_dice(tail, indexes)]
end
end
Future
Although my Yatzy implementation is currently functioning correctly, I plan to extend the project further in the future. In the current version, only three players are supported, with their names hard-coded into the program. I would like future versions to have a dynamic amount of players, along with the ability for the players to specify their own usernames.
Additionally, I am also planning to learn the basics of Phoenix LiveView in the near future. Once I have done this, I would like to write a frontend for the program, allowing the players to interact with a more readable, visually appealing graphical interface, rather than the current text-based version.
Conclusion
Overall, I would describe my experience with the project as positive and feel that it served as a good introduction to Elixir.
I was able to learn many of the basic features of the language naturally in order to fulfill the requirements of the game, and adjusted my ways of thinking about programming to better suit working with functional and concurrent programs. As a result, I feel like I have a good understanding of the basics of Elixir, and I am more confident about my ability to carry out other work with the language in the future.
It is an exciting time for the Elixir community. As you may have seen at ElixirConf or ElixirConf EU, we are celebrating the 10th anniversary of Elixir. Despite now being 10 years old, there is no slowdown in the number of exciting new features, frameworks, and improvements being made to the language.
One of the most exciting developments for Elixir is undoubtedly Phoenix. It is a project that is growing in both features and uses cases at an incredible pace. Phoenix 1.5 included some huge changes including the addition of LiveView to the framework, the creation of LiveDashboard, and the new version of PubSub (2.0.).
Next Phoenix 1.6 introduced even more exciting features, most notably the HEEx engine, the authentication and mailer generators, better integration with LiveView, and the removal of node and webpack, which was replaced with a more simplified esbuild tool.
For many of us, each new Phoenix framework release brings back the feeling of being a kid on Christmas, we wait with eager anticipation for Chris McCord to announce the new toys we have to play with for the upcoming year, but with these new toys also comes a challenge for those who want to keep their skills and their systems up-to-date. The migration nightmare. We will revisit that at the end of this post.
Roadmap
Since Phoenix 1.5 it is a noticeable trend to move into LiveView, as we progress, LiveView can replace more and more JavaScript code, allowing the Elixir developer to get better control of the HTML generation. In the latest release, this trend is continued with the following new features:
Verified Routes. This gives us the ability to define paths using a sigil that checks compilation time compared to defined routes.
Tailwind. In addition to answering our prayers concerning JavaScript and HTML, this new version also helps manage CSS.
Component-based generators. These features offer us a new and better way to write components.
Authentication generation code using LiveView. This lets us generate the code for the authentication code but using LiveView instead of the normal controllers, views, and templates.
We will go deeper into each of these features, but you can already see a trend, right? We are moving more and more to LiveView in the same way we are removing the need to manage things like HTML, JavaScript, and CSS.
First, let’s look more at LiveView specifically, for release 0.18, Chris McCord announced these improvements:
Declarative assigns/slots – which let us define information about attributes and slots which are included inside of the components.
HTML Formatter – which performs the format (mix format) for HEEx code even if it’s included inside of the sigil ~H.
Accessibility building blocks.
Now let’s look at each of these elements in deeper detail.
Verified Routes
The story is that Jason Stiebs (from the Phoenix team) has been requesting a better, less verbose way, to use the routes for the last 8 years. The 12th time he requested it Chris McCord agreed to this feedback and José Valim had a fantastic way to make that happen.
The basic idea is that if we have this:
This is generating the route which we could use in this way:
This is very verbose, but it could be even worse if we have a definition of the routes nested like this one:
And it is just as verbose when we use LiveView:
To avoid this, the Verified Routes provides us a shortcut using the path:
As you can see, using the sigil “~p” we can define the path where we want to go and it’s completely equivalent to using the previous Routes helper function.
The main advantage of this feature is that it allows us to write the path concisely and still check if that route is valid or not in the same way we would use the Route Helper function.
Tailwind
To understand this change let’s look at what Adam Wathan (creator of Tailwind) said about CSS and the use of CSS:
The use of CSS in a traditional way, that is using “semantic class names”, is hard to maintain and that’s why he created Tailwind. Tailwind is based on the specification of how the element should be shown. There can be different elements that are semantically the same, for example, two “Accept” buttons where we want one to appear big and the other a bit narrower. Under this paradigm, we’d be forced to use the class “accept-button” in addition to the classes which are modifying this case and which do not allow them to be reused.
The other approach is to implement small modifications to how we present the buttons. In this way, we can define a lot in HTML and get rid of the CSS.
The main idea, as I said previously, is to replace as much CSS as possible in the same way as LiveView replaced a lot of JavaScript:
For example, using Tailwind with HTML and getting rid of CSS, we could build a button like this one with the code shown in the image below:
It could be argued that it’s complex, but it’s indeed perfect from the point of view of LiveView and components because these classes can be encapsulated inside of the component and we can use it in this way:
And finally, in the template:
Easy, right?
Authentication generation code using LiveView
Big thank you to Berenice Medel on the Phoenix team, she had the great idea to have the generation of the authentication templates work with LiveView.
Declarative Assigns / Slots
Before going into this section, Chris McCord gave a big thank you to Marius Saraiva and Connor Lay. They are the people in charge of all of the improvements regarding declarative assigns, slots, and HEEx.
The idea behind slots and attrs is to provide us with a way to define attributes and sub-elements inside of a defined component. The example above, it’s defining a component with the name “table”. It’s defining the attributes “row_id” and “rest”, as you can see in the documentation, the attributes for the table are “rows”, “row_id”, and “class”. That means we can find “row_id”, then “rest” will feature a map with all of the remaining attributes.
As we said, the slot is a way to indicate we are going to use a sub-element “col” inside of the “table”. In the example, you can see two elements “col” inside of “table”. The “col” element has only defined one attribute “if” which is a boolean.
HTML Formatter
A big thank you to Felipe Renan who worked on the implementation of this for HEEx to be included in Phoenix. Now, it’s possible to have a “mix format” fixing the format of the code written inside of the templates, even inside of the ~H sigil.
Accessibility building blocks
Phoenix 1.7 includes some primitives for helping to create more accessible websites. One of them is “focus_wrap”:
This helps define the areas where you want to shift focus between multiple elements inside of a defined area instead of a whole website.
This works in combination with functions in the JS module which configure the focus like a stack. When you go into the modal it pushes the focus area that we use and when the modal is closed, we pop out from that area of the stack and stay with the previous one.
More improvements in the Roadmap
One of the improvements for LiveView is Storybook. Storybook is a visual UI creator which lets us define the components we want to be included in our websites and then generate the code to be implemented for it. Christian Blavier did great work starting this in his repository but he’s now off and the Phoenix team is going to be moving it forward and evolving it.
Streaming data for optimized handling of collections data is another priority in the roadmap. The work for this has already started, fingers are crossed that it might be announced for the next release.
During recent conferences, another speaker raised a concern about the messaging incompatibility between LiveView and LiveComponent, luckily, this is on the roadmap to be fixed shortly.
And is that all?
With all the developments in Phoenix, it would be easy to talk about at much greater length and in much greater detail. The pace of the Phoenix team’s progress is impressive and exciting.
As it continues to grow it is easy to imagine a future where we only need to write HEEx code inside of Elixir to get full control of generated HTML, CSS, and JavaScript for the browser. It’s exciting to imagine and will be sure to further grow the use and adoption of Elixir as a full-stack technology.
Read to adopt Elixir? Or need help with your implementation? Or contact us about our training options.
Feel free to check out my real-world implementation 
