Planet Jabber

December 05, 2022

The XMPP Standards Foundation

The XMPP Newsletter November 2022

Welcome to the XMPP Newsletter, great to have you here again! This issue covers the month of November 2022. This is the final release for this year and we will have a well-deserved winter break until the 5th of February 2023! Many thanks to all readers and all contributors!

Like this newsletter, many projects and their efforts in the XMPP community are a result of people’s voluntary work. If you are happy with the services and software you may be using, especially throughout the current situation, please consider saying thanks or help these projects! Interested in supporting the Newsletter team? Read more at the bottom.

Newsletter Contributors & Translations

This is a community effort, and we would like to thank translators for their contributions. Volunteers are welcome! Translations of the XMPP Newsletter will be released here (with some delay):

  • English (original): xmpp.org
    • General contributors: emus, Licaon_Kter, Ludovic Bocquet, MattJ, MSavoritias (fae,ve), wurstsalat, Zash
  • French: jabberfr.org and linuxfr.org
    • Translators: Adrien Bourmault (neox), alkino, anubis, Benoît Sibaud, Pierre Jarillon, Ppjet6, Ysabeau
  • German: xmpp.org and anoxinon.de
    • Translators: Jeybe, wh0nix
  • Italian: nicfab.it
    • Translators: nicfab
  • Spanish: xmpp.org
    • Translators: daimonduff, TheCoffeMaker

XSF Announcements

  • The XSF membership application period for the fourth quarter 2022 has ended. Voting results will be announced on Dec 13th.

Google Summer of Code 2022

XSF and Google Summer of Code 2022

XSF and Google Summer of Code 2022

The Google Summer of Code 2022 has finished! The two new contributors Patiga and PawBud worked on open-source software projects in the XMPP environment. Read the final collection blog posts:

XSF fiscal hosting projects

The XSF offers fiscal hosting for XMPP projects. Please apply via Open Collective. For more information, see the announcement blog post. Current projects:

Events

Articles

All outstanding Google Summer of Code articles are now complete and have been published. See above ^

From the JMP blog, an article about building an XMPP client from scratch, including worked examples of implementing many XEPs. Read also their Newsletter: New Cheogram Android Release, Chatwoot Instance

ejabberd turns 20! Congratulations and many thanks to all who contributed to this great project!

ejabberd turns 20

ejabberd turns 20

IT-Business writes about the most important internet protocols - XMPP is part of it [DE]!

Read about how Libervia bridges XMPP and ActivityPub as the gateway developed by Goffi (thanks to a NLnet/NGI0 grant) hit alpha state. Also announced is OMEMO:2 support, for messages, files, file transfers via Jingle and all Pubsub related features.

Software news

Clients and applications

Conversations 3.0 has been announced and is being supported by the NLnet foundation

Gajim 1.5.4 has been released, and it comes with a reworked file transfer interface, better URL detection, message selection improvements, and many fixes under the hood.

New Gajim file transfer interface

New Gajim file transfer interface

Spark 3.0.0 has been released with a new look and includes the Pade Meetings plugin for audio and video calls via Jitsi Meet.

Servers

Openfire 4.7.4 has been released, and it includes enhancements to cluster-specific implementation of Multi-User Chat functionality, improved websocket handling and improved statistics. Furthermore from the ignite realtime community:

As mentioned in the October 2022 Newsletter more info on the SASL2 related work starts to appear. The Prosody server developers talk about Bringing FASTer authentication to Prosody and XMPP and how “tokens” are the answer.

Libraries & Tools

python-omemo 1.0.2 has been released with support for the new SCE-based OMEMO. Interesting enough the Libervia progress above is based on this work.

And OMEMO news in the Go corner too, Mellium to receive grant for end-to-end encryption from the NGI Assure Fund.

Extensions and specifications

Developers and other standards experts from around the world collaborate on these extensions, developing new specifications for emerging practices, and refining existing ways of doing things. Proposed by anybody, the particularly successful ones end up as Final or Active - depending on their type - while others are carefully archived as Deferred. This life cycle is described in XEP-0001, which contains the formal and canonical definitions for the types, states, and processes. Read more about the standards process. Communication around Standards and Extensions happens in the Standards Mailing List (online archive).

xmpp.org features a page about XMPP RFCs as well.

Proposed

The XEP development process starts by writing up an idea and submitting it to the XMPP Editor. Within two weeks, the Council decides whether to accept this proposal as an Experimental XEP.

New

  • No new XEP this month.

Deferred

If an experimental XEP is not updated for more than twelve months, it will be moved off Experimental to Deferred. If there is another update, it will put the XEP back onto Experimental.

  • No XEPs deferred this month.

Updated

  • No XEPs updated this month.

Last Call

Last calls are issued once everyone seems satisfied with the current XEP status. After the Council decides whether the XEP seems ready, the XMPP Editor issues a Last Call for comments. The feedback gathered during the Last Call help improving the XEP before returning it to the Council for advancement to Stable.

  • No Last Call this month.

Stable

  • No XEP moved to stable this month.

Deprecated

  • No XEP deprecated this month.

Call for Experience

A Call For Experience - like a Last Call, is an explicit call for comments, but in this case it’s mostly directed at people who’ve implemented, and ideally deployed, the specification. The Council then votes to move it to Final.

  • No Call for Experience this month.

Spread the news!

Please share the news on other networks:

Subscribe to the monthly XMPP newsletter
Subscribe

Also check out our RSS Feed!

Looking for job offers or want to hire a professional consultant for your XMPP project? Visit our XMPP job board.

Help us to build the newsletter

This XMPP Newsletter is produced collaboratively by the XMPP community. Each month’s newsletter issue is drafted in this simple pad. At the end of each month, the pad’s content is merged into the XSF Github repository. We are always happy to welcome contributors. Do not hesitate to join the discussion in our Comm-Team group chat (MUC) and thereby help us sustain this as a community effort. You have a project and want to spread the news? Please consider sharing your news or events here, and promote it to a large audience.

Tasks we do on a regular basis:

  • gathering news in the XMPP universe
  • short summaries of news and events
  • summary of the monthly communication on extensions (XEPs)
  • review of the newsletter draft
  • preparation of media images
  • translations
  • communication via media accounts

License

This newsletter is published under CC BY-SA license.

December 05, 2022 00:00

December 03, 2022

Ignite Realtime Blog

Denial of Service Vulnerability in Smack 4.4 if XMPPTCPConnection is used with StAX

The fantastic folks behind Jitsi have discovered a Denial of Service (DoS) vulnerability in Smack (JSA-2022-0002, JSA-2022-0003), which is possible if a combination of Smack components is used. The root of the vulnerability is interesting because it is due to a countermeasure against DoS attacks, namely FEATURE_SECURE_PROCESSING of the Java API for XML Processing (JAXP).

The DoS is possible because the older XMPPTCPConnection implementation of Smack parses the XMPP stream as one large XML document. Suppose the connection instance uses a parser where FEATURE_SECURE_PROCESSING is enabled. In that case, it is easy for an attacker to craft a stanza that triggers one of the various limits imposed by FEATURE_SECURE_PROCESSING, causing an exception, leaving the parser in an unrecoverable state, and closing the connection.

This vulnerability was relatively recently introduced in Smack with the addition of the support for JAXP’s Streaming API for XML (StaX) parser. Historically, Smack only used XPP3 as XML pull parser. The default implementation of XPP3 is a fast, lightweight, and, to the best of our knowledge, secure parser. XPP3 is used, for example, by Android. However, with version 4.4.0 (SMACK-591), Smack gained support for using Java’s Streaming API for XML (StAX) in addition to XPP3, to facilitate code-reuse on Java SE platforms and avoiding the XPP3 dependency.

So this DoS is possible if the XMPP connection is of type XMPPTCPConnection and if the Smack connection instance uses a StAX parser for XMPP parsing.

On a related note, Smack’s newer modular connection architecture is not affected by this, because it splits the individual top-level XMPP stream elements and parses them as standalone document. The splitting is done very early in the input processing step by XmlSplitter (of jxmpp), which also enforces size limits for the XML elements. Therefore, the DoS is not possible over connections that are established via Smack’s modern ModularXmppClientToServerConnection.

If you are affected, then the following countermeasures are possible:

  1. Relax the FEATURE_SECURE_PROCESSING_LIMITS
  2. Switch to XPP3 (smack-xmlparser-xpp3)
  3. Use ModularXmppClientToServerConnection

Option A has the drawback that it is only possible to relax the limits globally. That is, it will affect XML processing regardless if Smack or some other component performs it. If you still want to go down that route, then

System.setProperty("jdk.xml.entityExpansionLimit", "0")
System.setProperty("jdk.xml.maxOccurLimit", "0")
System.setProperty("jdk.xml.elementAttributeLimit", "0")
System.setProperty("jdk.xml.totalEntitySizeLimit", "0")
System.setProperty("jdk.xml.maxXMLNameLimit", "524288")
System.setProperty("jdk.xml.entityReplacementLimit", "0")

2 posts - 2 participants

Read full topic

by Flow at December 03, 2022 14:42

Gajim

Gajim 1.5.4

Gajim 1.5.4 comes with a reworked file transfer interface, better URL detection, message selection improvements, and many fixes under the hood. Thank you for all your contributions!

What’s New

Gajim’s interface for sending files has been reworked, and should be much easier to use now. For each file you’re about to send, Gajim will generate a preview. This way, you can avoid sending the wrong file to somebody. Regardless of how you start a file transfer, be it drag and drop, pasting a screen shot, or simply clicking the share button, you’ll always be able to check what you’re about to send.

Gajim’s new file transfer interface

Gajim’s new file transfer interface

More Changes

  • Performance: Chat history is now displayed quicker
  • Support for Jingle XTLS has been dropped, since it hasn’t been standardized
  • geo:-URIs are now prettier (thanks, @mjk)
  • Dependencies: pyOpenSSL has been replaced by python-cryptography

Fixes

  • Fixes for message selection
  • Improvements for recognizing URLs (@mjk)
  • Many fixes to improve Gajim’s usability

Over 20 issues have been fixed in this release. Have a look at the changelog for a complete list.

Gajim

As always, don’t hesitate to contact us at gajim@conference.gajim.org or open an issue on our Gitlab.

December 03, 2022 00:00

December 01, 2022

Erlang Solutions

Advent of Code 2022 – Every Puzzle Solved in Erlang

Day 1

Christmas is getting closer and with that, the annual Advent of Code begins. For those who do not know, Advent of Code is a fun and inclusive event which provides a new programming puzzle every day. The fun is that these puzzles can be solved in any programming language and are accessible for varying levels of coding experience and skills. The real test is in your problem-solving. This year, we’ll be solving each of the problems in Erlang and publishing the results. We hope you enjoy it – if you’ve come up with a different solution and want to discuss it with us, we encourage you to comment on Twitter. 

The code will be added to the repo here: https://github.com/aleklisi/AdventOfCode2022 as I manage to solve each next puzzle.

Day 1

Full problem description: https://adventofcode.com/2022/day/1

The example input file:

1000
2000
3000

4000

5000
6000

7000
8000
9000

10000

Day 1- Puzzle 1

We are given a file with a list of values for calories in snacks carried by elves. The Elves take turns writing down the number of Calories contained in the various snacks that they’ve brought with them, one item per line. Each Elf separates its own inventory from the previous Elf’s inventory (if any) by a blank line. So the first task is to read and parse the data and decide how to represent it.

I decided to represent the input as a list of two-element tuples, where each tuple stores the elf’s number and a list of snacks. Here is an example data representation for the example file:

[
  {1,[1000,2000,3000]},
  {2,[4000]},
  {3,[5000,6000]},
  {4,[7000,8000,9000]},
  {5,[10000]}
]

Now we just need to define a max function, which compares elements based on the sum of elements in a list of the calories in the snacks. I assumed that the list of elves is not empty, so I start with its first element as the current max and then started comparisons with other elves. Every time I find an elf with a bigger calorie sum I replace my current elf with the new one so that I will end up with the elf with the highest calorie total. Once the elf with the most calories is found, we can return the sum of calories.
See the code: https://github.com/aleklisi/AdventOfCode2022/blob/main/day1_puzzle1/src/day1_puzzle1.erl#L48-L56 .

Day 1- Puzzle 2

The only difference in puzzle 2 compared to puzzle 1 is that now we need to find 3 elves with the most calories total instead of just 1 elf and sum their calories altogether.

We can heavily rely on solutions from puzzle 1.

To find the top 3 elves I will just:

  1. Find the elf with the highest calories and save this elf as the first one.
  2. Remove the first elf from the list of elves. 
  3. Find the next top elf with the highest calories from the list of elves (without the first elf) and save this elf as the second one.
  4. Remove the second elf from the list of elves.
  5. Find the next top elf with the highest calories from the list of elves (without the first and second elf) and save this elf as the third one.
  6. Return the sum of calories of the first, second and third elf.

Voila!

Day 2

Day 2 of Advent of Code sees us helping the Elves to score a game of Rock, Paper, Scissors. The Elves have provided us with a strategy guide and it’s our job to help them score their game. 

Day 2- Puzzle 1

To complete the task we need to calculate the results of the above games. Since the games are unrelated (the result of previous games does not impact the next games, the best way to approach the problem is to start by implementing a single game’s score count function and then map a list of games with this function, to get the scores for each of the games. To get the final score (which is a sum of the games’ scores) we then sum all of the elements of the list.

The data structure I decided to use to represent a single game is a two element tuple, where the first element is the opponent’s move and the second element is my move.

The list of games is parsed into something like this:

[{rock, rock},
              {scissors, rock},
              {scissors, rock},
              {scissors, rock},
              {paper, paper},
	…
]

Looking back (after solving the problem) I could have used maps with a structure like the one below:

	#{my_move = > rock, opponent_move => paper}

It might have helped me debug and avoid errors, which I did when first approaching the problem. That error was to confuse my move with my opponent’s move. In other words, I decoded A, B and C to be my moves and  X, Y, and Z to be my opponent’s moves. It is an obvious mistake when you spot it, but easy oversight when reading the puzzle’s description fast. I obviously had to read the description carefully 2 more times to spot my mistake, so as we say in Polish: “the cunning one loses twice”.

Both parsing and solving today’s puzzle heavily rely on pattern matching, so let’s see that in action.

Firstly, let’s take a look at how the data is decoded using pattern matching:

% "The first column is what your opponent is going to play:
% A for Rock,
% B for Paper,
% C for Scissors.
translate_opponent_move("A") -> rock;
translate_opponent_move("B") -> paper;
translate_opponent_move("C") -> scissors.

% The second column, you reason, must be what you should play in response:
% X for Rock,
% Y for Paper,
% Z for Scissors.
translate_my_move("X") -> rock;
translate_my_move("Y") -> paper;
translate_my_move("Z") -> scissors.

A smart observer might notice that I could have used a single function to handle that translation, but I find dividing the decoding into two separate functions much more readable.

Now let’s take a look at how scoring can be conveniently calculated:

count_games_score({OpponentMove, MyMove}) ->
    count_shape_score(MyMove) + count_result_score(OpponentMove, MyMove).

% The score for a single round is the score for the shape you selected (
% 1 for Rock,
% 2 for Paper,
% 3 for Scissors
count_shape_score(rock) -> 1;
count_shape_score(paper) -> 2;
count_shape_score(scissors) -> 3.

% ) plus the score for the outcome of the round (
% 0 if you lost,
% 3 if the round was a draw, 
% 6 if you won
% ).
count_result_score(rock, scissors) -> 0;
count_result_score(paper, rock) -> 0;
count_result_score(scissors, paper) -> 0;
count_result_score(OpponentMove, MyMove) when MyMove == OpponentMove -> 3;
count_result_score(scissors, rock) -> 6;
count_result_score(rock, paper) -> 6;
count_result_score(paper, scissors) -> 6.

Again a keen observer might notice that it could be done in a single function, but I think most people will agree that translating the specifications one-to-one is way more convenient and much easier to understand and possibly debug if the need arises. 

The solution can be found here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day2_puzzle1

Day 2- Puzzle 2

Puzzle two introduces a plot twist. It turns out that the second part of the input for each of the games is not what we are supposed to play, but the expected game result. We need to figure out what to play, based on what our opponent’s move is and the expected result. Notice that the game score count does not change, so if we determine what we have played based on the new understanding of input and provide parsing output to follow the same rules as we did in today’s puzzle 1 when doing the parsing, the rest of the code should work correctly without any change.

Let’s now see how to achieve that in practice.

In the `read_and_parse_data/1` function I modified the anonymous function inside a map function to translate the predicted result into my move:

        fun(RawGame) ->
            [OpponentMoveRaw, GameResultRaw] = string:split(RawGame, " "),
            OpponentMove = translate_opponent_move(OpponentMoveRaw),
            GameResult = translate_result(GameResultRaw),
            MyMove = find_my_move(OpponentMove, GameResult),
            {OpponentMove, MyMove}
        end

And this is the implementation of the translating functions:

% "The first column is what your opponent is going to play:
% A for Rock,
% B for Paper,
% C for Scissors.
translate_opponent_move("A") -> rock;
translate_opponent_move("B") -> paper;
translate_opponent_move("C") -> scissors.

% The second column says how the round needs to end:
% X means you need to lose,
% Y means you need to end the round in a draw,
% Z means you need to win.
translate_result("X") -> lose;
translate_result("Y") -> draw;
translate_result("Z") -> win.

find_my_move(OpponentMove, draw) -> OpponentMove;
find_my_move(rock, lose) -> scissors;
find_my_move(paper, lose) -> rock;
find_my_move(scissors, lose) -> paper;
find_my_move(rock, win) -> paper;
find_my_move(paper, win) -> scissors;
find_my_move(scissors, win) -> rock.

Again they heavily rely on pattern matching.

The solution can be found here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day2_puzzle2

Conclusions after completing day 2

Firstly, ALWAYS carefully read the requirements, and do not skip any part, because you find it “obvious”.

Secondly, pattern matching is a great tool to have, it allows us to easily implement readable code.

And last but not least, if you struggle or get stuck with something, it helps to add readable printing/logging to your code. When my implementation of `find_my_move/2` function (when solving puzzle 2) did not work. I added the following printing debug to the parsing data function:


…
MyMove = find_my_move(OpponentMove, GameResult),
            io:format("OpponentMoveRaw: ~p\t", [OpponentMoveRaw]),
            io:format("OpponentMove: ~p\t", [OpponentMove]),
            io:format("GameResultRaw: ~p\t", [GameResultRaw]),
            io:format("MyMove: ~p\t", [MyMove]),
            io:format("GameResult: ~p\t", [GameResult]),
            io:format("Result Score: ~p\n", [count_games_score({OpponentMove, MyMove})]),
            {OpponentMove, MyMove}
…

Which for the test file:

A X
A Y
A Z
B X
B Y
B Z
C X
C Y
C Z

Results with the following output:

OpponentMoveRaw: "A"	OpponentMove: rock	GameResultRaw: "X"	MyMove: scissors		GameResult: lose	Result Score: 3
OpponentMoveRaw: "A"	OpponentMove: rock	GameResultRaw: "Y"	MyMove: rock		GameResult: draw	Result Score: 4
OpponentMoveRaw: "A"	OpponentMove: rock	GameResultRaw: "Z"	MyMove: paper		GameResult: win	Result Score: 8
OpponentMoveRaw: "B"	OpponentMove: paper	GameResultRaw: "X"	MyMove: rock		GameResult: lose	Result Score: 1
OpponentMoveRaw: "B"	OpponentMove: paper	GameResultRaw: "Y"	MyMove: paper		GameResult: draw	Result Score: 5
OpponentMoveRaw: "B"	OpponentMove: paper	GameResultRaw: "Z"	MyMove: scissors		GameResult: win	Result Score: 9
OpponentMoveRaw: "C"	OpponentMove: scissors	GameResultRaw: "X"	MyMove: paper		GameResult: lose	Result Score: 2
OpponentMoveRaw: "C"	OpponentMove: scissors	GameResultRaw: "Y"	MyMove: scissors		GameResult: draw	Result Score: 6
OpponentMoveRaw: "C"	OpponentMove: scissors	GameResultRaw: "Z"	MyMove: rock		GameResult: win	Result Score: 7

Which I found extremely helpful when finding the mistake. It turned out that instead of:

% …
find_my_move(rock, lose) -> scissors;
find_my_move(paper, lose) -> rock;
find_my_move(scissors, lose) -> paper;
% …

I had:

% …
find_my_move(rock, lose) -> paper;
find_my_move(paper, lose) -> rock;
find_my_move(scissors, lose) -> paper;
% …

In the event that I was unable to locate my mistake, I would recommend implementing unit tests, hoping not to duplicate the mistake there.

That’s it for day 2. Come back on Monday for the solutions to the weekend’s puzzles as well as Monday’s solution.

Day 3

In today’s puzzles we are tasked with finding intersections of sets. I know that for 99% of my audience sets intersection is an obvious term, but for the 1% that do not know the intersection of sets is a fancy name for a collection of common elements of those sets.

An important note is that this definition works not only for two sets but for any number of sets.

In the diagram above, the intersection for the Sets’ is marked in green.

Day 3- Puzzle 1

In this puzzle, we need to divide each input line in half and then find the intersection (a single element) of the first and second half of each line. This can be done very simply using functions from Erlang’s sets (https://www.erlang.org/doc/man/sets.html) module:

Once we have the found intersections, we need to map them to priorities. This is one of the rare moments when using a dollar sign followed by character syntax makes the code much more readable. The one thing that we need to remember is that a letter is stored as a number, it’s ASCII code. So $a is stored internally in Erlang as number 97 etc. Having those 2 facts established we can implement mapping items (chars) to priorities:

For letters other than a, z, A and Z the code is not as obvious, but fortunately, we are provided with a description that can be translated directly into unit tests:

% Lowercase item types a through z have priorities 1 through 26.

% Uppercase item types A through Z have priorities 27 through 52.

priority_conversions_are_correct_for_edge_cases_test() ->

   ?assertEqual(1, item_to_priority($a)),

   ?assertEqual(26, item_to_priority($z)),

   ?assertEqual(27, item_to_priority($A)),

   ?assertEqual(52, item_to_priority($Z)).


% the priority of the item type that appears in both compartments of each rucksack is

% 16 (p),

% 38 (L),

% 42 (P),

% 22 (v),

% 20 (t),

% 19 (s)

priority_conversions_are_correct_test() ->

   ?assertEqual(16, item_to_priority($p)),

   ?assertEqual(38, item_to_priority($L)),

   ?assertEqual(42, item_to_priority($P)),

   ?assertEqual(22, item_to_priority($v)),

   ?assertEqual(20, item_to_priority($t)),

   ?assertEqual(19, item_to_priority($s)).

To run the tests execute: `rebar3 eunit` which should result in a message like this one:

$ rebar3 eunit

===> Verifying dependencies...

===> Analyzing applications...

===> Compiling day3_puzzle2

===> Performing EUnit tests...

..

Finished in 0.077 seconds

2 tests, 0 failures

Having the item-to-priority conversion implemented and tested is enough to get the intersections for the first and second compartments of each rucksack, map intersections to priorities and then sum priorities.

Full code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day3_puzzle1

Day 3 – Puzzle 2

Today’s second puzzle seems to actually be simpler than the first one. The task requires us to divide the input into groups of 3 lines (Elves group), where each line represents the Elf’s rucksack content. This can be achieved by a custom recursive function:

The next step is to find the intersection of those 3 collections (which is the Elve’s group badge, a single character) and reuse the item for the priority conversion function from puzzle 1. This will reveal the priorities of the badges. Last but not least, just sum up the priorities to get the answer.

Full code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day3_puzzle2

Day 4

Today’s puzzles are about cleaning up the camp. From a mathematical/algorithmic perspective, it is all about numerical ranges (discrete intervals, aka with integers only) and finding common elements between them.

For the input, we are given a list of paired ranges.

Day 4- Puzzle 1

The first task is to find how many assignment pairs in one range, fully contains the other. This is a simple enough task. After parsing the data we just need to apply a filter with a custom predicate of one range containing the other.

Take a second to notice that the first range contains the second one when both of the following conditions are met:

  • the smallest number in the second range is bigger or equal to the smallest number in the first range
  • the biggest number in the second range is smaller or equal to the biggest number in the first range

Also spot that the second range might be the one containing the first one, so we have to reverse the condition (that is the part after `orelse`).

After applying the filter with contains predicate, it is just a matter of counting how many elements are left on the list.

Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day4_puzzle1

Day 4 – Puzzle 2

The second puzzle today differs from the first one only in the nature of the predicate, because the second task is to find pairs which have any common part (overlap).

The only change that needs to be made is the predicate, this time instead of `contains/1` I defined a new predicate `overlap/1`. Notice that there can be a few approaches to this problem:

  1. Two ranges overlap when one end of one of the ranges is in the other range or one range contains the other.
  2. The size of the intersection of the two sets of integers that those ranges contain is bigger than zero (0), in other words, the two rangers overlap if they have at least 1 common element. 

Let’s see how to implement those two approaches. 

Firstly, the conditions of one range containing one of the ends of the other range. Notice that instead of checking if the `contains/` predicate we could alternatively reverse the roles and check if any of the ends of the first range is in the second range.

The second approach is leveraging the erlang sets module:

Playing around with multiple implementations and checking if they all work correctly I added some unit tests at the end of the module. If you want to play around with the conditions change the code and run `rebar3 eunit`.

If you are wondering how the test cases were constructed to cover all cases (I hope I did not miss anyone). I fixed one range (5 to 7) and placed the other range in different configurations, starting from both ends of the second range being smaller and ending on both ends of the second range being bigger (the only 2 cases when the two ranges do not overlap). This is how the code doing that looks:

do_overlap_sections_test() ->
   Cases = [
       {{2, 5},{5, 7}},
       {{2, 6},{5, 7}},
       {{2, 7},{5, 7}},
       {{2, 8},{5, 7}},
       {{2, 9},{5, 7}},
       {{4, 9},{5, 7}},
       {{5, 9},{5, 7}},
       {{6, 9},{5, 7}},
       {{7, 9},{5, 7}},
       {{2, 7},{3, 5}},
       {{3, 5},{2, 7}}
   ],
   [?assertEqual(true, overlap(#{
       first_section => #{from => FromF, to => ToF},
       second_section => #{from => FromS, to => ToS}}))
       ||{{FromF, ToF}, {FromS, ToS}} <- Cases].
 
do_not_overlap_sections_test() ->
   Cases = [
       {{2, 4},{5, 7}},
       {{8, 9},{5, 7}}
   ],
   [?assertEqual(false, overlap(#{
       first_section => #{from => FromF, to => ToF},
       second_section => #{from => FromS, to => ToS}}))
       ||{{FromF, ToF}, {FromS, ToS}} <- Cases].
Test Ranges Advent of Code 2022 Puzzle 2 Day 4

When there are many cases to be considered I think that the saying “a picture is worth a thousand words” applies. So I’ve provided the image above. 

Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day4_puzzle2

Day 5

Today’s puzzle is about implementing an interpreter for manipulating crates between stacks. The most difficult part of today’s puzzle turned out to be parsing and loading initial stacks into a reasonable data structure.

Here is an example input file:

First, I divided the input into initial stacks of cranes and a list of commands. Since parsing the list of commands is pretty straightforward and can be done in many ways I will skip describing how to parse commands. If you are interested in looking more into how I did that, take a look at the code here.

To parse initial stacks, I read the rows, divided them into crates and if there is an empty space (for lower crates stack), replace it with an atom `empty`.  Now the partially parsed crates look like this:

Next for a list of parsed rows, I picked the heads of those lists to a separate list (my first stack) and then reapplied the same function recursively for the tails of those rows (to deliver the following stacks).

Let’s see a simplified version of getting a single stack:

There were a few minor operations to be made to finally get the following data structure to represent my stacks: 

This creates a map where the key is the stack’s number, and the value is a list of crates from the top crate at the beginning of a list to the bottom, the last element at this list.

Day 5 – Puzzle 1

After creating the initial stacks of crates and a list of commands parsed, we can execute the operations. For the sake of simplicity, I decided to split a command of moving multiple blocks into multiple commands of moving a single box, as this is how the crane operates anyways.

This is the code to do exactly what I have described above:

Last but not least, when all of the commands are executed, we need to read the new tops of the stacks.

This is done by changing the map of stacks back into a list of tuples with stack numbers as the first element and stack as the second one. Then, a relatively rarely used function `lists:keysort/1` comes in handy, as we need to sort those tuples by stack number.

After sorting it remains to get only the top element for each stack.

Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day5_puzzle1

Day 5 – Puzzle 2

The only modification for this puzzle is how we move the crates. Previously they were moved one by one, now all of the crates are moved at once. So, the only change that needs to be made to the code is how the command is executed.  Now, I split the stack into parts using `lists:split/2` function, instead of just getting the top one element from the source stack, like it was done in today’s puzzle 1. Pay attention to the fact that now we need to concatenate a list of moving elements, instead of adding the moving elements as a single head of the destination stack like it was done in the previous puzzle.

The complete code for this is available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day5_puzzle2

As always if you want to discuss, or compare your solutions with us you are welcome to do so on Twitter. 

Day 6

Today, both puzzles are about finding the first subsequence of N (4 in the first puzzle and 14 in the second puzzle) characters that are all different in a given input sequence.

Unfortunately using mapping does not work, because we need to process more than one element at a time. Folding does not work either (obviously I bet someone can make it work, but it is way too complicated to be interested in doing) as it always needs to process the whole list. But we want to be able to stop processing as soon as we find the expected subsequence.

Day 6- Puzzle 1

For the first puzzle today, we need to find where in the given sequence of characters ends the first subsequence of 4 unique (all are different) characters. To find the first such sequence, I recursively check if the current 4 characters are all different. If they are then I return the result position, otherwise I remove the first character from the analysed sequence, increment the position counter by one and run the search again and so on until eventually the searched section is found.

find_start_of_packet_marker(Subroutine) ->
   find_start_of_packet_marker(4, Subroutine).
 
find_start_of_packet_marker(MarkerPos, [_ | Rest] = Subroutine) ->
   case first_4_chars_are_all_different(Subroutine) of
       true -> MarkerPos;
       false -> find_start_of_packet_marker(MarkerPos + 1, Rest)
   end.

In the snippet above, a very powerful concept of using recursion with an accumulator is shown. The idea of recursion with an accumulator is basically to add another parameter to the recursive function and to store partial results in this parameter. To see the difference, let’s take a look at 2 approaches to implementing finding the Nth element of the Fibonacci sequence. Firstly the classical implementation:

fib(0) -> 1;
fib(1) -> 1;
fib(N) -> fib(N - 1) + fib(N - 2).

And let’s compare it to the one using an accumulator:

fib_with_acc(0) -> 1;
fib_with_acc(N) ->
   fib_with_acc(N, {1, 1}).
 
fib_with_acc(0, {Prev, _Curr}) -> Prev;
fib_with_acc(N, {Prev, Curr}) -> fib_with_acc(N - 1, {Curr, Prev + Curr}).

I cannot disagree that at first glance the implementation with an accumulator is more complex, but much faster code. Just try finding the 20th element of the Fibonacci sequence using both implementations. And they give the same results (at least for the first few elements of the sequence) which I checked using this assertion:

fib_implementations_are_equivalents_test() ->
   [ ?assert(fib(N) == fib_with_acc(N)) || N <- lists:seq(0, 6) ].

Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day6_puzzle1

Day 6-Puzzle 2

The second puzzle only differs in the length of the subsequence where all elements must be different. In the first puzzle, I defined checking if all four elements are different “by hand”:

first_4_chars_are_all_different([A, B, C, D | _]) ->
   A /= B andalso A /= C andalso A /= D
          andalso B /= C andalso B /= D
                         andalso C /= D.

This would be ridiculously long and easy to make mistake code, therefore I decided that it makes much more sense to use a property of set (sets module). That is when an element is added to a set multiple times it is stored only once. In other words, sets do not keep the duplicates of elements. Knowing that, if I decide to build a set from a list of a given number of elements, the size of this set will only be equal to the length of the list it was constructed from if all of the elements on the source list were different from each other. I know it sounds complicated, but let’s see the code and everything should be clear.

first_14_chars_are_all_different(Lst) ->
   {First14Chars, _Rest} = lists:split(14, Lst),
   sets:size(sets:from_list(First14Chars)) == 14.

This solution can be easily generalised to work with a sequence of any length:

first_N_chars_are_all_different(N, Lst) ->
   {FirstNChars, _Rest} = lists:split(N, Lst),
   sets:size(sets:from_list(FirstNChars)) == N.

Then it would work for both the first and second puzzles by just changing parameter N, setting it to 4 or 14. 

Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day6_puzzle2

Day 7

Today, both puzzles are about finding directories and the sizes of files and subdirectories they contain. The biggest challenge was how to change input into data representation. The issue is mostly about using `cd ..` as when creating any recursive data structure, the inner part often does not know about the outer part. I tried 2 approaches, which failed. 

The first one was to use the Erlang digraph module: https://www.erlang.org/doc/man/digraph.html 

I found that I was not able to traverse the tree conveniently, therefore I gave it up. 

For the second approach, I was to create actual files and directories to represent the data given in the input. But then I came to the conclusion that it would be cheating, so I also gave up this idea quickly. I ended up parsing the given shell history into a list of files and directories, where each file and directory was represented by a tuple, like those:

[
   {dir,["/","a"]},
   {file,["/","b.txt"],14848514},
   {file,["/","c.dat"],8504156},
   {dir,["/","d"]}
]

With this representation of files and folders, there is a very easy way to check if a file should be added to the directories size: 

If the directory’s path is the first part of the file’s path ,then this file is a part of this directory. 

To make it more visually appealing, let’s see a few examples:

[
   {dir,["/"]},
   {dir,["/", "A"]},
   {dir,["/", "B"]},
 
   {file,["/","b.txt"],1},
   {file,["/","c.txt"],2},
   {file,["/", "A", "d.txt"],12},
   {file,["/", "C", "e.txt"],42}
]

Let’s consider input like the above. We can see that the “/” directory should consider all of the files in its size. The [“/”, “A”] directory should only consider “d.txt” file’s size to know its own size, as it never matches its path’s second element with any of the other files. And directory [“/”, “B”]  matches none of the files on its second element at all.

Such comparisons can be done by a surprisingly simple function:

contains([], _) -> true;
contains([Elem | DirRest], [Elem | FileRest]) ->
   contains(DirRest, FileRest);
contains(_, _) -> false.

Day 7 – Puzzle 1

We already know which files are to be counted when a directory size is calculated. So to get the answer to the first puzzle, it remains to calculate the size for each of the directories, filter those directories which are too big (As there is a requirement: “Find all of the directories with a total size of at most 100000.”) and then sum the sizes of the remaining once.

Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day7_puzzle1

Day 7 – Puzzle 2

The second part of the puzzle requires finding the smallest directory to be deleted to have enough free memory to install an update of the given size.

It is needed to know how much memory is taken. Fortunately, the amount of memory taken is the size of the “/” directory. To get how much memory is already available,we subtract taken memory from the total space available. To find the smallest folder that will release enough memory, it is necessary to calculate how much memory needs to be freed. To calculate that, subtract the free memory from how much memory needs to be freed. 

Then last but not least, filter directories which are too small (deleting them will not free enough memory) and from big enough directories we pick the one with the smallest size (the directory’s name is dropped as the puzzle asks only about this directory’s size).

Complete code available here: https://github.com/aleklisi/AdventOfCode2022/tree/main/day7_puzzle2

The post Advent of Code 2022 – Every Puzzle Solved in Erlang appeared first on Erlang Solutions.

by Aleksander Lisiecki at December 01, 2022 10:33

RabbitMQ – An Amazing Message Broker

In cloud architectures (or microservices), applications are broken down into smaller independent blocks that can be quickly developed, deployed, and maintained. Imagine you have a cloud architecture that has many services and many requests per second, you have to make sure that no requests are lost and your web service is always ready to receive new requests instead of locked by processing the previous request and must ensure that the services communicate with each other smoothly and efficiently.

So how do you? How can different applications communicate with each other? The answer is Message Broker!

Nowadays, there are many Message Broker software that can be listed as AWS Simple Queue Service (SQS), Apache Kafka, Apache ActiveMQ,… But the most popular among the names listed above are RabbitMQ!

So, What is a message broker? What is RabbitMQ? Why use RabbitMQ?
Your concerns will be answered by the RabbitMQ team right in the article!


What is a Message Broker?

A message broker (an integration broker or interface engine) is an intermediary module that transfers messages from sender to receiver. It is an architectural pattern for inspecting, relaying, and navigating messages; mediating between applications, simplifying communication between them, and maximizing the efficiency of splitting into smaller blocks. The main task of a Message broker is to receive messages from applications and perform some action. Let’s take a look at the sequence diagram below:

How does Message Broker work?

During the Initialize process, Service1 and Sevice2 load the proxy and register to the Broker. From there, the Broker will forward the messages to the pre-registered proxy.
The advantages can see here are:

  • Service1 and Service2 don’t need to know each other and don’t need to speak in the same language. They just need to send the message to the proxy, and from here proxy will forward the message to the Broker. The Broker will take care of validating, transforming, and routing messages between Service1 and Service2.
  • With this design pattern, we can set up \an asynchronous mechanism. Service1 doesn’t need to care when a message is delivered to Service2 and when Service2 finishes handling the message, everything Sevice1 should do is send the message to the Broker, Service2 will pick up the message whenever it wants.

To put it simply, a message broker is an intermediary program developed to serve the needs of easy communication between different applications. You can also understand message broker as a message broker software program.

Some highlights about RabbitMQ

RabbitMQ is an open-source message broker. In the beginning, it was used for Advanced Message Queueing Protocol (AMQP), and after that growing up to support the Streaming Text Oriented Messaging Protocol (STOMP), Message Queuing Telemetry Transport (MQTT), and other protocols. 

In a word, RabbitMQ is like an intermediary message carrier or queue manager. RabbitMQ presented in Erlang language gives programmers an intermediary means to exchange data between members of the system and securely stores the data before it is pushed to another location.

It can be understood simply, as a large-scale system, the exchange of messages between components is more and more complex. Therefore, RabbitMQ was born as an effective solution in the system structure. Not only that, but the capacity of RabbitMQ is also quite light, programmers can deploy it on both fixed and cloud environments. 

RabbitMQ also supports multiple Messaging protocols, develops in a distributed, federated environment, and meets the need for scalability. RabbitMQ also provides a wide range of tools for the most popular languages such as C++, C#, Java, PHP, Python,…

The most outstanding features of RabbitMQ

As the software of choice is used so much, it must contain many outstanding features:

  • Interface

RabbitMQ has a fairly simple interface, easy to use. Users can easily manage, monitor and control all problems in the programs.

  • Reliability

RabbitMQ offers a variety of features to let you trade off performance with reliability, including persistence, delivery acknowledgments, publisher confirms, and high availability.

  • When a connection fails, messages may be in transit between the client and server. They may be stuck in the middle of being decoded or encoded on either side, stuck in the TCP buffer, … In such events, messages in transit will not be delivered, they will need to be retransmitted. The RabbitMQ Acknowledgements and Confirms feature will let the server and clients know when to do this.
  • RabbitMQ can detect dead TCP connections with the Heartbeats feature.
  • With some messaging protocols supported by RabbitMQ, applications can control the durability of queues and messages. In order to avoid losing messages, durable queue is the recommended option, and messages are published as persistent by publishers (Delivery mode property).
  • In a RabbitMQ cluster, all definitions (of exchanges, bindings, users, etc) are replicated across the entire cluster. Queues may be located on a single node, or replicate their content for higher availability.

Quorum queues is a modern replicated queue type that focuses on data safety. They provide a more understandable, in some cases less resource intensive, new implementation for achieving replicated queues and high availability.

Streams is another replicated messaging data type that allows for repeatable consumption.

  • Flexibility:

The message is routed through the Exchange before reaching the Queue. RabbitMQ provides some commonly used Exchange types, we can also define our own Exchange as a plugin. For more complex routing you can bind exchanges together.

RabbitMQ enables users to control the trade-offs between messages, throughput, and performance. All the messages in the queues can specify where they should be saved to a disc before their delivery. Queues in a cluster can span multiple servers while ensuring that no messages are lost in the case of a server failure.

  • Multi-protocol, the multi-language feature creates a diversity of users.
  • Lightweight:

RabbitMQ is lightweight and requires less than 40MB of RAM to run the application core and plugins like the Management UI. 

  • High availability of queues: 

With queues in RabbitMQ, users can replicate it on several different machines in the same cluster. This will help to keep the messages safe even if the hardware fails.

  • Traceability

If messaging is not working properly, RabbitMQ will step in and take action. Thanks to its traceability, users can track the system’s operating status or tell if the system has any problems.

  • Plugins system

RabbitMQ also supports plugin extension through many forms. If you have the ability, you can also create these utilities yourself. For more information on how to develop a RabbitMQ Plugin, please refer to the Plugin Development Basics page.

  • Commercial Services

Support sales with training and consulting programs offered on Pivotal.

RabbitMQ’s application

RabbitMQ can be used when the web server needs to quickly respond to requests. This eliminates the need to perform resource-intensive operations while the user waits for results. RabbitMQ is also used to transport messages to many different recipients for processing or to share the load among highly loaded workers (20K+ messages/sec).

RabbitMQ can be used for:

  • Applications need to support legacy protocols, such as STOMP, MQTT, AMQP, 0-9-1.
  • Fine-grained control over consistency/set of guarantees on a per-message basis
  • Complex routing to consumers
  • Applications need multiple to publish/subscribe, point-to-point request/reply messaging capabilities.

Who is using RabbitMQ?

RabbitMQ is an open-source tool with 10.1K GitHub stars and 3.7K GitHub forks. Here’s a link to RabbitMQ’s open-source repository on GitHub.

Explore RabbitMQ’s Story.

1972 companies reportedly use RabbitMQ in their tech stacks, including Robinhood, Reddit, and Tech Stack.

Reference resources

https://www.rabbitmq.com/

https://github.com/rabbitmq/rabbitmq-server

https://www.erlang-solutions.com/blog/an-introduction-to-rabbitmq-what-is-rabbitmq/

https://www.erlang-solutions.com/blog/rabbitmq-quorum-queues-explained-what-you-need-to-know/

The post RabbitMQ – An Amazing Message Broker appeared first on Erlang Solutions.

by Bao Hoang at December 01, 2022 10:29

JMP

Writing a Chat Client from Scratch

There are a lot of things that go into building a chat system, such as client, server, and protocol.  Even for only making a client there are lots of areas of focus, such as user experience, features, and performance.  To keep this post a manageable size, we will just be building a client and will use an existing server and protocol (accessing Jabber network services using the XMPP protocol).  We’ll make a practical GUI so we can test things, but not spend too much time on polish, and look at getting to a useful baseline of features.

You can find all the code for this post in git.  All code licensed AGPL3+.

Use a Library

As with most large programming tasks, if we wanted to do every single thing ourselves we would spend a lot more time, so we should find some good libraries.  There is another reason to use a library: any improvements we make to the library benefits others.  While releasing our code might help someone else if they choose to read it, a library improvement can be picked up by users of that library right away.

We need to speak the XMPP protocol so let’s choose Blather.  We need a GUI so we can see this working, but don’t really want to futz with it much so let’s choose Glimmer.  The code here will use these libraries and be written in the Ruby programming language, but these ideas are general purpose to the task and hopefully we won’t get too bogged down in syntax specifics.

One little language-specific thing you will need to create is a description of which ruby packages are being used, so let’s make that file (named Gemfile):

Gemfile

source "https://rubygems.org"

gem "blather", git: "https://github.com/adhearsion/blather", branch: "develop"
gem "glimmer-dsl-libui", "~> 0.5.24"

Run this to get the packages installed:

bundle install --path=.gems

Let’s get the bare minimum: a connection to a Jabber service and a window.

client.rb

require "glimmer-dsl-libui"
require "blather/client"

BLATHER = self
include Glimmer

Thread.new do
	window("Contacts") {
		on_destroy {
			BLATHER.shutdown
		}
	}
end

When required in this way, Blather will automatically set up a connection with event processing on the main thread, and will process command line arguments to get connection details.  So we put the GUI on a second thread to not have them block each other.  When the window is closed (on_destroy), be sure to disconnect from the server too.  You can run this barely-a-client like this:

bundle exec ruby client.rb user@example.com password

The arguments are a Jabber ID (which you can get from many existing services), and the associated password.

You should get a blank window and no errors in your terminal.  If you wanted to you could even look in another client and confirm that it is connected to the account by seeing it come online.

Show a Contact List

Let’s fetch the user’s contacts from the server and show them in the window (if you use this with a new, blank test account there won’t be any contacts yet of course, but still).

$roster = [["", ""]]

Thread.new do
	window("Contacts") {
		vertical_box {
			table {
				button_column("Contact") {
				}
				editable false
				cell_rows $roster
			}
		}

		on_destroy {
			BLATHER.shutdown
		}
    }.show
end

after(:roster) do
	LibUI.queue_main do
		$roster.clear
		my_roster.each do |item|
			$roster << [item.name || item.jid, item.jid]
		end
	end
end

In a real app you would probably want some kind of singleton object to represent the contacts window and the contact list (“roster”) etc.  For simplicity here we just use a global variable for the roster, starting with some dummy data so that the GUI framework knows what it will look like, etc.

We fill out the window from before a little bit to have a table with a column of buttons, one for each contact.  The button_column is the first (and in this case, only) column definition so it will source data from the first element of each item in cell_rows.  It’s not an editable table, and it gets its data from the global variable.

We then add an event handler to our XMPP connection to say that once the roster has been loaded from the server, we hand control over to the GUI thread and there we clear out the global variable and fill it up with the roster as we now see it.  The first item in each row is the name that will be shown on the button (either item.name or item.jid if there is no name set), the second item is the Jabber ID which won’t be shown because we didn’t define that column when we made the window.  Any updates to the global variable will be automatically painted into the GUI so we’re done.

One Window Per Conversation

For simplicity, let’s say we want to show one window per conversation, like so:

$conversations = {}

class Conversation
	include Glimmer

	def self.open(jid, m=nil)
		return if $conversations[jid]

		($conversations[jid] = new(jid, m)).launch
	end

	def initialize(jid, m=nil)
		@jid = jid
		@messages = [["", ""]]
		new_message(m) if m
	end

	def launch
		window("Conversation With #{@jid}") {
			vertical_box {
				table {
					text_column("Sender")
					text_column("Message")
					editable false
					cell_rows @messages
					@messages.clear
				}

				horizontal_box {
					stretchy false

					@message_entry = entry
					button("Send") {
						stretchy false

						on_clicked do
							BLATHER.say(@jid, @message_entry.text)
							@messages << [ARGV[0], @message_entry.text]
							@message_entry.text = ""
						end
					}
				}
			}

			on_closing do
				$conversations.delete(@jid)
			end
		}.show
	end

	def format_sender(jid)
		BLATHER.my_roster[jid]&.name || jid
	end

	def message_row(m)
		[
			format_sender(m.from&.stripped || BLATHER.jid.stripped),
			m.body
		]
	end

	def new_message(m)
		@messages << message_row(m)
	end
end

message :body do |m|
	LibUI.queue_main do
		conversation = $conversations[m.from.stripped.to_s]
		if conversation
			conversation.new_message(m)
		else
			Conversation.open(m.from.stripped.to_s, m)
		end
	end
end

Most of this is the window definition again, with a table of the messages in this conversation sourced from an instance variable @messages.  At the bottom of the window is an entry box to type in text and a button to trigger sending it as a message.  When the button is clicked, send that message to the contact this conversation is with, add it to the list of messages so that it shows up in the GUI, and make the entry box empty again.  When the window closes (on_closing this time because it’s not the “main” window) delete the object from the global set of open conversations.

This object also has a helper to open a conversation window if there isn’t already one with a given Jabber ID (jid), some helpers to format message objects into table rows by extracting the sender and body (including format_sender which gets the roster item if there is one, uses &.name to get the name if there was a roster item or else nil, and if there was no roster item or no name just show jid) and a helper that adds new messages into the GUI.

Finally we add a new XMPP event handler for incoming messages that have a body.  Any such incoming message we look up in the global if there is a conversation open already, if so we pass the new message there to have it appended to the GUI table, otherwise we open the conversation with this message as the first thing it will show.

Getting from the Contact List to a Conversation

Now we wire up the contact list to the conversation view:

button_column("Contact") {
	on_clicked do |row|
		Conversation.open($roster[row][1].to_s)
	end
}

When a contact button is clicked, grab the Jabber ID from the hidden end of the table row that we had stashed there, and open the conversation.

horizontal_box {
	stretchy false

	jid_entry = entry {
		label("Jabber ID")
	}

	button("Start Conversation") {
		stretchy false

		on_clicked do
			Conversation.open(jid_entry.text)
		end
	}
}

And let’s provide a way to start a new conversation with an address that isn’t a contact too.  An entry to type in a Jabber ID and a button that opens the conversation.

Adding a Contact

Might as well add a button to the main window that re-uses that entry box to allow adding a contact as well:

button("Add Contact") {
	stretchy false

	on_clicked do
		BLATHER.my_roster << jid_entry.text
	end
}

Handling Multiple Devices

In many chat protocols, it is common to have multiple devices or apps connected simultaneously. It is often desirable to show messages sent to or from one device on all the others as well.  So let’s implement that.  First, a helper for creating XML structures we may need:

def xml_child(parent, name, namespace)
	child = Niceogiri::XML::Node.new(name, parent.document, namespace)
	parent << child
	child
end

We need to tell the server that we support this feature:

when_ready do
	self << Blather::Stanza::Iq.new(:set).tap { |iq|
		xml_child(iq, :enable, "urn:xmpp:carbons:2")
	}
end

We will be handling live messages from multiple event handlers so let’s pull the live message handling out into a helper:

def handle_live_message(m, counterpart: m.from.stripped.to_s)
	LibUI.queue_main do
		conversation = $conversations[counterpart]
		if conversation
			conversation.new_message(m)
		else
			Conversation.open(counterpart, m)
		end
	end
end

And the helper that will handle messages from other devices of ours:

def handle_carbons(fwd, counterpart:)
	fwd = fwd.first if fwd.is_a?(Nokogiri::XML::NodeSet)
	return unless fwd

	m = Blather::XMPPNode.import(fwd)
	return unless m.is_a?(Blather::Stanza::Message) && m.body.present?

	handle_live_message(m, counterpart: counterpart.call(m))
end

This takes in the forwarded XML object (allowing for it to be a set of which we take the first one) and imports it with Blather’s logic to become hopefully a Message object.  If it’s not a Message or has no body, we don’t really care so we stop there. Otherwise we can handle this extracted message as though we had received it ourselves.

And then wire up the event handlers:

message(
	"./carbon:received/fwd:forwarded/*[1]",
	carbon: "urn:xmpp:carbons:2",
	fwd: "urn:xmpp:forward:0"
) do |_, fwd|
	handle_carbons(fwd, counterpart: ->(m) { m.from.stripped.to_s })
end

Because XMPP is just XML, we can use regular XPath stuff to extract from incoming messages.  Here we say that if the message contains a forwarded element inside a carbons received element, then we should handle this with the carbons handler instead of just the live messages handler.  The XML that matches our XPath comes in as the second argument and that is what we pass to the handler to get converted into a Message object.

message(
	"./carbon:sent/fwd:forwarded/*[1]",
	carbon: "urn:xmpp:carbons:2",
	fwd: "urn:xmpp:forward:0"
) do |_, fwd|
	handle_carbons(fwd, counterpart: ->(m) { m.to.stripped.to_s })
end

This handler is for messages sent by other devices instead of received by other devices.  It is pretty much the same, except that we know the “other side of the conversation” (here called counterpart) is in the to not the from.

message :body do |m|
	handle_live_message(m)
end

And our old message-with-body handler now just needs to call the helper.

History

So far our client only processes and displays live messages.  If you close the app, or even close a conversation window, the history is gone.  If you chat with another client or device, you can’t see that when you re-open this one.  To fix that we’ll need to store messages persistently, and also fetch any history from while we were disconnected from the server.  We will need a few more lines in our Gemfile first:

gem "sqlite3"
gem "xdg"

And then to set up a basic database schema:

require "securerandom"
require "sqlite3"
require "xdg"

DATA_DIR = XDG::Data.new.home + "jabber-client-demo"
DATA_DIR.mkpath
DB = SQLite3::Database.new(DATA_DIR + "db.sqlite3")

if DB.user_version < 1
	DB.execute(<<~SQL)
		CREATE TABLE messages (
			mam_id TEXT PRIMARY KEY,
			stanza_id TEXT NOT NULL,
			conversation TEXT NOT NULL,
			created_at INTEGER NOT NULL,
			stanza TEXT NOT NULL
		)
	SQL
	DB.execute("CREATE TABLE data (key TEXT PRIMARY KEY, value TEXT)")
	DB.user_version = 1
end

user_version is a SQLite feature that allows storing a simple integer alongside the database.  It starts at 0 if never set, and so here we use it to check if our schema has been created or not.  We store the database in a new directory created according to the XDG Base Directory specification.  There are two relevant IDs for most XMPP operations: the MAM ID (the ID in the server’s archive) and the Stanza ID (which was usually selected by the original sender).  We also create a data table for storing basic key-value stuff, which we’ll use in a minute to remember where we have sync’d up to so far.  Let’s edit the Conversation object to store messages as we send them, updating the send button on_clicked handler:

def message
	Blather::Stanza::Message.new(@jid, @message_entry.text, :chat).tap { |m|
		m.id = SecureRandom.uuid
	}
end

on_clicked do
	m = message
	EM.defer do
		BLATHER << m
		DB.execute(<<~SQL, [nil, m.id, @jid, m.to_s])
			INSERT INTO messages
			(mam_id, stanza_id, conversation, created_at, stanza)
			VALUES (?,?,?,unixepoch(),?)
		SQL
	end
	@messages << message_row(m)
	@message_entry.text = ""
end

When we send a message we don’t yet know the server’s archive ID, so we set that to nil for now.  We set mam_id to be the primary key, but SQLite allows multiple rows to have NULL in there so this will work.  We don’t want to block the GUI thread while doing database work so we use EM.defer to move this to a worker pool.  We also want to store messages when we receive them live, so add this to the start of handle_live_message:

mam_id = m.xpath("./ns:stanza-id", ns: "urn:xmpp:sid:0").find { |el|
	el["by"] == jid.stripped.to_s
}&.[]("id")
delay = m.delay&.stamp&.to_i || Time.now.to_i
DB.execute(<<~SQL, [mam_id, m.id, counterpart, delay, m.to_s])
	INSERT INTO messages (mam_id, stanza_id, conversation, created_at, stanza)
    VALUES (?,?,?,?,?)
SQL

Here we extract the server archive’s ID for the message (added by the server in a stanza-id with by="Your Jabber ID") and figure out what time the message was originally sent (usually this is just right now for a live message, but if it is coming from offline storage because every client was offline or similar, then there can be a “delay” set on it which we can use).  Now that we have stored the history of message we received we need to load them into the GUI when we start up a Conversation so add this at the end of initialize:

EM.defer do
	mam_messages = []
	query = <<~SQL
		SELECT stanza
		FROM messages
		WHERE conversation=?
		ORDER BY created_at
	SQL
	DB.execute(query, [@jid]) do |row|
		m = Blather::XMPPNode.import(
			Nokogiri::XML.parse(row[0]).root
		)
		mam_messages << m
	end

	LibUI.queue_main do
		mam_messages.map! { |m| message_row(m) }
		@messages.replace(mam_messages + @messages)
	end
end

In the worker pool we load up all the stored messages for the current conversation in order, then we take the XML stored as a string in the database and parse it into a Blather Message object.  Once we’ve done as much of the work as we can in we worker pool we use queue_main to switch back to the GUI thread and actually build the rows for the table and replace them into the GUI.

With these changes, we are now storing all messages we see while connected and displaying them in the conversation.  But what about messages sent or received by other devices or clients while we were not connected?  For that we need to sync with the server’s archive, fetching messages at a reasonable page size from whatever we already have until the end.

def sync_mam(last_id)
	start_mam = Blather::Stanza::Iq.new(:set).tap { |iq|
		xml_child(iq, :query, "urn:xmpp:mam:2").tap do |query|
			xml_child(query, :set, "http://jabber.org/protocol/rsm").tap do |rsm|
				xml_child(rsm, :max, "http://jabber.org/protocol/rsm").tap do |max|
					max.content = (EM.threadpool_size * 5).to_s
				end
				next unless last_id

				xml_child(rsm, :after, "http://jabber.org/protocol/rsm").tap do |after|
					after.content = last_id
				end
			end
		end
	}

	client.write_with_handler(start_mam) do |reply|
		next if reply.error?

		fin = reply.find_first("./ns:fin", ns: "urn:xmpp:mam:2")
		next unless fin

		handle_rsm_reply_when_idle(fin)
	end
end

The first half of this creates the XML stanza to request a page from the server’s archive. We create a query with a max page size based on the size of our worker threadpool, and ask for messages only after the last known id (if we have one, which we won’t on first run). Then we use write_with_handler to send this request to the server and wait for a reply. The reply is sent after all messages have been sent down (sent seperately, not returned in this reply, see below), but we may still be processing some of them in the worker pool so we next create a helper to wait for the worker pool to be done:

def handle_rsm_reply_when_idle(fin)
	unless EM.defers_finished?
		EM.add_timer(0.1) { handle_rsm_reply_when_idle(fin) }
		return
	end

	last = fin.find_first(
		"./ns:set/ns:last",
		ns: "http://jabber.org/protocol/rsm"
	)&.content

	if last
		DB.execute(<<~SQL, [last, last])
			INSERT INTO data VALUES ('last_mam_id', ?)
			ON CONFLICT(key) DO UPDATE SET value=? WHERE key='last_mam_id'
		SQL
	end
	return if fin["complete"].to_s == "true"

	sync_mam(last)
end

Poll with a timer until the worker pool is all done so that we aren’t fetching new pages before we have handled the last one.  Get the value of the last archive ID that was part of the page just processed and store it in the database for next time we start up.  If this was the last page (that is, complete="true") then we’re all done, otherwise get the next page.  We need to make sure we actually start this sync process inside the when_ready handler:

last_mam_id = DB.execute(<<~SQL)[0]&.first
	SELECT value FROM data WHERE key='last_mam_id' LIMIT 1
SQL
sync_mam(last_mam_id)

And also, we need to actually handle the messages as they come down from the server archive:

message "./ns:result", ns: "urn:xmpp:mam:2" do |_, result|
	fwd = result.xpath("./ns:forwarded", ns: "urn:xmpp:forward:0").first
	fwd = fwd.find_first("./ns:message", ns: "jabber:client")
	m = Blather::XMPPNode.import(fwd)
	next unless m.is_a?(Blather::Stanza::Message) && m.body.present?

	mam_id = result.first["id"]&.to_s
	# Can't really race because we're checking for something from the past
	# Any new message inserted isn't the one we're looking for here anyway
	sent = DB.execute(<<~SQL, [m.id])[0][0]
		SELECT count(*) FROM messages WHERE stanza_id=? AND mam_id IS NULL
	SQL
	if sent < 1
		counterpart = if m.from.stripped.to_s == jid.stripped.to_s
			m.to.stripped.to_s
		else
			m.from.stripped.to_s
		end
		delay =
			fwd.find_first("./ns:delay", ns: "urn:xmpp:delay")
			&.[]("stamp")&.then(Time.method(:parse))
		delay = delay&.to_i || m.delay&.stamp&.to_i || Time.now.to_i
		DB.execute(<<~SQL, [mam_id, m.id, counterpart, delay, m.to_s])
			INSERT OR IGNORE INTO messages
			(mam_id, stanza_id, conversation, created_at, stanza)
			VALUES (?,?,?,?,?)
		SQL
	else
		DB.execute(<<~SQL, [mam_id, m.id])
			UPDATE messages SET mam_id=? WHERE stanza_id=?
		SQL
	end
end

Any message which contains a MAM (server archive) result will get handled here.  Just like with carbons we extract the forwarded message and import, making sure it ends up as a Blather Message object with a body.

Remember how when we stored a sent message we didn’t know the archive ID yet?  Here we check if there is anything in our database already with this stanza ID and no archive ID, if no we will insert it as a new message, but otherwise we can update the row we already have to store the server archive ID on it, which we now know.

And with that, our client now stores and syncs all history with the server, to give the user a full view of their conversation no matter where or when it happened.

Display Names

If a user is added to the contact list with a name, we already show that name instead of their address in conversations.  What if a user is not a contact yet, or we haven’t set a name for them?  It might be useful to be able to fetch any display name they advertise for themselves and show that.  First we add a simple helper to expose write_with_handler outside of the main object:

public def write_with_handler(stanza, &block)
	client.write_with_handler(stanza, &block)
end

We need an attribute on the Conversation to hold the nickname:

attr_accessor :nickname

And then we can use this in Conversation#initialize to fetch the other side’s nickname if they advertise one and we don’t have one for them yet:

self.nickname = BLATHER.my_roster[jid]&.name || jid
return unless nickname.to_s == jid.to_s

BLATHER.write_with_handler(
	Blather::Stanza::PubSub::Items.new(:get).tap { |iq|
		iq.node = "http://jabber.org/protocol/nick"
		iq.to = jid
	}
) do |reply|
	self.nickname = reply.items.first.payload_node.text rescue self.nickname
end

Inside the window declaration we can use this as the window title:

title <=> [self, :nickname]

and in format_sender we can use this as well:

return nickname if jid.to_s == @jid.to_s

Avatars

Names are nice, but what about pictures?  Can we have nice avatar images that go with each user?  What should we display if they don’t have an avatar set?  Well not only is there a protocol to get an avatar, but a specification that allows all clients to use the same colours to represent things, so we can use a block of that if there is no avatar set.  Let’s generate the colour blocks first.  Add this to Gemfile:

gem "hsluv"

Require the library at the top:

require "hsluv"

$avatars = {}

And a method on Conversation to use this:

def default_avatar(string)
	hue = (Digest::SHA1.digest(string).unpack1("v").to_f / 65536) * 360
	rgb = Hsluv.rgb_prepare(Hsluv.hsluv_to_rgb(hue, 100, 50))
	rgba = rgb.pack("CCC") + "xff".b
	image { image_part(rgba * 32 * 32, 32, 32, 4) }
end

This takes the SHA-1 of a string, unpacks the first two bytes as a 16-bit little-endian integer, converts the range from 0 to MAX_SHORT into the range from 0 to 360 for hue degrees, then passes to the library we added to convert from HSV to RGB colour formats.  The GUI library expects images as a byte string where every 4 bytes are 0 to 255 for red, then green, then blue, then transparency.  Because we want a square of all one colour, we can create the byte string for one pixel and then multiply the string by the width and height (multiplying a string by a number in Ruby make a new string with that many copies repeated) to get the whole image.

In Conversation#initialize we can use this to make a default avatar on the dummy message row then the window first opens:

@messages = [[default_avatar(""), "", ""]]

And we will need to add a new column definition to be beginning of the table { block:

image_column("Avatar")

And actually add the image to message_row:

def message_row(m)
	from = m.from&.stripped || BLATHER.jid.stripped
	[
		$avatars[from.to_s] || default_avatar(from.to_s),
		format_sender(from),
		m.body
	]
end

If you run this you should now see a coloured square next to each message.  We would now like to get actual avatars, so add this somewhere at the top level to advertise support for this:

set_caps(
	"https://git.singpolyma.net/jabber-client-demo",
	[],
	["urn:xmpp:avatar:metadata+notify"]
)

Then in the when_ready block make sure to send it to the server:

send_caps

And handle the avatars as they come in:

pubsub_event(
	"//ns:items[@node='urn:xmpp:avatar:metadata']",
	ns: "http://jabber.org/protocol/pubsub#event"
) do |m|
	id = m.items.first&.payload_node&.children&.first&.[]("id")
	next $avatars.delete(m.from.stripped.to_s) unless id

	path = DATA_DIR + id.to_s
	key = m.from.stripped.to_s
	if path.exist?
		LibUI.queue_main { $avatars[key] = image(path.to_s, 32, 32) rescue nil }
	else
		write_with_handler(
			Blather::Stanza::PubSub::Items.new(:get).tap { |iq|
				iq.node = "urn:xmpp:avatar:data"
				iq.to = m.from
			}
		) do |reply|
			next if reply.error?

			data = Base64.decode64(reply.items.first&.payload_node&.text.to_s)
			path.write(data)
			LibUI.queue_main { $avatars[key] = image(path.to_s, 32, 32) rescue nil }
		end
	end
end

When an avatar metadata event comes in, we check what it is advertising as the ID of the avatar for this user.  If there is none, that means they don’t have an avatar anymore so delete anything we may have in the global cache for them, otherwise create a file path in the same folder as the database based on this ID.  If that file exists already, then no need to fetch it again, create the image from that path on the GUI thread and set it into our global in-memory cache.  If the file does not exist, then use write_with_handler to request their avatar data.  It comes back Base64 encoded, so decode it and then write it to the file.

If you run this you should now see avatars next to messages for anyone who has one set.

Delivery Receipts

The Internet is a wild place, and sometimes things don’t work out how you’d hope.  Sometimes something goes wrong, or perhaps just all of a user’s devices are turned off.  Whatever the reason, it can be useful to see if a message has been delivered to at least one of the intended user’s devices yet or not.  We’ll need a new database column to store that status, add after the end of the DB.user_version < 1 if block:

if DB.user_version < 2
	DB.execute(<<~SQL)
		ALTER TABLE messages ADD COLUMN delivered INTEGER NOT NULL DEFAULT 0
	SQL
	DB.user_version = 2
end

Let’s advertise support for the feature:

set_caps(
	"https://git.singpolyma.net/jabber-client-demo",
	[],
	["urn:xmpp:avatar:metadata+notify", "urn:xmpp:receipts"]
)

We need to add delivery status and stanza id to the dummy row for the messages table:

@messages = [[default_avatar(""), "", "", false, nil]]

And make sure we select the status out of the database when loading up messages:

SELECT stanza,delivered FROM messages WHERE conversation=? ORDER BY created_at

And pass that through when building the message rows

mam_messages << [m, row[1]]

mam_messages.map! { |args| message_row(*args) }

Update the messages table to expect the new data model:

table {
	image_column("Avatar")
	text_column("Sender")
	text_column("Message")
	checkbox_column("Delivered")
	editable false
	cell_rows @messages
	@messages.clear if @messages.length == 1 && @messages.first.last.nil?
}

And update the row builder to include this new data:

def message_row(m, delivered=false)
	from = m.from&.stripped || BLATHER.jid.stripped
	[
		$avatars[from.to_s] || default_avatar(from.to_s),
		format_sender(from),
		m.body,
		delivered,
		m.id
	]
end

Inbound messages are always considered delivered, since we have them:

def new_message(m)
	@messages << message_row(m, true)
end

And a method to allow signalling that a delivery receipt should be displayed, using the fact that we now hide the stanza id off the end of the rows in the table to find the relevant message to update:

def delivered_message(id)
	row = @messages.find_index { |r| r.last == id }
	return unless row

	@messages[row] = @messages[row][0..-3] + [true, id]
end

In the Send button’s on_clicked handler we need to actually request that others send us receipts:

m = message
xml_child(m, :request, "urn:xmpp:receipts")

And we need to handle the receipts when they arrive:

message "./ns:received", ns: "urn:xmpp:receipts" do |m, received|
	DB.execute(<<~SQL, [received.first["id"].to_s])
		UPDATE messages SET delivered=1 WHERE stanza_id=?
	SQL

	conversation = $conversations[m.from.stripped.to_s]
	return unless conversation

	LibUI.queue_main do
		conversation.delivered_message(received.first["id"].to_s)
	end
end

When we get a received receipt, we get the id attribute off of it, which represents a stanza ID that this receipt is for.  We update the database, and inform any open conversation window so the GUI can be updated.

Finally, if someone requests a receipt from us we should send it to them:

message :body do |m|
	handle_live_message(m)

	if m.id && m.at("./ns:request", ns: "urn:xmpp:receipts")
		self << m.reply(remove_children: true).tap { |receipt|
			xml_child(receipt, :received, "urn:xmpp:receipts").tap { |received|
				received["id"] = m.id
			}
		}
	end
end

If the stanza has an id and a receipt request, we construct a reply that contains just the received receipt and send it.

Message Correction

Sometimes people send a message with a mistake in it and want to send another to fix it.  It is convenvient for the GUI to support this and render only the new version of the message.  So let’s implement that.  First we add it to the list of things we advertise support for:

set_caps(
	"https://git.singpolyma.net/jabber-client-demo",
	[],
	[
		"urn:xmpp:avatar:metadata+notify",
		"urn:xmpp:receipts",
		"urn:xmpp:message-correct:0"
	]
)

Then we need a method on Conversation to process incoming corrections and update the GUI:

def new_correction(replace_id, m)
	row = @messages.find_index { |r| r.last == replace_id }
	return unless row

	@messages[row] = message_row(m, true)
end

We look up the message row on the stanza id, just as we did for delivery receipts, and just completely replace it with a row based on the new incoming message.  That’s it for the GUI.  Corrections may come from live messages, from carbons, or even from the server archive if they happened while we were disconnected, so we create a new insert_message helper to handle any case we previously did the SQL INSERT for an incoming message:

def insert_message(
	m,
	mam_id:,
	counterpart: m.from.stripped.to_s,
	delay: m.delay&.stamp&.to_i
)
	if (replace = m.at("./ns:replace", ns: "urn:xmpp:message-correct:0"))
		DB.execute(<<~SQL, [m.to_s, counterpart, replace["id"].to_s])
			UPDATE messages SET stanza=? WHERE conversation=? AND stanza_id=?
		SQL
	else
		delay ||= Time.now.to_i
		DB.execute(<<~SQL, [mam_id, m.id, counterpart, delay, m.to_s])
			INSERT OR IGNORE INTO messages
			(mam_id, stanza_id, conversation, created_at, stanza, delivered)
			VALUES (?,?,?,?,?,1)
		SQL
	end
end

The else case here is the same as the INSERTs we’ve been using up to this point, but we also check first for an element that signals this as a replacement and if that is the case we issue an UPDATE instead to correct our internal archive to the new version.

Then in handle_live_message we also signal the possibly-open GUI:

if (replace = m.at("./ns:replace", ns: "urn:xmpp:message-correct:0"))
	conversation.new_correction(replace["id"].to_s, m)
else
	conversation.new_message(m)
end

We can now display incoming corrections, but it would also be nice to be able to send them.  Add a second button after the Send button in Conversation that can re-use the @message_entry box to correct the most recently sent message:

button("Correct") {
	stretchy false

	on_clicked do
		replace_row = @messages.rindex { |message|
			message[1] == format_sender(BLATHER.jid.stripped)
		}
		next unless replace_row

		m = message
		m << xml_child(m, :replace, "urn:xmpp:message-correct:0").tap { |replace|
			replace["id"] = @messages[replace_row].last
		}
		EM.defer do
			BLATHER << m
			DB.execute(<<~SQL, [m.to_s, @jid, @messages[replace_row].last])
				UPDATE messages SET stanza=? WHERE conversation=? AND stanza_id=?
			SQL
		end
		@messages[replace_row] = message_row(m, @messages[replace_row][-2])
		@message_entry.text = ""
	end
}

When the button is clicked we find the row for the most recently sent message, construct a message to send just as in the Send case but add the message correction replace child with the id matching the stanza id of the most recently sent message.  We send that message and also update our own local copy of the stanza both in the database and in the memory model rendered in the GUI.

Conclusion

There are a lot more features that a chat system can implement, but hopefully this gives you a useful taste of how each one can be incrementally layered in, and what the considerations might be for a wide variety of different kinds of features.  All the code for the working application developed in this article is available in git under AGPLv3+, with commits that corrospond to the path we took here.

by Stephen Paul Weber at December 01, 2022 04:23

November 30, 2022

Ignite Realtime Blog

HTTP File Upload plugin 1.2.0 released

We have now released version 1.2.0 of the HTTP File Upload plugin!

This plugin adds functionality to Openfire that allows clients to share files, as defined in the XEP-0363 ‘HTTP File Upload’ specification.

This release primarily enhances functionality when running in an Openfire cluster. All changes can be reviewed in the changelog for this release of the plugin.

As always, your instance of Openfire should automatically display the availability of the update. Alternatively, you can download the new release of the plugin at the HTTP File Upload plugin’s archive page.

For other release announcements and news follow us on Twitter

1 post - 1 participant

Read full topic

by guus at November 30, 2022 16:17

November 28, 2022

Prosodical Thoughts

Bringing FASTer authentication to Prosody and XMPP

As our work continues on modernizing XMPP authentication, we have some more new milestones to share with you. Until now our work has mostly been focused on internal Prosody improvements, such as the new roles and permissions framework. Now we are starting to extend our work to the actual client-to-server protocol in XMPP.

Prosody and Snikket are both regularly used from mobile devices, which have intermittent connectivity. Even if it’s only a change between networks, or when driving through a tunnel for a few minutes, these things can temporarily break your connection - requiring a new one to be established.

We’ve had solutions and optimizations in the XMPP protocol for this situation for years (really… the first version of XEP-0198 was published in 2004!). XEP-0198 allows a client to reconnect to the server as soon as the network comes back, easily discover if anything failed to be sent/received due to the network interruption, and then resync any lost packets in either direction.

This effectively allows resuming and repairing the session as if no disconnect occurred, while skipping a bunch of traffic that would usually be exchanged when establishing a new session (instead, everything is simply cached from the old session).

However, there is one important thing we don’t allow the client to skip. To keep this resumption step secure, we require authentication. It’s a new connection, and we need to prove it’s from who it claims to be from.

Authentication in XMPP today

The most common authentication method for XMPP connections today is SCRAM. This is a neat password-based authentication mechanism that has many nice properties, such as allowing both the client and the server to store only a hash of the password. It also allows the client to determine that the server really knows the user’s password, and supports channel binding. These features allow the client to detect various kinds of attack.

Even though we have been using SCRAM in XMPP for many years now, it still offers more protective features today than the vast majority of online services you use - which generally all send your password to the server in plain text, albeit within TLS or HTTPS.

A new SCRAM alternative is currently being developed, known as OPAQUE, which adds even more nice properties. But that’s for future blog post… :)

However, there are some drawbacks of SCRAM (and similar mechanisms, including OPAQUE) that can’t realistically be solved. To adequately protect your password, it requires some back-and-forth negotiation with the server. In protocol speak, we refer to such situations as “round trips” - every time the client sends something to the server and has to wait for a response before it can proceed. On a very slow network, round trips can add a bunch of latency, and as anyone who has used audio/video calls or gaming online knows, latency can be frustrating and hard to eliminate from a connection.

Simpler authentication methods just have the client say “here are my credentials”, and the server say “your credentials look great, you’re authenticated!“. That’s how HTTP and most websites work today. Such approaches are quick and easy, but they don’t protect your credentials as well as SCRAM does.

Passwords are the problem

SCRAM’s protections are important for passwords. Passwords are (unfortunately) often chosen by users to be the same or similar across multiple services, and even if they are strong and unique they can be vulnerable to phishing. If leaked, many memorable passwords contain private information about the user.

We don’t want to drop any of our important password security features just to improve connection speed. So instead we found a better solution: drop passwords!

Our new solution allows the client to log in initially using a password (or any other method the XMPP server supports). After that, it can upgrade to a strong unique authentication token provided by the server, which it can use to quickly re-authenticate on future connections.

Tokens are the answer

Tokens have many advantages compared to passwords:

  • They are unique to the service that generated them, so cross-service attacks like credential stuffing are useless against tokens.
  • Tokens don’t need to be memorable, so they can be very long and random (both desirable properties for increasing account security!).
  • As they are not memorized by the user, they can be rotated frequently without any inconvenience.
  • Different tokens can be generated for each of a user’s devices, instead of sharing the user’s password across all of them. This also allows selectively removing a device’s access from the user’s account, e.g. if it gets lost or stolen.

With these security advantages, we suddenly unlock the ability to use simpler authentication mechanisms without risking the security of the user’s account or password.

Still, we can do a bit better than just sending the token to the server as plain text. Fortunately, just the kind of modern token authentication method we need has already been in development by Florian Schmaus: the SASL HT mechanism family.

HT mechanisms have the following properties:

  • The actual token itself is not exchanged over the connection during authentication.
  • And yet, the server receives proof that the client has the full correct token.
  • The client also receives proof that the server has the full correct token (and isn’t just impersonating the real server).
  • Finally, if channel binding is used, both sides receive proof that no MITM or relay attack being performed.

And… all this can be completed within a single round trip!

The protocol to achieve this has been submitted to the XSF as “Fast Authentication Streamlining Tokens”. It is in the acceptance queue, so doesn’t have a XEP number assigned yet.

Updating and integrating with SASL2

If FAST authentication was the only thing we had been working on recently, we would be happy enough. But there’s more…

In collaboration with Thilo Molitor from the Monal project, a new version of XEP-0388 (SASL 2) has been submitted. SASL 2 was originally proposed back in 2017, and it defines a new authentication protocol for XMPP (still based on SASL, so we can reuse all the existing mechanisms we already have in place).

Several features of SASL 2 are very relevant to our work. For example, it allows negotiation of session features in parallel with the authentication process. The old way required the client to authenticate, and then proceed to negotiate whatever features and parameters it wanted for the new session. With SASL2 the client can provide this information at the same time it provides its credentials. This saves yet more round trips.

As well as SASL 2, we’ve also updated a related proposal from around the same time, XEP-0386 (Bind 2). This is also a critical piece of session establishment that integrates with SASL 2.

With the work we’ve done across these three specifications - XEP-0388, XEP-0386 and FAST - we’ve essentially overhauled the entire authentication and session establishment protocol of XMPP. Even with all our additional authentication security features, it’s now possible for a client to connect, authenticate, and resume or create a session in a single request and response.

This post shouldn’t be taken as being entirely about performance improvements. It’s nice to be able to (re)connect to the server in the blink of an eye. But there are other reasons to be working on this.

As anyone who used XMPP in 2012 and 2022 knows, XMPP has been continuously evolving as both the internet and the way people use it has changed. Over time we have “bolted on” various features to the connection process to achieve this evolution.

Now, with these new changes, we are bringing all these enhancements together into a single framework that was designed for them to fit neatly into. Not only are we reducing round trips, we are also simplifying connection establishment for the next generation of XMPP developers.

When can I use all this?

Even though this is all cutting edge stuff, you’ll be able to use it much sooner than you might think!

Prosody has support for the new SASL 2, Bind 2 and FAST protocols. They are all available as community modules right now, though we intend for them to become part of the main Prosody distribution eventually.

To get started, you’ll need a Prosody trunk nightly build, and simply enable the following community modules:

To take advantage of the new features, you’ll need a compatible client. FAST is already implemented in multiple clients, and will be available from Conversations 2.11 for Android, as well as the next major versions of Monal, Siskin and Beagle for iOS and MacOS.

Gajim already has SASL 2 implemented, and other client developers have also already expressed an interest in support.

If you’re a client or library developer interested in supporting any of this, we have a test server available that you are welcome to use. Just let us know!

Do remember that all this is still very new and experimental. The relevant protocol specifications are still working their way through the XSF standards process and there may be changes to come in the future. There may also be undiscovered bugs. We encourage brave souls to help test it all in real world deployments, but if your priority is keeping a stable setup, you should probably wait a little longer before deploying any of this.

TCP Fast Open

While this post is not just about performance improvements, we’ve talked a lot about performance improvements. Therefore it’s worth noting an extra little side feature at this point.

Prosody trunk builds, when used with the new LuaSocket 3.1.0, support something known as TCP Fast Open. This is a low-level TCP extension that allows new connections to skip a round trip, by exchanging initial data packets while the connection is being established.

It’s disabled for servers by default on Linux, but you can enable it on most modern systems by creating the file /etc/sysctl.d/tcp-fastopen.conf with the contents:

net.ipv4.tcp_fastopen=3

Run systemctl restart systemd-sysctl.service to apply the changes. More information on the sysctl configuration can be found in the Linux kernel documentation.

In Prosody’s config, add the following in the global section:

network_settings = {
    tcp_fastopen = 256;
}

Restart Prosody to apply these changes. Be aware that some networks and routers have been reported to be incompatible with TCP Fast Open (support was removed from Firefox for this reason). Although Linux has built-in recovery mechanisms that should work around such issues, if you experience trouble connecting to your server from certain networks, you may want to try turning this off again.

We’re also looking at support for TLS 1.3’s 0-RTT mode, which can be combined with FAST authentication and TCP Fast Open to achieve full connection establishment within a single round-trip. Pretty impressive!

Next steps

These protocol changes are yet another step on our XMPP authentication modernization journey. With the new protocols now written and implemented, we can start looking forward to the next milestones for the project.

In the coming months, we’ll be working on the ability to sign in to your XMPP account from third-party clients and services without sharing your password with them. Subscribe to our blog or Mastodon account and keep an eye out for that future post!

by The Prosody Team at November 28, 2022 17:30

November 24, 2022

Jérôme Poisson

Libervia progress note 2022-W45

Hello, it's time for a long overdue progress note.

I'll talk here about the work made on ActivityPub (AP) gateway and on end-to-end encryption around pubsub.

Oh, and if everything goes well, this blog post should be accessible from XMPP and ActivityPub (and HTTP and ATOM feed), using the same identifier goffi@goffi.org.

Forewords

The work made on the AP gateway has been possible thanks to a NLnet/NGI0 grant (with financial support from the European Commission's Next Generation Internet programme).

I especially appreciated that the team was really there to help bring the ideas to life, and not once did they get in the way: little paperwork, no unnecessary pressure, caring, contacts when help was needed, etc.

I wish there were more organizations like this one that really help develop libre projects for the common good.

So once again I want to thank them for all that.

XMPP ⬌ ActivityPub Gateway

There is probably no need to explain here what is ActivityPub, we can simply write that it is an open protocol that allows to do things that XMPP also allows doing, and that until now these 2 protocols could not communicate together. The work on the ActivityPub gateway aims to allow software implementing one of these 2 protocols to communicate as easily as possible. I firmly believe that all open protocols should be able to communicate which each other, to avoid creating more silos, proprietary software is already good enough at that.

To be useful, a gateway must use the full potential of both protocols. A simple bot transcribing messages as we see too often, using unsuitable features (such as instant messaging for blog posts), or using a very limited set of features to ensure compatibility are flaws that I have tried to avoid. Building a good gateway is a difficult and time-consuming task. If done right, the gateway should be as invisible as possible to the end user.

XMPP is featuring blogging since long before AP, however the set of features is not exactly the same. Current use of AP is clearly inspired from commercial "social" networks, and metadata such as subscribers/subscribed nodes (or followers/following in AP terms) are highlighted, feature such as like/favourite were missing in XMPP, and some implementation such as Pleroma do implement reactions. To integrate that in the gateway, I've been working on new specifications:

  • Pubsub Public Subscriptions: a way to publicly announce subscriptions, in an opt-in way. With this it's possible to implement followers/following features in a way respectful of privacy.

  • Pubsub Attachments: a generic way to attach any kind of data to a pubsub item. It's notably used to implements noticed/favourite button (see here and reactions.

  • Calendar Events: handling of events and all the RSVP mechanism. Libervia was handling events for years, but it was an experimental implementation, this specification is a next step in the effort to make it a standard.

Note that this XEP and the others linked below have been accepted but are not yet visible in official list.

You may wonder why there is a specification for Calendar Events… It's because the AP gateway also handles them, making it compatible with Mobilizon. The gateway may evolve in the future to support other non (micro)blogging use cases.

The gateway is now finished in terms of functionalities, however the code is clearly of an alpha quality for the moment. Now the goal in the coming months will be to stabilize and possibly implement other features if there is a demand for it.

Early adopters are encouraged to try and test it as long as they keep in mind that it's not stable. So if you do try it, I recommend keeping a separate ActivityPub account in whatever stable implementation you use at the moment, this way you can check if messages or media are missing, if there is any inconsistency or other bugs, and report them to me. If you test it, please join the XMPP room libervia@chat.jabberfr.org (click here to connect from your browser) for help and feedback. Stabilization will probably take weeks, but I hope to have it done by early 2023.

Installation instructions and details on how the conversion between protocols is done is available in the documentation and notably here

A question I've been asked a lot: yes, you can use the same identifier for XMPP (JID) and AP (WebFinger actor handle) as long as you use "simple" characters (i.e. alphanumeric ASCII chars, _, . and -). If you use something more complicated, you'll have to use the escaping mechanism explained in the doc (this is due to constraints with some AP implementations).

As for blogs on pubsub nodes (what Movim calls "communities"), I made it simple: you can use directly the name of the node that holds the blog in the local part (i.e. before the "@") of your actor handle: a blog named community_bog at the XMPP pubsub service pubsub.example.org can thus be addressed with the AP actor community_blog@pubsub.example.org. This way you can use a rather user-friendly identifier to share your blog with people who are only on ActivityPub.

This gateway should work with any XMPP server, and any client that implement blogging features (Only Libervia itself and Movim implement it for now, but I have heard that other clients are planning support for it). To enjoy the whole feature set of the gateway, the new specifications need to be implemented by the clients, so you can start to fill feature requests…

With this gateway, the door is open to have a client able to talk to the ActivityPub network, while having the feature of XMPP, including e2e encrypted private messages (e2e encrypted only if you communicate with an XMPP account, not with an AP one).

Oh, and please update your graphics, drawing and other texts to include XMPP in the fediverse ;)

End-to-End Encryption

Much effort has also gone into end-to-end encryption.

OMEMO implementation has been updated (OMEMO:2 is now used), including Stanza Content Encryption which allows encrypting arbitrary elements instead of only the \<body/> of the message, I believe that Libervia is the first XMPP client to implement it. OpenPGP for XMPP (or "OX") has also been implemented, all that thanks to the work of Tim Henkes "Syndace", the author of python-omemo.

Beside instant messaging, end-to-end encryption has also been introduced to pubsub. I've made specifications for two methods:

  • An OpenPGP profile for pubsub which is thought to encrypt a whole node, with a system of secret sharing/rotation/revocation. With it, it is easy to give access to new entities after publication, and to retrieve old items for newcomers. This specification can be used to encrypt any pubsub based features: (micro)blogging, calendar events, lists, etc.

  • Pubsub Targeted Encryption which is a way to apply the same cryptographic system used in instant messaging to pubsub. This way, OMEMO can be used with its forward secrecy property. It is not a good option to use this specification to encrypt a whole node, as archive is then not accessible to newcomers, and to add access to a new entity you have to re-encrypt all items, but it's an interesting option to encrypt an element occasionally, for instance to restrict access of a specific post in an otherwise public blog.

Specifications have also been written to sign a pubsub item in a backward compatible way (client which don't implement those specifications can still work normally):

All those specifications are already implemented in Libervia, but they are only usable from CLI frontend at the moment. All you have to do is to use the --encrypt and/or --sign options from pubsub or blog commands (check documentation for details).

Uploaded files were already encrypted with OMEMO Media Sharing which is what is commonly used these days, but this method has not been accepted as a standard as it was a workaround for limitation of legacy OMEMO implementation. The proper way is now specified with Stateless File Sharing and is encrypted with Encryption For Stateless File Sharing. Those methods are currently only usable when OMEMO:2 is implemented in the peer client, and with them metadata on the shared file can be attached, including thumbnails.

Encryption has also been implemented for Jingle (XEP-0391 and XEP-0396), which is notably used for Jingle File Transfer (specially useful for large files transfers).

So to summarize, nearly everything (instant messaging, files uploaded, large file transfers, all pubsub related features) can now be e2e encrypted with Libervia.

Possible Future

With the AP gateway permitting to reach the whole AP network, all the new features implemented, and the work done on e2e encryption, Libervia has everything to be a solid option for communication. After the recent events regarding a famous commercial network, we see a breakthrough of ActivityPub that will hopefully last over time. We can now access AP from XMPP, while having the possibility to have e2e encrypted private conversations or even blogs or calendar events.

As far as I know this is, so far, something unique for a Libre decentralized software. However, there is still work to do on stabilization on UI/UX update before this is really usable.

Those feature were planned for very long (years), but the lack of resources made them slow to come. The grant has made it possible to greatly accelerate the pace of development, and I doubt that it would have been possible to have all that without it.

Regarding how large the project is, and my family life, it's not possible any more to develop seriously this project on my free time alone (and I would like to do other things, sometimes, of my free time).

In other words, I need to find a way to sustain the development of Libervia for the years to come, so I can work full-time on it, and with some luck, build a team. I'm thinking very seriously about it these days, I'll probably write on this topic in a little while. If you are willing to help in any way, please contact me (on the Libervia room linked above for instance).

That's all for this progress note. I'm now working on stabilization and UI/UX update on the web frontend.

by goffi at November 24, 2022 11:58

November 22, 2022

Ignite Realtime Blog

Openfire Monitoring Service plugin 2.4.0 release

Earlier today, we have released version 2.4.0 of the Openfire Monitoring Service plugin. This plugin adds both statistics, as well as message archiving functionality to Openfire.

In this release, compatibility with future versions of Openfire is added. A bug that affects MSSQL users has been fixed, and the dreaded “Unable to save XML properties” error message has been resolved. A few other minor tweaks have been added.

As always, your instance of Openfire should automatically display the availability of the update. Alternatively, you can download the new release of the plugin at the Monitoring plugin’s archive page.

For other release announcements and news follow us on Twitter

4 posts - 2 participants

Read full topic

by guus at November 22, 2022 21:11

November 16, 2022

ProcessOne

ejabberd turns 20

ejabberd is a piece of software that was born 20 years ago. This is a long time, even at the scale of Internet. And yet, what ejabberd represents has not always been obvious. It took us a long time to realize what was so important about ejabberd. Why have we been developing it for 20 years? Why are we pushing it further even today? What makes it so special?


ejabberd is a scalable messaging server. That sums it all and that does not do justice to this critical piece of the Internet infrastructure. Sure, it is known to be the most scalable XMPP server, so scalable that it was used as a building brick to build Whatsapp messaging service. This is something that we have always been proud of, something you can easily brag about when meeting your friends.

But is that just it? Of course not. Today, with the troubles at Twitter, something appeared clearly.

ejabberd is important because it helped build much more than Whatsapp or any other big name high-profile projects we have built. It is important because it makes people communicate, in a federated way. It is important because it implements open protocols, and now several of them: XMPP, MQTT, SIP and now Matrix.

It’s about federation

ejabberd is about federation. It is helping people on different servers, domains, companies, communities or even countries chatting together. And today even more than 20 years ago, it really matters. We have built ejabberd for 20 years, because it is a critical building brick of what makes the Internet exists. Openness, interoperability, federation. It is one of the few software that prosper outside of the spotlights and make the Internet what it is, along for example with web and mail servers.

This is something we are pondering as we are thinking about the next steps, the next 20 years. But deep down, we know for sure, what we are about. ejabberd is about federation. You will read more from us here soon. It is a tradition. No birthday celebration speech is complete without looking back at the past.

It is hard to track all ejabberd usage, but we know that ejabberd empowers more than a billion users. Not bad for a piece of code we wrote. Trillions of messages went through our lines of code.

As mentioned in this post ten years ago:

Closed protocols come and go – ejabberd and XMPP remains

Happy 20th birthday, ejabberd!

Brief timeline

The very first public commit in ejabberd’s source code was done by Alexey Shchepin the 16th November of 2002. That was in the Jabber.ru CVS server. Later when that machine had technical problems, the development code moved to JabberStudio CVS.

The first official ejabberd release was ejabberd 0.5 in November 2003. The ejabberd home page at that time was a simple HTML. It’s also worth checking the early stage of the Ejabberd Installation and Operation Guide. Notice this first ejabberd logo represented a frog-like animal sitting on a “Jabber globe bulb,” with bat wings, dangerous-looking cogs, and an Erlang suit.

After the 0.7.5 release in October 2004, ejabberd home page moved from JabberStudio to ejabberd.jabber.ru, and the bug tracker to Jabber.ru’s Bugzilla. For this Drupal site, the logo changed to a hedgehog, and that would remain ten years until the final website and logo update in 2015.

At the beginning of 2005, JabberStudio CVS had technical difficulties and the development code moved to ProcessOne SVN. Notice that ProcessOne contracted Alexey to work in J-EAI, a project based in ejabberd specially designed for some business usages, and later extended that relationship to ejabberd.

In February 2005 the source code repository moved from SVN to Git, and the bug tracker to JIRA. Around October 2010 the source code repository and the bug tracker were finally moved to GitHub.

From around that time, there’s an interview to Alexey Shchepin which covers the initial concept and years of ejabberd development. By the way, there was another interview two years ago.

The ejabberd code base got a relevant massive change with the data binarization (use Erlang binaries instead of Erlang strings for data representation) in March 2013, which jumped ejabberd version from 2.1.12 to 13.03.

The next years followed another major source code change: the movement of many C/C++ code to independent external libraries.

Today, ejabberd is not just about XMPP. Even if it is mostly know for its great XMPP support, it also supports several other protocols:
– SIP support to connect SIP phone was added in 2014 (see ejabberd 14.05)
– Support for MQTT protocol, to better support Internet of Things use cases, was added initially in the Business Edition, and some months later added to the Community Server 19.02.
– Right now Matrix federation is being introduced to allow interop between ejabberd and Matrix servers, to ejabberd Business Edition internally or on Fluux ejabberd SaaS platform. It will come later to ejabberd Community Server.

ejabberd keeps on improving at a steady pace and is happy to open to other protocols and communities.

Some source code statistics

The oldest unchanged function in ejabberd is probably one of the least used: stop/0. And the oldest functional line is the SETS macro definition.

The ejabberd repository got 1,070,325 line insertions and 901,287 line deletions. When counting both ejabberd and the dependency libraries, they got 1.693.180 line insertions and 1.108.873 line deletions. With all this, the ejabberd source code went from 13 files to 868, from 1,448 lines of code to 480,961.

Looking at the programming languages of ejabberd and its libraries, Erlang is obviously the major one, and C comes as a relevant second:

Language Files Lines Code Comments Blanks
ABNF 3 128 110 3 15
ASN.1 1 14 10 0 4
Autoconf 14 696 544 33 119
Batch 4 31 21 0 10
C 20 187549 139764 39614 8171
C Header 8 11199 3783 6979 437
C++ 1 533 442 17 74
CSS 5 532 507 0 25
Elixir 32 1888 1469 130 289
Erlang 604 247901 208912 18368 20621
JavaScript 2 23 21 1 1
Lua 1 16 16 0 0
Makefile 21 848 635 10 203
Perl 3 1086 897 63 126
Python 1 53 49 0 4
RPM Specfile 3 5408 3928 1059 421
Shell 21 4031 3263 336 432
SQL 9 3857 2994 303 560
TCL 3 1179 1002 69 108
Plain Text 13 1870 0 1561 309
YAML 20 1448 1357 53 38
The post ejabberd turns 20 first appeared on ProcessOne.

by Mickaël Rémond at November 16, 2022 15:17

November 14, 2022

Ignite Realtime Blog

Spark 3.0.0 Released

The Ignite Realtime community is happy to announce the release of Spark 3.0.0 version.

We decided to increase major version to 3.x to coincide with a complete UI refresh of Spark which was contributed by Amos . Now Spark uses only FlatLaf Look and Feel. We are very much grateful for his incredible work. Along that Pade Meetings plugin was added by Dele. Spark now uses the latest version of the Smack 4.4 library, thanks Flow. Now Spark works on any Java 8 and more. Improved file transfer compared to Spark 2.9.4 and also updated the HttpFileUpload plugin for file transfer.

Pade Meetings (formerly SparkMeet) plugin is included in this release. You can find it in a chat window, a button with P letter. This plugin works with Openfire Meetings and enables audio and video chat. It was modified by Dele Olajide to use Electron technology. First time a user logs in using this version Spark will download required libraries into user’s profile (it is around 150 MB on Windows, so can take a minute to download). You can also manage all plugins in Spark via Client Control Pluigin and disable them if you want.

Installers:
For Windows has been added 64bit installer and MSI installer, which will be useful for use in a corporate environment.
For MacOS has been replaced the installer because modern versions require a digital signature.

The release also contains bug fixes and many improvements.

Full list of changes can be found in the changelog .

We encourage users and developers to get involved with Spark project by providing feedback in the forums or submitting pull requests on our GitHub page.

You can download Spark from the Downloads page. Below are the sha256 checksums:

5254d3559c74715f2ffebc4f30c55c873013e5dfb225d767577a8bda6434ffd9  spark_3_0_0-64bit.exe
69946b260155ffb5b2cd5cf6c6b44fe4cff7cf3d43a3583b4817bc7673d06205  spark_3_0_0-64bit.msi
79297356f681f4e470d4f99b29f110e93dce40b2cfd55845487cdc6e92f37d85  spark_3_0_0-arm.exe
6e21e29dd954df7bb38523ac620082fd13baf26477cf380307620721a14acb97  spark_3_0_0.deb
51f4cb414618a33f18f0b29002899ade2668b115c508d3579c2323d346211c6c  spark_3_0_0.dmg
b942e1a68bbe4dbc3641041d4e26c85926811f429cbb106653b22c838d96cdbb  spark_3_0_0.exe
9bb898ea398719ce548875c3447f3dcb18dc6af7a68ff521f6aeb211192ebc4d  spark_3_0_0.msi
6035bb35e8f52ea715de371620f855ab8975ea96e3edddc991eeff45c8e1c28e  spark-3_0_0.rpm
016d728edb6236aa8fc0bd77a2269cb0e39c3c2d587441b782f0e8cecb61b3b8  spark_3_0_0.sh
f5125d672424deec105066b29d36fd655e10b979213aab134a39edf0c9a92fbc  spark_3_0_0.tar.gz
84efa849883d5f96f67f3ac6c74fd3a42869167be6cc3f7b36da41b0c7ce4d46  spark_3_0_0-with-jre.dmg
f9beacca9b1633dc6be0b84baec065c9f876351ce19c16bf5077c6c8cbc59459  spark_3_0_0-with-jre.exe
96b853cb3e44aad1c9a01c3b42b9f85f564a53aaf877e7145b1314d3b2be8af3  spark_3_0_0-with-jre.msi

For other release announcements and news follow us on Twitter

18 posts - 6 participants

Read full topic

by ilyaHlevnoy at November 14, 2022 18:34

November 12, 2022

Dino

Stateless File Sharing: Source Attachment and Wrap-Up

Recap

Stateless file sharing (sfs) is a generic file sharing message which, alongside metadata, sends a list of sources where the file can be retrieved from. It is generic in the sense, that sources can be from different kinds of file transfer methods. HTTP, Jingle and any other file transfers can be encapsulated with it. The big idea is that functionality can be implemented for all file transfer methods at once, thanks to this wrapper.

Source Attaching

The idea is simple: Once somebody started sfs, anybody can contribute more sources to the list of sources. A sfs-attachment consists of the message id of the original sfs, as well as a list of sources that should be attached.

You might ask: “But why? I have used many messengers and never heard of such a feature, why should somebody else in the chat be able to do that?”

I was surprised as well, but there actually are some reasonable use cases.

  1. Peer-to-Peer file sharing in groups
  2. Reviving dead download links
  3. Announcing file uploads before the upload is finished

Security Considerations

We obviously have to prevent the possibility of other members attaching wrong files to other people’s sfs. To prevent this, sources can only be attached to sfs, which have checksums in their metadata.

Clients need to dismiss source attachments,

  1. if the sfs didn’t provide a checksum
  2. if the checksum algorithm is not trusted or unknown
  3. if the checksum of the downloaded file doesn’t match

Google Summer of Code Retrospect

This was the first time with Google Summer of Code and honestly, it’s been a nice ride. I’ve come to enjoy my stay and take pride in finishing the different components.

When I started earlier this year, I had developed a bit of an Imposter syndrome. For one I wasn’t sure if I understood the full scope of the project, but I was also uncertain if I had the skill to work on it.

Confidence came over time, when I managed to get an overview of everything that needed to be done. Everything suddenly looks much more manageable, when you understand the purpose and inner workings of each subcomponent that will be required over the way. Of course, it’s not completely straight forward. I often had to realize that I will need another component somewhere, but one thing at a time is manageable. Since I had my mentor to back me up whenever I got stuck, working on the project didn’t get frustrating.

Blog posts

Parallel to working on the project, we were expected to publish blog posts. Originally I was going for a weekly interval, but yea…

I don’t think writing the blog posts helped me in any meaningful way. However, I do think writing blogs is a good habit and hope that

  1. I improved my blog writing skills
  2. The blogs will someday help someone

Progress

The GSoC period has already finished. See my pull request.

November 12, 2022 00:00

November 09, 2022

Ignite Realtime Blog

Openfire 4.7.4 release

The Ignite Realtime Community is happy to announce the 4.7.4 release of Openfire. This release fixes a number of bugs and represents our effort to provide a stable 4.7.x series while work continues on the next feature release of Openfire.

Notable fixes include enhancements to cluster-specific implementation of Multi-User Chat functionality, improved websocket handling and improved statistics. Additionally, the dependency on Apache Commons Text has been upgraded, to prevent statistic analyzers raising unneeded alarms with regards to its recently published CVE.

You can find download artifacts available here with the following sha256sum values

a6a98540e3ab6da65916f630b7b22d04e7ec125be9d09ae98121f6075ef2ef77  openfire-4.7.4-1.noarch.rpm
5f4bd4e6390bdfe99a63e4e72b25200461854348a9f6039368cffe1c509782fa  openfire_4.7.4_all.deb
6bebb52b4828d9564b1e22f3d0aebaeec6eefb1e5c7899549747b48bc8d6e30d  openfire_4_7_4.dmg
142d923f0b17e4dff65b01f69d9d5885494798011157a64cfef8847063f503ee  openfire_4_7_4.exe
9ebcb9c15d38d4f8fb528a79f0f53440179d64c214176c2d9e578b11762258dd  openfire_4_7_4.tar.gz
ef233db999c8a18ac43edb7fd17657c7677c0ce2e29f09c9ccb53185c451ded3  openfire_4_7_4_x64.exe
ba0fb1d992c5169da466107e7752f5eee588d79987dcf052e5b68abc101f7173  openfire_4_7_4.zip

If you have any questions, please stop by our community forum or our live groupchat. We are always looking for volunteers interested in helping out with Openfire development!

For other release announcements and news follow us on Twitter

1 post - 1 participant

Read full topic

by guus at November 09, 2022 18:13

Hazelcast plugin version 2.6.1 released!

The Ignite Realtime community is happy to announce the immediate availability of version 2.6.1 of the Hazelcast plugin for Openfire! The Hazelcast plugin is what allows you to deploy Openfire as a clustered solution.

This release includes only one improvement, but that one can bring a significant performance improvement as compared to older versions of the plugin.

The updated plugin should become available for download in your Openfire admin console in the course of the next few hours. Alternatively, you can download the plugin directly, from the plugin’s archive page.

For other release announcements and news follow us on Twitter

1 post - 1 participant

Read full topic

by guus at November 09, 2022 18:02

REST API Openfire plugin 1.10.1 released!

We are happy to announce the immediate availability of version 1.10.1 of the REST API plugin for Openfire!

This is a bugfix release, that will improve the endpoints that make modifications to MUC rooms.

The updated plugin should become available for download in your Openfire admin console in the course of the next few hours. Alternatively, you can download the plugin directly, from the plugin’s archive page.

For other release announcements and news follow us on Twitter

1 post - 1 participant

Read full topic

by guus at November 09, 2022 17:57

November 07, 2022

ProcessOne

Matrix protocol added to ejabberd

ejabberd is already the most versatile and scalable messaging server. In this post, we are giving a sneak peak at what is coming next.

ejabberd just get new ace in it sleeve – you can now use ejabberd to talk with other Matrix servers, a protocol sometimes used for small corporate server messaging.

Of course, you all know ejabberd supports the XMPP instant messaging protocol with hundreds of XMPP extensions, this is what it is famous for.

The second major protocol in ejabberd is MQTT. ejabberd support MQTT 5 with clustering, and is massively scalable. ejabberd can be used to implement Internet of Things projects, using either XMPP or MQTT and it also supports hybrid workflow, where you can mix humans and machines exchanging messages on the same platform.

It also supports SIP, as you can connect to ejabberd with a SIP client, so that you can use a softphone directly with ejabberd for internal calls.

So far, so good, ejabberd leading both in terms of performance and number of messaging protocol it supports.

We always keep an eye on new messaging protocol. Recently, the Matrix protocol emerged as a new way to implement messaging for the small corporate servers.

ejabberd adds support for Matrix protocol

Of course, by design, the Matrix protocol cannot scale as well as XMPP or MQTT protocols. At the heart of Matrix protocol, you have a kind of merging algorithm that reminds a bit of Google Wave. It means that a conversation is conceptually represented as a sort document you constantly merge on the server. This is a consuming process that is happening on the server for each message received in all conversations. That’s why Matrix has the reputation to be so difficult to scale.

Even if it is not as scalable as XMPP, we believe that we can make Matrix much more scalable than what it is now. That’s what we are doing right now.

As a first step, we have been working on implementing a large subset of the Matrix protocol as a bridge in ejabberd.

It means that an ejabberd server will be able to act as a Matrix server in the Matrix ecosystem. XMPP users will be able to exchange messages with Matrix users, transparently.

To do that, we implemented the Matrix protocol for conversations and the server-to-server protocol to allow interop between XMPP and Matrix protocol.

This feature coming first for our customers, in the coming weeks, whether they are using ejabberd Business Edition internally or on Fluux ejabberd SaaS platform. It will come later to ejabberd Community Edition.

Interested? Let’s talk! Contact us.

The post Matrix protocol added to ejabberd first appeared on ProcessOne.

by Mickaël Rémond at November 07, 2022 16:44

November 05, 2022

The XMPP Standards Foundation

The XMPP Newsletter October 2022

Welcome to the XMPP Newsletter, great to have you here again! This issue covers the month of October 2022.

Like this newsletter, many projects and their efforts in the XMPP community are a result of people’s voluntary work. If you are happy with the services and software you may be using, especially throughout the current situation, please consider saying thanks or help these projects! Interested in supporting the Newsletter team? Read more at the bottom.

Newsletter Contributors & Translations

This is a community effort, and we would like to thank translators for their contributions. Volunteers are welcome! Translations of the XMPP Newsletter will be released here (with some delay):

  • English (original): xmpp.org
    • General contributors: emus, Licaon_Kter, Ludovic Bocquet, MattJ, MSavoritias (fae,ve), wurstsalat, Zash
  • French: jabberfr.org and linuxfr.org
    • Translators: Adrien Bourmault (neox), alkino, anubis, Benoît Sibaud, Ppjet6
  • German: xmpp.org and anoxinon.de
    • Translators: Jeybe, wh0nix
  • Italian: nicfab.it
    • Translators: nicfab
  • Spanish: xmpp.org
    • Translators: daimonduff, TheCoffeMaker

XSF Announcements

  • The XSF membership application period for the fourth quarter 2022 is currently open. If you are interested in becoming a XSF member then you can apply for membership. Please submit by November 27, 2022.

Google Summer of Code 2022

XSF and Google Summer of Code 2022

XSF and Google Summer of Code 2022

The Google Summer of Code 2022 has finished! The two new contributors Patiga and PawBud worked on open-source software projects in the XMPP environment. Read the final collection blog posts:

XSF fiscal hosting projects

The XSF offers fiscal hosting for XMPP projects. Please apply via Open Collective. For more information, see the announcement blog post. Current projects:

Events

Articles

ejabberd has added Matrix protocol support bringing their experience of scaling XMPP and MQTT to this new messaging avenue. At first a subset of the protocol will be implemented, acting like a bridge between protocols. Bussiness customers will get a taste of it first and later it will come to the Community Edition.

The JMP Newsletter details a big new release of the Cheogram Android client with features including XHTML-IM, Bits of Binary, a form of message retraction, contact tagging, and more. They also talk about an XMPP channel integration they have built for the Chatwoot support software, and offer to let community projects use their instance to try it out. This month they also wrote an article about SMS Account Verification and how it affects the ecosystem.

PGPainless: Implementing Packet Sequence Validation using Pushdown Automata

A new automated testing technique was recently adopted to help the Prosody developers during their daily development work: Mutation Testing in Prosody

Modernization of the authentication protocol is ongoing around the community, keyword “SASL”. Multiple hints of the work have been seen in the last two newsletters, in libs, XEPs updates and funding news. It spans multiple clients, multiple platforms and at least one server. Thilo Molitor, the Monal developer, blogged about the current broken state, the ideal properties of the system and the solutions being worked on. It’s a 30 minutes long read but will sure come handy as more news of the SASL implementations will start to appear from the other involved projects.

Software news

Clients and applications

Converse has released version 10.0.0 with a lot of polishing and an updated feature set.

Dino 0.3.1 has been released, which is a maintenance release for libsoup3 support and bug fixes.

Gajim 1.5.2 and 1.5.3 have been released. These releases bring another performance boost, better emojis, selection of multiple messages, and many bug fixes.

Gajim emoji shortcodes

Gajim emoji shortcodes

Profanity 0.13.1 is out with improved plugins management.

Psi+ 1.5.1641 (2022-09-24) has been released.

Monal 5.3.3 has been live in the AppStore for a while now, and even if it does not yet bring the SASL improvements mentioned above or the much awaited audio and video calls capabilities funded last month, it does feature several fixes for registrations, screenshots sharing and read markers.

aTalk has been released with an almost monthly cadence. Each new version brings more polish, like location/map views, calls, libraries updates and keeping up to date on the newer Android requirements. It’s biggest adoption impediment unfortunately still stays and stems from an outdated UI that precedes Android 5 Material UI.

Servers

ejabberd 22.10 release includes six months of work, over 140 commits, including relevant improvements in MIX, MUC, SQL, and installers, and bug fixes as usual.

Jackal 0.62.3 is out bringing 3 fixes for storage/archive and s2s stanzas.

Libraries & Tools

python-nbxmpp 3.2.4 and 3.2.5 have been released, bringing bug fixes and internal improvements.

Extensions and specifications

Developers and other standards experts from around the world collaborate on these extensions, developing new specifications for emerging practices, and refining existing ways of doing things. Proposed by anybody, the particularly successful ones end up as Final or Active - depending on their type - while others are carefully archived as Deferred. This life cycle is described in XEP-0001, which contains the formal and canonical definitions for the types, states, and processes. Read more about the standards process. Communication around Standards and Extensions happens in the Standards Mailing List (online archive).

xmpp.org features a page about XMPP RFCs as well.

Proposed

The XEP development process starts by writing up an idea and submitting it to the XMPP Editor. Within two weeks, the Council decides whether to accept this proposal as an Experimental XEP.

New

  • No new XEP this month.

Deferred

If an experimental XEP is not updated for more than twelve months, it will be moved off Experimental to Deferred. If there is another update, it will put the XEP back onto Experimental.

  • No XEPs deferred this month.

Updated

  • No XEPs updated this month.

Last Call

Last calls are issued once everyone seems satisfied with the current XEP status. After the Council decides whether the XEP seems ready, the XMPP Editor issues a Last Call for comments. The feedback gathered during the Last Call help improving the XEP before returning it to the Council for advancement to Stable.

  • No Last Call this month.

Stable

  • No XEP moved to stable this month.

Deprecated

  • No XEP deprecated this month.

Call for Experience

A Call For Experience - like a Last Call, is an explicit call for comments, but in this case it’s mostly directed at people who’ve implemented, and ideally deployed, the specification. The Council then votes to move it to Final.

  • No Call for Experience this month.

Spread the news!

Please share the news on other networks:

Subscribe to the monthly XMPP newsletter
Subscribe

Also check out our RSS Feed!

Looking for job offers or want to hire a professional consultant for your XMPP project? Visit our XMPP job board.

Help us to build the newsletter

This XMPP Newsletter is produced collaboratively by the XMPP community. Therefore, we would like to thank Adrien Bourmault (neox), anubis, Anoxinon e.V., Benoît Sibaud, cpm, daimonduff, emus, Gooya, Holger, IM, Ludovic Bocquet, martin, MattJ, MSavoritias (fae,ve), nicfab, Pierre Jarillon, Sam Whited, TheCoffeMaker, wh0nix, vanitasvitae, wurstsalat, Zash for their support and help in creation, review, translation and deployment. Many thanks to all contributors and their continuous support!

Each month’s newsletter issue is drafted in this simple pad. At the end of each month, the pad’s content is merged into the XSF Github repository. We are always happy to welcome contributors. Do not hesitate to join the discussion in our Comm-Team group chat (MUC) and thereby help us sustain this as a community effort. You have a project and want to spread the news? Please consider sharing your news or events here, and promote it to a large audience.

Tasks we do on a regular basis:

  • gathering news in the XMPP universe
  • short summaries of news and events
  • summary of the monthly communication on extensions (XEPs)
  • review of the newsletter draft
  • preparation of media images
  • translations

License

This newsletter is published under CC BY-SA license.

November 05, 2022 00:00

November 02, 2022

JMP

Newsletter: New Cheogram Android Release, Chatwoot Instance

Hi everyone!

Welcome to the latest edition of your pseudo-monthly JMP update!

In case it’s been a while since you checked out JMP, here’s a refresher: JMP lets you send and receive text and picture messages (and calls) through a real phone number right from your computer, tablet, phone, or anything else that has a Jabber client.  Among other things, JMP has these features: Your phone number on every device; Multiple phone numbers, one app; Free as in Freedom; Share one number with multiple people.

October saw the release of Cheogram Android 2.10.10-3, the largest release in awhile.  The app now stores data de-duplicated, so if you send or receive the same file multiple times only one copy will be stored.  This also lays the groundwork for some new file transfer improvements that will be coming in the future.  The app also now supports displaying rich text messages sent by clients which support that (such as Gajim), including the image protocol needed to display stickers sent by Movim users.  A form of message retraction is also supported, and should work with most Jabber clients out there.  A reminder that message retraction is a social convention and not a security feature – the target still has a full copy of your un-retracted message if they want it.

We know lots of you have big contact lists, across multiple accounts, and that’s why this release introduces the ability to edit tags on your contacts and a tag navigation widget integrated into contact search: to make finding the right conversation just a little bit easier.  We would love to hear feedback about this UI and how well it works for you.

For those of you who start a lot of group texts, there is an easy way to do that built into the app now as well.  When you start a “private group chat” and select only @cheogram.com contacts, you will be prompted to ask if you meant to start a group text instead.  This flow seemed necessary, as many have accidentally created private channels with their SMS contacts instead of the intended group text, so checking at this point was likely to be necessary anyway.

There are also some smaller quality of life improvements in this release, including the ability to copy any link in a message to the clipboard (not just the first one), dumping app logs to a special directory on your device after every call in order to make debugging issues easier, asking if you want to keep app data on uninstall (to make switching back and forth to custom builds possible without always needing export/re-import), a new first-start welcome screen, performance improvements, and more.

As JMP grows so does our support load.  Up until this month we have been managing all our support requests using normal Jabber clients (mostly Gajim and Dino), which worked well enough but less and less well as we grew.  It would be hard to know if someone else was handling a request, who had previously handled a request, or even what the status of some requests were (if they had been resolved elsewhere in the public channel or otherwise).  We’re a small enough team that we can just talk to each other to solve these things, but that does take time, and more time as there are more things to talk out.  So this month we built an XMPP channel integration for Chatwoot and have migrated our main support infrastructure to a Cheogram-hosted instance.  So far we like this a lot, and so much we’ve decided to share.  If you have a project that handles support using Jabber (or SMS with JMP!) you can use it on the Cheogram Chatwoot instance.  Just come by the chatroom and let us know you’re interested.  Only the XMPP channel works on our instance for now, but we’d be happy to enable other channels as well if that would be useful.

And finally, we know many of you are excited about the JMP Data Plan.  Roll out to the waiting list has gone a bit slower than we hoped, but many SIMs did go out in October.  There have been some bumps as you might expect with any test phase, but overall things are looking good and we hope to speed up the roll out and even move on to the next phase soon.

To learn what’s happening with JMP between newsletters, here are some ways you can find out:

Thanks for reading and have a wonderful rest of your week!

by Stephen Paul Weber at November 02, 2022 18:41

October 31, 2022

Gajim

Gajim 1.5.3

Gajim 1.5.3 brings back a feature many of you missed: selecting and copying multiple messages. Emoji shortcodes have been improved and cover even more emojis now. Gajim also lets you mark workspaces as read, so you don’t have to go through all conversations. Thank you for all your contributions!

What’s New

Since we changed the way Gajim displays messages in Gajim 1.4, selecting multiple messages to copy them was not possible anymore. With Gajim 1.5.3 you can now select multiple messages via message menu, followed by marking each message you would like to copy.

Thanks to @mjk, Gajim now offers even more emojis when using shortcodes. You will now get suggestions for categories as well, which makes finding the right emoji easier 🎉

Emoji shortcodes

Emoji shortcodes

Gajim now fully supports message drafts. If you type a message and switch to another chat, you will now see your draft in the chat list as well.

More Changes

  • You can now select a default workspace for new chats (for each account)
  • Workspaces can now be marked as read

Fixes

  • Improvements for recognizing URLs (@mjk)
  • Fix message correction behavior when switching chats
  • Many fixes to improve Gajim’s usability

Over 30 issues have been fixed in this release. Have a look at the changelog for a complete list.

Gajim

As always, don’t hesitate to contact us at gajim@conference.gajim.org or open an issue on our Gitlab.

October 31, 2022 00:00

October 30, 2022

JMP

SMS Account Verification

Some apps and services (but not JMP!) require an SMS verification code in order to create a new account.  (Note that this is different from using SMS for authentication; which is a bad idea since SMS can be easily intercepted, are not encrypted in transit, and are vulnerable to simple swap scams, etc.; but has different incentives and issues.)  Why do they do this, and how can it affect you as a user?

Tarpit

In the fight against service abuse and SPAM, there are no sure-fire one-size-fits-all solutions.  Often preventing abusive accounts and spammers entirely is not possible, so targets turn to other strategies, such as tarpits.  This is anything that slows down the abusive activity, thus resulting in less of it.  This is the best way to think about most account-creation verification measures.  Receiving an SMS to a unique phone number is something that is not hard for most customers creating an account.  Even a customer who does not wish to give out their phone number or does not have a phone number can (in many countries, with enough money) get a new cell phone and cell phone number fairly quickly and use that to create the account.

If a customer is expected to be able to pass this check easily, and an abuser is indistiguishable from a customer, then how can any SMS verification possibly help prevent abuse?  Well, if the abuser needs to create only one account, it cannot.  However, in many cases an abuser is trying to create tens of thousands of accounts.  Now imagine trying to buy ten thousand new cell phones at your local store every day.  It is not going to be easy.

“VoIP Numbers”

Now, JMP can easily get ten thousand new SMS-enabled numbers in a day.  So can almost any other carrier or reseller.  If there is no physical device that needs to be handed over (such as with VoIP, eSIM, and similar services), the natural tarpit is gone and all that is left is the prices and policies of the provider.  JMP has many times received requests to help with getting “10,000 numbers, only need them for one day”.  Of course, we do not serve such customers.  JMP is not here to facilitate abuse, but to help create a gateway to the phone network for human beings whose contacts are still only found there.  That doesn’t mean there are no resellers who will work with such a customer, however.

So now the targets are in a pickle if they want to keep using this strategy.  If the abuser can get ten thousand SMS-enabled numbers a day, and if it doesn’t cost too much, then it won’t work as a tarpit at all!  So many of them have chosen a sort of scorched-earth policy.  They buy and create heuristics to guess if a phone number was “too easy” to get, blocking entire resellers, entire carriers, entire countries.  These rules change daily, are different for every target, and can be quite unpredictable.  This may help when it comes to foiling the abusers, but is bad if you are a customer who just wants to create an account.  Some targets, especially “big” ones, have made the decision to lose some customers (or make their lives much more difficult) in order to slow the abusers down.

De-anonymization

Many apps and services also make money by selling your viewing time to advertisers (e.g. ads interspersed in a social media feed, as pre-/mid-roll in a video, etc.) based on your demographics and behaviour.  To do this, they need to know who you are and what your habits are so they can target the ads you see for the advertisers’ benefit.  As a result, they have an incentive to associate your activity with just one identity, and to make it difficult for you to separate your behaviour in ways that reduce their ability to get a complete picture of who you are.  Some companies might choose to use SMS verification as one of the ways they try to ensure a given person can’t get more than one account, or for associating the account (via the provided phone number) with information they can acquire from other sources, such as where you are at any given time.

Can I make a new account with JMP numbers?

The honest answer is, we cannot say.  While JMP would never work with abusers, and has pricing and incentives set up to cater to long-term users rather than those looking for something “disposable”, communicating that to every app and service out there is a big job.  Many of our customers try to help us with this job by contacting the services they are also customers of; after all, a company is more likely to listen to their own customers than a cold-call from some other company.  The Soprani.ca project has a wiki page where users keep track of what has worked for them, and what hasn’t, so everyone can remain informed of the current state (since a service may work today, but not tomorrow, then work again next week, it is important to track success over time).

Part of why we can’t say whether you can make a new account with JMP numbers is because the reasons some companies choose to use SMS verification are opaque, so we may not know all of their criteria for sure.

Many customers use JMP as their only phone number, often ported in from their previous carrier and already associated with many online accounts.  This often works very well, but everyone’s needs are different.  Especially those creating new personas which start with a JMP number find that creating new accounts at some services for the persona can be frustrating to impossible.  It is an active area of work for us and all other small, easy-access phone network resellers.

by Stephen Paul Weber at October 30, 2022 00:53

October 29, 2022

ProcessOne

ejabberd 22.10

This ejabberd 22.10 release includes six months of work, over 140 commits, including relevant improvements in MIX, MUC, SQL, and installers, and bug fixes as usual.

jabberd 22.10 released
This version brings support for latest MIX protocol version, and significantly improves detection and recovery of SQL connection issues.

There are no breaking changes in SQL schemas, configuration, or commands API. If you develop an ejabberd module, notice two hooks have changed: muc_subscribed and muc_unsubscribed.

A more detailed explanation of those topics and other features:

Erlang/OTP 19.3

You may remember than in the previous ejabberd release, ejabberd 22.05, support for Erlang/OTP 25 was introduced, even if 24.3 is still recommended for stable deployments.

It is expected that around April 2023, GitHub Actions will remove Ubuntu 18 and it will not be possible to run automatic tests for ejabberd using Erlang 19.3, the lowest possible will be Erlang 20.0.

For that reason, the planned schedule is:

  • ejabberd 22.10
    • Usage of Erlang 19.3 is discouraged
    • Anybody still using Erlang 19.3 is encouraged to upgrade to 24.3, or at least 20.0
  • ejabberd 23.05 (or later)
    • Support for Erlang 19.3 is deprecated
    • Erlang requirement softly increased in `configure.ac`
    • Announce: no warranty ejabberd can compile, start or pass the Common Tests suite using Erlang 19.3
    • Provide instructions for anybody to manually re-enable it and run the tests
  • ejabberd 23.xx+1 (or later)
    • Support for Erlang 19.3 is removed completely in the source code

New log_burst_limit_* options

Two options were added in #3865 to configure logging limits in case of high traffic:

  • log_burst_limit_window_time defines the time period to rate-limit log messages by.

  • log_burst_limit_count defines the number of messages to accept in that time period before starting to drop them.

Support ERL_DIST_PORT option to work without epmd

The option ERL_DIST_PORT is added to ejabberdctl.cfg, disabled by default.

When this option is set to a port number, the Erlang node will not start epmd and will not listen in a range of ports for erlang connections (typically used for ejabberdctl and for clustering). Instead, the erlang node will simply listen in the configured port number.

Please note:

  • Erlang/OTP 23.1 or higher is required to use ERL_DIST_PORT
  • make relive doesn’t support ERL_DIST_PORT, neither rebar3 nor elixir
  • To start several ejabberd nodes in the same machine, configure a different port in each node

Support version macros in captcha_cmd option

Support for the @VERSION@ and @SEMVER@ macros was added to the captcha_cmd option in #3835.

Those macros are useful because the example captcha scripts are copied in a path like ejabberd-VERSION/priv/bin that depends on the ejabberd version number and changes for each release. Also, depending on the install method (rebar3 or Elixir’s mix), that VERSION may be in XX.YY or in SEMVER format (respectively).

Now, it’s possible to configure like this:

captcha_cmd: /opt/ejabberd-@VERSION@/lib/ejabberd-@SEMVER@/priv/bin/captcha.sh

Hook Changes

Two hooks have changed: muc_subscribed and muc_unsubscribed. Now they get the packet and room state, and can modify the sent packets. If you write source code that adds functions to those hooks, please notice that previously they were ran like:

ejabberd_hooks:run(muc_subscribed, ServerHost, [ServerHost, Room, Host, BareJID]);

and now they are ran like this:

{Packet2a, Packet2b} = ejabberd_hooks:run_fold(muc_subscribed, ServerHost, {Packet1a, Packet1b},
[ServerHost, Room, Host, BareJID, StateData]),

being Packet1b a copy of Packet1a without the jid attribute in the muc_subscribe element.

Translations Updates

Several translations were improved: Ukrainian, Chinese (Simplified), French, German, Russian, Portuguese (Brazil), Spanish and Catalan. Thanks to all this people that contribute in ejabberd at Weblate!

WebAdmin page for external modules

A new page is added in ejabberd’s WebAdmin to view available external modules, update their source code, install, upgrade and remove them. All this is equivalent to what was already available using API commands from the modules tag.

Many modules in the ejabberd-contrib git repository have been improved, and their documentation updated. Additionally, those modules are now automatically tested, at least compilation, installation and static code analysis.

Documentation Improvements

In addition to the normal improvements and fixes, two sections in the ejabberd Documentation are greatly improved:

ChangeLog

General

  • Add log_burst_limit_* options (#3865)
  • Support ERL_DIST_PORT option to work without epmd
  • Auth JWT: Catch all errors from jose_jwt:verify and log debugging details (#3890)
  • CAPTCHA: Support @VERSION@ and @SEMVER@ in captcha_cmd option (#3835)
  • HTTP: Fix unix socket support (#3894)
  • HTTP: Handle invalid values in X-Forwarded-For header more gracefuly
  • Listeners: Let module take over socket
  • Listeners: Don’t register listeners that failed to start in config reload
  • mod_admin_extra: Handle empty roster group names
  • mod_conversejs: Fix crash when mod_register not enabled (#3824)
  • mod_host_meta: Complain at start if listener is not encrypted
  • mod_ping: Fix regression on stop_ping in clustering context (#3817)
  • mod_pubsub: Don’t crash on command failures
  • mod_shared_roster: Fix cache invalidation
  • mod_shared_roster_ldap: Update roster_get hook to use #roster_item{}
  • prosody2ejabberd: Fix parsing of scram password from prosody

MIX

  • Fix MIX’s filter_nodes
  • Return user jid on join
  • mod_mix_pam: Add new MIX namespaces to disco features
  • mod_mix_pam: Add handling of IQs with newer MIX namespaces
  • mod_mix_pam: Do roster pushes on join/leave
  • mod_mix_pam: Parse sub elements of the mix join remote result
  • mod_mix_pam: Provide MIX channels as roster entries via hook
  • mod_mix_pam: Display joined channels on webadmin page
  • mod_mix_pam: Adapt to renaming of participant-id from mix_roster_channel record
  • mod_roster: Change hook type from #roster{} to #roster_item{}
  • mod_roster: Respect MIX “ setting
  • mod_roster: Adapt to change of mix_annotate type to boolean in roster_query
  • mod_shared_roster: Fix wrong hook type #roster{} (now #roster_item{})

MUC:

  • Store role, and use it when joining a moderated room (#3330)
  • Don’t persist none role (#3330)
  • Allow MUC service admins to bypass max_user_conferences limitation
  • Show allow_query_users room option in disco info (#3830)
  • Don’t set affiliation to none if it’s already none in mod_muc_room:process_item_change/3
  • Fix mucsub unsubscribe notification payload to have muc_unsubcribe in it
  • Allow muc_{un}subscribe hooks to modify sent packets
  • Pass room state to muc_{un}subscribed hook
  • The archive_msg export fun requires MUC Service for room archives
  • Export mod_muc_admin:get_room_pid/2
  • Export function for getting room diagnostics

SQL

  • Handle errors reported from begin/commit inside transaction
  • Make connection close errors bubble up from inside sql transaction
  • Make first sql reconnect wait shorter time
  • React to sql driver process exit earlier
  • Skip connection exit message when we triggered reconnection
  • Add syntax_tools to applications, required when using ejabberd_sql_pt (#3869)
  • Fix mam delete_old_messages_batch for sql backend
  • Use INSERT ... ON DUPLICATE KEY UPDATE for upsert on mysql
  • Update mysql library
  • Catch mysql connection being close earlier

Compile

  • make all: Generate start scripts here, not in make install (#3821)
  • make clean: Improve this and “distclean”
  • make deps: Ensure deps configuration is ran when getting deps (#3823)
  • make help: Update with recent changes
  • make install: Don’t leak DESTDIR in files copied by ‘make install’
  • make options: Fix error reporting on OTP24+
  • make update: configure also in this case, similarly to make deps
  • Add definition to detect OTP older than 25, used by ejabberd_auth_http
  • Configure eimp with mix to detect image convert properly (#3823)
  • Remove unused macro definitions detected by rebar3_hank
  • Remove unused header files which content is already in xmpp library

Container

  • Get ejabberd-contrib sources to include them
  • Copy .ejabberd-modules directory if available
  • Do not clone repo inside container build
  • Use make deps, which performs additional steps (#3823)
  • Support ERL_DIST_PORT option to work without epmd
  • Copy ejabberd-docker-install.bat from docker-ejabberd git and rename it
  • Set a less frequent healthcheck to reduce CPU usage (#3826)
  • Fix build instructions, add more podman examples

Installers

  • make-binaries: Include CAPTCHA script with release
  • make-binaries: Edit rebar.config more carefully
  • make-binaries: Fix linking of EIMP dependencies
  • make-binaries: Fix GitHub release version checks
  • make-binaries: Adjust Mnesia spool directory path
  • make-binaries: Bump Erlang/OTP version to 24.3.4.5
  • make-binaries: Bump Expat and libpng versions
  • make-packages: Include systemd unit with RPM
  • make-packages: Fix permissions on RPM systems
  • make-installers: Support non-root installation
  • make-installers: Override code on upgrade
  • make-installers: Apply cosmetic changes

External modules

  • ext_mod: Support managing remote nodes in the cluster
  • ext_mod: Handle correctly when COMMIT.json not found
  • Don’t bother with COMMIT.json user-friendly feature in automated user case
  • Handle not found COMMIT.json, for example in GH Actions
  • Add WebAdmin page for managing external modules

Workflows Actions

  • Update workflows to Erlang 25
  • Update workflows: Ubuntu 18 is deprecated and 22 is added
  • CI: Remove syntax_tools from applications, as fast_xml fails Dialyzer
  • Runtime: Add Xref options to be as strict as CI

Full Changelog

https://github.com/processone/ejabberd/compare/22.05…22.10

ejabberd 22.10 download & feedback

As usual, the release is tagged in the Git source code repository on GitHub.

The source package and installers are available in ejabberd Downloads page. To check the *.asc signature files, see How to verify ProcessOne downloads integrity.

For convenience, there are alternative download locations like the ejabberd DEB/RPM Packages Repository and the GitHub Release / Tags.

The Docker image is in Docker Hub, and there’s an alternative Container image in GitHub Packages.

If you suspect that you’ve found a bug, please search or fill a bug report on GitHub Issues.

The post ejabberd 22.10 first appeared on ProcessOne.

by Jérôme Sautret at October 29, 2022 12:38

October 26, 2022

Paul Schaub

Implementing Packet Sequence Validation using Pushdown Automata

This is part 2 of a small series on verifying the validity of packet sequences using tools from theoretical computer science. Read part 1 here.

In the previous blog post I discussed how a formal grammar can be transformed into a pushdown automaton in order to check if a sequence of packets or tokens is part of the language described by the grammar. In this post I will discuss how I implemented said automaton in Java in order to validate OpenPGP messages in PGPainless.

In the meantime, I made some slight changes to the automaton and removed some superfluous states. My current design of the automaton looks as follows:

If you compare this diagram to the previous iteration, you can see that I got rid of the states “Signed Message”, “One-Pass-Signed Message” and “Corresponding Signature”. Those were states which had ε-transitions to another state, so they were not really useful.

For example, the state “One-Pass-Signed Message” would only be entered when the input “OPS” was read and ‘m’ could be popped from the stack. After that, there would only be a single applicable rule which would read no input, pop nothing from the stack and instead push back ‘m’. Therefore, these two rule could be combined into a single rule which reads input “OPS”, pops ‘m’ from the stack and immediately pushes it back onto it. This rule would leave the automaton in state “OpenPGP Message”. Both automata are equivalent.

One more minor detail: Since I am using Bouncy Castle, I have to deal with some of its quirks. One of those being that BC bundles together encrypted session keys (PKESKs/SKESKs) with the actual encrypted data packets (SEIPD/SED). Therefore when implementing, we can further simplify the diagram by removing the SKESK|PKESK parts:

Now, in order to implement this automaton in Java, I decided to define enums for the input and stack alphabets, as well as the states:

public enum InputAlphabet {
    LiteralData,
    Signature,            // Sig
    OnePassSignature,     // OPS
    CompressedData,
    EncryptedData,        // SEIPD|SED
    EndOfSequence         // End of message/nested data
}
public enum StackAlphabet {
    msg,                 // m
    ops,                 // o
    terminus             // #
}
public enum State {
    OpenPgpMessage,
    LiteralMessage,
    CompressedMessage,
    EncryptedMessage,
    Valid
}

Note, that there is no “Start” state, since we will simply initialize the automaton in state OpenPgpMessage, with ‘m#’ already put on the stack.

We also need an exception class that we can throw when OpenPGP packet is read when its not allowed. Therefore I created a MalformedOpenPgpMessageException class.

Now the first design of our automaton itself is pretty straight forward:

public class PDA {
    private State state;
    private final Stack<StackAlphabet> stack = new Stack<>();
    
    public PDA() {
        state = State.OpenPgpMessage;    // initial state
        stack.push(terminus);            // push '#'
        stack.push(msg);                 // push 'm'
    }

    public void next(InputAlphabet input)
            throws MalformedOpenPgpMessageException {
        // TODO: handle the next input packet
    }

    StackAlphabet popStack() {
        if (stack.isEmpty()) {
            return null;
        }
        return stack.pop();
    }

    void pushStack(StackAlphabet item) {
        stack.push(item);
    }

    boolean isEmptyStack() {
        return stack.isEmpty();
    }

    public boolean isValid() {
        return state == State.Valid && isEmptyStack();
    }
}

As you can see, we initialize the automaton with a pre-populated stack and an initial state. The automatons isValid() method only returns true, if the automaton ended up in state “Valid” and the stack is empty.

Whats missing is an implementation of the transition rules. I found it most straight forward to implement those inside the State enum itself by defining a transition() method:

public enum State {

    OpenPgpMessage {
        @Overrides
        public State transition(InputAlphabet input, PDA automaton)
                throws MalformedOpenPgpMessageException {
            StackAlphabet stackItem = automaton.popStack();
            if (stackItem != OpenPgpMessage) {
                throw new MalformedOpenPgpMessageException();
            }
            swith(input) {
                case LiteralData:
                    // Literal Packet,m/ε
                    return LiteralMessage;
                case Signature:
                    // Sig,m/m
                    automaton.pushStack(msg);
                    return OpenPgpMessage;
                case OnePassSignature:
                    // OPS,m/mo
                    automaton.push(ops);
                    automaton.push(msg);
                    return OpenPgpMessage;
                case CompressedData:
                    // Compressed Data,m/ε
                    return CompressedMessage;
                case EncryptedData:
                    // SEIPD|SED,m/ε
                    return EncryptedMessage;
                case EndOfSequence:
                default:
                    // No transition
                    throw new MalformedOpenPgpMessageException();
            }
        }
    },

    LiteralMessage {
        @Overrides
        public State transition(InputAlphabet input, PDA automaton)
                throws MalformedOpenPgpMessageException {
            StackAlphabet stackItem = automaton.popStack();
            switch(input) {
                case Signature:
                    if (stackItem == ops) {
                        // Sig,o/ε
                        return LiteralMessage;
                    } else {
                        throw new MalformedOpenPgpMessageException();
                    }
                case EndOfSequence:
                    if (stackItem == terminus && automaton.isEmptyStack()) {
                        // ε,#/ε
                        return valid;
                    } else {
                        throw new MalformedOpenPgpMessageException();
                    }
                default:
                    throw new MalformedOpenPgpMessageException();
            }
        }
    },

    CompressedMessage {
        @Overrides
        public State transition(InputAlphabet input, PDA automaton)
                throws MalformedOpenPgpMessageException {
            StackAlphabet stackItem = automaton.popStack();
            switch(input) {
                case Signature:
                    if (stackItem == ops) {
                        // Sig,o/ε
                        return CompressedMessage;
                    } else {
                        throw new MalformedOpenPgpMessageException();
                    }
                case EndOfSequence:
                    if (stackItem == terminus && automaton.isEmptyStack()) {
                        // ε,#/ε
                        return valid;
                    } else {
                        throw new MalformedOpenPgpMessageException();
                    }
                default:
                    throw new MalformedOpenPgpMessageException();
            }
        }
    },

    EncryptedMessage {
        @Overrides
        public State transition(InputAlphabet input, PDA automaton)
                throws MalformedOpenPgpMessageException {
            StackAlphabet stackItem = automaton.popStack();
            switch(input) {
                case Signature:
                    if (stackItem == ops) {
                        // Sig,o/ε
                        return EncryptedMessage;
                    } else {
                        throw new MalformedOpenPgpMessageException();
                    }
                case EndOfSequence:
                    if (stackItem == terminus && automaton.isEmptyStack()) {
                        // ε,#/ε
                        return valid;
                    } else {
                        throw new MalformedOpenPgpMessageException();
                    }
                default:
                    throw new MalformedOpenPgpMessageException();
            }
        }
    },

    Valid {
        @Overrides
        public State transition(InputAlphabet input, PDA automaton)
                throws MalformedOpenPgpMessageException {
            // Cannot transition out of Valid state
            throw new MalformedOpenPgpMessageException();
        }
    }
    ;

    abstract State transition(InputAlphabet input, PDA automaton)
            throws MalformedOpenPgpMessageException;
}

It might make sense to define the transitions in an external class to allow for different grammars and to remove the dependency on the PDA class, but I do not care about this for now, so I’m fine with it.

Now every State has a transition() method, which takes an input symbol and the automaton itself (for access to the stack) and either returns the new state, or throws an exception in case of an illegal token.

Next, we need to modify our PDA class, so that the new state is saved:

public class PDA {
    [...]

    public void next(InputAlphabet input)
            throws MalformedOpenPgpMessageException {
        state = state.transition(input, this);
    }
}

Now we are able to verify simple packet sequences by feeding them one-by-one to the automaton:

// LIT EOS
PDA pda = new PDA();
pda.next(LiteralData);
pda.next(EndOfSequence);
assertTrue(pda.isValid());

// OPS LIT SIG EOS
pda = new PDA();
pda.next(OnePassSignature);
pda.next(LiteralData);
pda.next(Signature);
pda.next(EndOfSequence);
assertTrue(pda.isValid());

// COMP EOS
PDA pda = new PDA();
pda.next(CompressedData);
pda.next(EndOfSequence);
assertTrue(pda.isValid());

You might say “Hold up! The last example is a clear violation of the syntax! A compressed data packet alone does not make a valid OpenPGP message!”.

And you are right. A compressed data packet is only a valid OpenPGP message, if its decompressed contents also represent a valid OpenPGP message. Therefore, when using our PDA class, we need to take care of packets with nested streams separately. In my implementation, I created an OpenPgpMessageInputStream, which among consuming the packet stream, handling the actual decryption, decompression etc. also takes care for handling nested PDAs. I will not go into too much details, but the following code should give a good idea of the architecture:

public class OpenPgpMessageInputStream {
    private final PDA pda = new PDA();
    private BCPGInputStream pgpIn = ...; // stream of OpenPGP packets
    private OpenPgpMessageInputStream nestedStream;

    public OpenPgpMessageInputStream(BCPGInputStream pgpIn) {
        this.pgpIn = pgpIn;
        switch(pgpIn.nextPacketTag()) {
            case LIT:
                pda.next(LiteralData);
                ...
                break;
            case COMP:
                pda.next(CompressedData);
                nestedStream = new OpenPgpMessageInputStream(decompress());
                ...
                break;
            case OPS:
                pda.next(OnePassSignature);
                ...
                break;
            case SIG:
                pda.next(Signature);
                ...
                break;
            case SEIPD:
            case SED:
                pda.next(EncryptedData);
                nestedStream = new OpenPgpMessageInputStream(decrypt());
                ...
                break;
            default:
                // Unknown / irrelevant packet
                throw new MalformedOpenPgpMessageException();
    }

    boolean isValid() {
        return pda.isValid() &&
               (nestedStream == null || nestedStream.isValid());

    @Override
    close() {
        if (!isValid()) {
            throw new MalformedOpenPgpMessageException();
        }
        ...
    }
}

The key thing to take away here is, that when we encounter a nesting packet (EncryptedData, CompressedData), we create a nested OpenPgpMessageInputStream on the decrypted / decompressed contents of this packet. Once we are ready to close the stream (because we reached the end), we not only check if our own PDA is in a valid state, but also whether the nestedStream (if there is one) is valid too.

This code is of course only a rough sketch and the actual implementation is far more complex to cover many possible edge cases. Yet, it still should give a good idea of how to use pushdown automata to verify packet sequences 🙂 Feel free to check out my real-world implementation here and here.

Happy Hacking!

by vanitasvitae at October 26, 2022 16:14

October 19, 2022

Erlang Solutions

Learning functional and concurrent programming concepts with Elixir

If you are early in the process of learning Elixir or considering learning it in the future, you may have wondered a few things.  What is the experience like? How easy is it to pick up functional and concurrent programming concepts when coming from a background in languages which lack those features? Which aspects of the language are the most challenging for newcomers to learn?

In this article, I will relate my experience as a new Elixir developer, working to implement the dice game Yatzy as my first significant project with the language. 

So far in my education and career, I have worked primarily with Java. 

This project was my first extensive exposure to concepts such as recursive functions, concurrent processes, supervision trees, and finite state machines, all of which will be covered in more depth throughout this article.

The rules of Yatzy

Yatzy is a variation of Yahtzee, with slight but notable differences to the rules and scoring. Players take turns rolling a set of five dice. They have the option to choose any number of their dice to re-roll up to two times each turn. After this, they must choose one of fifteen categories to score in. The “upper half” of the scorecard consists of six categories- “ones” through “sixes”. The score for each simply is the sum of all dice with the specified number. The “lower half”, consisting of the remaining nine categories, has more specific requirements, such as “two pairs”, “three of a kind”, “full house”, etc..

If a player’s total score in the upper half is equal to or greater than 63, they receive a 50-point bonus. The player with the highest total across the whole scorecard once all categories have been filled wins the game. 

Requirements of the project

Given this ruleset, a functioning implementation of Yatzy would need to do the following: 

  • Simulate dice rolls, including those where certain dice are kept for subsequent rolls
  • Calculate the score a roll would result in for each category
  • Save each player’s scorecard throughout the entire game
  • Determine the winner at the end of the game, 
  • Allow the players to take these actions via a simple UI.

Due to my object-oriented background, my approach to this project in prior years would be to define classes to represent relevant concepts, such as the player, the scorecard, and the roll, and maintain the state via instances of these objects. 

Additionally, I would make sure of iteration via loops to traverse data structures. Working with Elixir requires these problems to be tackled in different ways. The concepts are instead represented by processes that can be run concurrently, and data structures are traversed with recursive functions. 

Adapting to this different structure and way of thinking was the most challenging and rewarding part of this project.

Score calculations and pattern matching

My first step in writing the project was to implement functions for rolling a set of five dice and calculating the potential scores of those dice rolls in each available category. The dice roll itself was fairly simple, but makes use of a notable feature of Elixir that I had not previously encountered: setting a default argument for a function. 

In this instance, the roll function takes a single argument, ‘keep’, representing the dice from a previous roll that the player has chosen to keep.

def roll(keep \\ []) do
  dice = 1..6
  number_of_dice = 5 - Enum.count(keep)
  func = fn -> Enum.random(dice) end
  roll = Stream.repeatedly(func) |> Enum.take(number_of_dice)
  keep ++ roll
end

Here ‘keep’ has a default value of an empty list that will be used if ‘roll’ is called with no arguments, as it would be for the first roll in any turn. If a list is passed to ‘roll’, the function will only generate enough new numbers to fill out the rest of the roll, and then combine this list with ‘keep’ for its final output. This allowed my code to be simpler, defining one function head that could be used in multiple different scenarios.

The score calculations themselves were far more complex and required making use of Elixir’s pattern-matching capabilities. 

In this case, testing for a valid score in each category required accounting for every possible configuration the dice could appear in when passed into the function. I was able to greatly reduce the number of cases by ensuring the dice were sorted descending when passed, but this still left a lot to account for. However, Elixir’s pattern matching makes this process easier than it would be otherwise: the cases can be handled entirely in the function heads, and each function can be written in a single line:

def two_pairs([x, x, y, y, _]) when x != y, do: x * 2 + y * 2
def two_pairs([x, x, _, y, y]) when x != y, do: x * 2 + y * 2
def two_pairs([_, x, x, y, y]) when x != y, do: x * 2 + y * 2
def two_pairs(_roll), do: 0

Processes and GenServers

The next step of building the game was to implement processes, starting with those for the player and the scorecard. Processes in Elixir are vital for maintaining state and allowing concurrency – as many of them can be run simultaneously. I was able to set up a process for each player in the game -one for the scorecard belonging to each of those players, as well as one more to handle the score calculations.

As processes are dissimilar to the object-oriented model, they were the aspect of Elixir that took me the longest time to adjust to. I became comfortable with them by first learning how to work with raw processes, in order to better understand the theory behind them. After this, I converted these processes into GenServers, which contain improved functionality and handle most of the client/server interactions automatically.

The supervision tree

Another benefit of GenServers over raw processes is that they can be used as part of a supervision tree. In Elixir, a supervisor is a process that monitors other processes and restarts them if they crash. A supervision tree is a branching structure consisting of multiple supervisors and their child processes. In my Yatzy application, the supervision tree consists of a head supervisor with the scoring process as a child, along with another child supervisor for each player in the game. Each of these player supervisors has two children: a player and a scorecard.

Due to supervisors being syntactically similar to GenServers, the majority of this step of the process was simple. I had already learned how to implement the relevant API and callback functions, however, one mistake that took some time to notice was accidentally using GenServer.start_link instead of Supervisor.start_link in the API for the player supervisor. This problem was particularly hard to diagnose as it resulted in no compile or runtime errors in the application but did result in the supervisor’s child processes not starting and the game not functioning.

Finite state machine

After setting up the supervision tree, I still needed to define one more process to handle the functions for running through a single player’s turn. This process was implemented as another child of the head supervisor. As this process needed to handle multiple different states representing different stages of the turn, I constructed it as a finite state machine using the GenStateMachine module.

In this case, I defined four states, representing how many rolls are remaining in the turn: three, two, one, and none. It contains functions handling calls that represent a roll of the dice, which will set the machine to its next state, and functions that will reset it to its initial state for the end of the turn, including if the player decides not to use all their rolls. 

Below is an example of one of the calls, representing a player making their second roll in a turn.

def handle_event({:call, from}, {:roll, keep}, :two_rolls, data) do
  data = data ++ keep
  {:next_state, :one_roll, data, [{:reply, from, data}]}
end

Compared to learning how to work with GenServers and Supervisors, this functionality was actually rather simple to pick up. I had never worked with finite state machines in other languages, but the examples of GenStateMachine in the Elixir documentation were easy to understand and contained all the information I needed in order to implement this process.

User interface and recursion

Once the required processes were in place in a supervision tree, it was time to implement a simple text-based interface allowing a full game of Yatzy to be played all the way through. 

This would require each player in turn to receive the results of a dice roll, be prompted to choose which, if any, dice to keep for their subsequent rolls, and then again prompt them to choose which category to score in for that turn. It should loop through the players in this way until the game is complete, at which point it should declare the winner and prompt the user to reset the scorecards and play again.

Implementing the interface was the most complex and time-consuming part of the project. This required a significant amount of trial-and-error and researching through the Elixir docs, in order to get something functioning. However, one aspect that was easier than expected was working with recursive functions. I had rarely used recursion while working in Java due to the language’s focus on iterative loops, and as such never became fully comfortable with the technique. Implementing the interface required me to use recursion in several different places, and I was surprised at how easy it was to pick up in this language, with the pattern matching on function parameters making it simple to account for the end of the loop. The following is one of the recursive functions I implemented, which maps the results of a dice roll to the letters a, b, c, d, and e, allowing the player to pick which of the five they want to keep in the text-based interface.

def map_dice([head | tail], indexes) do
  index = String.to_atom(head)
  key_in_indexes = Map.has_key?(indexes, index)
  case index do
    index when key_in_indexes ->
      value = Map.get(indexes, index)
      [value | map_dice(tail, indexes)]
    _index ->
      [map_dice(tail, indexes)]
  end
end

Future

Although my Yatzy implementation is currently functioning correctly, I plan to extend the project further in the future. In the current version, only three players are supported, with their names hard-coded into the program. I would like future versions to have a dynamic amount of players, along with the ability for the players to specify their own usernames.

Additionally, I am also planning to learn the basics of Phoenix LiveView in the near future. Once I have done this, I would like to write a frontend for the program, allowing the players to interact with a more readable, visually appealing graphical interface, rather than the current text-based version.

Conclusion

Overall, I would describe my experience with the project as positive and feel that it served as a good introduction to Elixir. 

I was able to learn many of the basic features of the language naturally in order to fulfill the requirements of the game, and adjusted my ways of thinking about programming to better suit working with functional and concurrent programs. As a result, I feel like I have a good understanding of the basics of Elixir, and I am more confident about my ability to carry out other work with the language in the future.

The post Learning functional and concurrent programming concepts with Elixir appeared first on Erlang Solutions.

by Rhys Davey at October 19, 2022 10:19

October 13, 2022

Erlang Solutions

What we expect from Phoenix Framework 1.7

It is an exciting time for the Elixir community. As you may have seen at ElixirConf or ElixirConf EU, we are celebrating the 10th anniversary of Elixir. Despite now being 10 years old, there is no slowdown in the number of exciting new features, frameworks, and improvements being made to the language.

One of the most exciting developments for Elixir is undoubtedly Phoenix. It is a project that is growing in both features and uses cases at an incredible pace. Phoenix 1.5 included some huge changes including the addition of LiveView to the framework, the creation of LiveDashboard, and the new version of PubSub (2.0.).

Next Phoenix 1.6 introduced even more exciting features, most notably the HEEx engine, the authentication and mailer generators, better integration with LiveView, and the removal of node and webpack, which was replaced with a more simplified esbuild tool.

For many of us, each new Phoenix framework release brings back the feeling of being a kid on Christmas, we wait with eager anticipation for Chris McCord to announce the new toys we have to play with for the upcoming year, but with these new toys also comes a challenge for those who want to keep their skills and their systems up-to-date. The migration nightmare. We will revisit that at the end of this post.

Roadmap

Since Phoenix 1.5 it is a noticeable trend to move into LiveView, as we progress, LiveView can replace more and more JavaScript code, allowing the Elixir developer to get better control of the HTML generation. In the latest release, this trend is continued with the following new features: 

  • Verified Routes. This gives us the ability to define paths using a sigil that checks compilation time compared to defined routes.
  • Tailwind. In addition to answering our prayers concerning JavaScript and HTML, this new version also helps manage CSS.
  • Component-based generators. These features offer us a new and better way to write components.
  • Authentication generation code using LiveView. This lets us generate the code for the authentication code but using LiveView instead of the normal controllers, views, and templates.

We will go deeper into each of these features, but you can already see a trend, right? We are moving more and more to LiveView in the same way we are removing the need to manage things like HTML, JavaScript, and CSS.

First, let’s look more at LiveView specifically, for release 0.18, Chris McCord announced these improvements:

  • Declarative assigns/slots – which let us define information about attributes and slots which are included inside of the components.
  • HTML Formatter – which performs the format (mix format) for HEEx code even if it’s included inside of the sigil ~H.
  • Accessibility building blocks.

Now let’s look at each of these elements in deeper detail.

Verified Routes

The story is that Jason Stiebs (from the Phoenix team) has been requesting a better, less verbose way, to use the routes for the last 8 years. The 12th time he requested it Chris McCord agreed to this feedback and José Valim had a fantastic way to make that happen.

The basic idea is that if we have this:

This is generating the route which we could use in this way:

This is very verbose, but it could be even worse if we have a definition of the routes nested like this one:

And it is just as verbose when we use LiveView:

To avoid this, the Verified Routes provides us a shortcut using the path:

As you can see, using the sigil “~p” we can define the path where we want to go and it’s completely equivalent to using the previous Routes helper function.

The main advantage of this feature is that it allows us to write the path concisely and still check if that route is valid or not in the same way we would use the Route Helper function. 

Tailwind

To understand this change let’s look at what Adam Wathan (creator of Tailwind) said about CSS and the use of CSS:

The use of CSS in a traditional way, that is using “semantic class names”, is hard to maintain and that’s why he created Tailwind. Tailwind is based on the specification of how the element should be shown. There can be different elements that are semantically the same, for example, two “Accept” buttons where we want one to appear big and the other a bit narrower. Under this paradigm, we’d be forced to use the class “accept-button” in addition to the classes which are modifying this case and which do not allow them to be reused.

The other approach is to implement small modifications to how we present the buttons. In this way, we can define a lot in HTML and get rid of the CSS.

The main idea, as I said previously, is to replace as much CSS as possible in the same way as LiveView replaced a lot of JavaScript:

For example, using Tailwind with HTML and getting rid of CSS, we could build a button like this one with the code shown in the image below:

It could be argued that it’s complex, but it’s indeed perfect from the point of view of LiveView and components because these classes can be encapsulated inside of the component and we can use it in this way:

And finally, in the template:

Easy, right?

Authentication generation code using LiveView

Big thank you to Berenice Medel on the Phoenix team, she had the great idea to have the generation of the authentication templates work with LiveView. 

Declarative Assigns / Slots

Before going into this section, Chris McCord gave a big thank you to Marius Saraiva and Connor Lay. They are the people in charge of all of the improvements regarding declarative assigns, slots, and HEEx.

The idea behind slots and attrs is to provide us with a way to define attributes and sub-elements inside of a defined component. The example above, it’s defining a component with the name “table”. It’s defining the attributes “row_id” and “rest”, as you can see in the documentation, the attributes for the table are “rows”, “row_id”, and “class”. That means we can find “row_id”, then “rest” will feature a map with all of the remaining attributes.

As we said, the slot is a way to indicate we are going to use a sub-element “col” inside of the “table”. In the example, you can see two elements “col” inside of “table”. The “col” element has only defined one attribute “if” which is a boolean.

HTML Formatter

A big thank you to Felipe Renan who worked on the implementation of this for HEEx to be included in Phoenix. Now, it’s possible to have a “mix format” fixing the format of the code written inside of the templates, even inside of the ~H sigil.

Accessibility building blocks

Phoenix 1.7 includes some primitives for helping to create more accessible websites. One of them is “focus_wrap”:

This helps define the areas where you want to shift focus between multiple elements inside of a defined area instead of a whole website. 

This works in combination with functions in the JS module which configure the focus like a stack. When you go into the modal it pushes the focus area that we use and when the modal is closed, we pop out from that area of the stack and stay with the previous one.

More improvements in the Roadmap

One of the improvements for LiveView is Storybook. Storybook is a visual UI creator which lets us define the components we want to be included in our websites and then generate the code to be implemented for it. Christian Blavier did great work starting this in his repository but he’s now off and the Phoenix team is going to be moving it forward and evolving it.

Streaming data for optimized handling of collections data is another priority in the roadmap. The work for this has already started, fingers are crossed that it might be announced for the next release.

During recent conferences, another speaker raised a concern about the messaging incompatibility between LiveView and LiveComponent, luckily, this is on the roadmap to be fixed shortly.

And is that all?

With all the developments in Phoenix, it would be easy to talk about at much greater length and in much greater detail. The pace of the Phoenix team’s progress is impressive and exciting. 

As it continues to grow it is easy to imagine a future where we only need to write HEEx code inside of Elixir to get full control of generated HTML, CSS, and JavaScript for the browser. It’s exciting to imagine and will be sure to further grow the use and adoption of Elixir as a full-stack technology. 

Read to adopt Elixir? Or need help with your implementation? Or contact us about our training options. 

The post What we expect from Phoenix Framework 1.7 appeared first on Erlang Solutions.

by Manuel Rubio at October 13, 2022 11:44