This is my update post for the 7th week of GSoC. The next evaluation phase is slowly approaching and it is still a lot of work to do.

This week I started to work on integrating encryption in my Jingle File Transfer code. I’m very pleased, that only very minor changes are required to my OMEMO code. The OmemoManager just has to implement a single interface with two methods and that’s it. The interface should be pretty forward to implement in other encryption methods as well.

Unfortunately the same is not true for the code I wrote during the GSoC. There are many things I haven’t thought about, which require major changes, so it looks like I’ll have to rethink the concept once again. My goal is to make the implementation so flexible, that description (eg. video chat, file transfer…), transport (eg. Socks5, IBB…) and security (XTLS, my Jet spec etc.) can be mixed in arbitrary ways without adding in glue code for new specifications. Flow told me, this is going to get complicated, but I want to try anyway . For “safety” reasons, I’ll keep a seperate working branch in case the next iteration does not turn out as I want.

Yesterday Flow found an error in smack-omemo, which caused bundles not getting fetched. The mistake I made was, that a synchronous CarbonListener was registered in the packet-reader thread. This caused the packet-reader thread to timeout on certain messages, even though the stanzas arrived. It is nice to have this out of the way and it was a good lesson about the pitfalls of parallel programming.

While reading the Jingle File Transfer XEP I also found some missing XML schemes and proposed to replace the usage of xs:nonNegativeInteger and xs:positiveNumber with xs:unsignedLong to simplify/unify the process of implementing the XEP.

Thats basically it for this week. Unfortunately I have an upcoming exam at university next week, which means even less free time for me, but I’ll manage that. In the worst case I always have a second try on the exam

Happy Hacking!

I have become progressively disenchanted with, indeed actively opposed to, politics. More and more, I see it as a disease that has grown to take over not only the "body politic" but also the individual mind and soul. My enemy is the assertion that "the personal is the political", and that there is no domain of life outside of politics - not art or music, not love or friendship, not philosophy or religion - nothing. This idea emerged in the radical Marxist-feminist movement of the late 1960s, and was given canonical expression in an essay by Carol Hanisch....

Hello,

a short message to give you the link to our talk explaining Salut à Toi and Libervia at Linuxwochen in Vienna (Austria), 2 weeks ago.
The talk is in english, and we do a live demo of an installation with docker images, and publishing to the blog graphically with Libervia or directly in console with Vim. Thanks to the Linuxwochen team for inviting us and organising this nice event!

Talking about Vim, a new Emacs frontend has been started by Xavier Maillard (screenshot below), so nobody is jealous ;)

We have also started working on Cagou, our new desktop/mobile device frontend.

For more details, you can come to talk in our room: sat@chat.jabberfr.org.

First image of Sententia, the new Emacs frontend:

Hi to you,

It's time for a blog post on Cagou development.
Cagou is the new desktop/Android frontend for "Salut à Toi" (SàT), the full-featured social network based on standards (XMPP) and with a strong emphasis on ethics.

This frontend is using the very nice Kivy framework, and is already usable on desktop.

Let's see a video on the current state:

As you can see, the interface can be divided easily. This idea is inspired by the excellent Blender interface, and make it easy to adapt to a small or big screen by choosing the number of widgets accordingly.
Instead of being a discreet software that you'll use only when you want to answer to somebody, Cagou is an application which should be used full-screen, with a layout suiting the current needs.

Other important point: the contact list is not the central element (as it is usually in instant messaging software). You can access it by selecting the widget, or put it on the side to simulate traditional layout, but it is not visible by default and you can spend a whole session without using it.

About notifications, it will be slighly different on desktop and on platforms with their own notification system (like Android). In the later case, the native way will be used.
On desktop, a specific header is used. 2 cagou icons (thanks to Muchoslava for its contributions) can appear: one on the left for notes (i.e. short messages which will automatically disappear), and one of the right for more important notifications, needing attention from the user.
A click on the left icon will show the last 10 notes, and one the right one show the next important notification.

This was made to avoid popups: they are displayed only after an interaction of user, and will not appear all of a sudden while you are writting a message.

Thanks to the architecture of SàT, menus are already available with a couple of useful actions.

There are several ways to start a chat: you can click on an item in the contact list, or enter a jid (XMPP identifier) in a chat widget.
Discussions are not closed when you switch widget, and they can be later accessed by swiping the current one (see video below).

There have been a lot of work done and the interface starts to be really usable. Concerning the desktop the greatest part is already finished, the most important remaining thing is to adapt interface to features (e.g. displaying of delivery reciept) and to make new widgets.

The other big thing to cope with would be displaying of HTML content and of course the Android port. An Ubuntu touch port is an eventuality.

And for those who wonder: Cagou will as well benefit from end-to-end encryption (for now OTR, but OpenPGP is planned for next release, OMEMO is considerated), and other SàT features

The project is still in active progress, but it is really demanding. If you can contribue in any way – code, distributions packages, tests, graphics, translations, etc. – please contact us by any mean shown on our website.

Last but not least, we reminds all of the Aziruka contributors of the crowdfunding campaign, that they can claim their counterpart if not done yet (or request it later).

See you soon :-)

We have just launched a crowdfunding campaign to develop a new frontend to Libervia, and port it to Android (in a native application!).

http://www.arizuka.com/en/projects/libervia (subtitles available in many languages).

If the campaign is successful, we'll have a unique tool. Here are only a few features managed by Libervia:

• blogging/microblogging: we have a decentralized blogging engine based on XMPP, no need to create an account/validate an email to post a comment
• instant messaging, single or multi-user, with a lot of features
• end to end encryption: we already manage OTR for single chat, there is a good probability that we also implement OMEMO/Axolotl
• file sharing: being file uploading or P2P transfer
• group permissions: similarly to what others call “circle” or “aspect”, you can share with only your friends or your family
• a lot more, check http://salut-a-toi.org/features.html (and don't miss the experimental ones)

In addition we are a non-profit association focusing on ethic, we want to make a good tool, not a place for advertisements!

So if you are looking for next-gen communication tool, please support us and share the link: http://www.arizuka.com/en/projects/libervia

Thanks !

The 0.6.1 release of Libervia (Salut à Toi) is out, and it is a big one (despite of the minor version number). The one which will follow — and is already well advanced — will be the « general public » release, or in other words we'll do what is necessary to have a release stable enough to be used seriously, and easy enough to install so the large public can enjoy it.

You can have a look at the changelog or the repository for more details, but here are the main changes and fixes:

• import plugin which can be extended, with "importers" for Dotclear and DokuWiki. Please note that, thanks to the use of standards, this can be useful even if you're not using Libervia to display your blog (it's compatible with Movim for instance)

• in addition to the Dotclear import, its wiki syntax is available for SàT too, it comes with the already available Markdown and XHTML.

• automatic subscription to PubSub feeds for new accounts, this avoids an empty pages for newcomers

• MAM (XEP-0313) is implemented for PubSub, this allows to go back in publication history and to search in a more flexible and powerful way. Note that MAM is not available yet for instant messaging, it's planed for 0.7

• better detection of repository revision

• jp (CLI) has now an option --output with who you can specify the output format (e.g. JSON so parsing by an other command is easy).

• jp: new command to import a blog (import), (re-)edit a new/old publication (edit) or have a preview in real time (preview). We have published a tutorial in french to explain how to publish on your XMPP blog with Vim, Emacs or whatever else, any help to translate to English welcome

• jp: commands to manage your roster: get it (get), have statistic on it (stats), or clean it (purge)

• jp: the command "message" has been moved to "message send" to be coherent with other commands, and because a "message get" should come in the future

• Primitivus (console interface): paste detection (a validation is needed in this case, this avoid sending a bad text like a password by mistake), and handling of "bracketed paste mode"

• Libervia (web interface): a new option "allow_registration" allows to deactivate the new account registration page, specially useful if you are alone on your instance, or if you create accounts by your own means

• Libervia: a new pop-up with a counter is displayed if connection with server is lost

• Libervia: new favicon with a notification counter

• Libervia: connection with an external jid is fixed

• Libervia: it is now possible to redirect pages or to integrate a local directory in Libervia. That's used to display directly the blog instead of the login page on www.goffi.org.

• Libervia: TLS chains are now handled (hello Let's Encrypt)

• Libervia (blog): tags are now handled, including search

• we have also updated and improved our Docker images, so you can easily install and test Libervia. Please check the wiki page with the explanations.

This is a summary of 0.6.1, but it is not covering everything!

Salut à Toi is an ethical project is staying away from big proprietary companies, if you feel like helping it contact us by email (contact at salut-a-toi d.o.t org) or on our XMPP MUC room. We need more hands!

The 0.6.1 is the last one before the "general public" one and has its issues to be improved. Don't hesitate to report them so that the 0.7 will be rock solid :)

The 0.7 will go through beta testing phase, and will bring in particular:

• refactoring of messages, which will be needed to implement a lot of extensions which were blocked until now, like last message correction, or server side messages archives (MAM). This part is already finished, and you can see below an example of an experimental language detection plugin for Primitivus (based on langid):

• Cagou, the new desktop/mobile devices frontend that we have promised after the success on our crowdfunding campain. The development is advanced, a post about it should arrive soon.

• blog and messaging gateways

Beside all this, there is a good probability that we implement new end 2 end encryption system, in addition to the already available OTR. We specially think about OpenPGP and OMEMO.

Salut à Vous !

It's been a little more that one year now that the crowdfunding campain has been successfuly completed, and that we have promised to develop a new Desktop/Android frontend for "Salut à Toi", our Multipurpose, multi frontend communication tool based on XMPP.

It's time for an overview of the state of the development. You'll find below a link to the first .apk (pre-alpha), FOR TESTING PURPOSE ONLY.

Cagou (SàT)

As we've already announced, the new Desktop/mobiles (Android only for now) frontend will be named "Cagou", a wind to the Kivy framework that we're using, and to this endemic bird of New Caledonia, which bark and can't fly.

Oh, and yes we know that "Cagou" has different meaning in different languages.

Development

This part is technical, you can go directly to the next one if you're not interested.

After a short time to get familiar with the Kivy ecosystem, the first step has been to integrate "Quick Frontend", which is a common base that we are using, as you guess, to quickly develop a frontend and factorise the code (cache handling, contact list – or "roster" in XMPP world –, widgets, etc.), then to integrate the "bridge" which is the name that we are using for the IPC and which handle the communication between the backend (which is the heart of SàT) and the frontends.

This phase went relatively well, and was followed by a thought on the architecture and user interface.

Once all of this was more or less usable, the Android port could began.

Things have been a big more complicated there. The Kivy community has created several tools to develop on this platform, including python-for-android (compilation and archive creation tool), and Buildozer (multi-platform tool to facilitate the use of the first one). Using these tools take some efforts, specially for a project already well advanced (it's far more easy if you start directly with Kivy and the Android port).

There a 2 "chains" for developing on Android: the old and the new one. After the first unsuccessful tries with the new one, it has been temporarly put aside for the old one, the time to build the foundations of the port.

Pure Python dependencies can be imported easily, but as soon as it get more complicated, you'll have to creates "recipes" to tell how to build something to python-for-android. Fortunately, most of those needed for SàT (Twisted in particular) were already available, and just needed to be updated.

After all this dependencies and building chain problems solved, and after the joy to see the first .apk (no working), 2 other big troubles showed up: D-Bus which is the main "bridge" is not usable on Android, and how to have the backend and the frontend running together?

Being my first Android development, I've had to read a lot or documentation (which luckily is not lacking), and after a first try with a bridge "embedded", allowing to have backend and frontend in the same process, it's finally a new "pb" bridge which solved the issue. "pb" stands for "Perspective Broker", the Twisted IPC. Android native IPC is an other option to be evaluated in the future.

To launch the backend, Kivy comes with modules to start it as an Android service. With it, the backend can stay in background and process messages and other activities when the frontend is not visible to the user (which means frozen until the user show it again on Android).

This section is already long, so I'll skip other problems (like the lack of HTML widget), and let's now talk about the UI.

UI

At the moment Cagou is usable on desktop (GNU/Linux, but other platforms will most certainly follow), and on Android.

The current release is a pre-alpha, the .apk is available below, only to have a rough idea of the software. It is really bugged, doesn't check yet server certificates, doesn't handle SRV record on Android, etc. This is linked for 2 reasons: showing the progress, and having feedbacks early enought to go in the right direction.

You can't create account from the application (this will come before the release), so if you have no account yet you can use Libervia(SàT), the demo instance of our web frontend, to create one.

main concepts

Cagou's UI take inspiration from the excelent Blender. The widget selection and splitting are the most obvious examples. The huge bars that you can see should disappear later in favor of a small button, which may follow Blender example again. The goal here is that a newcomer can switch widgets intuitively, and an advanced user can use this splitting feature.

The contacts list is not the central element in the interface, it can be displayed if wanted, but it's not needed to use Cagou.

The upper menu, which is for now always visible, will certainly be only available on Desktop. On Android the menu button or a floating touch one should replace it before the release.

If you have notifications, they should appear for a couple of second on the top, but you can read them later by touching the cagou head on the upper left.

If there is an event needing user action (for instance a website needs your authorisation with XMPP), an other cagou will appear on the right, and the dialog will only appear after touching it. The idea is to never have an unwanted popup stealing the focus when you are doing something else: popups are displayed only with explicit user action.

In Android case, it may be replaced by the native notification system in the future, but it's not decided yet because notifications history would not be available anymore.

To change the mode (widget), you just have to click on the upper left button in the current widget. Only 4 widgets are available so far: the selector which display all widgets available, the settings, the contact list, and the chat. Other are planed, notabily the blogging one.

Inside a widget (only for chat so far), you can swipe horizontaly to switch between active conversations.

For now it's not easy to use the first time (you have to do a very quick swipe), it needs some tuning.

As for other SàT parts, Cagou is thought since the beginning to work with plugins and being easy to extend. All the widgets and file transmitting system (see below) are plugins.

instant messaging (chat)

As we want a frontend usable on small screens, which is simple but without making sacrifice on features, we have to find a compromise between data displayed on the screen and the elements/buttons needed for actions. Too many elements will make the UI more complicated and take room, but not enough will make the actions more difficult to do.

The current design (which can evolve) has a header with an input bar and a button (in addition to widget selection button), the body with messages, and a bottom input bar with a button too.

To talk with one or more contact(s), type some letters belonging to its name (anywhere in the name). For now, only identifiers ("jid") and already opened talks are looked for, but later the search will include names, nicknames and bookmarks.

Cagou detect if you want to talk to a single person, or to a group chat room, and will act consequently.

The end 2 end encryption is there, but only with OTR (v2) for the moment. OX (modern OpenPGP) and OMEMO are planed, but it's not sure that they will be available for the next release (they may be there for the following one, 0.8). To use it, you just have to click on the lock, it will be closed if the chat is encrypted, and will have a question mark if your contact is not authenticated.

Let's go directly to the input bar. You'll see on the right a "+" button, click on it and you'll have a sending dialog.

This dialog has 2 buttons on the top, with which you can choose between uploading the file or sending it directly using peer 2 peer. A text under them explain in simple language where your file will go, and if encryption is done (for now all files are sent unencrypted).

This text message is important to let the user understand where the data are transiting, it's the kind of information we plan to put in several locations.

The buttons below are the various transmitting options. On desktop, you can only use a file browser (for now), but on Android you can also send a picture from your gallery, take a new photo or a video, or record a voice message.

He is a screenshot of voice recording:

around Cagou

In addition to the work on Cagou itself, other things have been done.

A now mandatory feature with the explosion of mobile devices, carbon copy has been implemented. Server archives is implemented for long fot blogging, but not yet for chat messages, it will be done before the stable release.

Small binary files handling ("BoB" for "Bits of Binary") is now available. Implentation has been motivated by its use on the friend project Movim.

Delivery receipts and HTTP authentification have been contributed by Chteufleur.

Since the 0.6.1 version, messages handling has been improved, making now possible to implement features like last message correction, planed for the release.

Lastly, component (for gateways) and static blogs have also been implemented, but we'll talk about this later.

help

SàT is a huge project with strong ethical values (check the social contract), and it need help! It can be as easy as coming to talk with us (our room is at sat@chat.jabberfr.org, also availble by clicking here).

If you can, financial aid would of course be useful, we have recently opened an account on the excellent Liberapay.

You can also join us in our association, all informations are available here

And of course contributions are useful, specially development but also translations, icons, CSS, etc. Most of the coding is done in Python, and working on SàT is working on a tool you may use everyday. You'll not find us on big centralised forges, but we are in the process of modernising our development tools (more to come on that).

Talk about our association and project around you is always useful.

I think the most important things have been said, I'll try to keep you up to date with shorted posts.

Oh, and the link with the pre-alpha version (once again: FOR TESTING PURPOSE ONLY): https://www.goffi.org/public/Cagou-0.1-debug.apk

Today is the 200th anniversary of Henry David Thoreau's birth. To celebrate, I've published two books on his philosophy of life: The Upland Farm (a 50-page original exploration in the form of a journal) and Seasons of Thoreau (a 300-page compilation of quotes from his writings). Both books are available for free on my website, and can also be purchased in paperback and ebook formats from Amazon....

Time is flying by. The sixth week is nearly over. I hope I didn’t miscounted so far

This week I made some more progress working on the file transfer code. I read the existing StreamInitialization code and found some typos which I fixed. I than took some inspiration from the SI code to improve my Jingle implementation. Most notably I created a class FileTransferHandler, which the client can use to control the file transfer and get some information on its status etc. Most functionality is yet to be implemented, but at least getting notified when the transfer has ended already works. This allowed me to bring the first integration test for basic Jingle File transfer to life. Previously I had the issue, that the transfer was started as a new thread, which was then out of scope, so that the test had no way to tell if and when the transfer succeeded. This is now fixed

Other than that integration test, I also worked on creating more junit tests for my Jingle classes and found some more bugs that way. Tests are tedious, but the results are worth the effort. I hope to keep the code coverage of Smack at least on a constant level – already it dropped a little bit with my commits getting merged, but I’m working on correcting that again. While testing, I found a small bug in the SOCKS5 Proxy tests of Smack. Basically there were simultaneous insertions into an ArrayList and a HashSet with a subsequent comparison. This failed under certain circumstances (in my universities network) due to the properties of the set. I fixed the issue by replacing the ArrayList with a LinkedHashSet.

Speaking of tests – I created a “small” test app that utilizes NIO for non-blocking IO operations. I put the word small in quotation marks, because NIO blows up the code by a factor of at least 5. My implementation consists of a server and a client. The client sends a string to the server which than 1337ifies the string and sends it back. The goal of NIO is to use few Threads to handle all the connections at once. It works quite well I’d say. I can handle around 10000 simultaneous connections using a single thread. The next steps will be working NIO into Smack.

Last but not least, I once again got excited about the XMPP community
As some of you might now, I started to dig into XMPP roughly 8 months ago as part of my bachelors thesis about OMEMO encryption. Back then I wrote a mail to Daniel Gultsch, asking if he could give me some advice on how to start working on an OMEMO implementation.
Now eight months later, I received a mail from another student basically asking me the same question! I’m blown away by how fast one can go from the one asking to the one getting asked. For me this is another beautiful example of truly working open standards and free software.

Thank you

ejabberd 17.07 includes an important security fix. Except this fix, the release is completely equivalent to 17.06. If you run any version from 17.03 to 17.06, it’s possible to consume all available ports regardless of ERL_MAX_PORTS. You should upgrade to 17.07 as soon as possible if you are running a public server.

Please, note that the security of users, data, conversation, etc. is not at stake. Data / privacy is safe.

Users of ejabberd Business Edition and ejabberd SaaS are safe and have nothing to do as the issue was not in those code base.

Changes

Core

• Close accepted socket if sockname/peername has failed

Feedback

As usual, the release is tagged in the Git source code repository on Github.

The source package and binary installers are available at ProcessOne.

If you suspect that you’ve found a bug, please search or fill a bug report on Github.

<p>MongooseIM is a platform for businesses to build messaging and social apps, or add messaging and social features to existing apps. When you need to adapt to growing user demand, we&rsquo;re here to guarantee that your service scales, grows with new functionalities and remains reliable for all.</p> <p>The fullstack <a href="http://www2.erlang-solutions.com/l/23452/2017-07-06/4t7t1v">MongooseIM platform</a> offers various components, both for the backend (Erlang, Elixir) and the frontend (iOS, Android, Web) for you to pick up and assemble like pieces of a puzzle.</p> <h2>Latest beta release of MongooseIM 2.1.0 is here!</h2> <p>This time, we focused on improving the source code and documentation quality. Everything to make your <a href="http://www2.erlang-solutions.com/l/23452/2017-07-06/4t7t1v">MongooseIM</a> deployment easier to maintain and more stable.</p> <p>Our team wants to ensure that the next version will be as mature as possible, with accurate and easy-to-read documentation, and friendly source code for developers; free of known bugs and issues. </p> <p>Beta2 is also time to welcome our new ICE/STUN/TURN component, which has almost reached its stable version by now.</p> <p>And by the way&hellip; Hi! I&rsquo;m Piotr and I&rsquo;m going to be your guide in the tour of beta2!</p> <h2>Read The Fine Manual</h2> <p>It&rsquo;s easy to forget about documentation when you&rsquo;re chasing after new features, improvements and world-changing pull requests. When that does happen, you run the risk of users not appreciating the new shiny module you prepared for them - mainly because they do not know what it is or how to configure it.</p> <p>We understand it is a challenge to maintain up-to-date, high quality documentation of a project of MongooseIM&rsquo;s size. It is one of our primary tasks for 2.1.0 release to pay off this debt and ensure that no important information piece is missing. It involves revising docs about every module and configuration description correctness. We are verifying the validity of high level concepts and guides.</p> <p>Every GitHub issue, that narrows down to a configuration problem, is a clear indication what we should improve. It&rsquo;s an ultimate goal to ensure that every MongooseIM user will be able to configure the server without any doubts on which option to use and how! It is our priority to ensure that our users always find the information they seek, no matter if these are abstract concepts, design decisions and development guidelines.</p> <p>In beta2 we continue to work towards a high quality documentation with 50% of docs being reviewed and already improved. We aim at 100% by the 2.1.0 release freeze!</p> <h2>MongooseICE</h2> <p>Nowadays it&rsquo;s not so easy to have each user occupy public IP addresses. In most cases both users are behind NATs, which effectively prevents them from establishing a peer-to-peer connection. It means no file transfers (except for e.g. inefficient <a href="https://xmpp.org/extensions/xep-0047.html">In-Band Bytestreams</a>), no video or voice calls. What can an application provider (or just two users connecting to public XMPP clusters) do under these conditions? ICE/STUN/TURN enables two clients to establish a peer-to-peer connection and exchange data, in a manner that is cheap and efficient.</p> <p>You can find more information about our new backend service in the MongooseIM family <a href="https://github.com/esl/fennec">esl/fennec</a>. Yes, the link is correct. Fennec is simply the codename until its first stable release.</p> <h2>Bloated-stanza-proof</h2> <p>Beta2 features an important security update, which moves the stanza size check from Erlang code to the XML parser. It means it&rsquo;s no longer possible to fill the server&rsquo;s memory with an enormous stanza. Before this update, an oversized XML was discarded of course, but only after parsing, so the memory was already allocated and the damage was already done.</p> <p>Because of this vulnerability, a malicious user could trigger out of memory exception and crash the server. Beta2 prevents it.</p> <h2>Goodbye, deprecation!</h2> <p>Erlang/OTP evolves over time, so does its API. Some functions become deprecated for good reasons. In beta2 we&rsquo;ve got rid of <code>erlang:now/0</code> and <code>crypto:rand_bytes/1</code> calls.</p> <p>The highlights of this change are:</p> <ul> <li>Better performance (less Erlang VM locks)</li> <li>More secure tokens and authentication salts</li> </ul> <p>OK, now it gets technical. :)</p> <p>The former function has been in use for a very long time but, as server hardware provided more and more CPU cores, this BIF&rsquo;s weaknesses had to be repaired. First of all, <code>now()</code> call guarantees returning strictly monotonic values on a single machine, so it requires synchronisation between all CPU cores. Obviously it means a performance penalty. What is more, in extreme cases simultaneous calls to this function from multiple schedulers could shift the timer more and more into the future. We&rsquo;ve replaced it with new time API, choosing appropriate BIFs for every use case (<code>now()</code> used to be &ldquo;one size fits all&rdquo;).</p> <p>It is a bit tricky in case of the Message Archive Management though. Message IDs are heavily based on timestamps and we couldn&rsquo;t risk ID duplication, so the microsecond part is replaced with an arbitrary integer (in the 0-999 range, of course) to reduce a chance of collision. It limits message ordering precision to a millisecond, so we are still exploring other possible timestamp sources. We&rsquo;d appreciate all feedback from MAM users on this new method. For those who would like to learn technical details of these solutions: feel free to check the link to a pull request in <a href="https://github.com/esl/MongooseIM/releases/tag/2.1.0beta2">the changelog</a>. Look for the <code>mod_mam_utils.erl</code> file.</p> <p>The latter change, <code>crypto:strong_rand_bytes/1</code> is all about security. Instead of an internal pseudo-random number generator, it calls a <code>RAND_bytes</code> method from OpenSSL.</p> <p>One deprecation remains: <code>random</code> module. It will be replaced with <code>rand</code> in MongooseIM source code once we stop supporting OTP 17.5 in favour of the latest version from the 20.x line.</p> <h2>True beauty lies within</h2> <h3>A stanza&rsquo;s tale</h3> <p>Amongst many internal technical changes, <code>mongoose_acc</code> is one of the most significant ones. With beta2, stanza&rsquo;s whole life&rsquo;s history is recorded inside <a href="http://mongooseim.readthedocs.io/en/latest/developers-guide/accumulators/">a common structure</a>. Although it was introduced quite some time ago, at last it is not unpacked in the middle of processing. It is created when a stanza arrives and lives until the end of processing.</p> <p>If you haven&rsquo;t heard about it yet: what if I told you that a stanza is not just some internal XML representation passed around by processes? What if the hook execution results were not lost? These can be privacy &amp; AMP check results, response to an IQ or just some info extracted for convenience (e.g. <code>from</code>, <code>to</code> attributes). Handy, right? That&rsquo;s what <code>mongoose_acc</code> is!</p> <p>For now it provides more information in logs and easier access to some parameters, but in the future it will, for example, allow us to perform advanced stanza tracing on demand.</p> <p>The cost of passing a bigger structure around is balanced by the fact that some operations are faster (e.g. no need to query the XML structure to get some commonly accessed attribute).</p> <h3>Rock &lsquo;n&rsquo; Roll</h3> <p>Some of you, especially contributors, may have already met Elvis - our <a href="http://inaka.net/blog/2014/09/05/erlang-standards-and-elvis-rock/">King of Code Style</a>. Obviously we&rsquo;ve always paid attention to our coding style but we&rsquo;re humans - it&rsquo;s not difficult to nest expressions too much or abuse macros. Now we have an automatic checker that ensures we don&rsquo;t omit anything. It verifies every pull request and mercilessly points out every mistake.</p> <p>We not only care for the new code, but for the existing one as well. Thus, we&rsquo;ve refactored a large part of the existing codebase to meet these standards as well. It obviously brings value to us, the developers. It includes not only the MongooseIM team, but all other contributors as well. By eliminating ugly or overly complicated parts we increase the pace of future work and make all these modules and functions easier to read and understand. This has a direct benefit for our whole community, as all the new features will be developed faster and bugs will appear less and less frequently!</p> <h2>It&rsquo;s the little things&hellip;</h2> <h3>Deeper into REST world</h3> <p>MongooseIM&rsquo;s client REST API grows. With beta2 the roster management is now possible via this interface. This improvement is yet another result of our efforts to help the XMPP world cross the boundaries between protocols. You can find more information in our beautiful <a href="http://mongooseim.readthedocs.io/en/latest/swagger/index.html?client=true#/">Swagger documentation</a> (see &ldquo;Contacts&rdquo; group).</p> <h3>Silent push notifications</h3> <p>MongooseIM is able now to send “silent” push notifications. They enable client applications to receive data via this channel without any information shown to the human user. It is fully up to the application to handle this data and decide on the action.</p> <p>This feature requires MongoosePush 0.9.0 component or newer.</p> <h3>Remaining technical improvements</h3> <p>There are other new features worth checking out. Some of them are beneficial to MongooseIM users, others can make MongooseIM developers&rsquo; and contributors&rsquo; lives easier.</p> <ol> <li><code>mod_vcard</code> supports RSM in queries now. It means that vCard search results can be paginated now.</li> <li>You can choose whether Message Archive Management should archive groupchat messages in users&rsquo; private archives or not. A default behaviour so far was to store them anyway.</li> <li>One of our Travis CI jobs executes unit &amp; integration tests on OTP 19.3, effectively ensuring MongooseIM compatibility with latest stable OTP version.</li> <li>New command available in mongooseimctl: <code>import_users</code>. It takes a CSV file (with <code>user,domain,password</code> columns) as input and I think it&rsquo;s pretty obvious what it does then. :)</li> <li>Requesting a roster entry from <code>mod_roster</code> backend may now return a <code>does_not_exist</code> atom.</li> <li><code>gen_mod</code> will print a warning message when a module&rsquo;s <code>start/2</code> callback links to the caller. It is a discouraged practice, because it can cause problems when starting modules via RPC or in console.</li> </ol> <h2>Load tests</h2> <ul> <li><a href="http://tide.erlang-solutions.com/public/93b81248-fc27-4e4c-bb8d-7aa79b9f8934">2.1.0beta1</a></li> <li><a href="http://tide.erlang-solutions.com/public/b2032f52-09e4-4207-a4a9-772ceaf73f04">2.1.0beta2</a></li> </ul> <p>Our standard Tide tests results in 1-1 messaging show no difference in terms of latency between beta1 and beta2.</p> <h2>Changelog</h2> <p>Please feel free to read the detailed <a href="https://github.com/esl/MongooseIM/releases/tag/2.1.0beta2">changelog</a>, where you can find a full list of source code changes and useful links.</p> <h2>Contributors</h2> <p>Special thanks to our contributors: <a href="https://github.com/astro">@astro</a> <a href="https://github.com/strugee">@strugee</a> <a href="https://github.com/msantos">@msantos</a> <a href="https://github.com/daniel-e">@daniel-e</a> <a href="https://github.com/deadjdona">@deadjdona</a>!</p> <h2>What&rsquo;s next?</h2> <h3>Final release is coming!</h3> <p>It&rsquo;s the last stretch for bug fixing or improving performance, documentation and style. We anticipate the feature set to remain stable until our final release, but surprises do happen so stay tuned for <a href="https://twitter.com/MongooseIM/">new tweets</a> and <a href="https://github.com/esl/MongooseIM/pulls">new PRs</a>!</p> <p>We strongly encourage everyone to join our wonderful circle of contributors. Every PR is welcome, no matter whether it&rsquo;s a fix in Erlang code, Makefile fix or new documentation section.</p> <h3>Spoilers!</h3> <ul> <li>ODBC backend for <code>mod_muc</code>.</li> <li>SSL support for MySQL connection.</li> <li>Better JWT authentication support.</li> </ul> <p>But ssssssh! You haven&rsquo;t heard it from me, allright? :)</p> <h2>Test our work on MongooseIM 2.1.0beta2 and share your feedback</h2> <p>Help us improve the MongooseIM platform:</p> <ul> <li>Star our repo: <a href="https://github.com/esl/MongooseIM">esl/MongooseIM</a></li> <li>Report issues: <a href="https://github.com/esl/MongooseIM/issues">esl/MongooseIM/issues</a></li> <li>Share your thoughts via Twitter: <a href="https://twitter.com/MongooseIM/">twitter.com/MongooseIM</a></li> <li><a href="https://hub.docker.com/r/mongooseim/mongooseim/">Download Docker image</a> with new release</li> </ul> <p><a href="http://www2.erlang-solutions.com/emailpreference?utm_source=ESL&amp;utm_medium=Products%20Page&amp;utm_campaign=Pref%20center">Sign up to our dedicated mailing list</a> to stay up to date about MongooseIM, messaging innovations and industry news.</p> <p>Check out our <a href="http://www2.erlang-solutions.com/l/23452/2017-07-06/4t7t1v">MongooseIM product page</a> for more information on the MongooseIM platform.</p>

This is my blog post for the 5th week of the Google Summer of Code. I passed the first evaluation phase, so now the really hard work can begin.
(Also the first paycheck came in, and I bought a new Laptop. Obviously my only intention is to reduce compile times for Smack to be more productive *cough*).

The last week was mostly spent writing JUnit tests and finding bugs that way. I found that it is really hard to do unit tests for certain methods, which might be an indicator that there are too many side effects in my code and that there is room to improve. Sometimes when I need to save a state as a variable from within a method, I just go the easy way and create a new attribute for that. I should really try to improve on that front.

Also I did try to create an integration test for my jingle file transfer code. Unfortunately the sendFile method creates a new Thread and returns, so I have no real way of knowing when the file transfer is complete for now, which hinders me from creating a proper integration test. My plans are to go with a Future task to solve this issue, but I’ll have to figure out the most optimal way to bring Futures, Threads, (NIO) and the asynchronous Jingle protocol together. This will probably be the topic of the second coding phase

The implementation of the Jingles transport fallback functionality works! When a transport method (eg. SOCKS5) fails for some reason (eg. proxy servers are not reachable), then my implementation can fallback to another transport instead. In my case the session switches to an InBandBytestream transport since I have no other transports implemented yet, but theoretically the fallback method will try all available transports in the future.

I started on creating a small client/server application that will utilize NIO to handle multiple connections on a single thread as a small apprentice piece. I hope to get more familiar with NIO to start integrating non-blocking IO into the jingle filetransfer code.

In the last days I got some questions on my OMEMO module, which very nicely illustrated to me, that developing a piece of software does not only mean to write the code, but also maintain it and support the people that end up using it. My focus lays on my GSoC project though, so I mostly tell those people how to fix their issues on their own.

Last but not least a sort of political remark: In the end of the GSoC, students will receive a little gift from Google (typically a Tshirt). Unfortunatelly not all successful students will receive one due to some countries being “restricted”. Among those countries are Russia, Ukraine, Kazakhstan, but also Mexico. It is sad to see, that politics made by few can affect so many.

Working together, regardless of where we come from, where we live and of course who we are, that is something that the world can and should learn from free software development.

Happy Hacking.

ejabberd 17.06 includes a lot of improvements over the previous 17.04 release. To name the most important ones: new caching system, Riak support for several modules and introduction of Certificate Manager.

Certificate Manager is a feature that has been requested by many organisations, allowing administrators to manage their certificate more easily. From now, starting ejabberd with an invalid certificate will dump a clear entry in ejabberd log file, explaining what’s wrong. Upcoming ACME support will further refine these improvements we’ve worked on early this year to give our users a great experience with certificate management.

The new cache system is also a new component that allows fine tuning of ejabberd performance for either small systems or large scale servers. To use data cache for a supported module, you need to set the module option use_cache. You also have the possibility to define a maximum number of cache entries and/or maximum life time of cached data, so you keep control on your memory use. Example:

modules:
  mod_roster:
    use_cache: true
    cache_size: 10000
    cache_life_time: 3600  # 1 hour

The cleanup tasks on all ejabberd API also continue, consider checking against few methods rename.

Changes

API

• Deprecate misc:encode_base64/1 and misc:decode_base64/1
• Rename is_user_exists function to user_exists
• Allow api access on both ipv4 and 6 loopback addresses
• Fix invalid argument in get_messages_susbset

Admin

• Refactor ejabberdctl
• Improve ejabberdctl parameters parsing
• Quote all paths to allow spaces in directory names
• update-deps-releases.pl: Show operations to perform before asking to apply them
• Fix Salt import from prosody SCRAMmed password

Configuration

• Validate module options on start_module/2
• Validate new options before module reloading
• Validate second-level options
• Introduce iqdisc global option
• stream_management listen option deprecated, use mod_stream_mgmt
• Check presence of some files during option validation
• Speedup configuration options lookup
• Validate all certfiles on startup
• Only validate certfiles if public_key:short_name_hash/1 is available
• Introduce Certficate Manager

Commands

• Add clear_cache admin command
• Parse correctly presence_broadcast option in change_room_option command
• Describe command arguments and results in mod_muc_admin
• Improve export2sql explanation; remove obsolete and duplicated command
• Fix and document push_roster_all command
• Fix mod_muc_admin command set_room_affiliation
• Fix invalid {args,result}_examples in mod_muc_admin
• Fix srg_user_add/del for non-Mnesia database backends
• Make ext_mod api return rescode

Compilation

• Erlang 17.5 or higher is required
  (ejabberd_c2s needs latest maps fixes from OTP-17, see OTP-12402 and OTP-12623)
• Add --enable-system-deps configure option
• Add --enable-stun and --enable-sip configure options

Core

• Speedup Mnesia tables initialization
• Improve Mnesia tables creation and transformation
• Improve ejabberd_c2s:close()
• ejabberd_c2s: Don’t close session on stream resume
• Speedup loading of translation files
• Fix ejabberd_router:is_my_route/1
• Don’t let a receiver crash if a controller is unavailable
• Fix OTP-17.5 support
• websocket: Catch exception that may happen when sending data over websocket

Databases

• New sql_connect_timeout option
• New sql_query_timeout option
• Get rid of sql_queries.erl
• Use round-robin algorithm when selecting worker from DB pool
• Add Riak as BOSH RAM backend
• Add Riak as mod_proxy65 RAM backend
• Add Riak as mod_carboncopy RAM backend
• Add Riak as router RAM backend
• Add Riak as session manager RAM backend
• Fix cleaning of Riak route table
• Add pubsub import from prosody/metronome
• Fix username in mam export
• Fix Salt import from prosody SCRAMmed password
• In offline export to SQL, first write all DELETE, later all INSERT

Cache

• Implement cache for mod_announce
• Implement cache for mod_private
• Implement cache for mod_privacy/mod_blocking
• Implement cache for mod_last
• Implement cache for mod_vcard and mod_vcard_xupdate
• Implement cache for roster
• Add cache options to the validator
• Use cache for authentication backends
• Use new cache API in mod_shared_roster_ldap
• Use new cache API in ejabberd_oauth
• Use new cache API in mod_mam
• Use new cache API in mod_caps
• Use cache in front of Redis/SQL RAM backends

Modules

• mod_http_upload: Add support for HTTP File Upload 0.3.0
• mod_mam: Added export function
• mod_metrics: Don’t leak with UDP sockets
• mod_metrics: New options ip and port
• mod_muc: Allow a room admin to un/subscribe another JID
• mod_offline: Don’t store messages via a single process
• mod_offline: Make sure only jabber:x:event tag is present in offline event
• mod_register: New option ‘access_remove’ ACL
• mod_stream_mgmt: Preserve stanza count on timeout
• mod_vcard_ldap: Parse ldap_uids like in eldap_utils
• mod_client_state: Reset state on session resume
• mod_metrics: Fix IP address parsing
• mod_pubsub: Avoid useless calls on simples subscriptions
• mod_block_strangers: Add allow_local_users option

Elixir

• Update elixir to v1.4.4

Installer

• Upgrade OTP to 19.3

Feedback

As usual, the release is tagged in the Git source code repository on Github.

The source package and binary installers are available at ProcessOne.

If you suspect that you’ve found a bug, please search or fill a bug report on Github.

Many software developers are familiar with realtime, but we believe that realtime concepts and user experiences are becoming increasingly important for less technical individuals to understand.

At Fanout, we power realtime APIs to instantly push data to endpoints – which can range from the actual endpoints of an API (the technical term) to external businesses or end users. We use the word in this post loosely to refer to any destination for data.

We’re here to share our experience with realtime: we’ll provide a definition and current examples, peer into the future of realtime, and try and shed some light on the eternal realtime vs. real-time vs. real time semantic debate.

...

This is a quick update post to my post from yesterday. The GSoC part came a little short because of my OMEMO reflections and because I only worked on one topic the last week, so this is intended as a kind of supplement.

Today I got my SOCKS5 code working again! I can send and receive both from and to local and external proxy servers. It was previously not working because I missread the XEP. I thought that one party only connects to a proxy after the activation element was received, while in fact the connection must be established beforehand. Also on the sender side I closed my OutputStream after writing the data on it. This caused the receiver to fail opening an InputStream because the socket was closed. It took some time and Googling to find out that closing the OutputStream also closes the socket. Unfortunately I’m not having a “established” implementation which I can test my code against. If you know of an app/messenger which supports Jingle FT 5, please let me know

My next steps will be writing integration tests. At some point I lost track of tests, so I’m a little bit behind on that front. Creating those tests should greatly improve the stability of the code and make breaking changes more obvious. Also I’ll create a little NIO test application to make myself familiar with the NIO framework. Currently my focus lays on getting things to work in the first place, but in the end it would be nice to have data streams implemented using NIO. Another thing that I’ll tackle this week is getting jingles fallback mechanism in case of a failed transport to work. This should be pretty straight forward.

Then again I’ll hopefully soon spent some more time again on my plans for Jingle Encrypted Transfers. I hope to get a working prototype after the general Jingle File Transfer is finished.

Thats all for now. Happy Hacking

GSoC

Evaluation phase is there! Time went by faster than I expected. That’s a good sign, I really enjoy working on Smack (besides when my code does not work ).

I spent this week to work on the next iteration of my Jingle code. IBB once again works, but SOCKS5 still misses a tiny bit, which I struggle to find. For some reason the sending thread hangs up and blocks just before sending begins. Its probably a minor issue, which can be fixed by changing one line of code, nevertheless I’m struggeling to find the solution.

Apart from that it turned out, that a bug with smack-omemo which I was earlier (unsuccessfully) trying to solve resurrected from the land of closed bug reports. Under very special conditions pubsub results seem to only arrive in Smack exactly after the result listener timed out. This keeps smack-omemo from fetching device bundles on some servers. I originally thought this was a configuration issue with my server, but it turned out that aTalk (a Jitsi for Android fork which is working on including OMEMO support) faces the exact same issue. Looks like I’ll have to investigate this issue once more.

OMEMO – Reflections

It appears that soon the council will decide on the future of OMEMO. I really hope, that the decision will make everyone happy and that the XMPP community (and – perhaps more important – the users) will benefit from the outcome.

There are multiple options for the direction, which the development of OMEMO can take. While everybody aggrees, that something should happen, it is not clear what.

OMEMO is already rather well deployed in the wild, so it is obviously not a good idea to break compatibility with existing implementations. A few months ago, OMEMO underwent a very minor protocol change to mitigate a vulnerability and even though Conversations (birthplace to OMEMO) made a smooth transition over multiple versions, things broke and confused users. While things like this probably cannot be avoided in a protocol which is alive and changing, it is desirable to avoid breaking stuff for users. Technology is made for users. Keeping them happy should be highest priority.

However, at the moment not every user can benefit from OMEMO since the current specification is based on libsignal, which is licensed under the GPLv3. This makes the integration in premissively licensed software either expensive (buying a license from OWS), or laborious (substitute libsignal with other double ratchet library). While the first option is probably unrealistic, the second option has been further investigated during discussions on the standards mailing list.

Already the “official” specification of OMEMO is based on the Olm library instead of libsignal. Olm more or less resembles OWSs double ratchet specification, which has been published by OWS roughly half a year ago. Both Olm and libsignal got audited, while libsignal got significantly more attention both in the media and among experts. One key difference between both libraries is, that libsignal uses the Extended Triple Diffie Hellman (X3DH) key exchange using the XEdDSA signature scheme. The later allows a signature key to be derived from an encryption key, which spares one key. In order to use this functionality, a special conversion algorithm is used. While apparently this algorithm is not too hard to implement, there is no permissive implementation available in any established, trusty crypto library.

In order to work around this issue, one proposal suggests to replace X3DH completely and switch to another form of key exchange. Personally I’m not sure, whether it is desirable to change a whole (audited) protocol in order to replace a single conversion algorithm. Given the huge echo of the Signal protocol in the media, it is only a matter of time until the conversion algorithm makes its way into approved libraries and frameworks. Software follows the principle of supply and demand and as we can conclude from the mailing list discussion, there is quite a lot of demand even alone in the world of XMPP.

I guess everybody aggrees that it is inconvenient, that the “official” OMEMO XEP does not represent, what is currently implemented in the real world (siacs OMEMO). It is open to debate now, how to continue from here on. One suggestion is to document the current state of siacs OMEMO in a historical XEP and continue development of the protocol in a new one. While starting from scratch offers a lot of new possibilities and allows for a quicker development, it also most definitely implies to completely break compatibility to siacs OMEMO at users expense. XMPP is nearly 20 years old now. I do not believe that we are in a hurry .

I think siacs OMEMO should not get frozen in time in favor of OMEMO-NEXT. Users are easily confused with names, so I fear that two competing but incompatible OMEMOs are definitely not the way to go. On the other hand, a new standard with a new name that serves the exact same use case is also a bad idea. OMEMO works. Why sould users switch? Instead I’d like to see a smooth transition to a more developer friendly, permissively implementable protocol with broad currency. Already there are OTR and OpenPGP as competitors. We don’t need even more segmentation (let alone the same name for different protocols). I propose to ditch the current official OMEMO XEP in favor of the siacs XEP and from there on go with Andreas’ suggestion, which allows both libsignal as well as Olm to be used simultaneously. This allows permissive implementations of OMEMO, drives development of a permissively licensed conversion algorithm and does not keep users standing in the rain.

To conclude my reflections: Users who use OMEMO will use OMEMO in two or three years. It is up to the community to decide, whether it will be frozen “historical” siacs OMEMO, or collectively developed, smoothly transitioned and unified OMEMO.

Thank you for your time