Planet Jabber

Remko Tronçon: Trying out Git

Remko Tronçon — 2008-08-28T01:25:06+00:00

A while ago, the Psi development team switched from Darcs to Subversion for version control, because the Darcs pros (distributed, extremely simple and elegant) did not compensate for the cons any longer (slowness, non-scalability, ‘infinite’ merges, lack of community and tools, …). Our development was pretty central anyway at that time, so we decided that Subversion was good enough. However, we started to miss local commits more than we thought we would, and some of us are working on their own forks, which makes Subversion a suboptimal choice. We are therefore currently trying out Git as a replacement, which should bring us all the good stuff from Darcs, combined with the speed and portability of Subversion. Note that during the experiment, we will not be updating our Subversion branch any more (which will soon cause breakage, since Subversion automatically updates changes to the external Iris repository).

If you’re interested in following our latest developments from our experimental Git tree, simply clone our official repository through the following steps:

git clone git://git.psi-im.org/psi.git
cd psi
git submodule init
git submodule update

To get the latest changes after an initial clone, execute the following commands

git pull
git submodule update

If you’re interested in knowing more about using Git, check out Scott Chacon’s excellent Git talk (and accompanying book).

For fans of social networking, we also have a mirror of the official Psi repository on GitHub. And while we’re on the topic of web interfaces: while waiting for an official web interface to our Git repository, you can check out an experimental one here.

Paweł Wiejacha: End of the Google Summer of Code

Paweł Wiejacha — 2008-08-27T20:45:16+00:00

Google Summer of Code 2008 ended last week. I think it would be nice to write a new blog entry about my project. So here is the mandatory report.
Continue reading "End of the Google Summer of Code"

Jack Moffitt: Migrating To Ejabberd: The Gory Details

metajack — 2008-08-27T16:51:03+00:00

Yesterday I blogged about choosing an XMPP server. Once you’ve made this decision, it is time to get it set up and running great. If this is your first XMPP server, this is often just some software installation and a little bit of configuration. However, for those who already have an existing set up, or who have special data or run time requirements, this can involve a substantial amount of work. It took us about a week to move from jabberd2 to ejabberd at Chesspark, and this is how we did it. While I talk mostly about ejabberd and jabberd2, many of the issues are similar to those any deployment or redeployment would face.

What We Knew At The Start

There were three main things that we need we needed to work on in order to transition to ejabberd. These were authentication, data migration, and porting a custom server module.

Authentication. Many sites already have a user database they they want to reuse for the XMPP service. This almost never looks like the bare bones schema that the XMPP servers are configured to use by default. The first task then becomes to configure the XMPP server to authenticate from this custom user data.

Jabberd2 is a lot more configurable for database authentication than ejabberd. For starters, it allows you to define the SQL query used to authenticate users right in the configuration file. With ejabberd the choices are to write your own authentication module, modify the database authentication module to provide your own queries, or use external authentication via a script.

External scripts are pretty easy in ejabberd, but we opted for modifying the queries used for authentication in the ejabberd source. For anyone trying this themselves, the queries are all in src/odbc/odbc_queries.erl.

The Chesspark service is open to the entire federated XMPP network, and we store member information for everyone in the same place. This means that we must store the bare JID as opposed to just the node portion. Some users are local to our server, like jack@chesspark.com, and some are not, such as metajack@gmail.com. Only the local users will be used for XMPP authentication, but almost every XMPP server is written to expect usernames without domains, and ejabberd is no exception.

Here is what ejabberd’s default authentication query looks like:

get_password(LServer, Username) ->
    ejabberd_odbc:sql_query(
        LServer,
        ["select password from user "
         "where username='", Username, "';"]).

We changed it to:

get_password(LServer, Username) ->
    ejabberd_odbc:sql_query(
        LServer,
        ["select password from member "
         "where lower(username)='", Username, "@", LServer, "';"]).

This wasn’t very difficult at all. We don’t need in-band registration, so we didn’t change all of the user creation and deletion queries, but those would be very similar to this one.

Data Migration. Each XMPP server has its own format for storing data, and if you are switching from one server to another, you will probably want to take your data with you. At Chesspark, we already had 400,000 users worth of rosters, private XML storage, and privacy lists, and we didn’t want our users to lose this information. Our task was to figure out how to migrate all the jabberd2 data to a form ejabberd could use.

Most XMPP server schemas are quite simple and quite similar. We achieved 95% of the data migration by using SQL’s ALTER TABLE command to add, remove, and rename columns. The only trouble we ran into was figuring out which columns in jabberd2 mapped to which columns in ejabberd and dealing with virtual hosts.

Figuring out the mapping was pretty straightforward. We made some users in a test database under the default ejabberd configuration and did some common operations. Then we inspected the database tables and determined what the columns were for. For example, we saved some private XML as a user, then looked at how ejabberd’s private_storage table stored and formatted the resulting data. In almost every case, it was extremely simple to migrate the data. The notable exception is privacy lists, which were stored in two tables in ejabberd instead of just one like jabberd2. This was easily fixed up with a small Python script.

Dealing with virtual hosts was a bit harder. We run multiple servers at Chesspark to handle our branded chess sites like WuChess.com. We wanted to move from multiple jabberd2 servers to a single ejabberd installation with virtual hosting. Unfortunately, since ejabberd stores user names without domains, this enforces unique user names across all virtual hosts, a requirement we weren’t able to meet with our existing user data. We decided to patch ejabberd to use the domain along with the user name to make this easier.

These changes were very similar to the changes for authentication above and we modified nearly every query in odbc_queries.erl. Unfortunately, not every function takes the parameter LServer which is the stringprepped domain, and this made the patch more difficult. We had to add the LServer variable where it was missing, and then modify the ejabberd internals to pass this along in places where it didn’t already do so. This was pretty easy, except for one or two places where ejabberd passed data in a giant list and we had to fix specific entries in the lists.

Unfortunately, not all the SQL queries are in odbc_queries.erl as the code would have you believe. We also had to patch the queries in other modules of the code where they had not yet been moved to odbc_queries.erl. Hopefully the ejabberd team will finish moving these to one place in the future.

After all this patching, it required a few days of testing and tuning to make sure everything was running smoothly. Supporting virtual hosts by adding the domain to usernames is definitely something I feel should be in upstream ejabberd, and I will send my patch upstream to see if they will add this.

The Custom Module. Chesspark is built from a set of components that implement and understand chess rules, player matching rules, and various other things. These components must receive presence notifications to know when users are on and offline. It can happen that a presence notification is slightly delayed and stanzas are sent to an offline client. These don’t cause any harm, but they do take up disk space. With a lot of users in the database, this disk space can start to be substantial, so we wrote a custom server module in jabberd2 to ignore traffic to offline storage if it came from certain components.

Porting this module turned out to be very simple. It took me a few hours to write and test the module. It would have taken a bit longer had this been my first ejabberd module instead of my second. Most of the work involved figuring out various bits of the gen_mod API in ejabberd. For those of you interested in writing or understand ejabberd modules, I highly recommend Anders Conbere’s blog where he has provided several tutorials.

Putting Ejabberd Into Production

After all this work and testing, it was time to put ejabberd into production. We gave notice to the users, took the site offline, and began the data migration process. This process took longer than our trial runs for some reason, totaling a little bit over an hour.

Once the site came back up, we immediately noticed some problems that we had not caught in our testing. First, database traffic was extremely high and load was about 4 times higher than we expected. Next, we discovered that one of our client applications had an infinite loop bug triggered by receiving an echo of your own presence stanzas. Finally, memory use on the ejabberd server was extreme, and over the next day or so it ran out of memory and crashed 3 times.

Solving The Problems

The database problems were the easiest to diagnose thanks to our database profiling tools. It was easy to spot a query that was missing an index, and this immediately caused load to drop. Query traffic was still much to high, and we found two causes for this.

The first cause was our buggy client application which was in an infinite loop sending presence. This caused a presence storm on the server, causing thousands of useless packets to be sent both to the server and to the user’s contacts. This affected database queries because of the second problem.

The other problem is that mod_privacy_odbc in ejabberd is just terrible. It does two database queries, one for rosters and one for roster groups, for every single packet that is delivered to a local user. It does this for the silly purpose of letting stanzas through from your roster contacts that would otherwise be blocked by your privacy settings. Why this exists, I have no idea, but coupled with the presence storms, it caused a massive amount of database traffic.

A combination of solutions was used to solve these problems. First we pushed out an upgraded client that did not have the presence storm bug. Next, we disabled mod_privacy_odbc temporarily to ease the database traffic. After these were implemented, the server was in good shape, but our users who had contacts still running buggy clients were not.

To solve the problem of people who refused to upgrade, we wrote a new ejabberd module called mod_sunshine, which stopped the presence storms. mod_sunshine keeps track of duplicate presence sent from users and terminates their session if they send more than X duplicate presence stanzas in Y seconds. Configured mod_sunshine to kill clients that sent more than 10 stanzas in 60 seconds removed nearly all of the presence storm traffic, and we dealt with the confused users who could not connect with apologetic emails and some upgrade hand holding.

After some new indexes, a new client, and mod_sunshine, we were able to stabilize the database usage and turn back on mod_privacy_odbc.

The last problem was memory use, which was much too high. We are not sure yet of the true reason for ejabberd’s hungry memory needs, but we were able to get past the immediate problem of server crashes by upgrading from 2GB to 8GB of RAM on our XMPP server. Also, moving from the latest stable release, ejabberd 2.0.1, to the trunk of ejabberd development removed a lot of the memory use. Memory use is still quite high, about 1.4GB for ~250 simultaneous users, and we’ll continue to investigate. There does seem to be one reported issue that TLS connections take up substantially more memory than non-TLS connections. Thankfully, unlike jabberd2, we have not detected any major memory leaks, although we are still keeping a close eye on this.

The End Result - Happy Users

After a week of patching and preparation and a few frantic days of chasing down glitches, ejabberd is running quite well. Our users are extolling the lower latency of the new server, which was our ultimate aim with the upgrade. It seems we have achieved some success for now, but as always, we continue to measure and analyze performance and dream of new improvements we can make.

If you have a story about an XMPP server upgrade gone right or wrong, anecdotes about your favorite or least favorite XMPP server, or just comments or question, please share them with us below.

Jack Moffitt: Strophe Update: Public Repo, Authentication Testing, and More

metajack — 2008-08-26T21:21:12+00:00

It’s taking a bit longer than planned to whip Strophe into releasable shape, but work is progressing well. Since the last update, I have made a new and better Trac instance for the project, made a public SVN repository for the code, added SASL ANONYMOUS support to both libstrophe and Strophe.js, fixed DIGEST-MD5 authentication with ejabberd, did compatibility testing for both libraries, and wrote up a basic Strophe.js example. Still to come are TLS fixes for libstrophe, porting the libstrophe documentation to Natural Docs, working around a BOSH bug with Tigase, and writing up some examples.

On a related note, Matthew Wild is working on an XMPP library for Lua built on top of libstrophe.

Also, remember to vote for my XMPP talk and panel at SXSW 2009.

UPDATED 8/26: It turns out Tigase and Openfire are smarter than the other XMPP servers with DIGEST-MD5 authentication, and that it wasn’t a bug at all. My apologies (kudos) to the Tigase and Openfire teams. The issue is now fixed in trunk.

Jack Moffitt: Choosing An XMPP Server

metajack — 2008-08-26T15:49:01+00:00

Choosing an XMPP server is a big decision. Should you go with the popular one or the one written in the most popular language? Perhaps you don’t plan to become a systems administrator and you need one which is easy to set up and maintain. Unfortunately for people making this important choice, there is not much guidance published beyond features comparisons. What follows is an account of our decision making process on XMPP servers - how we came to pick jabberd2 originally, and how we switched to ejabberd.

A Brief History Of XMPP Servers

XMPP began as Jabber and had only one server, jabberd. As popularity of the protocol grew, more servers appeared, and now there are half a dozen major contenders, both commercial and open source.

The main players these days seem to be:

jabberd - The original server. It started in C, but now appears to be a mix of C and C++ code. The main users for years was jabber.org itself. The code may have changed substantially in the last couple of years, but I remember it being rather crufty. Matthias Wimmer has been maintainer at least since 2004, and he continues to maintain it today, although it does go through some long periods of inactivity.
jabberd 2.x - A rewrite of jabberd, originally by Rob Norris. The code is pure C, modular, and fairly easy to understand. Chesspark picked this to start with in 2006. Another notable user of jabberd 2.x is Meebo. Unfortunately it was abandoned by its original maintainers some time ago. Tomas Sterna stepped up to the plate and took over the project, and it is now actively maintained again.
ejabberd - An XMPP server written in Erlang which claims to be quite scalable. Erlang is the language created decades ago by Ericcson to power their telephone switches. It has many features that make it well suited for XMPP servers. ejabberd has been around and active since early 2005, and is supported officially by Process One. It also has a growing developer community. Jabber.org switched from jabberd to ejabberd some time ago, and continues to run ejabberd today.
Openfire - A Java server written by JIVE Software. This was formerly a commercial product called Wildfire. I have no personal experience with it, but it appears to have an active community.
Tigase -Another Java server that started in late 2004. It is actively maintained. I have no personal experience with Tigase other than meeting Artur at the recent XMPP Summit, but the Seesmic folks speak very highly of the project.

This list is incomplete. Notable omissions include Google Talk (not publically available), Jabber XCP (the commercial offering from Jabber.com), and djabberd (Danga’s jabber server written in Perl).

Chesspark’s Initial Decision

Chesspark chose jabberd2 as its XMPP server about 3 years ago. I recall being impressed with the clean and modular code base as well as its ability to change the SQL queries right in the configuration file. It also supported PostgreSQL which was the RDBMS we preferred. I don’t think that Tigase or ejabberd were considered; they were likely too young at the time. Jabberd was the only other real choice, but we were not impressed with the code.

One major factor which influenced our decision was code readability and maintainability. We wanted an XMPP server that we could patch ourselves if needed, and we didn’t want to be stuck in case the project was abandoned down the road. This turned out to be a wise decision - the jabberd 2.x project was unmaintained for a long period while we used it. Over the last few years we’ve made patches and assisted others with patches as best we can.

Jabberd 2.x Disappointment

Over time, Chesspark’s user base got larger as the site became more popular, and jabberd 2.x’s warts began to show. Here are the main ones in the approximate order we discovered them.

Database transactions abused. Jabberd 2.x does no queries out of the box that require the use of database transactions. By default it is configured to do every query in a transaction, even if it is a simple SELECT. This is a common problem with many libraries we’ve used at Chesspark. Thankfully, jabberd 2.x can be configured to turn this off. Normally this would not affect anything, but the small amount of overhead caused can add up fast when jabberd 2.x does lots of queries, which it certainly does.
Memory leaks. There are several memory leaks that persist to this day. Even with a small userbase like Chesspark has, this forces us to restart the server about once a week. As memory usage climbs, the server latency gets higher. Our attempts to find this leak have been unsuccessful to date.
Non-blocking design inconsistent. The server uses a non-blocking design common to scalable daemons; this is great. Unfortunately, all database calls use the blocking database API and a single database connection. This means that even with light load, packet latencies can be quite high if the database isn’t ridiculously fast.

For us, the show stopper was latency. Games depend on near real-time performance, and latency destroys the user experience. We generated test load which logged in a few hundred users, did roster operations, logged out, and repeated. We then measured latency of chess moves on the same server. We were shocked when we saw the numbers - over 3 seconds of lag between an IQ query and its response.

All of these things could theoretically be fixed, and I hope that they are fixed eventually. Diversity in server choices is a feature of XMPP - the more the merrier.

Finding A New Server

Once we decided to abandon jabberd2, we needed to find a new server.

Feature wise, all the current servers support the stack we need - authentication with TLS and SASL, ability to use PostgreSQL as a backend, private XML storage, external components, and privacy lists. Years ago, some of these features were hard to come by, but today they are common. ejabberd, Openfire, and Tigase have pubsub and BOSH support as well, neither of which was available in any open source server when we started.

We knew right off the bat that we didn’t want to be writing C. While we have a lot of C experience, we like to reserve C for the few times it is actually needed and spend the rest of our time in more productive and higher level languages. This removed jabberd from the list.

From here the language choice is Erlang or Java. Erlang is a dynamically typed, functional language - quite a radical departure from the norm for people most C and Java hackers. We work a lot in Python, so Erlang was the closer fit. Many people make the decision to work in Java, and from there they will need to pick between Tigase and Openfire.

One thing to note is that some people seem scared away by the Erlang language. Don’t be one of these people. Erlang is well documented and pretty easy to learn. We knew nothing about Erlang a few months ago. That did not slow us down too much when we needed to write ejabberd modules or make changes in ejabberd internals. Even without knowing Erlang, we were able to write extensions to ejabberd much faster than for jabberd2.

The last part of our decision was to test server latency with ejabberd. We ran the same test that we ran on jabberd2, and ejabberd didn’t flinch. The measured latency at idle was twice as fast in some cases with ejabberd, and there was very little change even as we pounded the database to levels that would have made jabberd2 cry.

Life With Ejabberd

ejabberd is not perfect; no server is. Here’s a list of our current gripes:

Memory hog - Erlang uses a lot of memory for basic string handling since a string is represented as a list of integers. There also seems to be a bug in TLS that causes it to use quite a bit more memory than non-TLS connections. These add up to quite a bit of memory usage. For Chesspark, we use over a gig of RAM for a few hundred connections. Jabber.org uses about 2.7GB of RAM for its 10k+ connections. I’m not sure what the discrepancy is between these numbers; we are still looking. I expect the TLS memory issue to be solved soon, and the Process One folks told me that they were going to switch the string handling to use Erlang binaries which are more memory efficient.
Lots of database queries - As with jabberd2, ejabberd does an enormous amount of database queries. With mod_privacy enabled, two roster related queries are done for every packet sent. ejabberd also uses the database inappropriately, with idioms like SELECT, DELETE, INSERT which can lead to race conditions. Thankfully, this does not seem to be a big problem with a correctly tuned database, as ejabberd doesn’t block on the queries.
Lack of comments in the code - The code is often quite clear, but comments would be helpful. There is some basic, but very helpful, developer documentation, but the code contains virtually no comments. Luckily, this is not as bad as it seems,because many of the idioms in the server are really Erlang and OTP idioms, so reading up a bit on Erlang and OTP answers a lot of questions. I’m also not sure other servers are better. Jabberd2 probably had more comments, but it also had less documentation in general; I’ve found working with ejabberd to be easier.

There are many excellent things about ejabberd that make up for these and other shortcomings, and have made us very happy we made the switch.

Hot code loading - After we write a new ejabberd module, we can deploy it in production without pausing or restarting the server. We can also redeploy it later if we find a bug. We have even redeployed core server pieces this way with success. This can even be done to some degree right in the Web interface.
Live console - It is possible to open an Erlang shell inside the running ejabberd node. This makes it really easy to poke around to see which processes are running, how much memory they are using, and which internal database tables are getting full. Java, C, Python, and Ruby have nothing quite like this, although Twisted Python’s manhole is similar.
Very low CPU usage - You’d think a C based XMPP server would be a winner in this area, but that is not always true. ejabberd uses very little CPU usage, except when things go wrong. Chesspark’s XMPP server is sitting around a load of 0.1 to 0.2; Jabber.org sits at 0.1. This is exactly what you want. An XMPP server should be I/O bound, not CPU bound.

So far we’re pretty happy. How did you pick your XMPP server?

Process One: Instant Messaging as a social research tool: Study revives six degrees theory

Mickaël Rémond — 2008-08-25T07:49:26+00:00

According to a recent study by Microsoft into instant messaging habits it takes only six steps to link everyone together.

The research carried out by Microsoft researchers Eric Horvitz and Jure Leskovec studied 30 billion instant messages sent using Microsoft Messenger during June 2006 and found that any two people were linked by seven or fewer acquaintances.

As reported by BBC News, Horvitz says "What we are seeing suggests that there may be a social connectivity constant for humanity."

This research is further evidence of the social and collaborative benefits instant messaging can offer both in and outside of the enterprise through linking more and more people together.

You can get more details on the article from BBC News: Study revives six degrees theory.

Kevin Smith: XMPP: The almost-definitive-guide to-be

Kev — 2008-08-21T18:27:26+00:00

You’ll already have read, or be about to read, similar blog posts from Remko and Peter, I imagine, here’s my take:

Remko Tronçon, Peter Saint-Andre and I are writing a book on XMPP for O’Reilly, with a working title of “XMPP: The Definitive Guide”, expected to hit the shelves (and hopefully fly off them soon after) early 2009.

If you’re wondering how things like this come about, the story goes something like this:

Every so often, someone comes into the Jabber Development room, and asks if there’s any getting started documentation, or they ask if there’s a decent book, or they ask if there’s a guide to what Jabber/XMPP can do for them as a developer. For some reason, it seems to usually be me that ends up letting them know that there isn’t really, unless you want to go and read the RFCs, or the XEPs. I got the daft idea that writing such a book would be helpful, so I poked Remko and asked if he was interested in co-authoring something if O’Reilly would publish it - he was, so off we went and did very little for quite a while. As it happens, O’Reilly were interested in publishing a book about XMPP; realising an XMPP book wasn’t an XMPP book without Peter, we set about persuading him that joining us in the venture was a Smart Thing™, and things started moving.

We met up for a pizza at FOSDEM in February, hashed out where we wanted to go with the book, wrote out the overview, sent it off to O’Reilly, who didn’t hate it, and started writing.

Since then, we’ve made some decent progress on it, and also found out that writing books is hard work (even given that each of us has written at least either a doctoral thesis or RFCs). Still, we’re chugging along and things should be really taking shape in the next month or two
The rest will, someday, be history — we’ll post an update later when we’re closer to the inevitable fame and fortune.

Remko Tronçon: We’re writing an XMPP book

Remko Tronçon — 2008-08-21T18:23:47+00:00

I’m excited to announce that Peter, Kevin, and I recently got the green light from O’Reilly to start writing a book about Jabber/XMPP. The book will be targeted at a diverse public: on one hand, people who want to get acquainted with XMPP and will get an introduction and a general overview of XMPP, its workings, and its possibilities. On the other hand, software engineers who want to integrate XMPP into their products will get a guide to implementing different use cases of XMPP through a series of different developer stories. The book is expected to be available in 2009, so start making some room on your bookshelf!

Peter Saint-Andre: Don’t Wait For The Movie!

stpeter — 2008-08-21T18:21:30+00:00

In case you hadn’t noticed, I like to write. In fact, it’s pretty much all I do, whether professionally or recreationally. So when Kevin Smith and Remko Tronçon said they were interested in writing a book about XMPP, I was intrigued. And when the folks at O’Reilly said they were interested too, I was doubly intrigued. So Kevin, Remko, and I have committed to writing XMPP: The Definitive Guide for O’Reilly, and we’re now hard at work. Expect to see it published sometime in 2009. And remember: don’t wait for the movie!

Safa Sofuoğlu: GSoC Project Successfully Completed

Safa Sofuoğlu — 2008-08-20T23:22:24+00:00

I am proud to announce that I have successfully completed my Google Summer of Code Project. As we hit the official pencils down date, I thought it might be good to publish results and final toughts.

I started the project in time and completed it 3 working days later than planned, though it could require more effort if we didn't change our goals. I cooperated with Tomas and Tobias to fix the flaws I couldn't notice during development.

It was a wonderful experience to work on Openfire and SparkWeb, especially with my mentor Gaston. Even if my GSoC project is complete, I feel there'll always be something to do for me with Jabber. I am having fun with Jabber, and planning to continue working on Jabber development as a community contributor.

I would like to thank Google for giving me such a great opportunity. I also thank David Smith and Peter Saint-Andre for their excellent support.

See you around!

Tobias Markmann: Writing a HTTP 1.1 Stack And Implementing BOSH

tm — 2008-08-19T16:39:16+00:00

Hi,

here a short and long awaited update on my Google Summer of Code work. I've been finishing the update of libpurple's Entity Capabilities implementation to the latest version and added a bit functionality so that 3rd-party plugins for Pidgin can query for some contact's capabilities or adding capabilities to Pidgin's feature list which they implement. It's been tested and seems to work. Thanks to Arne König for testing all this and giving useful feedback.

During the last weeks I've added TXT record lookup to libpurple's asynchronous DNS lookup system and make libpurple automatically query for Alternate Connection Methods for XMPP. The end user won't notice much of the underlying mechanics but if she/he is behind some firewall with HTTP proxy and the XMPP service administrator has setup the right records and services libpurple will automatically use BOSH to login and everything will be nice.

Onto new business. I've started implementing a HTTP 1.1 stack which is nearly finished. I've already started implementing BOSH which is based on this new HTTP stack. The great thing about it is that I use Safa Sofuoğlu's Google Summer of Code project, which is about improving OpenFire's BOSH support, for testing. So if I implement according the spec and something doesn't work I can directly report the errors, discuss them and get them fixed quite fast. The best of all will be a XMPP server supporting latest version, version 1.6, of BOSH in form of a connection manager part in OpenFire and a client supporting the latest BOSH.

The next week I'll try to get BOSH at least working and do some debugging and cleaning here and there. If I don't get it finished till GSoC deadline I'll finish it later on.

Cheers and happy coding,
Tobias

Tobias Markmann: What Were Our XEPs Doing The Last Years?

tm — 2008-08-19T16:33:17+00:00

Here I present some charts on XEPs, XMPP Enhancement Proposals, and their development. You can see, most XEPs are either in Experimental or Draft status. The one and only final XEPs are currently:

XEP-0077: In-Band Registration
XEP-0030: Service Discovery
XEP-0009: Jabber-RPC
XEP-0004: Data Forms

There have been two XEP source files unprocessable in XMPP's repository because of invalid XML and since the reposity only exists since end 2006 no longer history was accessable.

These charts have been generated by OpenOffice.org and some custom python script which worked on XSF's svn repository.

Adam Michał Strzelecki: jabberd2 2.2SVN for Windows is out

Adam — 2008-08-18T15:27:45+00:00

jabberd2 win32 project is finally updated and synced with changes made in the generic Unix code. New changes should bring improved compatibility & stability and new 2.2 features for Windows users. Binary installer and optional build instructions available as usual at jabberd-win32 page. Click more for changes.

Changes:

FIX: Setup files were not anymore compatible with latest WiX 3 beta
FIX: Router failing opening users and filters configuration in Windows
FIX: Mysql missing crypt is replaced with DES_crypt from OpenSSL in Windows
NEW: Compressed stream support via zlib
NEW: Keep server.pem on upgrades
NEW: Using SubWCRev from TortoiseSVN for version files generation (instead of xsltproc)
NEW: All configuration files are now generated on build by Perl from generic distribution ones
NEW: Using udns library for Windows builds too (resolver component was removed)
UPDATE: Synced config.h with the one from generic builds

All changes are visible as SVN revisions 644-688 in the repository.

Please note new build prerequisites such as udns, zlib, Perl and TortoiseSVN for manual builds.

Tobias Markmann: Inter-Project Collaboration during Google Summer of Code™ 2008

tm — 2008-08-16T19:02:38+00:00

Since XMPP is (becomming) the biggest player from all the instant messaging protocols out there, there are a lot Google Summer of Code™ projects in the XMPP field this year. BOSH, the highly discussed dream team for connecting to XMPP from mobile or other limited network environments, is covered by a lot projects this year.
My GSoC project is, not only, about adding BOSH support to libpurple, the C instant messaging library which powers desktop clients like Pidgin and Adium and web clients like Meebo. libpurple doesn't only cover nearly any proprietary instant messaging protocol but also some open protocols like IRC, SILC and of course XMPP. For XMPP, as the (future) major instant messaging protocol, it's most important that XEPs get implemented, coded and used in real life. There is a huge number of XEPs which aren't implemented and may never be, who knows.
I will implement BOSH from the beginning since there is no codebase in libpurple in the BOSH field and just contacted Safa Sofuoğlu, a GSoC student for the XSF mentoring organization, who updates Openfire's BOSH implementation. We plan to test our implementations of the two XEPs, XEP-0124 and XEP-0206, against each other since he'll write a server side implementation and I'm writing a client side implementation.

Our aim is to have not just working and good performing implementation but moreover implementations according to the two XMPP Enhancement Proposals. I'm looking forward to the inter-project collaboration.

Cheers and good luck to all Google Summer of Code™ students,

Tobias

Google Talkabout: New transliteration bots make it easy to chat in Indian languages

Brian Hutchins (noreply@blogger.com) — 2008-08-12T17:38:47+00:00

Have you ever wished that you could chat with your family and friends in your native language? Sometimes there's just no substitute for expressing a thought in your own language. Google Talk now has transliteration bots that will convert text from English to Hindi, Kannada, Malayalam, Tamil or Telugu. Think of a bot as an invited guest to your chat session that will transliterate what you type in English to the right local script. For those who are not familiar with transliteration, it is a service provided by Google India that allows you to type in Indian languages using phonetically equivalent English script (it is also available on our labs page, orkut scraps and blogger). If you're chatting in Hindi, when you type 'haal kaisa hai janab ka?' the en2hi.translit@bot.talk.google.com bot will reply in Hindi as 'हाल कैसा है जनाब का?'

There are currently 5 transliteration bots - Hindi (en2hi.translit), Kannada (en2kn.translit), Malayalam (en2ml.translit), Tamil (en2ta.translit) and Telugu (en2te.translit), and remember that their names end with "@bot.talk.google.com". To use one of these bots follow these three steps:

1) First add the bot that you want to your friend's list. (For example, add en2hi.translit@bot.talk.google.com for Hindi). You just need to do this once.
2) Start a chat session with your friend
3) Convert the chat session to a group chat and invite the bot to it.

Read this to know more about the bots, and let us know what you think.

तो शुरू हो जाइए...

Kuntal Loya
Software Engineer

Artur Hefczyc: Tigase LiveCD 4.0.0

kobit — 2008-08-12T09:06:52+00:00

A few days ago I have silently uploaded tigase-livecd-4.0.0.iso file into our download section. Even there was no announcement the download counter shows now 145. I wonder if there was such a big number if people knew what they are downloading....

Anyway. This ISO file is the first release of the Tigase LiveCD version. It is a complete environment runnable from the CD with pre-configured Tigase server, Drupal CMS (Blog and Forums) and Dovecot - IMAP4 server integrated together. The Drupal CMS runs with the Minichat on the example website. The live CD is based on the Gentoo Linux.

In this particular case integration means all services (Tigase, Drupal, Email) use user accounts from a single database. The user account management is done via Drupal website.

You can also post short news on the website directly from your XMPP client, you will also receive notifications about comments and posts on the website to your XMPP client.

The live CD contains also 2 nice XMPP clients: Psi and Coccinella in the most recent versions. Hm, Psi has just increased version number so it is not the most recent.

Everything installed and pre-configured, starting up automatically when the system boots from the CD.

Christopher Zorn: WuChess Video Contest

2008-08-11T15:36:15+00:00

If you have not seen this yet, you really need to check it out. Or even give it a try. ;)

http://www.youtube.com/video_response_view_all?v=3YAIVQ_oh88

Peter Saint-Andre: Where Is All The Jabber Spam?

stpeter — 2008-08-11T13:56:20+00:00

Over on the JUser discussion list, someone asked why there’s less spam on the Jabber network than there is on the email network. I answered as follows:

I’ll have to write a paper about this sometime, but here are some points
to consider:

In XMPP, the sender’s address is not asserted by the sender’s client but instead is stamped by the sender’s server. So a client can’t fake the “from” address. (Naturally if you run the server you could fake addresses at your domain, so as the admin of jabber.org I could send messages from any address at jabber.org — but I can’t fake messages from other domains, see #2.)

In XMPP, servers check each other’s identities, either through a DNS-based “dialback” protocol (RFC 3920 / XEP-0220) or real server certificates. So if I run a server at jabber.org I can’t send messages putatively “from” microsoft.com or whitehouse.gov or whatever. (Also we don’t have multi-hop routing, so modifications to the addresses can’t happen between the sending server and receiving server.)

So far, server dialback has been sufficient to prevent most address spoofing on the network, but we have a certificate authority in place (visit https://www.xmpp.net/ for details) and we could fairly easily upgrade the network to certificate-based authentication between servers if needed.

XMPP is pure XML, and attackers can’t easily attach malware like scripts and viruses to Jabber messages. This helps us avoid the unholy alliance between virus writers and spammers that has occurred on the email network.

A great deal of email spam (or spam+malware) is directed against a single platform: Outlook running on Windows. In the XMPP world we have a much more diverse software ecosystem.

In IM systems, people are accustomed to sharing presence / adding someone to their buddy list. There’s less of a culture of “I must be able to accept messages from anyone in the world” as in email. You can say this is good or bad, but that’s how it is — so if someone bothers you, you can delete them from your friend list or block them at the server side (see RFC 3921 / XEP-0016) or the client side.

All XMPP server codebases have rate limiting in place to prevent a single client from sending a large number of messages (especially a large number of large messages) in a short period of time.

Although we have not seen very much one-to-one spam on the Jabber network (our biggest problem so far is abusive behavior in groupchat rooms), we are actively planning for the arrival of spam and have designed some spam-fighting measures such as challenge-response (CAPTCHA) forms to join groupchat rooms or add someone to your contact list — see XEP-0158.

IM systems have traditionally been quite fragmented (and in many ways still are — as witness ICQ, AIM, MSN, Yahoo!, Skype, etc.) so there isn’t the expectation that you’ll necessarily be able to send a message to any random person on the Internet. This probably makes IM less appealing to spammers than email is. (Remember, spam is a matter of economics, and there may simply not be enough money to be made via IM.)

XMPP is not perfect. Spam is possible on our network, but it’s not very easy. By design, spam is harder on the XMPP network than it is on the SMTP network, and if spam does start to occur more widely we will design and deploy even better spam-fighting tools (or, for instance, tighten up or turn off in-band registration, which is user-friendly but also makes it possible to create lots of accounts at multiple servers).

However, XMPP does not need to be perfect. You don’t need to be the fastest antelope in the herd to avoid being eaten by the lion, you just need to be faster than the slow antelope who get caught.

Peter

UPDATE: A lively discussion has ensued on the list — you can participate even without joining the list by visiting JabberForum.org. I’ve also received a number of private messages about the topic, so I may post about it again soon. :)

Peter Saint-Andre: What About BoB?

stpeter — 2008-08-09T16:23:09+00:00

Jabber technologies have always been optimized for sending many small pieces of structured data, which we call XML stanzas. As a result we’ve struggled with ways to send binary data — even small bits of binary. Out of that need emerged XEP-0231, which I recently renamed “bits of binary” (a.k.a. “BoB”) because I like catchy spec titles (hey, it’s one of the things that keep me going). Folks like Ralph Meijer have had vague worries about the approach we’d taken in that spec, but we needed it for anti-spam CAPTCHA forms so we kind of brushed those aside because we need to fight spam, right?

Then over the last few weeks Pavel Šimerda raised some further concerns and, more productively, suggested solutions. So Pavel and I have been collaborating on revisions in a longish email thread, in a groupchat the other day, and in follow-up email thread. Pavel has been a pleasure to work with and it was fun to make such significant progress in such a short period of time.

Best of all, I think “BoB” is one of those building block technologies that we’ll be able to use for features like emoticons, thumbnails for file transfer, in-line images in messages and whiteboarding sessions, and yes those CAPTCHA forms for joining chatrooms or initiating presence subscriptions.

So here’s to BoB! :-)

Peter Saint-Andre: Got Examples?

stpeter — 2008-08-09T04:35:20+00:00

I try to put a lot of examples in the protocol specs I write (as I like to say, “we put the example in example.com”). So when Jeff Williams of Gadgetworks recently noted on the jingle@xmpp.org discussion list that a number of the examples in the Jingle specs are not valid in accordance with the XML schemas (or even well-formed), I was chagrined but also intrigued. Off-list, Jeff and I have worked a bit on the Jingle examples and schemas, but he’s found it painful to manually extract all the examples from the specs for testing purposes. So late this afternoon I wrote a small XSLT to automatically extract all the examples from a XEP (wrapped by a <stream></stream> element) and write them to a file. I’ve run the transformation on all the XMPP extensions we’ve published so far, and the results are available at <http://www.xmpp.org/extensions/examples/>. I’m sure there are many errors to be found (e.g., I know some examples don’t parse because they include things like “[...]” in place of elided markup), but we’ll work to clean those up over time. Feel free to report any errors to me directly, or on the appropriate discussion list. Have fun!

Peter Saint-Andre: OAuth over XMPP

stpeter — 2008-08-09T02:06:34+00:00

The results of the XMPP Summit a few weeks ago continue to leak out. I’ve been so busy with action items that I haven’t had time to blog about it all, but I did want to note that over the last few days we’ve worked out the remaining details about how to use OAuth to access protected resources over XMPP. Read all about it in the latest version of XEP-0235. Special thanks to Ralph Meijer and Blaine Cook for setting me straight about a number of issues on the social@xmpp.org discussion list.

Jack Moffitt: Learn About XMPP At SXSW 2009

metajack — 2008-08-08T16:28:18+00:00

I submitted two XMPP themed proposals for SXSW 2009. The interactive panel picker is now open, so please go vote for my entries if you have any interest in XMPP at SXSW. As far as I can tell, I’m the only one speaking on the topic.

The first submission is a panel called “The XMPP Powered Web“.

XMPP is being used to build the next generation of Web applications. The open messaging standard has crossed from the desktop to the Web and is powering applications for microblogging, gaming, social networks, communication, and data portability. Learn how others are using XMPP and how you can use it, too.

I am planning to invite some well known developers using XMPP for interesting Web projects to join me. Google, Flickr, Seesmic, Chesspark, and others are already building on XMPP technology into the Web, and it is time for the rest of the world to get into the game as well. Vote here.

The second proposal is a solo talk about XMPP basics called “X Is For XMPP: An Open Messaging Primer“.

XMPP is an open, extensible messaging standard designed for instant messaging and presence. The protocol is based on XML and is simple and easy to understand. XMPP can be used to power next generation Web apps. Learn how XMPP works and how you can use it in your own applications.

I’ll cover the basics of the protocol and dive into various ways XMPP can be used. I’ll cover the big XEPs like Pubsub, BOSH, and MUC, as well as topics like using components and writing bots. Vote here.

If any of you are planning to be at SXSW 2009, let me know. I’d love to meet more people building cool things with XMPP.

Coccinella: Tcl/Tk - Recent Advances

matben — 2008-08-08T12:04:25+00:00

Credit: freebird4, License: by-nc

There has been a lot of news in the Tcl/Tk community the last month or so, and I thought it's time to compile them here since Coccinella is written in Tcl. Tcl's windowing toolkit, Tk, has been "known" to be ugly and outdated. With the 8.5 release last December the tile package, now named ttk (Themed Tk), is included in the core which brings true native widgets on Windows (yes, Vista too) and Mac. Since Mac is my native platform, I can tell you that Tk beats both Qt and gtk, but perhaps I haven't seen the latest of them.

The question is then, where does this put Linux/Unix? There are a number of themes included in ttk which looks OK on these platforms, but there is nothing exactly like Qt or gtk. Instead, there have been a tileqt package written by Georgios Petasis which has delivered true Qt widgets. With ttk in the core, there have been some initial problems getting this to work due to missing stub tables etc., but it should work with the latest Tcl/Tk 8.5.3 release.

Just a few days ago the very same person announced tilegtk which does the same thing for the gtk toolkit. This is very, very, good news. Imagine that you can switch theme, and toolkits, on the fly without any program restart. This already works with the other themes and will likely work with tilegtk as well. My question to you is: which other toolkit has this flexibility?

I have already mentioned my recent advances with tkpath in this blog thread, and I will mention it again. This is the super canvas which is lacking in Tk, and which will bring not only tight SVG support, but also make Tk come on par with canvas like widgets from other toolkits. Still more to do.

Recently my attention turned to WebKit which has its history of its own. Dual licensed as LGPL/BSD it fits the Tcl community well. I have looked at some of the sources, and judging from the wxWidgets backend, it doesn't look terribly difficult to make it a Tk widget, but there could be show stoppers, like event loop integration etc.

When all this comes together it will make Tk a very competitive toolkit.

Aleksey Palazchenko: History: Reloaded

AlekSi — 2008-08-07T21:53:45+00:00

Too much time have passed since my last post... Huh? It was the beginning of my last report month ago. Sadly, not too much progress due to personal problems. Time for real speed-up. Anyway, Psi will have new history system no later this Autumn. :)
Two recent days was great. 2/3 of code was refactored or rewritten. Now it's much better and clear. You can see it and add comments at gihub. And new GUI looks better too:

Do you think it looks good? or bad? Please download and try. Git's black magic:
git clone git://github.com/AlekSi/psi.git cd psi git branch --track fake_backend origin/fake_backend git checkout fake_backend git submodule init git submodule update cd proto qmake (or qmake-qt4 for Debian-based distros) make ./proto
Comments are needed. :)

Hal Rottenberg: New Version of Psi is Out!

halr9000 — 2008-08-07T21:06:22+00:00

We do like the long release cycles, it seems. :) Kev the project lead has written it up on his blog. You can find download links on the Psi Download page.