keyboard layout is switched to default after window change

At least two keyboard layouts configured.
Different keyboard layouts for every application.
Let's say "en" is a default layout (loaded on start-up, in new window...)
Sometimes this happens:

  Window is opened and layout different to default is set. Then switch to another window and work there for a while (no need for typing there). Then come back to the former window and find out that default layout is set which is incorrect.

I was not able to find steps to reproduce it since it has never happened after immediate switch of windows. When you just open them and spend a while by switching them waiting for the bug, it won't happen (or at least not for me). Its after some time (few minutes is enough) when it does.

I am running xfce now installed over gnome unity. The bug is in unity environment too.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: gnome-settings-daemon 3.2.2-0ubuntu2.1
ProcVersionSignature: Ubuntu 3.0.0-16.29-generic 3.0.20
Uname: Linux 3.0.0-16-generic i686
ApportVersion: 1.23-0ubuntu4
Architecture: i386
Date: Thu Mar 15 01:28:43 2012
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release i386 (20111012)
SourcePackage: gnome-settings-daemon
UpgradeStatus: No upgrade log present (probably fresh install)