X can't be started without login manager

On Ubuntu 16.04 LTS

X runs fine from the login manager, but if I remove the login manager (apt-get remove sddm), X can't be run from the text login: running startx don't even change to graphic mode and causes errors (log attached).

This used to work at least up to 15.04

This happens on various VM vith different video card emulation (vmware, cirrus, qemu vga) and on both ubuntu and kubuntu. Log changes slightly, but it still seems a permission-related problem: depending by the (emulated) video HW, it can complain about
xf86EnableIOPorts: failed to set IOPL for I/O (Operation not permitted)
or can segfault in libpciaccess.

This happens both on system updated from older version (15.04) and on a new clean install from scratch.

While the installed-from-scratch system obviously has the default, untouched, kernel, on the machine I updated from 15.04 I kept my custom kernel, that is the same that worked before update to 16.04, so it's probably not due to anything related to the kernel). Never tested outside a VM on real HW.

On the system I updated from 15.04, I also tried to run startx as root, and it seems to make the difference, because in this way the graphic is initialized correctly. (I see some windows with errors, and I can't reach the desktop, but this is probably another unrelated problem, possibly due to my local configuration).

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: xorg 1:7.7+13ubuntu3
Uname: Linux 4.5.2 x86_64
ApportVersion: 2.20.1-0ubuntu2
Architecture: amd64
Date: Sat May 21 11:01:12 2016
InstallationDate: Installed on 2012-03-02 (1540 days ago)
InstallationMedia: Kubuntu 11.10 "Oneiric Ocelot" - Release amd64 (20111012)
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: xorg
UpgradeStatus: Upgraded to xenial on 2016-05-16 (4 days ago)