dns resolition failure occured, let's say within  04:02:10-04:02:20

dnsmasq log showed a lot of incoming SIGTERMs
http://paste.openstack.org/show/256285/

looks like 2 SIGTERMS within 2 secs could lead to dns outage
Jun  1 04:02:12 dnsmasq[1994]: exiting on receipt of SIGTERM
Jun  1 04:02:14 dnsmasq[2103]: exiting on receipt of SIGTERM
Jun  1 04:02:20 dnsmasq[2132]: exiting on receipt of SIGTERM

cobbler produced warning with error
http://paste.openstack.org/show/256283/

looks like those warning/errors in cobbler trigger dnsmasq restart.

I've searched on '$tCD3X7ji' and found it in 

./node-4.test.domain.local/root/anaconda-ks.cfg:rootpw  --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33b4TpQAjRiCeDZx0jmgMhDYhfB9KuGfqO9OcMaKyUxnGGWslEDQ4HxTw7vcAMP85NxQe61
./node-4.test.domain.local/root/cobbler.ks:rootpw --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33b4TpQAjRiCeDZx0jmgMhDYhfB9KuGfqO9OcMaKyUxnGGWslEDQ4HxTw7vcAMP85NxQe61
./node-4.test.domain.local/root/cobbler.ks:sshpw --username root --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33k2jjklHSDG2hg2234kJHESJ3hwhsjHshSJshHSJSh333je34DHJHDr4je4AMP85NxQe61
./node-5.test.domain.local/root/anaconda-ks.cfg:rootpw  --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33b4TpQAjRiCeDZx0jmgMhDYhfB9KuGfqO9OcMaKyUxnGGWslEDQ4HxTw7vcAMP85NxQe61
./node-5.test.domain.local/root/cobbler.ks:rootpw --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33b4TpQAjRiCeDZx0jmgMhDYhfB9KuGfqO9OcMaKyUxnGGWslEDQ4HxTw7vcAMP85NxQe61
./node-5.test.domain.local/root/cobbler.ks:sshpw --username root --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33k2jjklHSDG2hg2234kJHESJ3hwhsjHshSJshHSJSh333je34DHJHDr4je4AMP85NxQe61
./node-3.test.domain.local/root/anaconda-ks.cfg:rootpw  --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33b4TpQAjRiCeDZx0jmgMhDYhfB9KuGfqO9OcMaKyUxnGGWslEDQ4HxTw7vcAMP85NxQe61
./node-3.test.domain.local/root/cobbler.ks:rootpw --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33b4TpQAjRiCeDZx0jmgMhDYhfB9KuGfqO9OcMaKyUxnGGWslEDQ4HxTw7vcAMP85NxQe61
./node-3.test.domain.local/root/cobbler.ks:sshpw --username root --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33k2jjklHSDG2hg2234kJHESJ3hwhsjHshSJshHSJSh333je34DHJHDr4je4AMP85NxQe61
./node-2.test.domain.local/root/anaconda-ks.cfg:rootpw  --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33b4TpQAjRiCeDZx0jmgMhDYhfB9KuGfqO9OcMaKyUxnGGWslEDQ4HxTw7vcAMP85NxQe61
./node-2.test.domain.local/root/cobbler.ks:rootpw --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33b4TpQAjRiCeDZx0jmgMhDYhfB9KuGfqO9OcMaKyUxnGGWslEDQ4HxTw7vcAMP85NxQe61
./node-2.test.domain.local/root/cobbler.ks:sshpw --username root --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33k2jjklHSDG2hg2234kJHESJ3hwhsjHshSJshHSJSh333je34DHJHDr4je4AMP85NxQe61
./node-1.test.domain.local/root/anaconda-ks.cfg:rootpw  --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33b4TpQAjRiCeDZx0jmgMhDYhfB9KuGfqO9OcMaKyUxnGGWslEDQ4HxTw7vcAMP85NxQe61
./node-1.test.domain.local/root/cobbler.ks:rootpw --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33b4TpQAjRiCeDZx0jmgMhDYhfB9KuGfqO9OcMaKyUxnGGWslEDQ4HxTw7vcAMP85NxQe61
./node-1.test.domain.local/root/cobbler.ks:sshpw --username root --iscrypted $6$tCD3X7ji$1urw6qEMDkVxOkD33k2jjklHSDG2hg2234kJHESJ3hwhsjHshSJshHSJSh333je34DHJHDr4je4AMP85NxQe61

So it's cobbler issue. Finally triaged.

fuel has hardcoded root password in kickstart template for cobbler. Cobbler accidentally tries to render template from a part of password recognizing password salt $tCD3X7ji as a valid template.

Under certain circumstances it could lead to provisioning failure totally depending on how often cobbler renders kickstart files for nodes in a short period of time.