Squid service was not migrated to new ci.linaro.org

<fabo> pfalcon: john issue link -> https://ci.linaro.org/jenkins/view/All/job/linux-linaro-tracking_panda-omap2plus/16/console

Loic's response:

 Alexander noticed too; new ci.linaro.org setup was launching jobs which
 were actually pointing at old ci.linaro.org setup as web proxy. When
 the old instance was shut down, it exposed the problem. Problem is
 that new instance doesn't have squid installed/configured. We need to
 restore the config and any other missing bits.

My response:

Oh my, that's why I wanted Deepti to have another look before stopping
the old instance. But as she went on a leave, I had a look myself,
noted the squid difference and explicitly wrote it off as not in use.
But well, new instance couldn't automatically use squid on the old -
old had his IP/domain name changed, so stale config wouldn't explain
it. I also thought that Loic might have restarted the old instance, bit
it wasn't, so doesn't explain why builds went thru. So, maybe it's not
squid after all. We'll be looking into the issue immediately.

Alexander noted that he disable configuring system proxy in kernel_cloud slave init script. It was:

git config --system http.proxy http://ip-10-195-115-143.ec2.internal:3128

That explains the situation with accessing old ci.linaro.org: it was accessed by internal EC2 IP, so when we moved Elastic IP (i.e. public IP and domain name), it stayed the same.

Btw, James noted that we excluded squid from consideration of memory issues on old ci.* instance, which is good points as squid is known to be memory hog. On the other hand, I don't remember it popping up on top output. Anyway, worth watching performance closely once we bring squid up.

apt-get install squid

cp /etc/squid/squid.conf /etc/squid/squid.conf.org

cp /mnt-ci.linaro.org-system/etc/squid/squid.conf /etc/squid/squid.conf
mkdir -p /mnt/spool/squid/
chown proxy.proxy -R /mnt/spool/squid/
/etc/init.d/squid restart

Re-enabled proxy for test build https://ci.linaro.org/jenkins/view/All/job/linux-linaro-tracking_panda-omap2plus/19/

During the clone of that build:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1246 jenkins 20 0 2716m 1.0g 13m S 52.6 26.7 352:30.57 java
12965 proxy 20 0 18020 15m 1896 S 0.3 0.4 0:17.75 squid

It took watching for 30s to catch squid in that position, it's not visible otherwise at all, so I guess it doesn't much cpu/mem pressure, at least so far.

Ok, seems to be working as in "just working".

Now, I don't really know if a decent way to easily get cache stats out of squid, man, dpkg -L, FAQ, google, stackoverflow don't immediately help. So, here's poor man approach (after just 2 builds of the same job):

root@ip-10-99-37-240:/var/log/squid# grep -E 'TCP_HIT|TCP_MEM_HIT' * | wc -l
305
root@ip-10-99-37-240:/var/log/squid# grep -E 'TCP_MISS' * | wc -l
2034

After yet another build:

root@ip-10-99-37-240:/var/log/squid# grep -E 'TCP_HIT|TCP_MEM_HIT' * | wc -l
610
root@ip-10-99-37-240:/var/log/squid# grep -E 'TCP_MISS' * | wc -l
2756

So, caching itself sorta works, but not ideal. Anyway, topic of this ticket is to restore squid service as it was on old ci.linaro.org, which is done. Watching for a day before closing.

(Short) docs added at https://wiki.linaro.org/Platform/Infrastructure/LinaroCIBuildTestService#Additional_mirror.2BAC8-caching_services