Ubuntu CI Services

There should only be one, documented, way to run tests on devices

Bug #1262879 reported by Michał Sawicz
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ubuntu CI Services
Confirmed
Undecided
Unassigned

Bug Description

There's at least three different ways that are in use to do testing on devices:

* otto
* utah
* phablet-test-run

All of them are in constant use (otto in upstream merger, utah in smoke testing, phablet-test-run in manual testing), and cause issues like bug #1262743.

We *really* need a common way (for infrastructure and developers) to run the test in exactly the same environment, so that we can all reliably reproduce any issues arising.

Revision history for this message
Michael Zanetti (mzanetti) wrote :

There is also

phablet-click-test-setup --click com.ubuntu.notes; phablet-test-run notes_app

vs

phablet-test-run -p notes-app-autopilot notes_app

which are supposed to do the same thing, but cause different results for some reason.

Changed in ubuntu-ci-services-itself:
status: New → Confirmed
Revision history for this message
Robert Bruce Park (robru) wrote : Re: [Bug 1262879] Re: There should only be one, documented, way to run tests on devices

On Thu, Dec 19, 2013 at 2:01 PM, Michael Zanetti
<email address hidden> wrote:
> phablet-click-test-setup --click com.ubuntu.notes; phablet-test-run notes_app

This one installs the autopilot tests from the project's launchpad trunk branch

> phablet-test-run -p notes-app-autopilot notes_app

This one installs the autopilot tests from the debian package, which
is likely to be unmaintained if the project has switched to using
click packages as the primary delivery mechanism; it depends on the
app which one is the correct way to invoke the tests.

I would guess that the first one is preferable when it works, but not
all apps are shipped as click yet.

Revision history for this message
Michał Sawicz (saviq) wrote :

BTW, phablet-click-test-setup has bitten us more than once now after having pulled unity8 tests into /home/phablet/autopilot, when people expected the unity8-autopilot package to be used.

It's relatively easy to see - autopilot spits out where it takes the tests from:

Loading tests from: /home/phablet/autopilot

But it's also relatively easy to miss - especially if you're not looking for it specifically.

Revision history for this message
Alan Pope 🍺🐧🐱 🦄 (popey) wrote :

The following wiki page seeks to detail the One True Way™

https://wiki.ubuntu.com/Touch/Testing

If there are issues with that page, please edit to fix.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.