Ubuntu CI Services

run-tests not working correctly for cli

Bug #1285675 reported by Chris Johnston
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu CI Engine
Invalid
High
Vincent Ladeuil
Ubuntu CI Engine backlog
Ubuntu CI Services
Invalid
High
Vincent Ladeuil
Ubuntu CI Services backlog

Bug Description

running ./run-tests provides the following output related to the cli:

tests.test_cli.MainScriptTestCase.test_add_new_ticket ... You have successfully submitted a ticket to the Ubuntu CI Engine. Your ticket number is 38.
OK (0.007 secs)
Not checking signature
INFO:dput:Not checking signature
Not checking signature
INFO:dput:Not checking signature
tests.test_cli.MainScriptTestCase.test_add_new_ticket_multiple_subtickets ... You have successfully submitted a ticket to the Ubuntu CI Engine. Your ticket number is 39.
OK (0.006 secs)
tests.test_cli.MainScriptTestCase.test_cli_arguments_are_required ... OK (0.013 secs)
tests.test_cli.MainScriptTestCase.test_cli_changes_file_is_something_else ... OK (0.004 secs)
Not checking signature
tests.test_cli.MainScriptTestCase.test_cli_changes_file_is_unreleased ... FAIL (0.005 secs)
tests.test_cli.MainScriptTestCase.test_cli_dsc_file_in_disguise ... OK (0.004 secs)
tests.test_cli.MainScriptTestCase.test_cli_only_source_files_pass ... OK (0.002 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status ... OK (0.003 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status_bad_status_line ... OK (0.004 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status_internal_server_error ... OK (0.003 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status_other_httperror ... OK (0.003 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status_url_not_found ... OK (0.003 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status_urlerror ... OK (0.003 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_all_files_found ... ERROR (0.000 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_file_to_upload_not_found ... ERROR (0.000 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_files_to_upload_dir_found ... ERROR (0.000 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_files_to_upload_dir_not_provided ... ERROR (0.000 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_parse ... ERROR (0.000 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_parse_changes_file_is_a_dsc ... ERROR (0.000 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_parse_changes_file_is_something_else ... ERROR (0.000 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_parse_changes_file_not_found ... OK (0.000 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_process ... ERROR (0.000 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_signature_check_fails ... ERROR (0.000 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_upload_dir_not_found ... ERROR (0.000 secs)
tests.test_file_handler.ChangesProcessorTestCase.test_validate_distribution_unreleased ... ERROR (0.000 secs)
tests.test_get_ticket_status.GetTicketStatus.test_get_all_ticket_status ... OK (0.003 secs)
tests.test_get_ticket_status.GetTicketStatus.test_get_single_ticket_status ... OK (0.002 secs)
tests.test_get_ticket_status.GetTicketStatus.test_get_status_404_response ... OK (0.002 secs)
tests.test_style.TestPep8.test_pep8_conformance ... OK (0.039 secs)
tests.test_style.TestPyflakes.test_pyflakes_conformance ... OK (0.013 secs)
tests.test_ticket.SubTicketTestCase.test_create_artifact ... OK (0.001 secs)
Not checking signature
INFO:dput:Not checking signature
tests.test_ticket.SubTicketTestCase.test_create_sourcepackage ... OK (0.002 secs)
Not checking signature
INFO:dput:Not checking signature
tests.test_ticket.SubTicketTestCase.test_create_spu ... OK (0.003 secs)
Not checking signature
INFO:dput:Not checking signature
tests.test_ticket.SubTicketTestCase.test_create_spu_sourcepackage_not_found ... OK (0.003 secs)
tests.test_ticket.SubTicketTestCase.test_create_subticket ... OK (0.000 secs)
tests.test_ticket.TicketTestCase.test_create_ticket ... OK (0.003 secs)
tests.test_utils.UtilsTestCase.test_get_sourcepackage_uri ... OK (0.001 secs)
tests.test_utils.UtilsTestCase.test_get_sourcepackage_uri_not_found ... OK (0.001 secs)
tests.test_utils.UtilsTestCase.test_load_config ... OK (0.023 secs)
tests.test_utils.UtilsTestCase.test_parse_id ... OK (0.000 secs)
tests.test_utils.UtilsTestCase.test_parse_id_invalid_response ... OK (0.000 secs)
tests.test_utils.UtilsTestCase.test_parse_id_not_int ... OK (0.000 secs)

That's a bunch of tests... When running ./run-tests cli I would expect that all tests related to the cli are run..

./run-tests cli
Tests running...
TRACE:dput:No clojure support :(
TRACE:dput:No hython support :(
Not checking signature
tests.test_cli.MainScriptTestCase.test_add_new_ticket ... You have successfully submitted a ticket to the Ubuntu CI Engine. Your ticket number is 38.
OK (0.007 secs)
Not checking signature
Not checking signature
tests.test_cli.MainScriptTestCase.test_add_new_ticket_multiple_subtickets ... You have successfully submitted a ticket to the Ubuntu CI Engine. Your ticket number is 39.
OK (0.007 secs)
tests.test_cli.MainScriptTestCase.test_cli_arguments_are_required ... OK (0.013 secs)
tests.test_cli.MainScriptTestCase.test_cli_changes_file_is_something_else ... OK (0.004 secs)
Not checking signature
tests.test_cli.MainScriptTestCase.test_cli_changes_file_is_unreleased ... FAIL (0.006 secs)
tests.test_cli.MainScriptTestCase.test_cli_dsc_file_in_disguise ... OK (0.004 secs)
tests.test_cli.MainScriptTestCase.test_cli_only_source_files_pass ... OK (0.002 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status ... OK (0.003 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status_bad_status_line ... OK (0.003 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status_internal_server_error ... OK (0.003 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status_other_httperror ... OK (0.003 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status_url_not_found ... OK (0.003 secs)
tests.test_cli.MainScriptTestCase.test_ticket_status_urlerror ... OK (0.003 secs)
Ran 13 tests in 0.626s

Instead, only 13 tests are run. This is alot less than what was run when running ./run-tests

Tags: airline
Chris Johnston (cjohnston)
tags: added: airline
Evan (ev)
Changed in ubuntu-ci-services-itself:
assignee: nobody → Vincent Ladeuil (vila)
Chris Johnston (cjohnston)
Changed in ubuntu-ci-services-itself:
importance: Undecided → High
milestone: none → phase-0
Revision history for this message
Vincent Ladeuil (vila) wrote :

From './run-tests -h':

usage: uci-run-tests [-h][--exclude EXCLUDE] [--list]
                     [--format {text,subunit}]
                     [INCLUDE [INCLUDE ...]]

Load and run tests.

positional arguments:
  INCLUDE All tests matching the INCLUDE regexp will be run. Can
                        be repeated.

'./run-tests cli' runs all the tests with an id that matches the regexp 'cli'.

The following tests, for example, have ids that don't match 'cli':
tests.test_utils.UtilsTestCase.test_load_config ... OK (0.023 secs)
tests.test_utils.UtilsTestCase.test_parse_id ... OK (0.000 secs)
tests.test_utils.UtilsTestCase.test_parse_id_invalid_response ... OK (0.000 secs)
tests.test_utils.UtilsTestCase.test_parse_id_not_int ... OK (0.000 secs)

If you want to run them, you need to select them by providing the relevant regexp.

The 'cli' component use a different layout than the other components which produces test ids that don't mention the component (you can explore the test suite name space with --list to refine your regexps). The 'tests' module is imported from './cli/tests/' so their ids don't mention 'cli'. Arguably we should fix the 'cli' layout, but in the mean time, you can use that different layout at your advantage and run those tests with:

$ ./run-tests ^tests --list | wc -l
42

That's for revno 297.

This seems to match what you pasted but I don't know which revno you were using.

In general, when searching for the right regexp(s) to use, using '--list' provides a fast way to test the regexps without running the tests themselves. That is:

$ ./run-tests <regexp1> <regexp2> --list | wc -l

iterated with finer regexps until you've narrowed the test suite you're interested in.

Changed in ubuntu-ci-services-itself:
status: New → Invalid
Revision history for this message
Chris Johnston (cjohnston) wrote : Re: [Bug 1285675] Re: run-tests not working correctly for cli

We should be able to run ./run-tests <component> for any component and all
of the tests are run for that component. I shouldn't have to know that in
order to run the tests for the test runner that I actually have to run
./run-tests tstrun

Revision history for this message
Vincent Ladeuil (vila) wrote :

> I shouldn't have to know that in order to run the tests for the test runner that I actually have to run ./run-tests tstrun

Running the tests for a single component is a specific (yet useful) use case. But in general, you need to learn what tests exist before selecting them.

The point of having a single test runner is to allow a dev to define different subsets tailored to his current need. ./run-tests pep8 will run tests across several components for example, that's the point of using regexps.

In the long term we may want to support aliases to make the most common cases easier to type but for now, it's hard to beat '^tstrun' even if it requires a bit of learning as it's supported out of the box.

Adding aliases for all the tests of a given component won't scale to select less tests in that component or will make the UI more complex.

Revision history for this message
Evan (ev) wrote :

Why is this invalid? Don't we still need to fix the cli layout so that ./run-tests cli works?

Revision history for this message
Vincent Ladeuil (vila) wrote :

> Why is this invalid?

because run-tests works as designed in that case.

> Don't we still need to fix the cli layout so that ./run-tests cli works?

There are several issues with the current layout of the project:
- we have 2 levels of directories for each component: one inside the project and one below to hold python files for the component,
- we have tests that involves several components and cannot be stored in a component dir.

So it's more about making sure we have unique test IDs than making 'run-tests cli' works since run-tests doesn't care about how the tests are named, leaving the user control the layout.

Since we have a different directory for each component, we have to add them all to PYTHONPATH which can create collisions.
You encountered one with run_worker, I encountered some others while working on the integration tests.

Hard-coding relationships between tests and components inside run-tests is not the way to go, that will just create additional maintenance for no benefits.

Vincent Ladeuil (vila)
Changed in ubuntu-ci-services-itself:
milestone: phase-0 → none
Andy Doan (doanac)
Changed in ubuntu-ci-services-itself:
milestone: none → backlog
Chris Johnston (cjohnston)
Changed in uci-engine:
importance: Undecided → High
assignee: nobody → Vincent Ladeuil (vila)
milestone: none → backlog
Ursula Junque (ursinha)
Changed in uci-engine:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.