I'm currently trying to set up UTAH metadata for my new desktop integration tests, according to https://wiki.ubuntu.com/QATeam/AutomatedTesting/UbuntuTestAutomationHarness/UTAHCompatibleTC and the existing example. I do test runs with "utah -r master.run", but it always just says
Master runlist failed to validate: '/home/martin/ubuntu/desktop-tests/master.run'
Detailed information: {'testsuites': [{'name': 'tests'}]} is not of type [{'items': {'type': 'object', 'properties': {'fetch_cmd': {'required': True, 'type': 'string'}, 'exclude_tests': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'required': True, 'type': 'string'}, 'include_tests': {'items': {'type': 'string'}, 'type': 'array'}}}, 'type': 'array'}, {'type': 'object', 'properties': {'repeat_count': {'minimum': 0, 'type': 'integer'}, 'type': {'enum': ['smoke', 'kernel-sru', 'bootspeed', 'upgrade'], 'type': 'string'}, 'publish': {'type': 'object', 'properties': {'url': {'required': True, 'type': 'string'}, 'token': {'required': True, 'type': 'string'}, 'name': {'required': True, 'type': 'string'}}}, 'timeout': {'minimum': 0, 'type': 'integer'}, 'testsuites': {'required': True, 'type': [{'items': {'type': 'object', 'properties': {'fetch_cmd': {'required': True, 'type': 'string'}, 'exclude_tests': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'required': True, 'type': 'string'}, 'include_tests': {'items': {'type': 'string'}, 'type': 'array'}}}, 'type': 'array'}]}}}]
This is not a helpful error message. I am doing changes over changes in the various master.run, testcase.run, ts_control, tc_control etc. files that this requires me to write, but there is no real way to tell me whether I'm going in the right direction.
Can the error messages please be more specific in what files, keys, or values are missing or wrong?
The error message comes from the jsonschema library itself. This is something we are already aware of, but that we cannot fix in our code. Anyway, I took a look at the jsonschema code and it wouldn't be to hard to at least pretty print the dictionaries to have something more readable.