LAVA result regex can't be validated prior to a run

Mike,
Did you try this tool: http://regexpal.com/
It doesn't meet all your requirements, but should do the some of the job.

Yes I use two such tools like that, on A HO with Tyler yesterday we
concluded that all the external tools did pass the regex, but LAVA did not,
that is the case presented here.
In addition I think Tyler implied that such a regex tester is not the
complete LAVA story, and that other validation of the result could be
combined in a LAVA specific tool.

Mike

On 25 March 2014 06:00, Milosz Wasilewski <email address hidden>wrote:

> Mike,
> Did you try this tool: http://regexpal.com/
> It doesn't meet all your requirements, but should do the some of the job.
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Project.
> https://bugs.launchpad.net/bugs/1296880
>
> Title:
> LAVA result regex can't be validated prior to a run
>
> Status in LAVA Project:
> New
>
> Bug description:
> When developing a test, python regex tools can provide some validation
> of the results string parsing outside of LAVA.
>
> However the results are not always the same or complete, and this is a
> problem with tests that take hours to run.
>
> A tool - ideally web based and thus always current, that could
> validate a results string pasted into a window similar to how JSON
> scripts can be run would be very valuable. It should return the same
> results that LAVA would be expecting such as test_case_id, results
> etc.
>
> e.g.
> If I were to paste in the regex
> ^(?P<test_case_id>[\s\S]+)\s+Result:(?P<result>(pass|fail))
> and
> a results string such as
> test atomic inc/dec of signed word_1_2 Result:pass
>
> I would expect to see
>
> LAVA results = OK
> test_case_id = "test atomic inc/dec of signed word_1_2"
> result = "pass"
>
> Or some helpful diagnosis if the result would not have produced any
> test result had this been a real run.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-project/+bug/1296880/+subscriptions
>

OK, understood.
I'm using a simple script for checking my regexps:

import sys
import re

r = re.compile("^(?P<result>(PASS|FAIL))\s(?P<test_case_id>[a-zA-Z\.]+)")
f = open(sys.argv[1], "r")
for line in f.readlines():
    w = r.search(line)
    if w:
         print w.group('result'), w.group('test_case_id')
f.close()

This is how I understand LAVA parses the output. I copy the output from the full log and save it to file. Than I pass this file to the script. The only difference is that the single backslashes have to be replaced with double ones in YAML file. That works perfectly fine for me. I guess it might save your day in a short term (while the LAVA regexp parser tool is not available).

Your test implies that python regex should work, let my try my snippet with
your test, I expect it to pass your test also.

On 25 March 2014 07:49, Milosz Wasilewski <email address hidden>wrote:

> OK, understood.
> I'm using a simple script for checking my regexps:
>
> import sys
> import re
>
> r = re.compile("^(?P<result>(PASS|FAIL))\s(?P<test_case_id>[a-zA-Z\.]+)")
> f = open(sys.argv[1], "r")
> for line in f.readlines():
> w = r.search(line)
> if w:
> print w.group('result'), w.group('test_case_id')
> f.close()
>
> This is how I understand LAVA parses the output. I copy the output from
> the full log and save it to file. Than I pass this file to the script.
> The only difference is that the single backslashes have to be replaced
> with double ones in YAML file. That works perfectly fine for me. I guess
> it might save your day in a short term (while the LAVA regexp parser
> tool is not available).
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Project.
> https://bugs.launchpad.net/bugs/1296880
>
> Title:
> LAVA result regex can't be validated prior to a run
>
> Status in LAVA Project:
> New
>
> Bug description:
> When developing a test, python regex tools can provide some validation
> of the results string parsing outside of LAVA.
>
> However the results are not always the same or complete, and this is a
> problem with tests that take hours to run.
>
> A tool - ideally web based and thus always current, that could
> validate a results string pasted into a window similar to how JSON
> scripts can be run would be very valuable. It should return the same
> results that LAVA would be expecting such as test_case_id, results
> etc.
>
> e.g.
> If I were to paste in the regex
> ^(?P<test_case_id>[\s\S]+)\s+Result:(?P<result>(pass|fail))
> and
> a results string such as
> test atomic inc/dec of signed word_1_2 Result:pass
>
> I would expect to see
>
> LAVA results = OK
> test_case_id = "test atomic inc/dec of signed word_1_2"
> result = "pass"
>
> Or some helpful diagnosis if the result would not have produced any
> test result had this been a real run.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-project/+bug/1296880/+subscriptions
>

On 25 March 2014 12:02, Mike Holmes <email address hidden> wrote:
> Your test implies that python regex should work, let my try my snippet with
> your test, I expect it to pass your test also.

hmm, no. When the regexp string is incorrect (invalid expression) you
get the python exception on the line with re. When it's a valid
expression but doesn't match anything in your log, you get empty
output.

Now I'm just curious, can you point us to the specific problem text/parser?

https://pythex.org/?regex=(%3FP%3Ctest_case_measurement%3E.*%3F)%23%5Cs(%3FP%3Cmeasurement%3E%5Cw*%3F)%5Cs&test_string=Late%20%23%200%0Anone%20drafting%20%23%2011%0Aavg%20days%20sched%20%23%200%0Aup%20development%20%23%201%0A&ignorecase=0&multiline=0&dotall=0&verbose=0

https://pythex.org/?regex=(%3FP%3Ctest_case_id%3E(%5B%5Cs%5CS%5D%2B))%5Cs%2BResult%3A(%3FP%3Cresult%3E(pass%7Cfail))&test_string=test%20atomic%20inc%2Fdec%20of%20signed%20word_1_2%20Result%3Apass&ignorecase=0&multiline=0&dotall=0&verbose=1

This passes Milosz test, and all the parssers online, but fails in LAVA

I think this run used this regex exactly, I have tried so many combinations
https://validation.linaro.org/dashboard/streams/anonymous/mike-holmes/bundles/791e0a80bc7a9948e8c672a6ac065b5bc355e10a/7140c443-b37c-4b33-bc39-7552da7fee49/

On 26 March 2014 00:37, Alan Bennett <email address hidden> wrote:

> Now I'm just curious, can you point us to the specific problem
> text/parser?
>
>
> https://pythex.org/?regex=(%3FP%3Ctest_case_measurement%3E.*%3F)%23%5Cs(%3FP%3Cmeasurement%3E%5Cw*%3F)%5Cs&test_string=Late%20%23%200%0Anone%20drafting%20%23%2011%0Aavg%20days%20sched%20%23%200%0Aup%20development%20%23%201%0A&ignorecase=0&multiline=0&dotall=0&verbose=0
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Project.
> https://bugs.launchpad.net/bugs/1296880
>
> Title:
> LAVA result regex can't be validated prior to a run
>
> Status in LAVA Project:
> New
>
> Bug description:
> When developing a test, python regex tools can provide some validation
> of the results string parsing outside of LAVA.
>
> However the results are not always the same or complete, and this is a
> problem with tests that take hours to run.
>
> A tool - ideally web based and thus always current, that could
> validate a results string pasted into a window similar to how JSON
> scripts can be run would be very valuable. It should return the same
> results that LAVA would be expecting such as test_case_id, results
> etc.
>
> e.g.
> If I were to paste in the regex
> ^(?P<test_case_id>[\s\S]+)\s+Result:(?P<result>(pass|fail))
> and
> a results string such as
> test atomic inc/dec of signed word_1_2 Result:pass
>
> I would expect to see
>
> LAVA results = OK
> test_case_id = "test atomic inc/dec of signed word_1_2"
> result = "pass"
>
> Or some helpful diagnosis if the result would not have produced any
> test result had this been a real run.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-project/+bug/1296880/+subscriptions
>