Option to temporarily pause the run from the test code.

For anyone interested, I think I found a workaround. The following adding seems to do the trick:

   [NUnitAddin(Description = "Stop Upon Failure")]
   public class StopUponFailureAddIn : IAddin, EventListener
   {
      #region IAddin Members

      public bool Install(IExtensionHost host)
      {
         if (host == null)
            throw new ArgumentNullException("host");

         IExtensionPoint listeners = host.GetExtensionPoint("EventListeners");
         if (listeners == null)
            return false;

         listeners.Install(this);
         return true;
      }

      #endregion

      #region EventListener Members

      public void RunFinished(Exception exception)
      {
      }

      public void RunFinished(TestResult result)
      {
      }

      public void RunStarted(string name, int testCount)
      {
      }

      public void SuiteFinished(TestResult result)
      {
      }

      public void SuiteStarted(TestName testName)
      {
      }

      public void TestFinished(TestResult result)
      {
         if (result.IsError || result.IsFailure)
         {
            Trace.WriteLine("STOPPING TEST EXECUTION");
            Thread.Sleep(Timeout.Infinite);
         }
      }

      public void TestOutput(TestOutput testOutput)
      {
      }

      public void TestStarted(TestName testName)
      {
      }

      public void UnhandledException(Exception exception)
      {
      }

      #endregion
   }

I haven't tried the code, but it doesn't look like it should work. The Sleep
command will put the thread that sends events into a wait state. However,
that won't have any impact on the test thread. I think it will just continue
to run tests and queue events that you never see because of the Sleep.

Charlie

On Tue, Jul 12, 2011 at 3:20 PM, Nicolas Rivera
<email address hidden> wrote:
> For anyone interested, I think I found a workaround.  The following
> adding seems to do the trick:
>
>   [NUnitAddin(Description = "Stop Upon Failure")]
>   public class StopUponFailureAddIn : IAddin, EventListener
>   {
>      #region IAddin Members
>
>      public bool Install(IExtensionHost host)
>      {
>         if (host == null)
>            throw new ArgumentNullException("host");
>
>         IExtensionPoint listeners = host.GetExtensionPoint("EventListeners");
>         if (listeners == null)
>            return false;
>
>         listeners.Install(this);
>         return true;
>      }
>
>      #endregion
>
>      #region EventListener Members
>
>      public void RunFinished(Exception exception)
>      {
>      }
>
>      public void RunFinished(TestResult result)
>      {
>      }
>
>      public void RunStarted(string name, int testCount)
>      {
>      }
>
>      public void SuiteFinished(TestResult result)
>      {
>      }
>
>      public void SuiteStarted(TestName testName)
>      {
>      }
>
>      public void TestFinished(TestResult result)
>      {
>         if (result.IsError || result.IsFailure)
>         {
>            Trace.WriteLine("STOPPING TEST EXECUTION");
>            Thread.Sleep(Timeout.Infinite);
>         }
>      }
>
>      public void TestOutput(TestOutput testOutput)
>      {
>      }
>
>      public void TestStarted(TestName testName)
>      {
>      }
>
>      public void UnhandledException(Exception exception)
>      {
>      }
>
>      #endregion
>   }
>
> --
> You received this bug notification because you are subscribed to NUnit
> Framework.
> https://bugs.launchpad.net/bugs/730891
>
> Title:
>  GUI runner option to stop upon failure
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunit-3.0/+bug/730891/+subscriptions
>

It actually works great so far with version 2.5.9.10348. It is already helping me -- I have two PCs that stopped at two different tests in the same test suite, and now I can go back and look at the last portion of the log to easily analyze if the problem is the production code or the test. The only real drawback I have found is that sometimes I have to kill the GUI process to close it. But the GUI still repaints and I can switch between the tabs, which I wasn't sure if I was going to be able to do.

It sounds from your comments I may have uncovered a bug in the way the addins work. If I did, could you please consider Not changing the current behavior, at least not until you have had time to consider my feature request? I really do find this feature helpful for what I am doing (automated regression tests of whole subsystems including hardware), and this workaround suits me just as well.

I'm not sure why it appears to work, but if it's helping you go for it!

We won't change any of the current behavior in NUnit 2.x unless
somebody reports a bug. NUnit 3.0 will have an entirely different
approach to addins.

Charlie

On Wed, Jul 13, 2011 at 7:40 AM, Nicolas Rivera
<email address hidden> wrote:
> It actually works great so far with version 2.5.9.10348.  It is already
> helping me -- I have two PCs that stopped at two different tests in the
> same test suite, and now I can go back and look at the last portion of
> the log to easily analyze if the problem is the production code or the
> test.  The only real drawback I have found is that sometimes I have to
> kill the GUI process to close it.  But the GUI still repaints and I can
> switch between the tabs, which I wasn't sure if I was going to be able
> to do.
>
> It sounds from your comments I may have uncovered a bug in the way the
> addins work.  If I did, could you please consider Not changing the
> current behavior, at least not until you have had time to consider my
> feature request?  I really do find this feature helpful for what I am
> doing (automated regression tests of whole subsystems including
> hardware), and this workaround suits me just as well.
>
> --
> You received this bug notification because you are subscribed to NUnit
> Framework.
> https://bugs.launchpad.net/bugs/730891
>
> Title:
>  GUI runner option to stop upon failure
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunit-3.0/+bug/730891/+subscriptions
>

There is a more flexible way to do this that I just found out about.

In TearDown() one could write:

if (TestContext.CurrentContext.Result.Status == TestStatus.Failed)
  { Thread.Sleep(Timeout.Infinite); }

The problems with that approach are:

1. The error call stack does not get printed in the "Error and Failures" tab (which makes sense, since TearDown() has not completed). Ideally the CurrentContext would include the exception also, so that the call stack could be output to the "Text Output" tab.
2. Thread.Sleep(Timeout.Infinite) does not allow the GUI to close cleanly, so the process has to be killed through task manager. Ideally there would be a way to tell any NUnit client (whether GUI or console) to stop running a batch of tests from inside teardown.

Could the above two changes be considered instead of my original feature request? I rather like those than a GUI-centric feature. Plust they allow for a higher control and granularity than a simple global GUI setting to "Stop Upon Failure".

Are TextContexts something carried over to NUnit 3.0?

Hi Nicolas,

On Mon, Jul 25, 2011 at 6:47 PM, Nicolas Rivera
<email address hidden> wrote:
> There is a more flexible way to do this that I just found out about.
>
> In TearDown() one could write:
>
> if (TestContext.CurrentContext.Result.Status == TestStatus.Failed)
>  { Thread.Sleep(Timeout.Infinite); }

The main problem here is that "Sleep" is not the same as "Stop." It's
more like you're hanging the Gui. As a result, the test run never ends
and NUnit never does any of the stuff it does at the end of a test run
like saving the XML output.

> The problems with that approach are:
>
> 1. The error call stack does not get printed in the "Error and Failures" tab (which makes sense, since TearDown() has not completed).  Ideally the CurrentContext would include the exception also, so that the call stack could be output to the "Text Output" tab.

Yup... because that test never ended either.

> 2. Thread.Sleep(Timeout.Infinite) does not allow the GUI to close cleanly, so the process has to be killed through task manager.  Ideally there would be a way to tell any NUnit client (whether GUI or console) to stop running a batch of tests from inside teardown.

Yup... again, as above.

>
> Could the above two changes be considered instead of my original feature
> request?  I rather like those than a GUI-centric feature.  Plust they
> allow for a higher control and granularity than a simple global GUI
> setting to "Stop Upon Failure".

I'm not 100% sure what you are asking for. If it's to have "Sleep" work as
a "Stop" that's not possible. If it's to expose some way for a test to
individually
request that no further tests should be run, it's technically doable in a few
different ways. All the approaches probably involve a special exception
being thrown. That could be through an Assert or a method call to
the TestContext. Throwing an exception allows the test to complete
correctly before stopping the Gui. Stopping it would require completion
of all the teardowns from the test itself up to the top of the tree as well
as any relevant After actions, so it's non-trivial (but I think doable).

Whether we want to do this at all is what I'm wondering about. I'll
start a discussion on the mailing list about that question.

Charlie

> Are TextContexts something carried over to NUnit 3.0?
>
> --
> You received this bug notification because you are subscribed to NUnit
> Framework.
> https://bugs.launchpad.net/bugs/730891
>
> Title:
>  GUI runner option to stop upon failure
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunit-3.0/+bug/730891/+subscriptions
>

Hi Nicolas,

I neglected to answer this question in my last reply...

> Are TextContexts something carried over to NUnit 3.0?

Yes, they and they can have much more capability than is possible with
NUnit 2.6. That's because the 3.0 framework contains the classes that
run the tests, which are part of nunit.core in the 2.6/2.6 releases.
This means that a framework class (like TestContext or Assert) can
actually call into the classes that drive test execution, which is not
now possible.

Charlie

Hi Charlie,

Thanks for your prompt replies!

No, I am not asking for stopping tests through Sleep. I recognize what I did was a hack -- but it is a hack that gives me more control than my previous hack (I can now stop the unit tests without executing teardown, and I can select which unit tests I want to stop upon failure).

What I was wondering is if it would make more sense to be able to stop the a batch of unit tests programmatically through some new mechanism added to NUnit, instead of my original request of a GUI setting of "Stop upon failure". Stopping a batch of unit tests from inside TearDown() programmatically is just far more flexible for the unit test writer -- and it would not depend on the client (i.e., it could be used while running through the console also).

Let me give you a user scenario for this feature. In my case, I am running subsystem-level automated tests. Furthermore, because I write unmanaged code to interface with cellular devices, the unit tests are actually excercising code that communicates with hardware (and networks). There are some errors that are not easily reproducible. The one I am working on now is such an example. Sometimes it can be reproduced quite often in spurts, but othertimes I can run all the unit tests for hours without any error. Being able to stop the unit test programmatically depending on some condition (in this case a failure), and being able to skip the TearDown() code upon that condition (so that the current state stays in memory), allows me to make a memory dump with windbg that shows pretty much what is going on in the system at the time of the failure.

I realize my scenario is not very common. But I do suspect that any type of subsystem/integration testing would benefit from being able to do that. Oftentimes integration testing deals with multiple threads and environmental conditions that make the tests appear somewhat undeterministic. Of course, if all you are doing is strict TDD at the method level, then there is no need to do that. But I thought one of the reasons for NUnit 3.0 was to allow more flexibility for other types of testing (e.g., integration testing).

Thanks,

Nicolas

I posted to the mailing list asking for opinions on this. You may want to join that thread.

From one comment, I began to wonder if your situation could be modeled using some level of setup. If a setup
fails, then all the tests under the control of that setup are immediately failed without trying to run them.

Of course, unrelated tests would continue to run but if you put the code in a high-level setup method, you could prevent lots of tests from running.

See the discussion of this request at http://groups.google.com/group/nunit-discuss/browse_thread/thread/21279aaa01a78001

As a result of the discussion, the request is being changed to ask for a way to pause execution of the tests, allowing the developer to take whatever debugging action is needed at that point. The title and description are being changed accordingly.