Running tests without calling `environment.InitFlags()` func leads to failed test being skipped instead of failure

[cardil]

The environment.InitFlags() is required to be executed. Without it, the flags are not set properly. Their values are outside of possible known values - s == 0 and l == 0.

https://github.com/knative-sandbox/reconciler-test/blob/816f2192fec9b6363c2b9ed79875aa54d472eede/pkg/environment/flags.go#L29-L30

This leads to nasty behavior - the shouldFail(s *feature.Step) bool func returns false, and instead of failures, the tests are skipped. This behavior is hard to understand.

https://github.com/knative-sandbox/reconciler-test/blob/816f2192fec9b6363c2b9ed79875aa54d472eede/pkg/environment/magic.go#L306-L309

I think, we should amend the test framework to ensure flags have valid values. Either by forcing people to call environment.InitFlags(), or setting flag values with reasonable defaults.

[cardil]

/kind bug

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[cardil]

/reopen
/remove-lifecycle stale

[knative-prow]

@cardil: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle stale

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.