Fuzzing - Required but nullable

[thim81]

Currently Portman, fuzzes “required” properties by removing them and expecting a 4xx response
But what if in OpenAPI the property is “required” but allowed to be “nullable”, what should Portman do?

Example:

components:
  schemas:
    Pet:
      type: object
      required:
        - url
      properties:
        name:
          type: string
          nullable: true
          example: 'doggie'
        url:
          type: string
          nullable: true
          example: 'https://www.example.net'

What would be the expected fuzzing behaviour?

Options:

1. Remove the URL property and expect a 4xx response?
2. Remove the URL property and expect a 2xx response?
3. Keep the property but with null value and expect a 4xx response?
4. Skip the variation, since we are testing for 4xx variations, which is not applicable for a null value, since that would return a 200

[thim81]

@savage-alex I'm curious which option you would expect from Portman fuzzing?

[thim81]

The problems with "type-flipping" is that Postman is not really suited for this, since it would generate an exponential amount of Postman request variations for each type flip.

1 POST request with a request body with 5 properties, could easily result in +100 Postman requests.

[thim81]

@savage-alex On the 4 options above with required but nullable, what would be your expected option?

[nicklloyd]

The problems with "type-flipping" is that Postman is not really suited for this, since it would generate an exponential amount of Postman request variations for each type flip.

1 POST request with a request body with 5 properties, could easily result in +100 Postman requests.

Yea I guess if it was enableTypeFlipping on the whole request that could get messy fast. In the end, dev would be testing that their schema validation is ok, so wondering if 'one' is caught by server we can go as far as to assume any would be.

What I mean is we could enable type flip on prop level instead of whole request to verify schema. Then if dev wants to do 20 props in the same request, it's up to them...

Also, we only need to check that server catches when prop is not what schema says, not that it catches all other types.

[thim81]

Not sure what that I fully understand?
How would changing the types for the JSON schema validation, result in a logical test?

Is it like the flip we do with the "additionalProperties" in the Contract test schema validation?

[nicklloyd]

Sorry when I said schema validation, I meant server side.

So we're talking about overwriting a request body with something like:

{
   isActive: 42
}

And then test that the server responds with (something like) 'BadRequest / InvalidInput error, isActive should be a boolean'.

[savage-alex]

Re the exact options (sorry to take it somewhere else)

Question: But what if in OpenAPI the property is “required” but allowed to be “nullable”, what should Portman do?

Have we got an example of this in the wild? I didn't know you could do both required and nullable

apimatic had a write up that was interesting https://www.apimatic.io/blog/2021/09/using-optional-and-nullable-properties-in-api-requests/

My gut says:

• If its nullable and required
  ** Sending property:null should return a 200 as it looks like its legit. otherwise it would be 400
  ** Not sending a property at all that is required should be a 400