Add unset field semantics for structs ("strict structs")

[AdamGlustein]

Cleanup and a few fixes to #574, thanks @sim15 for the contribution!

[ptomecek]

Previously, when all fields on structs were optional, we hard-coded these two flags for pydantic validation

• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L125
• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L130

Can we only apply this logic when allow_unset=True?
I think that when allow_unset is False, pydantic should infer it correctly based on the declared type and the default value, though we should add a unit test.

[AdamGlustein]

Previously, when all fields on structs were optional, we hard-coded these two flags for pydantic validation

• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L125
• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L130

Can we only apply this logic when allow_unset=True? I think that when allow_unset is False, pydantic should infer it correctly based on the declared type and the default value, though we should add a unit test.

I tested out this approach and it doesn't validate correctly in the case of a required field with a default value.

class MyStrictStruct(csp.Struct, allow_unset=False):
            req_default_str: str = "default"

E       pydantic_core._pydantic_core.SchemaError: Error building "list" validator:
E         SchemaError: Error building "function-wrap" validator:
E         SchemaError: Error building "typed-dict" validator:
E         SchemaError: Field 'req_default_str': a required field cannot have a default value

[ptomecek]

Previously, when all fields on structs were optional, we hard-coded these two flags for pydantic validation

• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L125
• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L130

Can we only apply this logic when allow_unset=True? I think that when allow_unset is False, pydantic should infer it correctly based on the declared type and the default value, though we should add a unit test.

I tested out this approach and it doesn't validate correctly in the case of a required field with a default value.

class MyStrictStruct(csp.Struct, allow_unset=False):
            req_default_str: str = "default"

E       pydantic_core._pydantic_core.SchemaError: Error building "list" validator:
E         SchemaError: Error building "function-wrap" validator:
E         SchemaError: Error building "typed-dict" validator:
E         SchemaError: Field 'req_default_str': a required field cannot have a default value

I think this just means that if there is a default value, we leave it as "optional" (as the user doesn't have to provide it), but if there isn't, then it's not. I guess I was wrong about pydantic inferring it. However, the behavior for non-defaulted fields will be different for the two types of Struct.

[AdamGlustein]

Previously, when all fields on structs were optional, we hard-coded these two flags for pydantic validation

• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L125
• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L130

Can we only apply this logic when allow_unset=True? I think that when allow_unset is False, pydantic should infer it correctly based on the declared type and the default value, though we should add a unit test.

I tested out this approach and it doesn't validate correctly in the case of a required field with a default value.

class MyStrictStruct(csp.Struct, allow_unset=False):
            req_default_str: str = "default"

E       pydantic_core._pydantic_core.SchemaError: Error building "list" validator:
E         SchemaError: Error building "function-wrap" validator:
E         SchemaError: Error building "typed-dict" validator:
E         SchemaError: Field 'req_default_str': a required field cannot have a default value

I think this just means that if there is a default value, we leave it as "optional" (as the user doesn't have to provide it), but if there isn't, then it's not. I guess I was wrong about pydantic inferring it. However, the behavior for non-defaulted fields will be different for the two types of Struct.

I think this just means that if there is a default value, we leave it as "optional" (as the user doesn't have to provide it)

I am not sure what you mean by this. In the semantics of structs, a: str= 'default' != a: Optional[str] = 'default'. The latter means the field is nullable, the former does not.

[robambalu]

Looks gtg, lets wait on Pascal for that open conversation though

[AdamGlustein]

Previously, when all fields on structs were optional, we hard-coded these two flags for pydantic validation

• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L125
• https://github.com/Point72/csp/blob/strict-structs/csp/impl/struct.py#L130

Can we only apply this logic when allow_unset=True? I think that when allow_unset is False, pydantic should infer it correctly based on the declared type and the default value, though we should add a unit test.

I tested out this approach and it doesn't validate correctly in the case of a required field with a default value.

class MyStrictStruct(csp.Struct, allow_unset=False):
            req_default_str: str = "default"

E       pydantic_core._pydantic_core.SchemaError: Error building "list" validator:
E         SchemaError: Error building "function-wrap" validator:
E         SchemaError: Error building "typed-dict" validator:
E         SchemaError: Field 'req_default_str': a required field cannot have a default value

I think this just means that if there is a default value, we leave it as "optional" (as the user doesn't have to provide it), but if there isn't, then it's not. I guess I was wrong about pydantic inferring it. However, the behavior for non-defaulted fields will be different for the two types of Struct.

I think this just means that if there is a default value, we leave it as "optional" (as the user doesn't have to provide it)

I am not sure what you mean by this. In the semantics of structs, a: str= 'default' != a: Optional[str] = 'default'. The latter means the field is nullable, the former does not.

Added logic to require fields without a default value in the case of a strict struct: 80f26d3

[ptomecek]

Looks gtg, lets wait on Pascal for that open conversation though

All good based on 80f26d3 and bd403e5