Improved Literal? types

[randolf-scholz]

This PR aims to formalize the behavior of Literal? types (Instance with last_known_value) by treating them as-if they were AnyOf-types (see python/typing#566)

For example, Literal["x"]? is treated as-if it were AnyOf[str, Literal["x"]].

• Fixes Specify behavior for Literal? types. #19625 (specification)
• Fixes meet_types gives unexpected results when meeting literal and Instance #19560
• Fixes Regression: false positive avoid redundant cast with literal union #19055

New Tests

• JoinSuite.test_mixed_literal_types tests join between Literal? and other types.
• MeetSuite.test_mixed_literal_types tests meet between Literal? and other types
• TypeOpsSuite.test_simplified_union_with_mixed_str_literals2 tests simplified unions containing Literal? types
• SubtypingSuite.test_literal tests subtype checks with Literal? types
• RestrictionSuite: new test suite for testing the restrict_subtype_away method.
• testJoinLiteralInstanceAndEnum tests join between Literal? and StrEnum

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pyinstrument (https://github.com/joerick/pyinstrument)
- pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'disabled', 'strict'] | float | bool | None]"; expected "float"  [arg-type]
+ pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'strict'] | float | str | bool | None]"; expected "float"  [arg-type]
- pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'disabled', 'strict'] | float | bool | None]"; expected "Literal['enabled', 'disabled', 'strict']"  [arg-type]
+ pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'strict'] | float | str | bool | None]"; expected "Literal['enabled', 'disabled', 'strict']"  [arg-type]
- pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'disabled', 'strict'] | float | bool | None]"; expected "bool | None"  [arg-type]
+ pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'strict'] | float | str | bool | None]"; expected "bool | None"  [arg-type]

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/server/services/task_run_recorder.py:183: error: "Collection[Any]" has no attribute "state"  [attr-defined]
+ src/prefect/server/services/task_run_recorder.py:183: error: "object" has no attribute "state"  [attr-defined]
- src/prefect/server/services/task_run_recorder.py:186: error: "Collection[Any]" has no attribute "id"  [attr-defined]
+ src/prefect/server/services/task_run_recorder.py:186: error: "object" has no attribute "id"  [attr-defined]
- src/prefect/server/services/task_run_recorder.py:192: error: "Collection[Any]" has no attribute "keys"  [attr-defined]
+ src/prefect/server/services/task_run_recorder.py:192: error: "object" has no attribute "keys"  [attr-defined]
- src/prefect/concurrency/_asyncio.py:112: error: R? has no attribute "json"  [attr-defined]
+ src/prefect/task_worker.py:370: error: Argument 1 to "submit" of "Executor" has incompatible type "Callable[[Callable[_P, _T], **_P], _T]"; expected "Callable[[Callable[[Task[P, R], UUID | None, TaskRun | None, dict[str, Any] | None, PrefectFuture[Any] | Any | Iterable[PrefectFuture[Any] | Any] | None, Literal['state', 'result'], dict[str, set[RunInput]] | None, dict[str, Any] | None], R | State[Any] | None], Task[[VarArg(Any), KwArg(Any)], Any], UUID, TaskRun, dict[Any, Any], list[Any], str, Any | None], Any | State[Any] | None]"  [arg-type]

operator (https://github.com/canonical/operator)
+ ops/model.py:809: error: Unsupported operand types for - ("set[tuple[Literal['tcp', 'udp', 'icmp'], int | None]]" and "set[tuple[str, int] | tuple[Literal['tcp', 'udp', 'icmp'], int | None]]")  [operator]
+ ops/model.py:811: error: Incompatible types in assignment (expression has type "str", variable has type "Literal['tcp', 'udp', 'icmp']")  [assignment]

discord.py (https://github.com/Rapptz/discord.py)
- discord/app_commands/transformers.py:139: error: Incompatible types in assignment (expression has type "list[dict[str, Any]]", target has type "str | int")  [assignment]
+ discord/app_commands/transformers.py:139: error: Incompatible types in assignment (expression has type "list[dict[str, Any]]", target has type "bool | str | int")  [assignment]
- discord/app_commands/transformers.py:141: error: Incompatible types in assignment (expression has type "list[int]", target has type "str | int")  [assignment]
+ discord/app_commands/transformers.py:141: error: Incompatible types in assignment (expression has type "list[int]", target has type "bool | str | int")  [assignment]
- discord/app_commands/transformers.py:149: error: Incompatible types in assignment (expression has type "int | float", target has type "str | int")  [assignment]
+ discord/app_commands/transformers.py:149: error: Incompatible types in assignment (expression has type "int | float", target has type "bool | str | int")  [assignment]
- discord/app_commands/transformers.py:151: error: Incompatible types in assignment (expression has type "int | float", target has type "str | int")  [assignment]
+ discord/app_commands/transformers.py:151: error: Incompatible types in assignment (expression has type "int | float", target has type "bool | str | int")  [assignment]

dedupe (https://github.com/dedupeio/dedupe)
- dedupe/api.py:1547: error: Redundant cast to "Literal['match', 'distinct']"  [redundant-cast]

[randolf-scholz]

Repro of the operator issue:

from typing import Literal, reveal_type

class Port:
    protocol: Literal["tcp", "udp", "icmp"]
    port: int | None

def show(*ports: int | Port) -> None:
    # master: set[tuple[L['tcp', 'udp', 'icmp'], int | None]]
    # PR:     set[tuple[L['tcp']?, int] | tuple[L['tcp', 'udp', 'icmp'], int | None]]
    reveal_type(  
        {         
            ( "tcp", port ) if isinstance(port, int) else ( port.protocol, port.port )
            for port in ports
        }
    )

On master the set comprehension combines both branches into set[tuple[L['tcp', 'udp', 'icmp'], int | None]].
With this PR, Literal['tcp']? is a subtype but not a proper subtype of Literal['tcp'], so the tuples are not simplified away in make_simplified_union, which checks proper subtyping. For context, here are other type checker results:

checker	result
mypy master	set[tuple[L['tcp', 'udp', 'icmp'], int | None]]
mypy PR	set[tuple[L['tcp']?, int] | tuple[L['tcp', 'udp', 'icmp'], int | None]]
pyright	set[tuple[str, int] | tuple[str, int | None]]
ty	set[tuple[str, int | None] | unknown]
pyrefly	set[tuple[str, int] | tuple[str, int | None]]

RFC: would this be considered a regression, or is the behavior proposed by the PR OK?

[randolf-scholz]

Maybe it's possible to do something smarter here, so that e.g. list[Literal["x"]] and list[Literal["x"]?] would be combined as well

[davidhalter]

Just in case you're interested, I have implicit literals implemented as well in Zuban. However it works in a different way, I store the implicit information on the literal. This makes it a bit easier to deal with literals in my opinion. I understand that that might cause a lot of changes in Mypy and it also might cause other issues, but I feel like it's a bit better to store it that way, because it essentially is a literal.