Prophet
Jeremy's
notes

Mocking modules in Python§

When writing unit tests in Python there is an issue that pops up quite often: You are testing something that depends on many of libraries that are not available on your development environment. Normally you do not need that libraries functionality, you are going to mock objects imported from those libraries and assert that they are properly called.

I have recently come up with a solution for this problem that ensures that everything imported from a user-defined list of libraries (independent from the import level) is a MagicMock, lets see how this can be achieved:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
# Stdlib
import sys
import importlib
from unittest import mock
from types import ModuleType
from typing import Optional, Sequence
from importlib.machinery import ModuleSpec
from importlib.abc import MetaPathFinder, Loader


class MockModule(mock.MagicMock):
    """Mocks a module so anything that is imported from it is also a
    :class:`MockModule`
    `"""

    class MockModuleLoader(Loader):
        """Dummy loader to be used when defining :class:`MockModule` modules"""

        def create_module(self, spec: ModuleSpec):
            return sys.modules[spec.name]

        def exec_module(self, module: ModuleType):
            pass


    def __init__(self, name: str = '', prefix: str='', *args, **kwargs):
        """Creates an object that you can import things from.

        Args:
            name: Name of the module.
            prefix: Fully-qualified name prefix, what is before ``name`` removing
                last dot ('.')
        """
        name = name if name else self.__class__.__name__
        kwargs['name'] = name
        super().__init__(ModuleType(name), *args, **kwargs)
        self.name = name
        self.__all__ = []
        self.__path__ = []
        self.__name__ = self.get_fully_qualified_name(name, prefix)
        self.__loader__ = self.MockModuleLoader()
        self.__spec__ = ModuleSpec(
            name = self.__name__,
            loader = self.__loader__,
            is_package = True
        )
        sys.modules[self.__name__] = self
        globals()[self.__name__] = importlib.import_module(self.__name__)

    def __getattr__(self, name: str):
        if name.startswith('_'):
            raise AttributeError(f"MockModule has no attribute {name}")
        self.__all__.append(name)
        child_module = sys.modules.get(self.get_fully_qualified_name(name, self.__name__))
        if child_module is None:
            child_module = MockModule(name=name, prefix="{self.__name__}")
        return child_module

    @staticmethod
    def get_fully_qualified_name(mod_name: str, prefix: str = '') -> str:
        """Built fully qualified either for top or lower level packages and
        modules"""
        suffix = ('.' if prefix else '') + mod_name
        return f"{prefix}{suffix}"


class MockModuleMetaPathFinder(MetaPathFinder):
    """Finder for :class:`MockModule` so those can be imported"""

    def find_spec(
        self,
        fullname: str,
        path: Optional[Sequence[str]],
        target: Optional[ModuleType] = None,
    ) -> Optional[ModuleSpec]:
        *prefixes, mod_name = fullname.split('.')
        mod_parent = sys.modules.get('.'.join(prefixes))
        if isinstance(mod_parent, MockModule):
            return getattr(mod_parent, mod_name).__spec__
        return None


# Set the hook so MOCKED_MODULES return mocks for everything requested
sys.meta_path.insert(0, MockModuleMetaPathFinder())

# -----------------------------------------------------------------------------
# Moodules that would provide MagicMocks instead of in-use Python objects
MOCKED_PACKAGES = [
    'unavailable_lib',
]

for pkg_name in MOCKED_PACKAGES:
    sys.modules[pkg_name] = MockModule(pkg_name)

Lets understand what is going on. Until line 85 we are monkey-patching the Python import machinery so every time it looks for a module that comes from a MockedModule it is imported also as a MockedModule. When you import something in Python what Python does is:

  1. Look for its fully-qualified name in sys.modules (it is here if it has been imported before). If it is found then import finishes here. If not found…

  2. The find_spec method of every MetaPathFinder in sys.meta_path list is called. What you are doing is registering a new MetaPathFinder so, if a module or class is being imported from a MockModule then find_spec returns a new MockModule spec.

    If all find_spec methods return None then ImportError is raised, but if anyone has returned a ModuleSpec then it is loaded.

  3. The way a module is loaded is described by this code chunk. What is happening in our code is we are abusing that the MetaPathFinder has already registered the module in sys.modules so we are just getting it.

Rest of the code is just the MockModule and MockModuleLoader definition which is quite straight forward.

After executing this code, every of the following statements is perfectly fine:

from unavailable_lib.external_apis import ExternalApiClient
from unavailable_lib.external_databases import ExternalDatabaseHandler
...

Everything imported this way is going to be a MagicMock, so you can check and assert calls as you would normally do.

Note

If you are a pytest user as you should, adding this code to the conftest.py file might solve many problems

References§