Mocking modules in Python§
When writing unit tests in Python there is an issue that pops up quite often: You are testing something that depends on many of libraries that are not available on your development environment. Normally you do not need that libraries functionality, you are going to mock objects imported from those libraries and assert that they are properly called.
I have recently come up with a solution for this problem that ensures that everything imported from a user-defined list of libraries (independent from the import level) is a MagicMock
, lets see how this can be achieved:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | # Stdlib import sys import importlib from unittest import mock from types import ModuleType from typing import Optional, Sequence from importlib.machinery import ModuleSpec from importlib.abc import MetaPathFinder, Loader class MockModule(mock.MagicMock): """Mocks a module so anything that is imported from it is also a :class:`MockModule` `""" class MockModuleLoader(Loader): """Dummy loader to be used when defining :class:`MockModule` modules""" def create_module(self, spec: ModuleSpec): return sys.modules[spec.name] def exec_module(self, module: ModuleType): pass def __init__(self, name: str = '', prefix: str='', *args, **kwargs): """Creates an object that you can import things from. Args: name: Name of the module. prefix: Fully-qualified name prefix, what is before ``name`` removing last dot ('.') """ name = name if name else self.__class__.__name__ kwargs['name'] = name super().__init__(ModuleType(name), *args, **kwargs) self.name = name self.__all__ = [] self.__path__ = [] self.__name__ = self.get_fully_qualified_name(name, prefix) self.__loader__ = self.MockModuleLoader() self.__spec__ = ModuleSpec( name = self.__name__, loader = self.__loader__, is_package = True ) sys.modules[self.__name__] = self globals()[self.__name__] = importlib.import_module(self.__name__) def __getattr__(self, name: str): if name.startswith('_'): raise AttributeError(f"MockModule has no attribute {name}") self.__all__.append(name) child_module = sys.modules.get(self.get_fully_qualified_name(name, self.__name__)) if child_module is None: child_module = MockModule(name=name, prefix="{self.__name__}") return child_module @staticmethod def get_fully_qualified_name(mod_name: str, prefix: str = '') -> str: """Built fully qualified either for top or lower level packages and modules""" suffix = ('.' if prefix else '') + mod_name return f"{prefix}{suffix}" class MockModuleMetaPathFinder(MetaPathFinder): """Finder for :class:`MockModule` so those can be imported""" def find_spec( self, fullname: str, path: Optional[Sequence[str]], target: Optional[ModuleType] = None, ) -> Optional[ModuleSpec]: *prefixes, mod_name = fullname.split('.') mod_parent = sys.modules.get('.'.join(prefixes)) if isinstance(mod_parent, MockModule): return getattr(mod_parent, mod_name).__spec__ return None # Set the hook so MOCKED_MODULES return mocks for everything requested sys.meta_path.insert(0, MockModuleMetaPathFinder()) # ----------------------------------------------------------------------------- # Moodules that would provide MagicMocks instead of in-use Python objects MOCKED_PACKAGES = [ 'unavailable_lib', ] for pkg_name in MOCKED_PACKAGES: sys.modules[pkg_name] = MockModule(pkg_name) |
Lets understand what is going on. Until line 85 we are monkey-patching the Python import machinery so every time it looks for a module that comes from a MockedModule
it is imported also as a MockedModule
. When you import something in Python what Python does is:
Look for its fully-qualified name in
sys.modules
(it is here if it has been imported before). If it is found then import finishes here. If not found…The
find_spec
method of everyMetaPathFinder
insys.meta_path
list is called. What you are doing is registering a newMetaPathFinder
so, if a module or class is being imported from aMockModule
thenfind_spec
returns a newMockModule
spec.If all
find_spec
methods returnNone
thenImportError
is raised, but if anyone has returned aModuleSpec
then it is loaded.The way a module is loaded is described by this code chunk. What is happening in our code is we are abusing that the
MetaPathFinder
has already registered the module insys.modules
so we are just getting it.
Rest of the code is just the MockModule
and MockModuleLoader
definition which is quite straight forward.
After executing this code, every of the following statements is perfectly fine:
from unavailable_lib.external_apis import ExternalApiClient
from unavailable_lib.external_databases import ExternalDatabaseHandler
...
Everything imported this way is going to be a MagicMock
, so you can check and assert calls as you would normally do.
Note
If you are a pytest user as you should, adding this code to the conftest.py
file might solve many problems
References§
Python docs: The import statement
Python docs: The import system