Automating specific microservice python dependencies

[mrvanes]

We are contemplating extraction of our micro_services from our SATOSA fork, just like this project but I'm unsure how we would cater for microservice specific python dependencies?

I guess this repo is supposed to be copied into the SATOSA virtualenv manually, but that would break if any of the included (and used) microservices required extra dependencies. What would a setup.py for this project look like if it was to install it's microservices into an existing SATOSA venv, while adding the necessary requirements on the fly? Can packages share names, so that files land in the same site-packages dir?

[c00kiemon5ter]

A similar discussion had started on satosa-dev mailing list. At the time I had said the following:

having each microservice as a separate (python) package, with its own dependencies and deployment process, is the way to go.

which aligns with the general meaning of microservice – ofcourse, one could argue that these are not self standing microservices but plugins.

However, we have a problem: some microservices depend on satosa itself. This means that some microservices may try to install a different satosa version than the one you're running.

---

If we don't go that way, then a setup.py for the repo would have to cater for all microservice's dependencies, or list them as extra_requires.

---

I really think that this is problem that has come up due to bad separation of concerns between the satosa modules and microservices. It shouldn't be that way.

The other option is to revisit the whole "microservices" concept and convert it to proper semantics. This could be plugins, or hooks, or callbacks, or signals, or a mix as needed (ie asynchronous invocation might be a valid case).

[mrvanes]

Ok, so I tested a little and this setup seems to do the trick:

base_project/
├── my_test_project
│   ├── extensions
│   │   ├── __init__.py
│   │   └── main.py
│   ├── __init__.py
│   └── more
│       ├── base.py
│       └── __init__.py
└── setup.py

overlay/
├── my_test_project
│   └── extensions
│       └── overlay.py
└── setup.py

using this setup.py in base_project:

from setuptools import setup, find_packages

setup(
    name = "my_test_project",
    version = "0.0.1",
    author = "Me",
    author_email = "***@gmail.com",
    description = ("Test base packages."),
    license = "Apache 2.0",
    keywords = "example test project",
    url = "https://github.com/***/",
    packages=find_packages(),
    install_requires=[
        #"base_requirement",
    ],
    zip_safe=False,
)

and this setup.py in overlay:

from setuptools import setup

setup(
    name = "my_test_project_overlay",
    version = "0.0.1",
    author = "Me",
    author_email = "***@gmail.com",
    description = ("Test overlay packages."),
    license = "Apache 2.0",
    keywords = "example test project",
    url = "https://github.com/***/",
    packages=['my_test_project.extensions'],
    install_requires=[
        #"overlay_requirement",
    ],
    zip_safe=False,
)

This creates the following structures (a.o.) under site-packages/ when I pip install both packages to the same venv.

my_test_project
├── extensions
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── main.py
│   ├── main.pyc
│   ├── overlay.py
│   └── overlay.pyc
├── __init__.py
├── __init__.pyc
└── more
    ├── base.py
    ├── base.pyc
    ├── __init__.py
    └── __init__.pyc
my_test_project-0.0.1-py2.7.egg-info
├── dependency_links.txt
├── installed-files.txt
├── not-zip-safe
├── PKG-INFO
├── SOURCES.txt
└── top_level.txt
my_test_project_overlay-0.0.1-py2.7.egg-info
├── dependency_links.txt
├── installed-files.txt
├── not-zip-safe
├── PKG-INFO
├── SOURCES.txt
└── top_level.txt

All __init__.py files in my_test_project are still the base_project versions and overlay.py is added on top.

Since micro_services are loaded on-demand by plugins configuration and have an empty __init__.py, I don't see how this wouldn't work? Am I missing something or some python magic that stops working this way?

Only obvious problem would be the non-unique packagename of overlay, so it can't be distributed via pip? But for a local distribution repository it would suffice?

[mrvanes]

So, I went ahead and this is what our repo now looks like
https://github.com/SURFscz/SATOSA-micro_services