API improvements, changes, adaptations:

[blythed]

We would like to make it maximally easy to support developers in creating their own superduperdb integrations/ porting their own models to our framework.

Here are a few ideas which might make that easier:

1. Support for decorating classes

@superduper(preprocess=my_preprocess)
class MyClass:
    def __init__(self, *args, **kwargs):
        ...

Under the hood this would auto-identify the class type, and wrap the class instance with a superduperdb wrapper.

Any other ideas?

[jieguangzhou]

• Provide a decorator or base class to enable lazy loading of models.
• In combination with a Ray computational backend, this could allow sending all computations to a remote Ray cluster.
• When registering model instances in metadata, it could exclude the model itself and only include class attributes, initialization parameters, etc.

from functools import wraps

class SuperDuper(type):
    def __new__(cls, name, bases, dct):
        cls.apply_initialization_hook(dct)
        cls.apply_ray_hook(dct)

        return super().__new__(cls, name, bases, dct)

    @staticmethod
    def apply_initialization_hook(dct):
        if 'predict' in dct:
            original_predict = dct['predict']

            @wraps(original_predict)
            def wrapped_predict(self, *args, **kwargs):
                if not self._is_initialized:
                    self.init()
                return original_predict(self, *args, **kwargs)

            dct['predict'] = wrapped_predict

    @staticmethod
    def apply_ray_hook(dct):
        import ray
        if 'on_ray' in dct:
            original_class = .......
            remote_class = ray.remote(original_class)

            def new_predict(self, *args, **kwargs):
                return ray.get(remote_class.predict.remote(self, *args, **kwargs))

            dct['predict'] = new_predict

def superduper(cls):
    return SuperDuper(cls.__name__, cls.__bases__, dict(cls.__dict__))

# The parent class
class MyBaseClass(metaclass=ExtendedMeta):
    ...

# Or use decorator
@superduper
class MyModel:
    ...

For convenience, the example code here involves metaprogramming, which will increase system complexity.
When implementing for real, alternative methods can be used to replace this approach.