Recently, I have been working on refactoring a pretty complex repository for a ML usecase. It has a complex configuration system, multiple applications reading from the central configuration and extensive usage of inheritance. Especially, the modeling part is quite diverse. The inference application needs to take care of all those different models with a generic and flexible code design. Under such complex projects, it is very important to strike a balance between readibility and flexibility.
In general, clear and explicit code is often prized for its maintainability and readability. However, there are scenarios where flexibility is equally important, and this is where **kwargs play a crucial role. It allows functions and methods to accept an arbitrary number of arguments, offering a level of flexibility that can be very beneficial in certain situations. In this post, I will explore various scenarios where **kwargs can enhance your classes and code structures, ranging from inheritance and futureproofing to factory patterns and argument forwarding.
When dealing with class hierarchies, it’s often necessary to pass arguments from a child class to its parent class. Using **kwargs, we can easily pass optional arguments to the base class without the child class having to be explicitly aware of these parameters. This approach enhances code maintainability and adaptability to changes in the base class.
Consider a base class Preprocessor that performs some preprocessing with optional filters:
class Preprocessor:
def __init__(self, format, validate=True):
self.format = format
self.validate = validate
def preprocess(self, data):
if self.validate:
self._validate_data(data)
# Preprocessing logic
print(f"Preprocessing data with format {self.format}")
def _validate_data(self, data):
# Validation logic
print("Validating data")
Now, let’s create a child class ImageProcessor that extends Preprocessor and adds additional functionality like image filters. The child class needs to pass the optional validate parameter to the base class:
class ImageProcessor(Preprocessor):
def __init__(self, format, filters, enhance=True, **kwargs):
super().__init__(format, **kwargs)
self.filters = filters
self.enhance = enhance
def preprocess(self, data):
data = super().preprocess(data)
data = self.apply_filter(data)
if self.enhance:
print("Adding some enhancing logic here")
return data
def apply_filter(self, data):
for f in self.filters:
data = f(data)
return data
By using **kwargs in the ImageProcessor constructor, we can flexibly initialize the object with any optional arguments defined in the Preprocessor class:
image_processor = ImageProcessor("csv", ["brightness", "contrast"], enhance=False, validate=False)
With this approach, the child class ImageProcessor can be initialized with the validate flag set to either True or False, showcasing the flexibility afforded by **kwargs.
A key advantage of using **kwargs is the ability to future-proof your classes. By including **kwargs in the constructor of a base class, you open the door to adding new features and settings without disrupting existing code. This approach is especially valuable in scenarios where your classes are a part of a library or framework used by others, as it ensures backward compatibility.
Initially, consider a Chart class with basic functionality, but without **kwargs:
class Chart:
def __init__(self, title, data_source):
self.title = title
self.data_source = data_source
def draw(self):
# Basic drawing logic
print(f"Drawing {self.title} chart with data from {self.data_source}")
Now, let’s say we develop a subclass BarChart:
class BarChart(Chart):
def __init__(self, title, data_source, bar_width, bar_color):
super().__init__(title, data_source)
self.bar_width = bar_width
self.bar_color = bar_color
def draw(self):
# Specialized drawing logic for bar chart
print(f"Drawing Bar Chart: {self.title}")
print(f"Width: {self.bar_width}, Color: {self.bar_color}")
As the codebase evolves, we decide to add a new feature, such as a background color. Without **kwargs, updating the Chart class and all its subclasses can lead to compatibility issues:
class Chart:
def __init__(self, title, data_source, background_color):
self.title = title
self.data_source = data_source
self.background_color = background_color
def draw(self):
print(f"Drawing {self.title} chart on {self.background_color} background with data from {self.data_source}")
The BarChart class must also be updated, which can break existing code that uses the older API:
class BarChart(Chart):
def __init__(self, title, data_source, bar_width, bar_color, background_color):
super().__init__(title, data_source, background_color)
self.bar_width = bar_width
self.bar_color = bar_color
In the absence of **kwargs, we need to update the function signature, which prevents the existing users from using the new feature unless they refactor their code. However, with **kwargs, we can elegantly solve this issue. By including **kwargs in the Chart class’s constructor, we can add new features without changing the function signature:
class Chart:
def __init__(self, title, data_source, **kwargs):
self.title = title
self.data_source = data_source
self.chart_settings = kwargs # Store additional settings
def draw(self):
print(f"Drawing {self.title} chart with data from {self.data_source}")
print("Additional settings:", self.chart_settings)
The BarChart class, with its own specific settings and those passed through **kwargs, becomes:
class BarChart(Chart):
def __init__(self, title, data_source, bar_width, bar_color, **kwargs):
super().__init__(title, data_source, **kwargs)
self.bar_width = bar_width
self.bar_color = bar_color
def draw(self):
print(f"Drawing Bar Chart: {self.title}")
print(f"Width: {self.bar_width}, Color: {self.bar_color}")
if 'background_color' in self.chart_settings:
print(f"Modify background color with {self.chart_settings['background_color']} ")
This design allows for a flexible and backward-compatible approach to class evolution. New features can be added without breaking existing implementations, and the **kwargs mechanism ensures that subclasses can adapt to changes in the base class without requiring code modifications.
The use of **kwargs is particularly beneficial in factory patterns, a common design pattern in software engineering, especially in scenarios like machine learning where different models require different initialization parameters. Utilizing **kwargs in such patterns allows for a single, unified interface to create diverse objects, each with its unique configuration, enhancing flexibility and code reusability.
Consider an example where we are building a machine learning system that requires various models, each with distinct initialization parameters:
from sklearn.neural_network import MLPClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
def create_model(model_type, **kwargs):
if model_type == 'neural_network':
return MLPClassifier(**kwargs)
elif model_type == 'decision_tree':
return DecisionTreeClassifier(**kwargs)
elif model_type == 'svm':
return SVC(**kwargs)
else:
raise ValueError("Unknown model type")
# Creating different models with specific parameters
neural_network = create_model('neural_network', hidden_layer_sizes=(100,), activation='relu', max_iter=300)
decision_tree = create_model('decision_tree', max_depth=5, criterion='entropy')
svm_model = create_model('svm', C=1.0, kernel='linear')
create_model.This scenario showcases how **kwargs facilitates the creation of a flexible and versatile factory function, capable of handling a wide range of object types and configurations. Such an approach is highly beneficial in fields like machine learning, where the diversity of models and their configurations is vast.
In Python, one of the practical uses of **kwargs is in argument forwarding. This technique is particularly useful when a function or method acts as a wrapper or forwarder to another function, simplifying the propagation of arguments. It’s especially handy when the wrapper function doesn’t need to handle those arguments directly.
Let’s illustrate this with a simple example. Suppose we have a basic function process_data, which accepts various parameters.
def process_data(data, format="JSON", optimize=False):
# Processing logic
print(f"Processing {data} as {format}, optimize={optimize}")
# Return some result
return "processed_data"
Now, imagine we want to create a wrapper function that logs the call details but doesn’t modify the original function’s signature.
**kwargs for Argument ForwardingThe wrapper function logging_wrapper can log the details and forward the arguments to the process_data function without needing to know the specifics of its parameters.
def logging_wrapper(*args, **kwargs):
print(f"Calling process_data with args: {args}, kwargs: {kwargs}")
return process_data(*args, **kwargs)
# Using the wrapper function
result = logging_wrapper("my_data", format="XML", optimize=True)
logging_wrapper can handle any arguments meant for process_data, making it versatile and adaptable.process_data.process_data remains unchanged, and additional functionality is added without modifying the original behavior.This example demonstrates how **kwargs can be effectively utilized for argument forwarding in wrapper functions. It’s a prevalent pattern in situations where additional functionality, like logging or preprocessing, needs to be added to existing functions without changing their signatures. The use of **kwargs maintains the cleanliness and flexibility of the code, making it a valuable tool in the Python programmer’s toolkit.
Avoid Overuse and Misuse: While **kwargs provides flexibility, its overuse can lead to code that is hard to understand and debug. It’s important to use **kwargs prudently. Over-reliance on **kwargs can obscure what your function or class actually requires or accepts, making the code less intuitive and more difficult for others (or even yourself at a later time) to work with.
Explicit is Better Than Implicit: In line with the Zen of Python, being explicit about what your functions and methods expect can often be more beneficial than using **kwargs. While **kwargs is useful for certain situations, explicitly defining parameters makes the code more readable and easier to understand.
I suggest using it with type hinting. This way, we can specify the type of arguments our functions are expected to receive. This can greatly enhance the readibility.
from typing import TypedDict
from typing import List
class ChartSettings(TypedDict, total=False):
title: str
data_source: str
background_color: str
bar_width: int
class Chart:
def __init__(self, title: str, data: List[float], **kwargs: ChartSettings):
self.title = title
self.data = data
self.settings = kwargs
By using TypedDict, we explicitly specify the allowed optional arguments. Also, total=False parameter in TypedDict indicates that not all keys are required, making it flexible for optional settings.
In this blog, we had a deep dive on the use of kwargs, exploring its utility and best practices. Through various scenarios, from enhancing class inheritance to designing futureproof and adaptable classes, we’ve seen how kwargs can significantly simplify and streamline our code. Note that these are merely the scenarios that I find particularly helpful. You are more than welcomed to share your own scenarios and usecases where kwargs is a game changer!