Closure magic in higher-order functions.

ghz 12hours ago ⋅ 6 views

It's easy to know how to assign a global variable to an inner function - which would make the global variable a function itself thats equal to the inner function - but how does the interpreter know how to call an argument from the inner function by using the code below?

def outer(arg):
    def inner(arg2):
        print(arg, ',', arg2):
    return inner

a = outer(‘outer arg’)

print(a) # .inner at 0x109bd0048
a('inner arg') # Output: outer arg, inner arg

Doing print(a), we see the variable/function a becomes the inner function.

What I don't understand, is how assigning the a variable to the outer function targets the inner function and argument with the code, a(‘inner argument’))

Is it somehow implicitly calling the inner function without explicitly stating it?

Is it doing something like this:

a = outer('outer arg')inner('inner arg')

Where is the python source code behind this magic?

Answer

Great question! The behavior you're describing is actually a feature of Python known as closures. It might seem like magic at first, but it's based on how Python handles nested functions and lexical scoping. Let's break it down step by step and see how the code works.

Key Concepts Involved:

  1. Closure: A closure is a function object that retains references to variables from its enclosing scope, even after the outer function has finished execution.
  2. Function Returning a Function: In Python, functions are first-class objects, meaning you can return them from other functions and assign them to variables.
  3. Lexical Scoping: When you define a function inside another function, the inner function can "see" the arguments and variables of the outer function because of lexical scoping (i.e., the scope is determined by where the function is defined, not where it's called).

How It Works:

Let's break down your code and explain how Python handles this:

def outer(arg):
    def inner(arg2):
        print(arg, ',', arg2)
    return inner

# Call the outer function with 'outer arg'
a = outer('outer arg')

# Print `a`
print(a)  # <function outer.<locals>.inner at 0x109bd0048>

# Now call `a`, which is the `inner` function returned by `outer`
a('inner arg')  # Output: outer arg, inner arg

Detailed Explanation:

  1. outer('outer arg'):

    • When you call outer('outer arg'), Python executes the outer function.
    • Inside outer, the inner function is defined. inner takes its own argument arg2, but it also has access to arg, which was passed to the outer function.
    • After inner is defined, outer returns the inner function, not its result, but the actual function object itself.
    • The returned function is assigned to the variable a. So, a is now referencing the inner function, with access to arg from outer.

  2. Why a('inner arg') works:

    • When you call a('inner arg'), you're actually calling the inner function that was returned by outer.
    • inner has access to its own argument arg2, but more importantly, it also retains access to the arg variable from outer through the closure mechanism. This is where the magic happens.
    • The print statement inside inner prints both arg (from the enclosing outer scope) and arg2 (passed to inner), which results in the output: outer arg, inner arg.

  3. Why does print(a) give <function outer.<locals>.inner at 0x109bd0048>?:

    • a is now a reference to the inner function. It’s a function object that resides in the local scope of outer. This is why the repr (representation) of a looks like outer.<locals>.inner, indicating that it's the inner function inside outer.
    • The hexadecimal number (0x109bd0048) is simply the memory address where the function is stored, which you see in the output.

What about this code?

a = outer('outer arg') inner('inner arg')

This is not valid syntax in Python. In Python, functions are called with parentheses, so a = outer('outer arg') inner('inner arg') would lead to a TypeError because the interpreter expects inner('inner arg') to be a function call, but the way it's written suggests you're trying to apply two functions in a way that isn't syntactically correct.

The Magic Behind It:

The "magic" happens because closures allow the inner function (inner) to "remember" its lexical environment, which includes the variables from the enclosing scope (outer in this case). When you assign the inner function to a, you are effectively getting a function that is bound to its lexical scope (i.e., arg from outer). This is implicit in how closures work, but it’s not exactly "magical" — it's a well-defined feature of Python's function handling.

Where's the Source Code Behind This?

If you're looking for the Python implementation details, the logic of closures is built into the core of Python. It's part of how Python handles function objects, scoping, and environments. The source code related to closures can be found in the Python interpreter's function handling and evaluation code, primarily in the ceval.c file in CPython (the standard Python implementation).

To see the exact details, you can look at the CPython GitHub repository or explore the Python documentation on closures.

To Summarize:

  • Python closures: Inner functions have access to variables in their enclosing scope, even after the outer function has returned.
  • a('inner arg'): When you assign a = outer('outer arg'), a becomes a reference to the inner function. Calling a('inner arg') is effectively calling inner('inner arg'), where inner has access to arg from the outer scope (via closure).
  • The magic: The "magic" is really just the closure concept, where the inner function retains access to its surrounding lexical environment.

I hope that clears up the confusion! Let me know if you need more details.

#py