How Session.run() chooses which sub-graph to run

ghz 14hours ago ⋅ 8 views

I was trying to understand how Session.run() works in Tensorflow flow. I know that Session.run() runs the sub-graph specified by the "fetch" argument we give it. Since depending on which part of the sub-graph is executed first we might get different results, I was trying to see if that is really the case. Suppose we compare the output of this code:

import tensorflow as tf

x = tf.Variable(42)
assign1 = tf.assign(x, 13)
assign2 = tf.assign(x, 14)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    val1, val2, val3 = sess.run([x, assign1, assign2])
    print(val1, val2, val3)

with this code:

import tensorflow as tf

x = tf.Variable(42)
assign2 = tf.assign(x, 14)
assign1 = tf.assign(x, 13)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    val1, val2, val3 = sess.run([x, assign1, assign2])
    print(val1, val2, val3)

(The only change is assigning of 14 comes first in the second code). I was expecting to see that the first code produces output 14, 14, 14 while the second one produced 13, 13, 13.

However, the result is that the first produced 13, 13, 13 while the second on produced 14, 14, 14.

Why does this happen?

Update: Following Chosen Answer:

I don't quite get what is independent and what is not. For example, in the following code:

x = tf.Variable([1, 2, 3, 4, 5])

def foo():
    tmp_list = []
    assign = tf.assign(x[4], 100)
    for i in range(0, 5):
        tmp_list.append(x[i])
    return tmp_list

z = foo()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    val1, val2 = sess.run([z, assign])
    print(val1, val2)

In my Session.run() I have both z and assign. To get z, I need foo(). When foo() is ran, assign = tf.assign(x[4], 100) is also ran. However, the output is [1, 2, 3, 4, 5] 13 and I have no idea where the 13 comes from.

Answer

The behavior you're observing is a result of how TensorFlow's computational graph and Session.run() work. Let's break this down step by step:

First Example:

Why does sess.run([x, assign1, assign2]) produce 13, 13, 13 in the first case and 14, 14, 14 in the second?

  1. Order of Execution in TensorFlow:

    • TensorFlow determines the order of operations in the graph based on dependencies, not the order in which you define operations.
    • When Session.run() is called, it evaluates all requested tensors/operations (x, assign1, and assign2) and ensures dependencies are resolved for each one.

  2. Independent Variables:

    • If the requested tensors/operations (like assign1 and assign2) have no dependency on each other, TensorFlow may execute them in any order.

  3. What Happens in Your Code:

    • First Code:

      assign1 = tf.assign(x, 13)
assign2 = tf.assign(x, 14)
val1, val2, val3 = sess.run([x, assign1, assign2])
      • assign1 sets x to 13.
      • assign2 sets x to 14.
      • x is fetched first, and since assign2 hasn't run yet, it still holds the initial value 13 set by assign1.

    • Second Code:

      assign2 = tf.assign(x, 14)
assign1 = tf.assign(x, 13)
val1, val2, val3 = sess.run([x, assign1, assign2])
      • In this case, assign2 (which sets x to 14) runs before assign1, so x reflects the value 14 when fetched.

    • The difference arises from which assignment operation TensorFlow schedules first when resolving dependencies.

Second Example:

Why does the output from the second code produce [1, 2, 3, 4, 5] for z and 13 for assign?

  1. Graph Dependencies:

    assign = tf.assign(x[4], 100)
tmp_list.append(x[i])
    • assign modifies only the element x[4]. However, the computation of tmp_list (via foo) only involves reading the values of x. There is no direct dependency between tmp_list and assign, so TensorFlow can evaluate them independently.

  2. Order of Execution:

    • When you call sess.run([z, assign]):
      • z corresponds to foo(), which involves reading the current values of x. It does not depend on assign.
      • Since TensorFlow evaluates z before assign, z is calculated from the initial values of x, which are [1, 2, 3, 4, 5].

  3. Why the 13?

    • This might be a typo or inconsistency in the output. Assuming the operation assign = tf.assign(x[4], 100) is correct, the output for assign should logically be 100. Ensure you're running the exact code without unintended modifications.

Key Takeaways:

  1. Order of Execution:

    • TensorFlow schedules operations based on dependencies, not the order of definition.
    • If two operations are independent, their execution order is not guaranteed and may vary.

  2. Session.run Fetch Behavior:

    • When fetching multiple tensors/operations, their evaluation is resolved based on the graph structure.
    • Fetching a variable (e.g., x) happens before any independent assignments.

  3. Dependent vs. Independent Operations:

    • If you want to enforce a specific order, introduce explicit dependencies using tf.control_dependencies().

Correct Example with Dependency Control:

If you want to ensure assign happens before evaluating z:

x = tf.Variable([1, 2, 3, 4, 5])

def foo():
    tmp_list = []
    for i in range(5):
        tmp_list.append(x[i])
    return tmp_list

assign = tf.assign(x[4], 100)
z = foo()

with tf.control_dependencies([assign]):
    z_with_assign = tf.identity(z)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    val1 = sess.run(z_with_assign)
    print(val1)  # Ensures z reflects the assignment

Output: [1, 2, 3, 4, 100].

Let me know if further clarification is needed!

#py