Question
I'm having a little trouble understanding the pass-by-reference properties of
data.table. Some operations seem to 'break' the reference, and I'd like to
understand exactly what's happening.
On creating a data.table from another data.table (via <-, then updating
the new table by :=, the original table is also altered. This is expected,
as per:
?data.table::copy and [stackoverflow: pass-by-reference-the-operator-in-the-
data-table-package](https://stackoverflow.com/questions/8030452/pass-by-
reference-the-operator-in-the-data-table-package)
Here's an example:
library(data.table)
DT <- data.table(a=c(1,2), b=c(11,12))
print(DT)
# a b
# [1,] 1 11
# [2,] 2 12
newDT <- DT # reference, not copy
newDT[1, a := 100] # modify new DT
print(DT) # DT is modified too.
# a b
# [1,] 100 11
# [2,] 2 12
However, if I insert a non-:= based modification between the <- assignment
and the := lines above, DT is now no longer modified:
DT = data.table(a=c(1,2), b=c(11,12))
newDT <- DT
newDT$b[2] <- 200 # new operation
newDT[1, a := 100]
print(DT)
# a b
# [1,] 1 11
# [2,] 2 12
So it seems that the newDT$b[2] <- 200 line somehow 'breaks' the reference.
I'd guess that this invokes a copy somehow, but I would like to understand
fully how R is treating these operations, to ensure I don't introduce
potential bugs in my code.
I'd very much appreciate if someone could explain this to me.
Answer
Yes, it's subassignment in R using <- (or = or ->) that makes a copy of
the whole object. You can trace that using tracemem(DT) and
.Internal(inspect(DT)), as below. The data.table features := and set()
assign by reference to whatever object they are passed. So if that object was
previously copied (by a subassigning <- or an explicit copy(DT)) then it's
the copy that gets modified by reference.
DT <- data.table(a = c(1, 2), b = c(11, 12))
newDT <- DT
.Internal(inspect(DT))
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
# @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,12
# ATTRIB: # ..snip..
.Internal(inspect(newDT)) # precisely the same object at this point
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
# @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,12
# ATTRIB: # ..snip..
tracemem(newDT)
# [1] "<0x0000000003b7e2a0"
newDT$b[2] <- 200
# tracemem[0000000003B7E2A0 -> 00000000040ED948]:
# tracemem[00000000040ED948 -> 00000000040ED830]: .Call copy $<-.data.table $<-
.Internal(inspect(DT))
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),TR,ATT] (len=2, tl=100)
# @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,12
# ATTRIB: # ..snip..
.Internal(inspect(newDT))
# @0000000003D97A58 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
# @00000000040ED7F8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040ED8D8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,200
# ATTRIB: # ..snip..
Notice how even the a vector was copied (different hex value indicates new
copy of vector), even though a wasn't changed. Even the whole of b was
copied, rather than just changing the elements that need to be changed. That's
important to avoid for large data, and why := and set() were introduced to
data.table.
Now, with our copied newDT we can modify it by reference :
newDT
# a b
# [1,] 1 11
# [2,] 2 200
newDT[2, b := 400]
# a b # See FAQ 2.21 for why this prints newDT
# [1,] 1 11
# [2,] 2 400
.Internal(inspect(newDT))
# @0000000003D97A58 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
# @00000000040ED7F8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040ED8D8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,400
# ATTRIB: # ..snip ..
Notice that all 3 hex values (the vector of column points, and each of the 2 columns) remain unchanged. So it was truly modified by reference with no copies at all.
Or, we can modify the original DT by reference :
DT[2, b := 600]
# a b
# [1,] 1 11
# [2,] 2 600
.Internal(inspect(DT))
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
# @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,600
# ATTRIB: # ..snip..
Those hex values are the same as the original values we saw for DT above.
Type example(copy) for more examples using tracemem and comparison to
data.frame.
Btw, if you tracemem(DT) then DT[2,b:=600] you'll see one copy reported.
That is a copy of the first 10 rows that the print method does. When wrapped
with invisible() or when called within a function or script, the print
method isn't called.
All this applies inside functions too; i.e., := and set() do not copy on
write, even within functions. If you need to modify a local copy, then call
x=copy(x) at the start of the function. But, remember data.table is for
large data (as well as faster programming advantages for small data). We
deliberately don't want to copy large objects (ever). As a result we don't
need to allow for the usual 3* working memory factor rule of thumb. We try to
only need working memory as large as one column (i.e. a working memory factor
of 1/ncol rather than 3).
