TopolearnUnexpected copies explain most memory blowups and slow loops in R.
In R, names point to objects; assignment y <- x doesn't copy, it binds a second name to the same object. R tracks how many references an object has (since R 4.0 with reference counting). When you modify an object that has more than one reference, R copies it first: this is copy-on-modify, which gives R value semantics without copying on every assignment.
If an object has a single reference, R can modify it in place. That's why a well-written loop updating out[i] on a preallocated vector is fast. But anything that adds a reference, such as passing the object to a regular function or storing it in a list, can trigger a full copy on the next modification.
Tools to see what's happening:
tracemem(x)prints a message wheneverxis copied (needs R compiled with memory profiling, which CRAN builds are).lobstr::obj_size(x)gives a size that accounts for sharing;object.sizedoes not.lobstr::ref(x)shows which objects a list or environment references.
Lists and data frames copy shallowly: modifying one column of a data frame copies the list of column pointers and the one column, not every column. Character vectors use a global string pool, so repeated strings are stored once.
Environments are the exception: they're never copied, so they're how you get reference semantics. data.table uses in-place modification by reference (:=) to avoid copies on large tables.
Example
x <- c(1, 2, 3)
y <- x # no copy: two names, one object
cat(tracemem(x) != "", "\n") #> TRUE (tracing on)
y[1] <- 10 #> tracemem[0x... -> 0x...]: copy happens now
untracemem(x)
print(x) #> [1] 1 2 3 (x unchanged)
df <- data.frame(a = runif(1e6), b = runif(1e6))
df2 <- df
df2$b <- 0 # shallow copy: column a still shared
print(identical(df$a, df2$a)) #> [1] TRUE
big <- runif(1e6)
lst <- list(big, big, big)
print(format(object.size(lst), units = "MB")) #> [1] "22.9 Mb" (ignores sharing)
e <- new.env(); e$v <- 1
modify <- function(env) env$v <- 99
modify(e)
print(e$v) #> [1] 99 (reference semantics)Remember
- Assignment shares; modification copies only when shared
- Objects with a single reference can be modified in place
- Data frames and lists copy shallowly, column by column
- Environments (and data.table's
:=) give reference semantics
Watch out for
- Trusting object.size() for objects that share data; use lobstr::obj_size().