Virenk's Blog

Recently i have been trying to read a large text file (1.8 G) in tcl using following code:

set fn [open “filename.txt” r];

set data [read $fn];

This works well for file sizes < 1 G. but crashes with larger files with core dump.

Another better approach recommended is to use command “file size” along with “read” so as to pre-allocate memory for required buffer. This should be faster than above approach:

set fn [open “filename.txt” r];

set data [read $fn [file size “$filename.txt”] ];

But this still does not read files > 1G.

Solution:

Instead of reading the entire file into buffer, read line by line:

set filename “myfile.xml”
set f [open $filename]
while {[gets $f  line] >= 0} {
# work with $line here …
}
close $f

Ref: http://en.wikibooks.org/wiki/Tcl_Programming/Working_with_files