Virenk's Blog

Reading large files in tcl

Posted on: March 15, 2011

Recently i have been trying to read a large text file (1.8 G) in tcl using following code:

set fn [open “filename.txt” r];

set data [read $fn];

This works well for file sizes < 1 G. but crashes with larger files with core dump.

Another better approach recommended is to use command “file size” along with “read” so as to pre-allocate memory for required buffer. This should be faster than above approach:

set fn [open “filename.txt” r];

set data [read $fn [file size “$filename.txt”] ];

But this still does not read files > 1G.

Solution:

Instead of reading the entire file into buffer, read line by line:

set filename “myfile.xml”
set f [open $filename]
while {[gets $f  line] >= 0} {
# work with $line here …
}
close $f

Ref: http://en.wikibooks.org/wiki/Tcl_Programming/Working_with_files

Advertisements
Tags: , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: