Virenk's Blog

A script to measure peak memory usage of a process on Linux shell.

Source : http://stackoverflow.com/questions/1080461/peak-memory-measurement-of-long-running-process-in-linux

#!/bin/bash

ppid=$$
maxmem=0

$@ &
pid=`pgrep -P ${ppid} -n -f $1` # $! may work here but not later
while [[ ${pid} -ne "" ]]; do
    #mem=`ps v | grep "^[ ]*${pid}" | awk '{print $8}'`
        #the previous does not work with MPI
        mem=`cat /proc/${pid}/status | grep VmRSS | awk '{print $2}'`
    if [[ ${mem} -gt ${maxmem} ]]; then
    	maxmem=${mem}
    fi
    sleep 1
    savedpid=${pid}
    pid=`pgrep -P ${ppid} -n -f $1`
done
wait ${savedpid} # don't wait, job is finished
exitstatus=$?   # catch the exit status of wait, the same of $@
echo -e "Memory usage for $@ is: ${maxmem} KB. Exit status: ${exitstatus}\n"

Recently i have been trying to read a large text file (1.8 G) in tcl using following code:

set fn [open “filename.txt” r];

set data [read $fn];

This works well for file sizes < 1 G. but crashes with larger files with core dump.

Another better approach recommended is to use command “file size” along with “read” so as to pre-allocate memory for required buffer. This should be faster than above approach:

set fn [open “filename.txt” r];

set data [read $fn [file size “$filename.txt”] ];

But this still does not read files > 1G.

Solution:

Instead of reading the entire file into buffer, read line by line:

set filename “myfile.xml”
set f [open $filename]
while {[gets $f  line] >= 0} {
# work with $line here …
}
close $f

Ref: http://en.wikibooks.org/wiki/Tcl_Programming/Working_with_files

Tags: , , ,

Debugging a crash point in a program is usually a very tedious job. Life gets even more difficult when you have to wait for hours to reach the crash point. And u fix something and again test if it worked. Imagine trying to debug a test case running for 100 hours! That happen quite often in routing/placement tools in EDA (Electronic Design Automation) domain.

It would be good to have a debugging tool that can help me in restarting my program just before the crash point (call it critical point) (a point where things were working fine).

Here i am assuming  that i have not changed anything in  functions/data objects  in my code, that were used in reaching the critical point of my choice.  Changes are permissible only  in functions/data called after the critical point.

We need to capture the entire state of the program for such a thing to work.  The state would include stack, opened files, files acessed/read/written to disk, network files and so on.  It would be interesting to see explore the possibility of this task. I need to do some R&D to here :P.

To my knowledge there in no tool as of now which can do that.  Imagine how much hours of debugging time will be saved if we had such a tool!

Some useful vim utilities.  Have been using these, thought people might find them useful. Will add more later.

—————————————————————–
#Paste contents of another file in current cursor location.
:r file2.txt
will substitute contents of file2 in current location.

————————————————————————-

#Deleting after a pattern till the end of the line. (including the pattern itself)
:g/pattern/normal nd$

pattern = pattern, normal is the mode
n => at each matching line look for next occurance of pattern matching word.
d => delete
$ => delete till the end of the line

example:
I am doing good.
You should be doing this right now.

:g/doing/normal nd$

will result in : (content after ‘doing’ is removed)
I am
You should be

To retain pattern word, you can do:
:g/doing/normal nwd$

Here, ‘w’ advances the cursor to next word.

—————————————————————–
#Deleting before a pattern till the beginning of the line.

:g/pattern/normal nd^
example:
I am doing good.
You should be doing this right now.

:g/doing/normal nd^

will result in : (content before ‘doing’ is removed)
doing good.
doing this right now.

—————————————————————–
# Map command:
map command can be used to map a sequence of steps into one single key mapping.

Example:
comment a line using map.

:map <F3> 0i/*<ESC>A*/<ESC><enter>

comment would look like:
>>before comment:
hello how are u.

>>after comment:
/*hello how are u.*/

Just place the cursor on the line that needs comment, and then press <F3>

Assuming we are in non-edit mode:
<F3> => the key used for mapping.
0i => 0 moves cursor to beginning of line.
i => start insert mode.
/* => write the beginning of comment
<ESC> the escape key to come out of insert mode.
A => moves cursor to end of line and enables the insert mode.
*/ => put the close of comment.
<ESC> come out of insert mode.
<enter> press enter key

—————————————————————–
#Execute a sequence of commands in a vim script file and apply to any no. of files.
Extremely useful when u have to do same editing on multiple files.

put the commands in script.vim and do
$vim  -u NONE -s script.vim <filename>

Example:
If u have to replace every occurance of ‘India’ with “INDIA’, in file1, then u can put this in script file (script.vim):
:%s/\<India\>/INDIA/g
:wq

and then execute following command in shell:
$vim -u NONE -s script.vim  file1

vim will execute the commands from script.vim and will save and close the file.

To execute same set of operations on multiple files :
Do above editing on all files which have .txt extension.
$ls *.txt | while read f ; do vim -u NONE -s script.vim $f ; done;

—————————————————————–
# Search for exactly 4 digit numbers in the file:
:\<\d\d\d\d\>

—————————————————————–
#Move things in newline
Hello how are u. I am fine . Hope everything is good.
:%s/\./\r\./g

>>will result in:
Hello how are u.
I am fine .
Hope everything is good.

——————————————————————————————-

#Search for exact match of a pattern
/\<world\>
will exclude lines where ‘world’ occurs as a sub string, like myworld.

—————————————————————————-

#Remove all lines which match a given pattern.
:g/pattern/d

#Delete a block of text enclosed in braces.
Example (input file):
(a b c d)
(
test 1
test 2
test 3
)
{ test test
test
test
}
test
{ test
test
test
}

Then:
:s/{/normal d%
will delete all {} braces and data between them.
:s/(/normal d%

will delete () braces and data between them.

% => matches the current open bracket to its matching close bracket.

—————————————————————–
# See the difference between two files in vim
$vim -d file1 file2
GVIM:
Select two (or more files) and select ‘diff with vim’ option.

# Execute a command and see its result.
:!hostname
will print out the name of your computer. <works on both linux and windows>
On windows try using GVIM.

Pls let me know if there are any mistakes in above.

Recently i was debugging a case taking a large amount of memory.  Reason : memory was not freed for some classes which had not been defined with virtual destructors.

While deriving a class from another class, it is important to make destructor of parent class virtual.

Consider following scenario:

class A
{
public:
string sa;
A(string sia);
// ~A();
virtual ~A(); // needed else there will be memory leak

};

class B: public A
{
public:
string sb;
B(string sia, string sib);
~B();

};

A::A(string sia):sa(sia)
{
}

B::B( string sia, string sib):A(sia),sb(sib)
{
}
A::~A() {
cout << “destroying A ” << endl;
}
B::~B() {
cout << “destroying B ” << endl;
}

int main(int argc, char*argv[])
{
string param1=”abcd”;
string param2=”efgh”;
A *b;
b = new B(param1, param2);
delete b; // memory leak if A does not have virtual destructor

B* b1 = new B(param1, param2);
delete b1; //this will automaticall call A’s destructor
return 0;
}

In above scenario, if A’s destructor is not defined as virtual,  “delete b” will  call destructor of A, as it is pointing to object of type A, so contents of B will not be deleted.

If you run valgrind (tool for debugging memory leaks) you will find memory leak for std::string::_Rep::_S_create.

Make sure of above else expect a lot of memory leaks in your program!. :P

Reference: http://www.daniweb.com/forums/thread119201.html

A good short tutorial on how to debug effectively using GDB.

http://betterexplained.com/articles/debugging-with-gdb/

Recently i came across a scenario where a pice of code was running extremely slow (taking approximately 7-8 hours), but with a minor change in algorithm approach it ran in less than a minute!.

Consider a scenario where you have to break a graph into smaller partitions.
s.t each partition must have size <= max_partition_size and cost <= max_partiton_cost.
where:
size = number of nodes in the graph
cost = sum of cost of each node in graph. (assuming each node has a cost associated with it)

Initially let graph (G) has N nodes.

First approach :
1. Cut out a partition P1 from graph (G) s.t P1.size <= max_partition_size, P1.cost <= max_partition_cost.
2. set G to G – P1.  (P1 is cut out and now G has remaining nodes).
3. Repeat above procedure on G.

The complexity of above appraoch is O(n^2). Since max_partiton_size and max_partition_cost are constants, with each iteration, we are processing data of size N, N-K, N-2K, … respectively (K is constant), for which complexity order is O(n^2).

This approach works well for small graph sizes, but for larger ones algorithm will run extremely slow. For example for a graph having 70K nodes, the algorithm approach would take 70K*70K = 4900M steps! Thats huge.

Better appraoch:
For G having large no. of nodes we change the strategy a bit. Instead of diretly forming partitions of required size, we form macro partitions first, and then further split up the macro partitions into partitions of required size recursively.

If G has more than 2000 nodes, we vary
macro_max_partition_cost = 20 * max_partition_cost
macro_max_partiton_size = 20 * max_partition_size

So the approach is :

1. Cut out a partition P1 from graph (G) s.t
if(G.size < 2000) {
P1 is cut out s.t P1.size < max_partition_size and P1.cost < max_partiton_cost
} else {
P1 is cut out s.t P1.size < macro_max_partiton_size and P1.cost < macro_max_partition_cost
}
2. if (P1.size <= max_partition_size && P1.cost <= max_partition_cost) {
//do nothing , as u already have required partition
} else {
Set G = P1 and repeat above procedure.
}
3. Set G = G – P1 (remaining portion of G). Repeat above procedure.

By forming larger cuts, we have reduced the total amount of calculations on data.

Using above appraoch i was able to get a piece of code run in less than a minute which was earlier taking 6-7 hours.

Ohter appraoches :
Binary partitioning , i.e split the graph into two pieces, and keep splitting each piece further until requited partition sizes are obtained.


  • None

Categories

Follow

Get every new post delivered to your Inbox.