THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

Linchi Shea

Checking out SQL Server via empirical data points

Add-Content and Out-File are not for performance

When you write Powershell scripts and need to write a text file, you have a number of ways to accomplish that. The often suggested approach is to use cmdlets Add-Content or Out-File. Well, this is not news.

 

But some may not notice that these cmdlets are not there for performance. Sure, they are convenient. But if you need to write to a file many times, you may want to consider something else. To demonstrate the point, try the following script, which tests the performance of writing to a text file 10,000 times:

 

# Measure Add-Content

$t = get-date;

for ($a=1; $a -le 10000; $a++)

{

   add-content -path "c:\junk\test1.txt" -value "Hello";

}

$s = $(get-date).subtract($t).TotalMilliseconds;

write-output ("Add-Content:  " + $s);

 

# Measure Out-File

$t = get-date;

for ($a=1; $a -le 10000; $a++)

{

   "Hello" | out-file -filepath "c:\junk\test2.txt" -append

}

$s = $(get-date).subtract($t).TotalMilliseconds;

write-output ("Out-File:     " + $s);

 

 

# Measure StreamWriter

$t = get-date;

$sw = New-Object System.IO.StreamWriter "c:\junk\test3.txt";

for ($a=1; $a -le 10000; $a++)

{

    $sw.WriteLine("Hello")

}

$sw.Close();

$s = $(get-date).subtract($t).TotalMilliseconds;

write-output ("StreamWriter: " + $s);

 

The output from this script is similar to what I obtained from my laptop and shown below:

 

Add-Content:  34015.8427

Out-File:     37781.4918

StreamWriter: 781.255

 

From running this scripts multiple times on my laptop, using StreamWriter was consistently about 4 ~ 5 times faster than either Add-Content or Out-File. This should not really be surprising because each time you call Add-Content or Out-File, you open and close the file, and that is expensive.

 

Performance wise, using Add-Content or Out-File is similar to using StreamWriter in the following way (i.e. including opening and closing the file inside the loop):

 

# Measure StreamWriter, open/close the file for every write

$t = get-date;

for ($a=1; $a -le 10000; $a++)

{

    $sw = New-Object -typename System.IO.StreamWriter("c:\junk\test4.txt", "true");

    $sw.WriteLine("Hello")

    $sw.Close();

}

$s = $(get-date).subtract($t).TotalMilliseconds;

write-output ("StreamWriter2:" + $s);

 

And if you run the above script, you should find its performance very similar to that of Add-Content or that of Out-File.

 

So what is the takeaway? Well, do not use Add-Content or Out-File if performance matters such as when you write a massive number of lines to a file.

 

Published Monday, January 04, 2010 5:31 PM by Linchi Shea

Comments

 

Nick Beagley said:

That's very handy to know and quiet timely. Thanks for the simple example.

January 6, 2010 9:02 PM
 

Peter said:

That came in really, really handy for something I was just doing. I can't believe the time difference. Thanks for sharing.

March 16, 2010 7:43 PM
 

Lonely programmer said:

You are the man!

I could bring down the creation time of a file from 14 minutes (!) to less than 10 seconds.

Great!

February 14, 2013 4:09 AM
 

Tim said:

Thank you, this is exactly the information that I needed!

September 16, 2013 3:15 PM
 

Chris said:

Brilliant!  I saw a 20x speed improvement with StreamWriter

May 27, 2015 10:00 AM
 

Jim Hark said:

You wrote

using StreamWriter was consistently about 4 ~ 5 times faster

But you meant (based on the numbers you provided and on my testing):

using StreamWriter was consistently about 40 ~ 50 times faster.

I just tested PS 5.0 on Windows 10 and for me the difference is even grater:

Add-Content:  45665.3372

Out-File:     59961.4252

StreamWriter: 164.8758

PS> 45665.3372 / 164.8758

276.968100837115

So StreamWriter is about 277 times faster than Add-Content!

Jim

September 21, 2015 8:34 PM
New Comments to this post are disabled

About Linchi Shea

Checking out SQL Server via empirical data points

This Blog

Syndication

Privacy Statement