Read file line by line in PowerShell

Question

I want to read a file line by line in PowerShell. Specifically, I want to loop through the file, store each line in a variable in the loop, and do some processing on the line.

I know the Bash equivalent:

while read line do
    if [[ $line =~ $regex ]]; then
          # work here
    fi
done < file.txt

Not much documentation on PowerShell loops.

Answer 1

Get-Content has bad performance; it tries to read the file into memory all at once.

C# (.NET) file reader reads each line one by one

Best Performace

foreach($line in [System.IO.File]::ReadLines("C:\path\to\file.txt"))
{
       $line
}

Or slightly less performant

[System.IO.File]::ReadLines("C:\path\to\file.txt") | ForEach-Object {
       $_
}

The foreach statement will likely be slightly faster than ForEach-Object (see comments below for more information).

Answer 2

Reading Large Files Line by Line

Original Comment (1/2021) I was able to read a 4GB log file in about 50 seconds with the following. You may be able to make it faster by loading it as a C# assembly dynamically using PowerShell.

[System.IO.StreamReader]$sr = [System.IO.File]::Open($file, [System.IO.FileMode]::Open)
while (-not $sr.EndOfStream){
    $line = $sr.ReadLine()
}
$sr.Close() 

Addendum (3/2022) Processing the large file using C# embedded in PowerShell is even faster and has less "gotchas".

$code = @"
using System;
using System.IO;

namespace ProcessLargeFile
{
    public class Program
    {
        static void ProcessLine(string line)
        {
            return;
        }

        public static void ProcessLogFile(string path) {
            var start_time = DateTime.Now;
            StreamReader sr = new StreamReader(File.Open(path, FileMode.Open));
            try {
                while (!sr.EndOfStream){
                    string line = sr.ReadLine();
                    ProcessLine(line);
                }
            } finally {
                sr.Close();
            }
            var end_time = DateTime.Now;
            var run_time = end_time - start_time;
            string msg = "Completed in " + run_time.Minutes + ":" + run_time.Seconds + "." + run_time.Milliseconds;
            Console.WriteLine(msg);
        }

        static void Main(string[] args)
        {
            ProcessLogFile("c:\\users\\tasaif\\fake.log");
            Console.ReadLine();
        }
    }
}
"@
 
Add-Type -TypeDefinition $code -Language CSharp

PS C:\Users\tasaif> [ProcessLargeFile.Program]::ProcessLogFile("c:\\users\\tasaif\\fake.log")
Completed in 0:17.109

Answer 3

The almighty switch works well here:

'one
two
three' > file

$regex = '^t'

switch -regex -file file { 
  $regex { "line is $_" } 
}

Output:

line is two
line is three