Register
Home   Ask Question Discuss Purchase Submit Events Give Solutions Share News
Automobiles Beauty & Styles Business Computers & Internet Sports Travel Consumer Electronics Education Entertainment & Music Family Society & Culture Food & Drinks Health
Computer Networking
Hardware
Internet
Programming & Design
Security
Software
Other - Computers
Script to split large log txt file into multiple files
-->

In a recent development work for one of my client I had to analyze a large log file. Client wanted to split the log files in multiple log files and wanted some 400 lines in each file. To achieve that I found/wrote a powershell script to split the large log text fie into multiple files. I am sharing the powershell script below.

Copy the code in a notepad and rename the extension of the file from .txt to .ps1 and then open the powershell and execute the script. It will ask you certain inputs are per your requirement. Once you do that you will see file getting split into multiple file.

#############################################
# Split a log/text file into smaller chunks #
#############################################
#
# WARNING: This will take a long while with extremely large files and uses lots of memory to stage the file
#
 
# Set the baseline counters
#
# Set the line counter to 0
$linecount = 0
# Set the file counter to 1. This is used for the naming of the log files
$filenumber = 1
 
# Prompt user for the path
$sourcefilename = Read-Host "What is the full path and name of the log file to split? (e.g. D:\mylogfiles\mylog.txt)"
 
# Prompt user for the destination folder to create the chunk files
$destinationfolderpath = Read-Host "What is the path where you want to extract the content? (e.g. d:\yourpath\)"
 
Write-Host "Please wait while the line count is calculated. This may take a while. No really, it could take a long time."
 
# Find the current line count to present to the user before asking the new line count for chunk files
Get-Content $sourcefilename | Measure-Object | ForEach-Object { $sourcelinecount = $_.Count }
 
#Tell the user how large the current file is
Write-Host "Your current file size is $sourcelinecount lines long"
 
# Prompt user for the size of the new chunk files
$destinationfilesize = Read-Host "How many lines will be in each new split file?"
 
# the new size is a string, so we convert to integer and up
# Set the upper boundary (maximum line count to write to each file)
$maxsize = [int]$destinationfilesize 
 
Write-Host File is $sourcefilename - destination is $destinationfolderpath - new file line count will be $destinationfilesize
 
# The process reads each line of the source file, writes it to the target log file and increments the line counter. When it reaches 100000 (approximately 50 MB of text data)
$content = get-content $sourcefilename | % {
 Add-Content $destinationfolderpath\splitlog$filenumber.txt "$_"
  $linecount ++
  If ($linecount -eq $maxsize) {
    $filenumber++
    $linecount = 0
  }
}
 
# Clean up after your pet
[gc]::collect() 
[gc]::WaitForPendingFinalizers()

--
Bookmark and Share
Related Posts
Comments
Add Comment
Name*
Email*
Comment*
Hot Categories: Security Tennis Other - Electronics Marriage & Divorce Music Jokes & Riddles Makeup Airlines
Contact Us About Us Privacy Policy Copyright Sitemap