SCRIPT LIBRARY · POWERSHELL
Keep One Copy of Each Game in Your ROM Backups, by Region
Group the USA, Europe, Japan and revised versions of each game in a folder of backups, keep the one you'd actually play, and move the rest to a _duplicates folder.
- What it does
- Reads game file names in the "Title (Region) (Rev x) [flags]" style, groups the versions of each game, keeps one by your region order and the highest revision, and moves the others into a _duplicates folder (or deletes them with -Delete).
- Requires
- Windows PowerShell 5.1 or PowerShell 7+
- No modules
- Files named in the No-Intro, Redump or GoodTools style
- Permissions
- Read and write access to the folders you point it at. No admin rights.
- Runs on
- Windows 10/11, macOS or Linux with PowerShell 7
- Tested
- Parse-checked and run in PowerShell 7.4 against folders of dummy files, with -WhatIf, for real, with -Recurse, -Delete, a custom -PreferRegion order, multi-disc sets, a translation and a bad dump
Part 4 of the thread Keeping a media library tidy
Dump your own cartridges for long enough, or back up a few discs in more than one region, and a system folder turns into a wall of near-twins. The USA release, the European one, the Japanese one, a Rev 1 that fixed a save bug, a [b1] that never booted in the first place. Your frontend lists every one of them, and you scroll past four copies to get to the game you wanted.
This started as a handful of one-off scripts for my own library, one per system, each with a hard-coded folder and its own idea of what a duplicate was. This is the general version. It reads the tags in the file name, groups the versions of each game, and keeps one: clean dumps first, then your region order, then the highest revision.
It doesn't delete anything unless you ask. The extras go into a _duplicates folder next to the originals, so if it picked wrong, you drag the file back.
<#
.SYNOPSIS
Keeps one copy of each game in a folder of backups and moves the other versions aside.
.DESCRIPTION
Reads file names in the usual "Title (Region) (Rev x) [flags]" style (No-Intro, Redump and most
GoodTools sets), groups the versions of the same game, and picks one to keep:
1. Clean dumps before bad dumps, betas, protos, demos and anything tagged with -Deprioritize
2. Your preferred region, in the order you give with -PreferRegion
3. The highest revision (Rev 2 beats Rev 1 beats no revision; v1.1 beats v1.0)
4. Verified [!] dumps, then the name with the fewest extra tags
Multi-disc sets (Disc 1, Disc 2) and translations stay in their own groups, so they're never
treated as duplicates of each other. The extras are moved to a _duplicates folder inside each
folder unless you ask for -Delete. Supports -WhatIf.
.PARAMETER Path
One or more folders of game files. Defaults to the current folder.
.PARAMETER Extension
File extensions to consider. Default: every file.
.PARAMETER Recurse
Also process subfolders. Each folder is deduplicated on its own.
.PARAMETER PreferRegion
Regions in order of preference. Default: USA, World, Europe, Japan.
.PARAMETER Deprioritize
Tags that make a version lose to any clean release. Default: Beta, Proto, Demo, Sample, Pirate, Hack.
.PARAMETER DuplicateFolder
Name of the folder the extras are moved into, created inside each folder. Default: _duplicates.
.PARAMETER Delete
Delete the extras instead of moving them. Honors -WhatIf and -Confirm.
.EXAMPLE
.\Remove-DuplicateRom.ps1 -Path D:\Games\snes -WhatIf
.EXAMPLE
.\Remove-DuplicateRom.ps1 -Path D:\Games -Recurse -PreferRegion Europe, World, USA | Export-Csv dedupe.csv -NoTypeInformation
#>
[CmdletBinding(SupportsShouldProcess, ConfirmImpact = 'Medium')]
param(
[Parameter(ValueFromPipeline, ValueFromPipelineByPropertyName)]
[Alias('FullName')]
[string[]]$Path = '.',
[string[]]$Extension,
[switch]$Recurse,
[ValidateNotNullOrEmpty()][string[]]$PreferRegion = @('USA', 'World', 'Europe', 'Japan'),
[string[]]$Deprioritize = @('Beta', 'Proto', 'Demo', 'Sample', 'Pirate', 'Hack'),
[ValidatePattern('^[^\\/:*?"<>|]+$')][string]$DuplicateFolder = '_duplicates',
[switch]$Delete
)
begin {
# GoodTools-style one-letter codes and a few common short forms, mapped to No-Intro region names.
$regionAlias = @{
'U' = 'USA'; 'US' = 'USA'; 'E' = 'Europe'; 'EU' = 'Europe'; 'J' = 'Japan'; 'JP' = 'Japan'; 'W' = 'World'; 'WW' = 'World'
'UE' = 'USA,Europe'; 'JU' = 'Japan,USA'; 'JUE' = 'World'; 'A' = 'Australia'; 'B' = 'Brazil'; 'K' = 'Korea'; 'F' = 'France'; 'G' = 'Germany'
}
$knownRegions = 'USA', 'Europe', 'Japan', 'World', 'Asia', 'Australia', 'Brazil', 'Canada', 'China', 'France', 'Germany', 'Hong Kong', 'Italy', 'Korea', 'Netherlands', 'Spain', 'Sweden', 'Taiwan', 'UK', 'Russia', 'Scandinavia', 'Latin America'
$extFilter = @($Extension | Where-Object { $_ } | ForEach-Object { if ($_ -like '.*') { $_.ToLower() } else { ".$_".ToLower() } })
function Get-RomInfo([IO.FileInfo]$File) {
$base = [IO.Path]::GetFileNameWithoutExtension($File.Name)
$tags = @([regex]::Matches($base, '\(([^\)]+)\)') | ForEach-Object { $_.Groups[1].Value.Trim() })
$flags = @([regex]::Matches($base, '\[([^\]]+)\]') | ForEach-Object { $_.Groups[1].Value.Trim() })
$title = ($base -replace '\s*[\(\[].*$', '').Trim()
if (-not $title) { $title = $base }
$regions = [System.Collections.Generic.List[string]]::new()
$revision = [version]'0.0'
$verified = $false; $penalty = 0; $groupExtra = @()
# Round-bracket tags: region, revision, disc number, beta/proto and so on.
foreach ($tag in $tags) {
$parts = $tag -split '\s*,\s*'
if ($regionAlias.ContainsKey($tag)) { $regionAlias[$tag] -split ',' | ForEach-Object { $regions.Add($_) } }
elseif (@($parts | Where-Object { $knownRegions -contains $_ }).Count -eq $parts.Count) { $parts | ForEach-Object { $regions.Add($_) } }
elseif ($tag -match '^Rev\s*([0-9]+|[A-Z])$') {
$n = $Matches[1]
$revision = [version]::new($(if ($n -match '^\d+$') { [int]$n } else { [int][char]$n.ToUpper() - 64 }), 0)
}
elseif ($tag -match '^v(\d+(\.\d+){1,3})$') { $revision = [version]$Matches[1] }
elseif ($tag -match '^(Disc|Disk|Side|Tape)\s*\w+' -or $tag -match 'Translat') { $groupExtra += $tag }
else { foreach ($word in $Deprioritize) { if ($tag -match "^$([regex]::Escape($word))\b") { $penalty = 1 } } }
}
# Square-bracket dump flags: [!] verified, [b] bad, [o] overdump, [h] hack, [p] pirate, [T+Eng] translation.
foreach ($flag in $flags) {
if ($flag -eq '!') { $verified = $true }
elseif ($flag -cmatch '^[bohp]\d*') { $penalty = 1 }
elseif ($flag -cmatch '^T[+-]') { $groupExtra += $flag }
}
$rank = $PreferRegion.Count
for ($i = 0; $i -lt $rank; $i++) { if ($regions -contains $PreferRegion[$i]) { $rank = $i } }
# Compare titles loosely so "Title - Subtitle" and "Title_ Subtitle" still match.
$key = (($title.ToLowerInvariant() -replace '[^a-z0-9]+', ' ').Trim()) + '|' + (($groupExtra | ForEach-Object { $_.ToLowerInvariant() }) -join '|')
[pscustomobject]@{
File = $File; Key = $key; Title = $title; Region = ($regions | Select-Object -Unique) -join ', '
RegionRank = $rank; Revision = $revision; Verified = $verified; Penalty = $penalty; TagCount = $tags.Count + $flags.Count
}
}
}
process {
foreach ($root in $Path) {
if (-not (Test-Path -LiteralPath $root -PathType Container)) { Write-Warning "Not a folder: $root"; continue }
$folders = @(Get-Item -LiteralPath $root)
if ($Recurse) { $folders += Get-ChildItem -LiteralPath $root -Directory -Recurse | Where-Object { $_.Name -ne $DuplicateFolder -and $_.FullName -notlike "*\$DuplicateFolder\*" -and $_.FullName -notlike "*/$DuplicateFolder/*" } }
foreach ($folder in $folders) {
$files = Get-ChildItem -LiteralPath $folder.FullName -File | Where-Object { -not $extFilter -or $extFilter -contains $_.Extension.ToLower() }
$groups = $files | ForEach-Object { Get-RomInfo $_ } | Group-Object Key | Where-Object Count -gt 1
foreach ($group in $groups) {
$ranked = @($group.Group | Sort-Object Penalty, RegionRank, @{ Expression = 'Revision'; Descending = $true }, @{ Expression = 'Verified'; Descending = $true }, TagCount, { $_.File.Name })
$keep = $ranked[0]
[pscustomobject]@{ Folder = $folder.FullName; Title = $keep.Title; Action = 'Keep'; Name = $keep.File.Name; Region = $keep.Region; Revision = $keep.Revision; Destination = $null; Status = 'Kept' }
foreach ($extra in $ranked | Select-Object -Skip 1) {
$result = [ordered]@{ Folder = $folder.FullName; Title = $keep.Title; Action = $(if ($Delete) { 'Delete' } else { 'Move' }); Name = $extra.File.Name; Region = $extra.Region; Revision = $extra.Revision; Destination = $null; Status = 'WhatIf' }
try {
if ($Delete) {
if ($PSCmdlet.ShouldProcess($extra.File.FullName, "Delete (keeping $($keep.File.Name))")) {
Remove-Item -LiteralPath $extra.File.FullName -Force -ErrorAction Stop
$result.Status = 'Deleted'
}
}
else {
$destDir = Join-Path $folder.FullName $DuplicateFolder
$dest = Join-Path $destDir $extra.File.Name
$n = 2
while (Test-Path -LiteralPath $dest) { $dest = Join-Path $destDir ('{0} ({1}){2}' -f $extra.File.BaseName, $n++, $extra.File.Extension) }
$result.Destination = $dest
if ($PSCmdlet.ShouldProcess($extra.File.FullName, "Move to $destDir (keeping $($keep.File.Name))")) {
if (-not (Test-Path -LiteralPath $destDir)) { $null = New-Item -ItemType Directory -Path $destDir -ErrorAction Stop }
Move-Item -LiteralPath $extra.File.FullName -Destination $dest -ErrorAction Stop
$result.Status = 'Moved'
}
}
}
catch {
$result.Status = "Failed: $($_.Exception.Message)"
Write-Warning "$($extra.File.Name): $($_.Exception.Message)"
}
[pscustomobject]$result
}
}
}
}
}
Parameters
| Parameter | Type | Default | What it's for |
|---|---|---|---|
-Path | string[] | . | One or more folders of game files. Takes pipeline input. |
-Extension | string[] | every file | Only look at these extensions, like .zip, .7z or .chd. Handy when a folder also holds saves or artwork. |
-Recurse | switch | — | Process subfolders too. Each folder is deduplicated on its own, so one run can cover every system. |
-PreferRegion | string[] | USA, World, Europe, Japan | Region order, best first. A file tagged (USA, Europe) counts as whichever of those ranks higher. |
-Deprioritize | string[] | Beta, Proto, Demo, Sample, Pirate, Hack | Tags that make a version lose to any clean release, whatever its region. |
-DuplicateFolder | string | _duplicates | Folder the extras are moved into, created inside each folder that has any. |
-Delete | switch | — | Delete the extras instead of moving them. Honors -WhatIf and -Confirm. |
Run it
See what it would keep and move, without touching anything.
.\Remove-DuplicateRom.ps1 -Path D:\Games\snes -WhatIfEvery system folder in one go, and a CSV of the decisions to look over later.
.\Remove-DuplicateRom.ps1 -Path D:\Games -Recurse | Export-Csv dedupe-log.csv -NoTypeInformationYou're in Europe and want PAL releases first.
.\Remove-DuplicateRom.ps1 -Path D:\Games\megadrive -PreferRegion Europe, World, USA, JapanOnly the disc images, and delete the extras once you trust it.
.\Remove-DuplicateRom.ps1 -Path D:\Games\psx -Extension .chd -DeleteWhat you'll see
Folder Title Action Name Region Revision Status
------ ----- ------ ---- ------ -------- ------
D:\Games\snes Super Mario World Keep Super Mario World (USA).zip USA 0.0 Kept
D:\Games\snes Super Mario World Move Super Mario World (Europe) (Rev 1).zip Europe 1.0 Moved
D:\Games\snes Super Mario World Move Super Mario World (Japan).zip Japan 0.0 Moved
D:\Games\snes Chrono Trigger Keep Chrono Trigger (USA) (Rev 1).zip USA 1.0 Kept
D:\Games\snes Chrono Trigger Move Chrono Trigger (USA).zip USA 0.0 Moved
D:\Games\snes Chrono Trigger Move Chrono Trigger (USA) (Beta).zip USA 0.0 Moved
D:\Games\snes Pilotwings Keep Pilotwings (USA, Europe).zip USA, Europe 0.0 Kept
D:\Games\snes Pilotwings Move Pilotwings (Japan).zip Japan 0.0 Moved
How it works
- Read the tags. For every file it splits the name into the title (everything before the first bracket), the round-bracket tags and the square-bracket dump flags. Short GoodTools codes like (U), (E) and (JU) are mapped to the No-Intro names, so old and new sets sort the same way.
- Group the versions. The title is lowercased and stripped of punctuation to make the key. Disc numbers and translation tags are added to the key, so they never collide with each other.
- Rank each group. It sorts on clean dump first (anything tagged Beta, Proto, Demo and so on, or flagged [b], [o], [h] or [p], sinks), then the best region in your
-PreferRegionlist, then the highest revision, then a verified [!] dump, then the name with the fewest extra tags. - Keep the winner, move the rest. The top file stays put. The others go to
_duplicatesthroughShouldProcess, so-WhatIfshows exactly what would move. A name clash in_duplicatesgets a (2) on the end instead of overwriting. - Report everything. Each file in a group comes back as an object with the action and status, which makes a CSV of the run easy to keep.
Take it further
- Clean the names first. If your set mixes (U) and (USA) styles, run Rename-RomFile over it before this, so the groups line up.
- Check the duplicates really are duplicates. Two files with different tags can still be byte-for-byte identical. Compare-FileHash will tell you.
- Trim a folder for a handheld. Run it with
-PreferRegion USAinto a copy of your library and you get one playable version per game, which is exactly what a small SD card wants. There's more on that in picking a handheld for emulation.
Things that'll trip you up
- Region beats revision, on purpose. A European Rev 1 loses to a USA original if USA is first in your list. That's usually what you want, since the region decides language and PAL speed. If a particular revision matters to you, move it back from _duplicates by hand.
- Titles have to match to be grouped. It compares the part of the name before the first bracket, ignoring case and punctuation. A Japanese release under its Japanese title won't be grouped with the English one, so it stays. That's a safe failure, but it means a mixed set won't shrink as much as you'd expect.
- Multi-disc games are grouped per disc. Disc 1 and Disc 2 are never duplicates of each other. Each disc picks its own winner, though, so if you only have Disc 2 in one region you can end up with a mixed set. Check multi-disc games before you use -Delete.
- Translations and hacks are handled differently. A [T+Eng] translation stays in its own group, so it's never swapped for the original. A [h] hack, [b] bad dump, [o] overdump or [p] pirate is treated as a duplicate and loses to any clean dump.