Wes Ellis./ a personal notebook
Technology. Stories. Side projects.
A few things worth writing down.
← Back to Script Library

SCRIPT LIBRARY · POWERSHELL

Keep One Copy of Each Game in Your ROM Backups, by Region

Group the USA, Europe, Japan and revised versions of each game in a folder of backups, keep the one you'd actually play, and move the rest to a _duplicates folder.

AT A GLANCERemove-DuplicateRom.ps1
What it does
Reads game file names in the "Title (Region) (Rev x) [flags]" style, groups the versions of each game, keeps one by your region order and the highest revision, and moves the others into a _duplicates folder (or deletes them with -Delete).
Requires
  • Windows PowerShell 5.1 or PowerShell 7+
  • No modules
  • Files named in the No-Intro, Redump or GoodTools style
Permissions
Read and write access to the folders you point it at. No admin rights.
Runs on
Windows 10/11, macOS or Linux with PowerShell 7
Tested
Parse-checked and run in PowerShell 7.4 against folders of dummy files, with -WhatIf, for real, with -Recurse, -Delete, a custom -PreferRegion order, multi-disc sets, a translation and a bad dump

Part 4 of the thread Keeping a media library tidy

Dump your own cartridges for long enough, or back up a few discs in more than one region, and a system folder turns into a wall of near-twins. The USA release, the European one, the Japanese one, a Rev 1 that fixed a save bug, a [b1] that never booted in the first place. Your frontend lists every one of them, and you scroll past four copies to get to the game you wanted.

This started as a handful of one-off scripts for my own library, one per system, each with a hard-coded folder and its own idea of what a duplicate was. This is the general version. It reads the tags in the file name, groups the versions of each game, and keeps one: clean dumps first, then your region order, then the highest revision.

It doesn't delete anything unless you ask. The extras go into a _duplicates folder next to the originals, so if it picked wrong, you drag the file back.

Remove-DuplicateRom.ps1Download
<#
.SYNOPSIS
    Keeps one copy of each game in a folder of backups and moves the other versions aside.
.DESCRIPTION
    Reads file names in the usual "Title (Region) (Rev x) [flags]" style (No-Intro, Redump and most
    GoodTools sets), groups the versions of the same game, and picks one to keep:
      1. Clean dumps before bad dumps, betas, protos, demos and anything tagged with -Deprioritize
      2. Your preferred region, in the order you give with -PreferRegion
      3. The highest revision (Rev 2 beats Rev 1 beats no revision; v1.1 beats v1.0)
      4. Verified [!] dumps, then the name with the fewest extra tags
    Multi-disc sets (Disc 1, Disc 2) and translations stay in their own groups, so they're never
    treated as duplicates of each other. The extras are moved to a _duplicates folder inside each
    folder unless you ask for -Delete. Supports -WhatIf.
.PARAMETER Path
    One or more folders of game files. Defaults to the current folder.
.PARAMETER Extension
    File extensions to consider. Default: every file.
.PARAMETER Recurse
    Also process subfolders. Each folder is deduplicated on its own.
.PARAMETER PreferRegion
    Regions in order of preference. Default: USA, World, Europe, Japan.
.PARAMETER Deprioritize
    Tags that make a version lose to any clean release. Default: Beta, Proto, Demo, Sample, Pirate, Hack.
.PARAMETER DuplicateFolder
    Name of the folder the extras are moved into, created inside each folder. Default: _duplicates.
.PARAMETER Delete
    Delete the extras instead of moving them. Honors -WhatIf and -Confirm.
.EXAMPLE
    .\Remove-DuplicateRom.ps1 -Path D:\Games\snes -WhatIf
.EXAMPLE
    .\Remove-DuplicateRom.ps1 -Path D:\Games -Recurse -PreferRegion Europe, World, USA | Export-Csv dedupe.csv -NoTypeInformation
#>
[CmdletBinding(SupportsShouldProcess, ConfirmImpact = 'Medium')]
param(
    [Parameter(ValueFromPipeline, ValueFromPipelineByPropertyName)]
    [Alias('FullName')]
    [string[]]$Path = '.',
    [string[]]$Extension,
    [switch]$Recurse,
    [ValidateNotNullOrEmpty()][string[]]$PreferRegion = @('USA', 'World', 'Europe', 'Japan'),
    [string[]]$Deprioritize = @('Beta', 'Proto', 'Demo', 'Sample', 'Pirate', 'Hack'),
    [ValidatePattern('^[^\\/:*?"<>|]+$')][string]$DuplicateFolder = '_duplicates',
    [switch]$Delete
)

begin {
    # GoodTools-style one-letter codes and a few common short forms, mapped to No-Intro region names.
    $regionAlias = @{
        'U' = 'USA'; 'US' = 'USA'; 'E' = 'Europe'; 'EU' = 'Europe'; 'J' = 'Japan'; 'JP' = 'Japan'; 'W' = 'World'; 'WW' = 'World'
        'UE' = 'USA,Europe'; 'JU' = 'Japan,USA'; 'JUE' = 'World'; 'A' = 'Australia'; 'B' = 'Brazil'; 'K' = 'Korea'; 'F' = 'France'; 'G' = 'Germany'
    }
    $knownRegions = 'USA', 'Europe', 'Japan', 'World', 'Asia', 'Australia', 'Brazil', 'Canada', 'China', 'France', 'Germany', 'Hong Kong', 'Italy', 'Korea', 'Netherlands', 'Spain', 'Sweden', 'Taiwan', 'UK', 'Russia', 'Scandinavia', 'Latin America'
    $extFilter = @($Extension | Where-Object { $_ } | ForEach-Object { if ($_ -like '.*') { $_.ToLower() } else { ".$_".ToLower() } })

    function Get-RomInfo([IO.FileInfo]$File) {
        $base = [IO.Path]::GetFileNameWithoutExtension($File.Name)
        $tags  = @([regex]::Matches($base, '\(([^\)]+)\)') | ForEach-Object { $_.Groups[1].Value.Trim() })
        $flags = @([regex]::Matches($base, '\[([^\]]+)\]') | ForEach-Object { $_.Groups[1].Value.Trim() })
        $title = ($base -replace '\s*[\(\[].*$', '').Trim()
        if (-not $title) { $title = $base }

        $regions = [System.Collections.Generic.List[string]]::new()
        $revision = [version]'0.0'
        $verified = $false; $penalty = 0; $groupExtra = @()

        # Round-bracket tags: region, revision, disc number, beta/proto and so on.
        foreach ($tag in $tags) {
            $parts = $tag -split '\s*,\s*'
            if ($regionAlias.ContainsKey($tag)) { $regionAlias[$tag] -split ',' | ForEach-Object { $regions.Add($_) } }
            elseif (@($parts | Where-Object { $knownRegions -contains $_ }).Count -eq $parts.Count) { $parts | ForEach-Object { $regions.Add($_) } }
            elseif ($tag -match '^Rev\s*([0-9]+|[A-Z])$') {
                $n = $Matches[1]
                $revision = [version]::new($(if ($n -match '^\d+$') { [int]$n } else { [int][char]$n.ToUpper() - 64 }), 0)
            }
            elseif ($tag -match '^v(\d+(\.\d+){1,3})$') { $revision = [version]$Matches[1] }
            elseif ($tag -match '^(Disc|Disk|Side|Tape)\s*\w+' -or $tag -match 'Translat') { $groupExtra += $tag }
            else { foreach ($word in $Deprioritize) { if ($tag -match "^$([regex]::Escape($word))\b") { $penalty = 1 } } }
        }

        # Square-bracket dump flags: [!] verified, [b] bad, [o] overdump, [h] hack, [p] pirate, [T+Eng] translation.
        foreach ($flag in $flags) {
            if ($flag -eq '!') { $verified = $true }
            elseif ($flag -cmatch '^[bohp]\d*') { $penalty = 1 }
            elseif ($flag -cmatch '^T[+-]') { $groupExtra += $flag }
        }

        $rank = $PreferRegion.Count
        for ($i = 0; $i -lt $rank; $i++) { if ($regions -contains $PreferRegion[$i]) { $rank = $i } }

        # Compare titles loosely so "Title - Subtitle" and "Title_ Subtitle" still match.
        $key = (($title.ToLowerInvariant() -replace '[^a-z0-9]+', ' ').Trim()) + '|' + (($groupExtra | ForEach-Object { $_.ToLowerInvariant() }) -join '|')

        [pscustomobject]@{
            File = $File; Key = $key; Title = $title; Region = ($regions | Select-Object -Unique) -join ', '
            RegionRank = $rank; Revision = $revision; Verified = $verified; Penalty = $penalty; TagCount = $tags.Count + $flags.Count
        }
    }
}

process {
    foreach ($root in $Path) {
        if (-not (Test-Path -LiteralPath $root -PathType Container)) { Write-Warning "Not a folder: $root"; continue }

        $folders = @(Get-Item -LiteralPath $root)
        if ($Recurse) { $folders += Get-ChildItem -LiteralPath $root -Directory -Recurse | Where-Object { $_.Name -ne $DuplicateFolder -and $_.FullName -notlike "*\$DuplicateFolder\*" -and $_.FullName -notlike "*/$DuplicateFolder/*" } }

        foreach ($folder in $folders) {
            $files = Get-ChildItem -LiteralPath $folder.FullName -File | Where-Object { -not $extFilter -or $extFilter -contains $_.Extension.ToLower() }
            $groups = $files | ForEach-Object { Get-RomInfo $_ } | Group-Object Key | Where-Object Count -gt 1

            foreach ($group in $groups) {
                $ranked = @($group.Group | Sort-Object Penalty, RegionRank, @{ Expression = 'Revision'; Descending = $true }, @{ Expression = 'Verified'; Descending = $true }, TagCount, { $_.File.Name })
                $keep = $ranked[0]

                [pscustomobject]@{ Folder = $folder.FullName; Title = $keep.Title; Action = 'Keep'; Name = $keep.File.Name; Region = $keep.Region; Revision = $keep.Revision; Destination = $null; Status = 'Kept' }

                foreach ($extra in $ranked | Select-Object -Skip 1) {
                    $result = [ordered]@{ Folder = $folder.FullName; Title = $keep.Title; Action = $(if ($Delete) { 'Delete' } else { 'Move' }); Name = $extra.File.Name; Region = $extra.Region; Revision = $extra.Revision; Destination = $null; Status = 'WhatIf' }
                    try {
                        if ($Delete) {
                            if ($PSCmdlet.ShouldProcess($extra.File.FullName, "Delete (keeping $($keep.File.Name))")) {
                                Remove-Item -LiteralPath $extra.File.FullName -Force -ErrorAction Stop
                                $result.Status = 'Deleted'
                            }
                        }
                        else {
                            $destDir = Join-Path $folder.FullName $DuplicateFolder
                            $dest = Join-Path $destDir $extra.File.Name
                            $n = 2
                            while (Test-Path -LiteralPath $dest) { $dest = Join-Path $destDir ('{0} ({1}){2}' -f $extra.File.BaseName, $n++, $extra.File.Extension) }
                            $result.Destination = $dest
                            if ($PSCmdlet.ShouldProcess($extra.File.FullName, "Move to $destDir (keeping $($keep.File.Name))")) {
                                if (-not (Test-Path -LiteralPath $destDir)) { $null = New-Item -ItemType Directory -Path $destDir -ErrorAction Stop }
                                Move-Item -LiteralPath $extra.File.FullName -Destination $dest -ErrorAction Stop
                                $result.Status = 'Moved'
                            }
                        }
                    }
                    catch {
                        $result.Status = "Failed: $($_.Exception.Message)"
                        Write-Warning "$($extra.File.Name): $($_.Exception.Message)"
                    }
                    [pscustomobject]$result
                }
            }
        }
    }
}

Parameters

ParameterTypeDefaultWhat it's for
-Pathstring[].One or more folders of game files. Takes pipeline input.
-Extensionstring[]every fileOnly look at these extensions, like .zip, .7z or .chd. Handy when a folder also holds saves or artwork.
-Recurseswitch—Process subfolders too. Each folder is deduplicated on its own, so one run can cover every system.
-PreferRegionstring[]USA, World, Europe, JapanRegion order, best first. A file tagged (USA, Europe) counts as whichever of those ranks higher.
-Deprioritizestring[]Beta, Proto, Demo, Sample, Pirate, HackTags that make a version lose to any clean release, whatever its region.
-DuplicateFolderstring_duplicatesFolder the extras are moved into, created inside each folder that has any.
-Deleteswitch—Delete the extras instead of moving them. Honors -WhatIf and -Confirm.

Run it

See what it would keep and move, without touching anything.

.\Remove-DuplicateRom.ps1 -Path D:\Games\snes -WhatIf

Every system folder in one go, and a CSV of the decisions to look over later.

.\Remove-DuplicateRom.ps1 -Path D:\Games -Recurse | Export-Csv dedupe-log.csv -NoTypeInformation

You're in Europe and want PAL releases first.

.\Remove-DuplicateRom.ps1 -Path D:\Games\megadrive -PreferRegion Europe, World, USA, Japan

Only the disc images, and delete the extras once you trust it.

.\Remove-DuplicateRom.ps1 -Path D:\Games\psx -Extension .chd -Delete

What you'll see

Example outputvalues are illustrative
Folder         Title              Action Name                                    Region         Revision Status
------         -----              ------ ----                                    ------         -------- ------
D:\Games\snes  Super Mario World  Keep   Super Mario World (USA).zip             USA            0.0      Kept
D:\Games\snes  Super Mario World  Move   Super Mario World (Europe) (Rev 1).zip  Europe         1.0      Moved
D:\Games\snes  Super Mario World  Move   Super Mario World (Japan).zip           Japan          0.0      Moved
D:\Games\snes  Chrono Trigger     Keep   Chrono Trigger (USA) (Rev 1).zip        USA            1.0      Kept
D:\Games\snes  Chrono Trigger     Move   Chrono Trigger (USA).zip                USA            0.0      Moved
D:\Games\snes  Chrono Trigger     Move   Chrono Trigger (USA) (Beta).zip         USA            0.0      Moved
D:\Games\snes  Pilotwings         Keep   Pilotwings (USA, Europe).zip            USA, Europe    0.0      Kept
D:\Games\snes  Pilotwings         Move   Pilotwings (Japan).zip                  Japan          0.0      Moved

How it works

  1. Read the tags. For every file it splits the name into the title (everything before the first bracket), the round-bracket tags and the square-bracket dump flags. Short GoodTools codes like (U), (E) and (JU) are mapped to the No-Intro names, so old and new sets sort the same way.
  2. Group the versions. The title is lowercased and stripped of punctuation to make the key. Disc numbers and translation tags are added to the key, so they never collide with each other.
  3. Rank each group. It sorts on clean dump first (anything tagged Beta, Proto, Demo and so on, or flagged [b], [o], [h] or [p], sinks), then the best region in your -PreferRegion list, then the highest revision, then a verified [!] dump, then the name with the fewest extra tags.
  4. Keep the winner, move the rest. The top file stays put. The others go to _duplicates through ShouldProcess, so -WhatIf shows exactly what would move. A name clash in _duplicates gets a (2) on the end instead of overwriting.
  5. Report everything. Each file in a group comes back as an object with the action and status, which makes a CSV of the run easy to keep.

Take it further

  • Clean the names first. If your set mixes (U) and (USA) styles, run Rename-RomFile over it before this, so the groups line up.
  • Check the duplicates really are duplicates. Two files with different tags can still be byte-for-byte identical. Compare-FileHash will tell you.
  • Trim a folder for a handheld. Run it with -PreferRegion USA into a copy of your library and you get one playable version per game, which is exactly what a small SD card wants. There's more on that in picking a handheld for emulation.

Things that'll trip you up

  • Region beats revision, on purpose. A European Rev 1 loses to a USA original if USA is first in your list. That's usually what you want, since the region decides language and PAL speed. If a particular revision matters to you, move it back from _duplicates by hand.
  • Titles have to match to be grouped. It compares the part of the name before the first bracket, ignoring case and punctuation. A Japanese release under its Japanese title won't be grouped with the English one, so it stays. That's a safe failure, but it means a mixed set won't shrink as much as you'd expect.
  • Multi-disc games are grouped per disc. Disc 1 and Disc 2 are never duplicates of each other. Each disc picks its own winner, though, so if you only have Disc 2 in one region you can end up with a mixed set. Check multi-disc games before you use -Delete.
  • Translations and hacks are handled differently. A [T+Eng] translation stays in its own group, so it's never swapped for the original. A [h] hack, [b] bad dump, [o] overdump or [p] pirate is treated as a duplicate and loses to any clean dump.