Browse Source

Add TestTools with the nugetfuzz and decompdiff corpus tools

Both tools have found real decompiler defects (several merged fixes came out of
nugetfuzz sweeps), but they only existed in a private checkout, so nobody else
could run them and their setup knowledge lived in one head. They complement the
fixture suite from the other side: it decompiles code we wrote, these decompile
what the world ships.

They stay outside the solution - file-based apps, run by hand, never by CI - and
the near-empty Directory.Build.props/Directory.Packages.props keep the repo-wide
warnings-as-errors, lock-file and central-package-management settings from
reaching them.

The catalog sweep driver is PowerShell rather than bash so it runs on Windows as
well, which also drops its curl/jq dependency; staging falls back to copying when
Windows withholds symlink privileges, and report file names are hash-truncated to
stay inside the 260-character path limit.

Assisted-by: Claude:claude-opus-5[1m]:Claude Code
pull/4020/head
Siegfried Pammer 1 month ago
parent
commit
1da62d9e08
  1. 1
      CLAUDE.md
  2. 12
      TestTools/.gitignore
  3. 5
      TestTools/Directory.Build.props
  4. 7
      TestTools/Directory.Packages.props
  5. 95
      TestTools/README.md
  6. 876
      TestTools/decompdiff.cs
  7. 147
      TestTools/nugetfuzz-all.ps1
  8. 884
      TestTools/nugetfuzz.cs

1
CLAUDE.md

@ -37,6 +37,7 @@ Cross-platform core (decompiler engine + shared support):
Test support: Test support:
- `TestPlugin/` — sample plugin exercising the plugin-loading system (`net10.0`) - `TestPlugin/` — sample plugin exercising the plugin-loading system (`net10.0`)
- `TestFixtures.Resources/` — generates resource fixtures consumed by the decompiler tests - `TestFixtures.Resources/` — generates resource fixtures consumed by the decompiler tests
- `TestTools/` — file-based apps run by hand against real-world assemblies: `nugetfuzz` (crash/assert sweep over nuget.org packages) and `decompdiff` (output diff between two decompiler builds). In no solution and not run by CI; see `TestTools/README.md`
Windows-only frontends, packaging, and tests: Windows-only frontends, packaging, and tests:
- `ILSpy.AddIn/`, `ILSpy.AddIn.VS2022/` — Visual Studio add-ins (`net472`) - `ILSpy.AddIn/`, `ILSpy.AddIn.VS2022/` — Visual Studio add-ins (`net472`)

12
TestTools/.gitignore vendored

@ -0,0 +1,12 @@
# Run artifacts: crawl state and per-package logs are regenerated by every sweep
# and grow into the tens of megabytes.
crawl/
logs/
# Generated reports (nugetfuzz-report.html, decompdiff report dirs)
*-report.html
decompdiff-report/
# dotnet file-based app build output
bin/
obj/

5
TestTools/Directory.Build.props

@ -0,0 +1,5 @@
<Project>
<!-- Intentionally empty: its presence stops MSBuild's upward Directory.Build.props search at
this folder, so the repo-wide build settings (warnings-as-errors, lock files) do not apply
to the file-based apps here. They are standalone tools, not part of any solution. -->
</Project>

7
TestTools/Directory.Packages.props

@ -0,0 +1,7 @@
<Project>
<PropertyGroup>
<!-- The tools declare their package versions inline with #:package directives, which central
package management forbids. Opting out here keeps the repo's central versions untouched. -->
<ManagePackageVersionsCentrally>false</ManagePackageVersionsCentrally>
</PropertyGroup>
</Project>

95
TestTools/README.md

@ -0,0 +1,95 @@
# TestTools
Two standalone tools that run the decompiler over real-world assemblies, to find defects the
in-repo test suite cannot: it decompiles fixtures we wrote, these decompile what the world ships.
| tool | question it answers |
|---|---|
| `nugetfuzz.cs` | Does the decompiler *crash* on real code? (asserts, exceptions, IL warnings) |
| `decompdiff.cs` | Did a change make the *output* better or worse? (readability across two builds) |
Both are [file-based apps](https://learn.microsoft.com/dotnet/core/whats-new/dotnet-10/sdk#file-based-apps):
single `.cs` files run directly by the SDK, no project, no solution entry. They are not built by
`ILSpy.sln` and not run by CI. Requirements: the .NET SDK from `global.json` (or newer) and, for
the sweep script, PowerShell 7 (`pwsh`) - both cross-platform.
`Directory.Build.props` / `Directory.Packages.props` in this folder are intentionally near-empty:
they stop MSBuild's upward search, so the repo-wide warnings-as-errors, lock-file and central
package management settings do not reach these tools.
## nugetfuzz
Downloads NuGet packages, resolves their dependency closure, picks a matching lib TFM, and
decompiles every type of every assembly, reporting `Debug.Assert` failures, exceptions and
`//IL_xxxx:` warning comments. Exit code 0 means no finding.
```pwsh
dotnet run nugetfuzz.cs -- Newtonsoft.Json Serilog@3.1.1
dotnet run nugetfuzz.cs -- @packagelist.txt # one id per line, # comments allowed
dotnet run nugetfuzz.cs -- --report crawl/findings.jsonl [out.html]
```
Reference assemblies are fetched as needed: `Microsoft.NETCore.App.Ref` and the Windows-desktop /
ASP.NET packs for .NET Core targets, `Microsoft.NETFramework.ReferenceAssemblies` for classic
net4x. Getting these right matters - binding a WPF assembly against the stub facades in
`NETCore.App.Ref` collapses whole type hierarchies to `Unknown` and invents hundreds of bogus
warnings, so treat a sudden warning spike as a reference problem until proven otherwise.
Environment variables: `NUGETFUZZ_VERBOSE` (per-type progress), `NUGETFUZZ_DUMP=<dir>` (write the
decompiled C#), `NUGETFUZZ_LEDGER=<file>` (append findings as JSONL instead of writing a
per-run HTML report), `NUGETFUZZ_HTML=<file>` (report path), `NUGET_PACKAGES` (package cache).
### Sweeping the whole catalog
`nugetfuzz-all.ps1` walks the nuget.org catalog and runs `nugetfuzz.cs` on every package id it
has not seen. It is resumable - the page cursor and the seen-id list live in `crawl/`, so an
interrupted sweep continues where it stopped:
```pwsh
./nugetfuzz-all.ps1 # everything, from the cursor
./nugetfuzz-all.ps1 -MaxPages 5 -MaxPackages 50
```
Findings from every package land in `crawl/findings.jsonl`; render the aggregate at any time,
including while the sweep is still running, with `--report`. Logs of failed runs are kept in
`logs/`, successful ones are deleted. The package cache (`~/.cache/nugetfuzz`) is capped at 20 GB
by default (`-CacheCapMB`) and pruned least-recently-used, because `decompdiff` uses it as a corpus.
## decompdiff
Decompiles a corpus with **two** builds of `ICSharpCode.Decompiler` side by side (separate
`AssemblyLoadContext`s, driven through the stable `CSharpDecompiler(string, DecompilerSettings)`
API, so arbitrary version pairs work) and reports how the output differs. Correctness is what the
round-trip tests check; this checks readability. Exit code 1 means the new side has errors the old
side did not.
```pwsh
dotnet run decompdiff.cs -- --old ../../ILSpy-master --new . -o report ~/.cache/nugetfuzz
dotnet run decompdiff.cs -- --old v9.1.dll --new v11.dll --refs <dir> corpus.dll
```
An `--old`/`--new` argument is either a path to `ICSharpCode.Decompiler.dll` or an ILSpy checkout,
which is restored and built in Release on demand. **Watch the timestamp in the header line**: an
existing Release build is reused as-is; pass `--build` to force a rebuild.
The report directory gets `summary.md`, a self-contained `index.html` with inline diffs, and the
changed types dumped under `old/` and `new/` for `git diff --no-index report/old report/new`.
Each assembly is decompiled from a staging directory holding its neighbours plus the transitive
closure of its references, found in `--refs` directories, the NuGet cache and the .NET Framework
reference packs. Both sides read the same staging directory, so anything still unresolved degrades
them identically and the diff stays meaningful. Unresolved references are listed in the summary.
## Windows notes
Both tools run on Windows, with two differences worth knowing:
- Staging uses symbolic links, which Windows only grants to elevated processes or with Developer
Mode enabled. Without it, the files are copied instead - correct, just slower and more disk.
- Report file names are truncated with a hash appended when a namespace-qualified type name would
push the path past the 260-character limit that applies unless long paths are enabled.
## Ignored output
`crawl/`, `logs/`, generated reports and `bin/`/`obj/` are gitignored: they are run artifacts that
grow into the gigabytes.

876
TestTools/decompdiff.cs

@ -0,0 +1,876 @@
// Copyright (c) 2026 Siegfried Pammer
//
// Permission is hereby granted, free of charge, to any person obtaining a copy of this
// software and associated documentation files (the "Software"), to deal in the Software
// without restriction, including without limitation the rights to use, copy, modify, merge,
// publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons
// to whom the Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all copies or
// substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
// INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
// PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
// FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
// OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
// DEALINGS IN THE SOFTWARE.
#:property PublishAot=false
// decompdiff: decompiles a corpus of assemblies with TWO builds of
// ICSharpCode.Decompiler (loaded side-by-side via AssemblyLoadContext) and
// reports how the output differs, to assess quality/readability of decompiler
// changes across real-world code. The textual complement to the Windows
// round-trip tests, which verify correctness but not output quality.
//
// usage: dotnet run decompdiff.cs -- --old <ILSpy-checkout|Decompiler.dll> --new <ILSpy-checkout|Decompiler.dll>
// [-o <report-dir>] [--build] [--refs <dir>]... <dll|dir>...
//
// - checkout args are built on demand (Release; restore keeps packages.lock.json
// whole via -p:RestoreEnablePackagePruning=false); pass --build to force rebuild.
// - corpus dirs are scanned recursively for *.dll (e.g. ~/.cache/nugetfuzz).
// - changed/errored types are written to <report-dir>/{old,new}/...; inspect with
// `git diff --no-index <report-dir>/old <report-dir>/new`, or open the generated
// <report-dir>/index.html, which carries the same data with inline diffs.
// - exit code 1 when the new version has errors the old one didn't (regressions).
//
// Reference handling: every assembly is decompiled out of a staging directory that
// holds symlinks to itself, its original neighbours, and the transitive closure of
// its references as found in --refs directories, the machine-wide NuGet cache, and
// the .NET Framework reference-assembly packs. UniversalAssemblyResolver searches
// that directory first (ResolveInternal -> SearchDirectory), so staging fixes both
// classic failure modes - a sibling package that does not sit next to the assembly,
// and the Windows-only mscorlib lookup that throws "Version not supported" on Linux
// - while still driving the stable 2-arg CSharpDecompiler ctor. Both sides read the
// SAME staging directory, so whatever stays unresolved degrades them identically
// and the diffs remain meaningful.
using System.Collections;
using System.Diagnostics;
using System.Reflection;
using System.Reflection.Metadata;
using System.Reflection.PortableExecutable;
using System.Runtime.Loader;
using System.Text;
using System.Text.RegularExpressions;
Trace.Listeners.Clear();
Trace.Listeners.Add(new ThrowOnAssert());
try
{
Debug.Fail("self-test");
Console.Error.WriteLine("FATAL: assert hook not active, Debug.Assert in Debug decompiler builds would kill the process");
return 2;
}
catch (AssertionFailedException)
{
// hook works
}
string? oldSpec = null, newSpec = null, reportDir = null;
bool forceBuild = false;
var corpus = new List<string>();
var refDirs = new List<string>();
for (int i = 0; i < args.Length; i++)
{
switch (args[i])
{
case "--old":
oldSpec = args[++i];
break;
case "--new":
newSpec = args[++i];
break;
case "-o":
reportDir = args[++i];
break;
case "--build":
forceBuild = true;
break;
case "--refs":
refDirs.Add(args[++i]);
break;
default:
corpus.Add(args[i]);
break;
}
}
if (oldSpec == null || newSpec == null || corpus.Count == 0)
{
Console.Error.WriteLine("usage: decompdiff --old <ILSpy-checkout|Decompiler.dll> --new <...> [-o report-dir] [--build] [--refs <dir>]... <dll|dir>...");
return 1;
}
reportDir ??= "decompdiff-report";
if (Directory.Exists(reportDir))
{
if (!File.Exists(Path.Combine(reportDir, "summary.md")) && Directory.EnumerateFileSystemEntries(reportDir).Any())
{
Console.Error.WriteLine($"refusing to reuse {reportDir}: exists, non-empty, and not a decompdiff report");
return 1;
}
Directory.Delete(reportDir, true);
}
Directory.CreateDirectory(reportDir);
var oldSide = Side.Create("old", oldSpec, forceBuild);
var newSide = Side.Create("new", newSpec, forceBuild);
Console.WriteLine($"old: {oldSide.Description}");
Console.WriteLine($"new: {newSide.Description}");
var assemblies = corpus
.SelectMany(a => Directory.Exists(a)
? Directory.EnumerateFiles(a, "*.dll", SearchOption.AllDirectories)
: [a])
.Where(f => !f.EndsWith(".resources.dll", StringComparison.OrdinalIgnoreCase))
.Distinct()
.OrderBy(f => f)
.ToList();
Console.WriteLine($"corpus: {assemblies.Count} assemblies");
var refIndex = new RefIndex(refDirs, corpus);
Console.WriteLine($"references: {refIndex.Description}");
var stageRoot = Path.Combine(reportDir, ".staging");
var unresolvedRefs = new SortedDictionary<string, List<string>>(); // assembly -> missing refs
int asmCount = 0, unchanged = 0;
var skipped = new List<string>();
var changed = new List<ChangedType>();
var transitions = new List<string>(); // fixed-error / NEW-ERROR / only-old / only-new lines
int newErrors = 0, fixedErrors = 0, bothErrors = 0;
var oldTotals = default(Metrics);
var newTotals = default(Metrics);
foreach (var dll in assemblies)
{
var asmName = Path.GetFileNameWithoutExtension(dll);
var (staged, missing) = RefIndex.Stage(dll, stageRoot, refIndex);
if (missing.Count > 0)
unresolvedRefs[asmName] = missing;
var oldTypes = oldSide.DecompileAssembly(staged);
if (oldTypes == null)
{
var why = missing.Count > 0 ? $"; unresolved refs: {string.Join(", ", missing.Take(5))}" : "";
Console.WriteLine($" skip {asmName}: not decompilable ({oldSide.LastAssemblyError}{why})");
skipped.Add($"{asmName}: {oldSide.LastAssemblyError}{why}");
continue;
}
var newTypes = newSide.DecompileAssembly(staged);
if (newTypes == null)
{
Console.WriteLine($" NEW-ERROR {asmName}: whole assembly failed only with new version ({newSide.LastAssemblyError})");
transitions.Add($"NEW-ERROR (assembly) {asmName}: {newSide.LastAssemblyError}");
newErrors++;
continue;
}
asmCount++;
int asmChanged = 0;
foreach (var name in oldTypes.Keys.Union(newTypes.Keys).OrderBy(n => n))
{
var o = oldTypes.GetValueOrDefault(name);
var n = newTypes.GetValueOrDefault(name);
var location = $"{asmName} / {name}";
if (o == null || n == null)
{
transitions.Add($"only-{(o != null ? "old" : "new")} {location}");
continue;
}
if (o.Error != null || n.Error != null)
{
if (o.Error != null && n.Error != null)
{
bothErrors++;
if (o.Error != n.Error)
transitions.Add($"error-changed {location}: {o.Error} -> {n.Error}");
}
else if (o.Error != null)
{
fixedErrors++;
transitions.Add($"fixed-error {location}: {o.Error}");
}
else
{
newErrors++;
transitions.Add($"NEW-ERROR {location}: {n.Error}");
DumpPair(location, o.Code!, $"// ERROR: {n.Error}");
}
continue;
}
oldTotals += o.Metrics;
newTotals += n.Metrics;
if (o.Code == n.Code)
{
unchanged++;
continue;
}
asmChanged++;
changed.Add(new ChangedType(location, o.Metrics, n.Metrics));
DumpPair(location, o.Code!, n.Code!);
}
Console.WriteLine($" {asmName}: {oldTypes.Count} types, {asmChanged} changed");
}
var summary = new StringBuilder();
summary.AppendLine("# decompdiff report");
summary.AppendLine();
summary.AppendLine($"- old: {oldSide.Description}");
summary.AppendLine($"- new: {newSide.Description}");
summary.AppendLine($"- corpus: {asmCount} assemblies, {unchanged + changed.Count} types compared");
summary.AppendLine();
summary.AppendLine($"| | old | new | delta |");
summary.AppendLine($"|---|---|---|---|");
summary.AppendLine(MetricRow("lines", oldTotals.Lines, newTotals.Lines));
summary.AppendLine(MetricRow("goto statements", oldTotals.Gotos, newTotals.Gotos));
summary.AppendLine(MetricRow("//IL_ warning comments", oldTotals.IlWarnings, newTotals.IlWarnings));
summary.AppendLine(MetricRow("compiler-generated name leaks (<>)", oldTotals.GeneratedNames, newTotals.GeneratedNames));
summary.AppendLine();
summary.AppendLine($"types: {unchanged} unchanged, {changed.Count} changed, {newErrors} NEW errors, {fixedErrors} fixed errors, {bothErrors} errored in both");
summary.AppendLine();
if (changed.Count > 0)
{
summary.AppendLine("## Changed types (largest line delta first)");
summary.AppendLine();
foreach (var c in changed.OrderByDescending(c => Math.Abs(c.New.Lines - c.Old.Lines)).Take(50))
summary.AppendLine($"- {c.Location}: {c.Old.Lines} -> {c.New.Lines} lines" + MetricNotes(c));
if (changed.Count > 50)
summary.AppendLine($"- ... {changed.Count - 50} more, see {reportDir}/{{old,new}}/");
summary.AppendLine();
}
if (transitions.Count > 0)
{
summary.AppendLine("## Error / presence transitions");
summary.AppendLine();
foreach (var t in transitions)
summary.AppendLine($"- {t}");
summary.AppendLine();
}
if (skipped.Count > 0)
{
summary.AppendLine("## Skipped assemblies (undecompilable on BOTH sides)");
summary.AppendLine();
foreach (var s in skipped)
summary.AppendLine($"- {s}");
summary.AppendLine();
}
if (unresolvedRefs.Count > 0)
{
// Point at what would widen the corpus: each name here is a reference no --refs
// directory, the NuGet cache, or a reference-assembly pack could supply.
summary.AppendLine("## Unresolved references (add --refs dirs to cover these)");
summary.AppendLine();
foreach (var (asm, refs) in unresolvedRefs)
summary.AppendLine($"- {asm}: {string.Join(", ", refs)}");
summary.AppendLine();
}
summary.AppendLine($"inspect changed output with: git diff --no-index {reportDir}/old {reportDir}/new");
summary.AppendLine($"or open {Path.Combine(reportDir, "index.html")}");
File.WriteAllText(Path.Combine(reportDir, "summary.md"), summary.ToString());
Report.WriteHtml(Path.Combine(reportDir, "index.html"), new ReportModel(
oldSide.Description, newSide.Description, asmCount, unchanged, changed, transitions,
skipped, unresolvedRefs, oldTotals, newTotals, newErrors, fixedErrors, bothErrors, reportDir));
Console.WriteLine();
Console.Write(summary);
return newErrors > 0 ? 1 : 0;
void DumpPair(string location, string oldCode, string newCode)
{
foreach (var (side, code) in new[] { ("old", oldCode), ("new", newCode) })
{
var path = Path.Combine(reportDir, side, Report.SanitizeFileName(location.Replace(" / ", "/")) + ".cs");
Directory.CreateDirectory(Path.GetDirectoryName(path)!);
File.WriteAllText(path, code);
}
}
static string MetricRow(string name, long oldValue, long newValue)
=> $"| {name} | {oldValue} | {newValue} | {newValue - oldValue:+#;-#;0} |";
static string MetricNotes(ChangedType c)
{
var notes = new List<string>();
if (c.New.Gotos != c.Old.Gotos)
notes.Add($"gotos {c.Old.Gotos}->{c.New.Gotos}");
if (c.New.IlWarnings != c.Old.IlWarnings)
notes.Add($"IL warnings {c.Old.IlWarnings}->{c.New.IlWarnings}");
if (c.New.GeneratedNames != c.Old.GeneratedNames)
notes.Add($"name leaks {c.Old.GeneratedNames}->{c.New.GeneratedNames}");
return notes.Count > 0 ? $" ({string.Join(", ", notes)})" : "";
}
// Locates reference assemblies by simple name and stages them next to the assembly
// being decompiled. Sources, in order: the --refs directories (indexed once), the
// machine-wide NuGet cache (probed per name, so nothing scans ~40k packages), and
// the .NET Framework reference-assembly packs restored under it - the last one is
// what makes classic net4x assemblies decompilable on Linux at all, since their
// mscorlib otherwise only exists behind a Windows path lookup.
sealed class RefIndex
{
readonly Dictionary<string, string> byName = new(StringComparer.OrdinalIgnoreCase);
readonly List<string> probeRoots = new();
public string Description { get; }
public RefIndex(List<string> refDirs, List<string> corpus)
{
foreach (var dir in refDirs.Concat(corpus).Where(Directory.Exists))
IndexDirectory(dir);
var nugetRoot = Environment.GetEnvironmentVariable("NUGET_PACKAGES")
?? Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".nuget", "packages");
if (Directory.Exists(nugetRoot))
probeRoots.Add(nugetRoot);
foreach (var pack in EnumerateFrameworkRefPacks(nugetRoot))
IndexDirectory(pack);
// On Windows the same reference assemblies also ship with the targeting packs, so a
// net4x corpus resolves there without restoring the NuGet package first.
var installedFxRefs = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFilesX86),
"Reference Assemblies", "Microsoft", "Framework", ".NETFramework");
if (Directory.Exists(installedFxRefs))
IndexDirectory(installedFxRefs);
Description = $"{byName.Count} assemblies indexed"
+ (probeRoots.Count > 0 ? $", NuGet cache probe at {string.Join(", ", probeRoots)}" : "");
}
// Reference assemblies for .NET Framework targets; the newest pack wins, and its
// Facades subdirectory carries the type-forwarding shims netstandard code needs.
static IEnumerable<string> EnumerateFrameworkRefPacks(string nugetRoot)
{
if (!Directory.Exists(nugetRoot))
yield break;
foreach (var pkg in Directory.EnumerateDirectories(nugetRoot, "microsoft.netframework.referenceassemblies.*")
.OrderBy(d => d))
{
foreach (var dir in Directory.EnumerateDirectories(pkg, "v*", SearchOption.AllDirectories))
{
yield return dir;
var facades = Path.Combine(dir, "Facades");
if (Directory.Exists(facades))
yield return facades;
}
}
}
void IndexDirectory(string dir)
{
foreach (var dll in Directory.EnumerateFiles(dir, "*.dll", SearchOption.AllDirectories))
{
var name = Path.GetFileNameWithoutExtension(dll);
// First indexed wins: --refs directories are added before the corpus, so an
// explicitly supplied reference is never shadowed by a corpus copy.
if (!byName.ContainsKey(name))
byName[name] = dll;
}
}
public string? Find(string simpleName)
{
if (byName.TryGetValue(simpleName, out var hit))
return hit;
foreach (var root in probeRoots)
{
// NuGet lays packages out as <root>/<id lowercased>/<version>/lib/<tfm>/<id>.dll,
// and the assembly name matches the package id often enough to be worth a look.
var pkgDir = Path.Combine(root, simpleName.ToLowerInvariant());
if (!Directory.Exists(pkgDir))
continue;
var candidate = Directory.EnumerateDirectories(pkgDir)
.OrderByDescending(d => Path.GetFileName(d), StringComparer.OrdinalIgnoreCase)
.SelectMany(v => Directory.EnumerateFiles(v, simpleName + ".dll", SearchOption.AllDirectories))
.FirstOrDefault(f => f.Contains($"{Path.DirectorySeparatorChar}lib{Path.DirectorySeparatorChar}")
|| f.Contains($"{Path.DirectorySeparatorChar}ref{Path.DirectorySeparatorChar}"));
if (candidate != null)
{
byName[simpleName] = candidate;
return candidate;
}
}
byName[simpleName] = null!; // negative cache: probing the filesystem twice buys nothing
return null;
}
// Builds the staging directory for one assembly and returns the path to decompile
// plus the reference names nothing could supply.
public static (string Staged, List<string> Missing) Stage(string dll, string stageRoot, RefIndex refs)
{
var dir = Path.Combine(stageRoot, StageName(dll));
Directory.CreateDirectory(dir);
// Whatever sat next to the assembly keeps sitting next to it, so staging never
// resolves LESS than decompiling in place would.
foreach (var sibling in Directory.EnumerateFiles(Path.GetDirectoryName(Path.GetFullPath(dll))!, "*.dll"))
Link(sibling, dir);
Link(dll, dir);
var missing = new List<string>();
var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
var queue = new Queue<string>();
foreach (var name in ReferencesOf(dll))
queue.Enqueue(name);
while (queue.Count > 0)
{
var name = queue.Dequeue();
if (!seen.Add(name))
continue;
var staged = Path.Combine(dir, name + ".dll");
if (!File.Exists(staged))
{
var found = refs.Find(name);
if (found == null)
{
missing.Add(name);
continue;
}
Link(found, dir);
}
// A staged reference brings its own references along: the type system follows
// base types and type-forwards across the whole closure, not just one hop.
foreach (var transitive in ReferencesOf(staged))
queue.Enqueue(transitive);
}
missing.Sort(StringComparer.OrdinalIgnoreCase);
return (Path.Combine(dir, Path.GetFileName(dll)), missing);
}
// Distinct per source path: two packages can ship the same assembly name with
// different contents, and they must not share a staging directory.
static string StageName(string dll)
{
var full = Path.GetFullPath(dll);
var hash = Convert.ToHexString(System.Security.Cryptography.MD5.HashData(Encoding.UTF8.GetBytes(full)))[..8];
return $"{Path.GetFileNameWithoutExtension(full)}-{hash}";
}
static void Link(string source, string targetDir)
{
var link = Path.Combine(targetDir, Path.GetFileName(source));
if (File.Exists(link))
return;
try
{
File.CreateSymbolicLink(link, Path.GetFullPath(source));
}
catch (Exception e) when (e is IOException or UnauthorizedAccessException)
{
// Windows only hands out symlink privileges to elevated processes or with
// Developer Mode enabled; copying costs disk but keeps staging working.
File.Copy(source, link, overwrite: true);
}
}
public static List<string> ReferencesOf(string dll)
{
var names = new List<string>();
try
{
using var stream = File.OpenRead(dll);
using var pe = new PEReader(stream);
if (!pe.HasMetadata)
return names;
var md = pe.GetMetadataReader();
foreach (var handle in md.AssemblyReferences)
names.Add(md.GetString(md.GetAssemblyReference(handle).Name));
}
catch (Exception ex) when (ex is BadImageFormatException or IOException)
{
// Native or corrupt file: it carries no managed references to follow.
}
return names;
}
}
// Self-contained HTML view of a run: the same numbers summary.md carries, plus the
// actual diff of every changed type inline, so a corpus sweep can be surveyed in a
// browser instead of by shelling out to `git diff --no-index` per type. No assets,
// no scripts from anywhere - the file opens straight off disk.
static class Report
{
public static void WriteHtml(string path, ReportModel m)
{
var html = new StringBuilder();
html.AppendLine("""
<!doctype html><html><head><meta charset="utf-8">
<title>decompdiff report</title>
<style>
:root { color-scheme: light dark; --bg:#fff; --fg:#1a1a1a; --muted:#666; --line:#d8d8d8;
--add:#e6ffec; --addfg:#0a5c1e; --del:#ffebe9; --delfg:#8a1c11; --chip:#f0f0f0; }
@media (prefers-color-scheme: dark) {
:root { --bg:#16181c; --fg:#e6e6e6; --muted:#9aa0a6; --line:#333;
--add:#12261a; --addfg:#7ee2a8; --del:#2b1416; --delfg:#ff9c92; --chip:#24262b; }
}
body { background:var(--bg); color:var(--fg); font:14px/1.5 system-ui,sans-serif; margin:0 auto; padding:24px; max-width:1100px; }
h1 { font-size:20px; margin:0 0 4px; } h2 { font-size:16px; margin:28px 0 8px; }
.meta { color:var(--muted); font-size:13px; }
table { border-collapse:collapse; margin:12px 0; } th,td { border:1px solid var(--line); padding:4px 10px; text-align:right; }
th:first-child,td:first-child { text-align:left; }
.pos { color:var(--delfg); } .neg { color:var(--addfg); }
details { border:1px solid var(--line); border-radius:6px; margin:6px 0; background:var(--chip); }
summary { cursor:pointer; padding:8px 10px; font-family:ui-monospace,monospace; font-size:13px; }
pre { margin:0; padding:10px; overflow-x:auto; background:var(--bg); font:12px/1.45 ui-monospace,monospace; }
ins { background:var(--add); color:var(--addfg); text-decoration:none; display:block; }
del { background:var(--del); color:var(--delfg); text-decoration:none; display:block; }
span.ctx { display:block; color:var(--muted); }
#filter { width:100%; padding:8px; margin:8px 0; border:1px solid var(--line); border-radius:6px;
background:var(--bg); color:var(--fg); font:13px ui-monospace,monospace; }
ul { padding-left:20px; } li { font-family:ui-monospace,monospace; font-size:12.5px; }
</style></head><body>
""");
html.AppendLine("<h1>decompdiff report</h1>");
html.AppendLine($"<div class=meta>old: {Esc(m.Old)}<br>new: {Esc(m.New)}<br>"
+ $"corpus: {m.Assemblies} assemblies, {m.Unchanged + m.Changed.Count} types compared</div>");
html.AppendLine("<table><tr><th>metric</th><th>old</th><th>new</th><th>delta</th></tr>");
html.AppendLine(Row("lines", m.OldTotals.Lines, m.NewTotals.Lines));
html.AppendLine(Row("goto statements", m.OldTotals.Gotos, m.NewTotals.Gotos));
html.AppendLine(Row("//IL_ warning comments", m.OldTotals.IlWarnings, m.NewTotals.IlWarnings));
html.AppendLine(Row("compiler-generated name leaks", m.OldTotals.GeneratedNames, m.NewTotals.GeneratedNames));
html.AppendLine("</table>");
html.AppendLine($"<div class=meta>{m.Unchanged} unchanged, {m.Changed.Count} changed, "
+ $"<b>{m.NewErrors} NEW errors</b>, {m.FixedErrors} fixed errors, {m.BothErrors} errored in both</div>");
if (m.Changed.Count > 0)
{
html.AppendLine($"<h2>Changed types ({m.Changed.Count})</h2>");
html.AppendLine("<input id=filter placeholder='filter by type or assembly name'>");
foreach (var c in m.Changed.OrderByDescending(c => Math.Abs(c.New.Lines - c.Old.Lines)))
{
var file = SanitizeFileName(c.Location.Replace(" / ", "/")) + ".cs";
var oldCode = ReadIfExists(Path.Combine(m.ReportDir, "old", file));
var newCode = ReadIfExists(Path.Combine(m.ReportDir, "new", file));
var delta = c.New.Lines - c.Old.Lines;
html.AppendLine($"<details><summary>{Esc(c.Location)} "
+ $"<span class=meta>({c.Old.Lines} &rarr; {c.New.Lines} lines, {delta:+#;-#;0})</span></summary>");
html.AppendLine($"<pre>{Diff(oldCode, newCode)}</pre></details>");
}
html.AppendLine("""
<script>
const box = document.getElementById('filter');
box.addEventListener('input', () => {
const needle = box.value.toLowerCase();
for (const d of document.querySelectorAll('details'))
d.style.display = d.querySelector('summary').textContent.toLowerCase().includes(needle) ? '' : 'none';
});
</script>
""");
}
AppendList(html, "Error / presence transitions", m.Transitions);
AppendList(html, "Skipped assemblies (undecompilable on both sides)", m.Skipped);
AppendList(html, "Unresolved references (pass --refs to cover these)",
m.UnresolvedRefs.Select(kv => $"{kv.Key}: {string.Join(", ", kv.Value)}").ToList());
html.AppendLine("</body></html>");
File.WriteAllText(path, html.ToString());
}
static void AppendList(StringBuilder html, string title, List<string> items)
{
if (items.Count == 0)
return;
html.AppendLine($"<h2>{Esc(title)} ({items.Count})</h2><ul>");
foreach (var item in items)
html.AppendLine($"<li>{Esc(item)}</li>");
html.AppendLine("</ul>");
}
static string Row(string name, long o, long n)
{
var delta = n - o;
var cls = delta == 0 ? "" : delta > 0 ? " class=pos" : " class=neg";
return $"<tr><td>{Esc(name)}</td><td>{o}</td><td>{n}</td><td{cls}>{delta:+#;-#;0}</td></tr>";
}
static string ReadIfExists(string path) => File.Exists(path) ? File.ReadAllText(path) : "";
// Line diff: common prefix/suffix are cheap to strip and usually account for nearly
// everything, leaving a middle small enough for an O(n*m) LCS. Beyond the cap the
// middle is shown as a plain replacement rather than spending minutes on alignment.
const int LcsCap = 1500;
static string Diff(string oldCode, string newCode)
{
var a = oldCode.ReplaceLineEndings("\n").Split('\n');
var b = newCode.ReplaceLineEndings("\n").Split('\n');
int start = 0;
while (start < a.Length && start < b.Length && a[start] == b[start])
start++;
int endA = a.Length, endB = b.Length;
while (endA > start && endB > start && a[endA - 1] == b[endB - 1])
{
endA--;
endB--;
}
var sb = new StringBuilder();
// A few lines of context on each side make the hunk readable on its own.
for (int i = Math.Max(0, start - 3); i < start; i++)
sb.Append("<span class=ctx>").Append(Esc(a[i])).Append("</span>");
int lenA = endA - start, lenB = endB - start;
if (lenA <= LcsCap && lenB <= LcsCap)
{
foreach (var (tag, line) in LcsDiff(a[start..endA], b[start..endB]))
sb.Append(tag switch { '+' => "<ins>", '-' => "<del>", _ => "<span class=ctx>" })
.Append(Esc(line))
.Append(tag switch { '+' => "</ins>", '-' => "</del>", _ => "</span>" });
}
else
{
for (int i = start; i < endA; i++)
sb.Append("<del>").Append(Esc(a[i])).Append("</del>");
for (int i = start; i < endB; i++)
sb.Append("<ins>").Append(Esc(b[i])).Append("</ins>");
}
for (int i = endA; i < Math.Min(a.Length, endA + 3); i++)
sb.Append("<span class=ctx>").Append(Esc(a[i])).Append("</span>");
return sb.ToString();
}
static List<(char Tag, string Line)> LcsDiff(string[] a, string[] b)
{
var lcs = new int[a.Length + 1, b.Length + 1];
for (int i = a.Length - 1; i >= 0; i--)
for (int j = b.Length - 1; j >= 0; j--)
lcs[i, j] = a[i] == b[j] ? lcs[i + 1, j + 1] + 1 : Math.Max(lcs[i + 1, j], lcs[i, j + 1]);
var result = new List<(char, string)>();
int x = 0, y = 0;
while (x < a.Length && y < b.Length)
{
if (a[x] == b[y])
{
result.Add((' ', a[x]));
x++;
y++;
}
else if (lcs[x + 1, y] >= lcs[x, y + 1])
{
result.Add(('-', a[x++]));
}
else
{
result.Add(('+', b[y++]));
}
}
while (x < a.Length)
result.Add(('-', a[x++]));
while (y < b.Length)
result.Add(('+', b[y++]));
return result;
}
static string Esc(string s) => s.Replace("&", "&amp;").Replace("<", "&lt;").Replace(">", "&gt;");
// '/' survives as a directory separator; everything the platform rejects becomes '_'.
// Long segments are truncated with a hash of the original appended, because a
// namespace-qualified generic type name can push a report path past the 260-character
// limit Windows applies unless long paths are enabled machine-wide.
public static string SanitizeFileName(string s)
=> string.Join('/', s.Split('/').Select(segment => {
var clean = string.Concat(segment.Select(
ch => Path.GetInvalidFileNameChars().Contains(ch) ? '_' : ch));
return clean.Length <= 80
? clean
: clean[..72] + Convert.ToHexString(
System.Security.Cryptography.MD5.HashData(Encoding.UTF8.GetBytes(clean)))[..8];
}));
}
record ReportModel(
string Old, string New, int Assemblies, int Unchanged, List<ChangedType> Changed,
List<string> Transitions, List<string> Skipped, SortedDictionary<string, List<string>> UnresolvedRefs,
Metrics OldTotals, Metrics NewTotals, int NewErrors, int FixedErrors, int BothErrors, string ReportDir);
record TypeResult(string? Code, string? Error, Metrics Metrics);
record ChangedType(string Location, Metrics Old, Metrics New);
record struct Metrics(int Lines, int Gotos, int IlWarnings, int GeneratedNames)
{
public static Metrics Measure(string code) => new(
code.Count(c => c == '\n') + 1,
Regex.Matches(code, @"\bgoto ").Count,
Regex.Matches(code, @"//IL_[0-9a-fA-F]+:").Count,
Regex.Matches(code, @"<>").Count);
public static Metrics operator +(Metrics a, Metrics b)
=> new(a.Lines + b.Lines, a.Gotos + b.Gotos, a.IlWarnings + b.IlWarnings, a.GeneratedNames + b.GeneratedNames);
}
// One decompiler version: locates/builds ICSharpCode.Decompiler.dll, loads it in
// its own AssemblyLoadContext, and drives it via `dynamic` through the stable
// CSharpDecompiler(string, DecompilerSettings) API so any two versions work.
class Side
{
readonly Assembly assembly;
public string Description { get; }
public string? LastAssemblyError { get; private set; }
Side(Assembly assembly, string description)
{
this.assembly = assembly;
Description = description;
}
public static Side Create(string name, string spec, bool forceBuild)
{
string dllPath;
string description;
if (File.Exists(spec))
{
dllPath = Path.GetFullPath(spec);
description = dllPath;
}
else if (Directory.Exists(spec))
{
var checkout = Path.GetFullPath(spec);
dllPath = BuildCheckout(checkout, forceBuild);
// The dll timestamp exposes stale pre-existing builds; --build forces a fresh one.
description = $"{checkout} ({GitDescribe(checkout)}, dll of {File.GetLastWriteTime(dllPath):yyyy-MM-dd HH:mm})";
}
else
{
throw new ArgumentException($"--{name} {spec}: no such file or directory");
}
var alc = new DecompilerLoadContext(name, dllPath);
return new Side(alc.LoadFromAssemblyPath(dllPath), description);
}
static string BuildCheckout(string checkout, bool forceBuild)
{
var csproj = Path.Combine(checkout, "ICSharpCode.Decompiler", "ICSharpCode.Decompiler.csproj");
if (!File.Exists(csproj))
throw new ArgumentException($"{checkout}: no ICSharpCode.Decompiler/ICSharpCode.Decompiler.csproj");
var binDir = Path.Combine(checkout, "ICSharpCode.Decompiler", "bin", "Release");
var existing = Directory.Exists(binDir)
? Directory.EnumerateFiles(binDir, "ICSharpCode.Decompiler.dll", SearchOption.AllDirectories)
.OrderByDescending(File.GetLastWriteTimeUtc).FirstOrDefault()
: null;
if (existing != null && !forceBuild)
return existing;
// A bare restore would prune the repo's packages.lock.json files; keep them whole.
Run("dotnet", $"restore \"{csproj}\" -p:RestoreEnablePackagePruning=false");
Run("dotnet", $"build \"{csproj}\" -c Release --no-restore");
return Directory.EnumerateFiles(binDir, "ICSharpCode.Decompiler.dll", SearchOption.AllDirectories)
.OrderByDescending(File.GetLastWriteTimeUtc).First();
}
static void Run(string exe, string arguments)
{
Console.WriteLine($" $ {exe} {arguments}");
var psi = new ProcessStartInfo(exe, arguments) {
RedirectStandardOutput = true,
RedirectStandardError = true,
};
using var p = Process.Start(psi)!;
var output = p.StandardOutput.ReadToEnd() + p.StandardError.ReadToEnd();
p.WaitForExit();
if (p.ExitCode != 0)
throw new InvalidOperationException($"{exe} {arguments} failed:\n{output}");
}
static string GitDescribe(string checkout)
{
try
{
var psi = new ProcessStartInfo("git", "describe --always --dirty --exclude *") {
WorkingDirectory = checkout,
RedirectStandardOutput = true,
RedirectStandardError = true,
};
using var p = Process.Start(psi)!;
var output = p.StandardOutput.ReadToEnd().Trim();
p.WaitForExit();
return p.ExitCode == 0 && output.Length > 0 ? output : "unknown";
}
catch
{
return "unknown";
}
}
// Decompiles every top-level type; null when the assembly itself cannot be
// opened (not managed, type system init failed) - see LastAssemblyError.
public Dictionary<string, TypeResult>? DecompileAssembly(string dllPath)
{
LastAssemblyError = null;
dynamic decompiler;
try
{
var settings = Activator.CreateInstance(assembly.GetType("ICSharpCode.Decompiler.DecompilerSettings", throwOnError: true)!)!;
decompiler = Activator.CreateInstance(
assembly.GetType("ICSharpCode.Decompiler.CSharp.CSharpDecompiler", throwOnError: true)!,
dllPath, settings)!;
}
catch (Exception ex)
{
LastAssemblyError = Unwrap(ex).Message;
return null;
}
var results = new Dictionary<string, TypeResult>();
foreach (object type in (IEnumerable)decompiler.TypeSystem.MainModule.TopLevelTypeDefinitions)
{
// The type definition's runtime type is internal, so dynamic cannot bind
// its members; go through the public interface property via reflection.
object fullTypeName = GetProperty(type, "FullTypeName");
string name = fullTypeName.ToString()!;
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(60));
try
{
decompiler.CancellationToken = cts.Token;
string code = decompiler.DecompileTypeAsString((dynamic)fullTypeName);
results[name] = new TypeResult(code, null, Metrics.Measure(code));
}
catch (Exception ex)
{
var inner = Unwrap(ex);
var error = inner is OperationCanceledException
? "timeout (60s)"
: $"{inner.GetType().Name}: {FirstLine(inner.Message)}";
results[name] = new TypeResult(null, error, default);
}
}
return results;
}
static object GetProperty(object obj, string name)
{
var type = obj.GetType();
var property = type.GetProperty(name)
?? type.GetInterfaces().Select(i => i.GetProperty(name)).FirstOrDefault(p => p != null)
?? throw new MissingMemberException(type.FullName, name);
return property.GetValue(obj)!;
}
static Exception Unwrap(Exception ex)
{
while (ex is TargetInvocationException { InnerException: not null } tie)
ex = tie.InnerException!;
return ex;
}
static string FirstLine(string s)
{
var i = s.IndexOfAny(['\r', '\n']);
return i < 0 ? s : s[..i];
}
}
// Resolves the decompiler's own dependencies from its build-output directory,
// falling back to the default context for framework assemblies. Each Side gets
// its own context so two ICSharpCode.Decompiler versions can coexist.
class DecompilerLoadContext(string name, string mainDllPath) : AssemblyLoadContext(name)
{
readonly string dir = Path.GetDirectoryName(Path.GetFullPath(mainDllPath))!;
protected override Assembly? Load(AssemblyName assemblyName)
{
var candidate = Path.Combine(dir, assemblyName.Name + ".dll");
return File.Exists(candidate) ? LoadFromAssemblyPath(candidate) : null;
}
}
class AssertionFailedException(string message) : Exception(message);
class ThrowOnAssert : TraceListener
{
public override void Fail(string? message, string? detailMessage)
=> throw new AssertionFailedException($"{message} {detailMessage}".Trim());
public override void Write(string? message)
{
}
public override void WriteLine(string? message)
{
}
}

147
TestTools/nugetfuzz-all.ps1

@ -0,0 +1,147 @@
#!/usr/bin/env pwsh
# Crawls the nuget.org catalog and runs nugetfuzz.cs on every package id.
# Resumable: page cursor + seen-id list live in ./crawl. Every call is recorded
# in crawl/history.log; the full per-package log is kept only when the run failed.
#
# usage: ./nugetfuzz-all.ps1 [-MaxPages n] [-MaxPackages n] [-CacheCapMB n]
# ponytail: sequential crawl; parallelize per-page if throughput matters
[CmdletBinding()]
param(
[int]$MaxPages = 0, # 0 = all
[int]$MaxPackages = 0, # 0 = unlimited
[int]$CacheCapMB = 20480,
[int]$TimeoutSeconds = 1800
)
$ErrorActionPreference = 'Stop'
Set-Location $PSScriptRoot
$state = Join-Path $PSScriptRoot 'crawl'
$logs = Join-Path $PSScriptRoot 'logs'
$cache = Join-Path ([Environment]::GetFolderPath('UserProfile')) '.cache/nugetfuzz'
# Findings from every package run land in one append-only ledger. Render the aggregate
# at any time, while the sweep keeps running:
# dotnet run nugetfuzz.cs -- --report crawl/findings.jsonl
if (-not $env:NUGETFUZZ_LEDGER) {
$env:NUGETFUZZ_LEDGER = Join-Path $state 'findings.jsonl'
}
# Required by the local OpenSSL configuration to validate the SHA-1 signed packages.
if (-not $env:OPENSSL_ENABLE_SHA1_SIGNATURES) {
$env:OPENSSL_ENABLE_SHA1_SIGNATURES = '1'
}
New-Item -ItemType Directory -Force -Path $state, $logs | Out-Null
$seenFile = Join-Path $state 'seen-ids.txt'
$cursorFile = Join-Path $state 'next-page.txt'
$historyFile = Join-Path $state 'history.log'
$seen = @{}
if (Test-Path $seenFile) {
foreach ($id in Get-Content $seenFile) { $seen[$id] = $true }
}
# Disk guard. Evicts least-recently-used <id>/<version> directories down to 80% of the
# cap rather than wiping the cache wholesale: the cache doubles as the corpus decompdiff
# runs against, and a wipe throws away packages that are expensive to re-download and
# that other tools are pointed at.
function Invoke-CachePrune {
if (-not (Test-Path $cache)) { return }
$dirSize = { param($d) (Get-ChildItem -LiteralPath $d -Recurse -File -EA SilentlyContinue
| Measure-Object -Property Length -Sum).Sum / 1MB }
$used = & $dirSize $cache
if ($used -le $CacheCapMB) { return }
$target = $CacheCapMB * 0.8
Write-Host ("cache {0:N0}MB over {1}MB, evicting least-recently-used down to {2:N0}MB" -f $used, $CacheCapMB, $target)
# NTFS disables last-access-time updates by default, so on Windows this degrades to
# least-recently-written, which for an extract-once cache means least recently added.
$victims = Get-ChildItem -LiteralPath $cache -Directory -EA SilentlyContinue
| ForEach-Object { Get-ChildItem -LiteralPath $_.FullName -Directory -EA SilentlyContinue }
| Sort-Object LastAccessTime
foreach ($dir in $victims) {
if ($used -le $target) { break }
$used -= & $dirSize $dir.FullName
Remove-Item -LiteralPath $dir.FullName -Recurse -Force -EA SilentlyContinue
$parent = $dir.Parent.FullName
if (-not (Get-ChildItem -LiteralPath $parent -Force -EA SilentlyContinue)) {
Remove-Item -LiteralPath $parent -Force -EA SilentlyContinue
}
}
Write-Host ("cache now ~{0:N0}MB" -f $used)
}
try {
$pages = (Invoke-RestMethod 'https://api.nuget.org/v3/catalog0/index.json').items
| Sort-Object commitTimeStamp | ForEach-Object { $_.'@id' }
}
catch {
Write-Error "failed to fetch catalog index: $_"
exit 1
}
$start = if (Test-Path $cursorFile) { [int](Get-Content $cursorFile -Raw).Trim() } else { 0 }
$i = 0
$donePages = 0
$donePkgs = 0
:pages foreach ($page in $pages) {
$i++
if ($i -le $start) { continue }
if ($MaxPages -gt 0 -and $donePages -ge $MaxPages) { break }
try {
$ids = (Invoke-RestMethod $page).items
| Where-Object { $_.'@type' -eq 'nuget:PackageDetails' }
| ForEach-Object { $_.'nuget:id'.ToLowerInvariant() }
| Sort-Object -Unique
}
catch {
Write-Warning "failed to fetch $page, stopping (resume with cursor)"
break
}
foreach ($id in $ids) {
if ($seen[$id]) { continue }
if ($MaxPackages -gt 0 -and $donePkgs -ge $MaxPackages) { break pages }
$seen[$id] = $true
Add-Content -LiteralPath $seenFile -Value $id
$log = Join-Path $logs "$id.log"
$errLog = "$log.err"
$p = Start-Process dotnet -ArgumentList 'run', 'nugetfuzz.cs', '--', $id `
-WorkingDirectory $PSScriptRoot -NoNewWindow -PassThru `
-RedirectStandardOutput $log -RedirectStandardError $errLog
if ($p.WaitForExit($TimeoutSeconds * 1000)) {
$rc = $p.ExitCode
}
else {
$p.Kill($true)
$rc = 124
}
# Start-Process cannot merge the two streams into one file; fold them afterwards.
if ((Get-Item -LiteralPath $errLog -EA SilentlyContinue).Length) {
Get-Content -LiteralPath $errLog -Raw | Add-Content -LiteralPath $log
}
Remove-Item -LiteralPath $errLog -Force -EA SilentlyContinue
$summary = Select-String -LiteralPath $log -Pattern '\d+ assemblies.*' -EA SilentlyContinue
| Select-Object -Last 1 | ForEach-Object { $_.Matches[0].Value }
Add-Content -LiteralPath $historyFile -Value "$(Get-Date -Format o) $id exit=$rc $(if ($summary) { $summary } else { 'no-summary' })"
if ($rc -eq 0) {
Remove-Item -LiteralPath $log -Force -EA SilentlyContinue
}
else {
Write-Host "FAILED ($rc): $id -> $log"
}
$donePkgs++
Invoke-CachePrune
}
Set-Content -LiteralPath $cursorFile -Value $i
$donePages++
Write-Host "--- page $i done ($donePkgs packages this run) ---"
}
Write-Host "run complete: $donePkgs packages processed, failures kept in $logs"
# Reaching here means the sweep ran to completion; a real failure throws (ErrorActionPreference
# is Stop) and exits non-zero on its own. Without this the exit code trails whatever the last
# cmdlet happened to set.
exit 0

884
TestTools/nugetfuzz.cs

@ -0,0 +1,884 @@
// Copyright (c) 2026 Siegfried Pammer
//
// Permission is hereby granted, free of charge, to any person obtaining a copy of this
// software and associated documentation files (the "Software"), to deal in the Software
// without restriction, including without limitation the rights to use, copy, modify, merge,
// publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons
// to whom the Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all copies or
// substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
// INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
// PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
// FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
// OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
// DEALINGS IN THE SOFTWARE.
#:project ../ICSharpCode.Decompiler/ICSharpCode.Decompiler.csproj
#:package NuGet.Packaging@*
#:property PublishAot=false
// nugetfuzz: downloads nuget packages (sequentially), resolves their dependency
// closure, picks a lib TFM (installed runtime, or classic .NET Framework via the
// Microsoft.NETFramework.ReferenceAssemblies packages), then decompiles every
// assembly type-by-type and reports Debug.Assert failures / exceptions.
//
// usage: dotnet run nugetfuzz.cs -- <PackageId[@Version]>... | @packagelist.txt
using System.Diagnostics;
using System.IO.Compression;
using System.Net.Http.Json;
using System.Text;
using System.Text.Json;
using System.Text.RegularExpressions;
using ICSharpCode.Decompiler;
using ICSharpCode.Decompiler.CSharp;
using ICSharpCode.Decompiler.Metadata;
using NuGet.Frameworks;
using NuGet.Packaging;
using NuGet.Versioning;
Trace.Listeners.Clear();
Trace.Listeners.Add(new ThrowOnAssert());
try
{
Debug.Fail("self-test");
Console.Error.WriteLine("FATAL: assert hook not active, Debug.Assert would go unreported");
return 2;
}
catch (AssertionFailedException)
{
// hook works
}
var http = new HttpClient();
http.DefaultRequestHeaders.UserAgent.ParseAdd("nugetfuzz/1.0");
var cacheRoot = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".cache", "nugetfuzz");
var globalPackagesRoot = Environment.GetEnvironmentVariable("NUGET_PACKAGES")
?? Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".nuget", "packages");
var installedTfm = NuGetFramework.Parse($"net{Environment.Version.Major}.{Environment.Version.Minor}");
var net48 = NuGetFramework.Parse("net48");
var reducer = new FrameworkReducer();
var failures = new Dictionary<string, Finding>();
int assemblyCount = 0, typeCount = 0;
long charCount = 0, refsResolved = 0, refsTotal = 0;
bool verbose = Environment.GetEnvironmentVariable("NUGETFUZZ_VERBOSE") != null;
var dumpDir = Environment.GetEnvironmentVariable("NUGETFUZZ_DUMP");
if (dumpDir != null)
Directory.CreateDirectory(dumpDir);
var versionCache = new Dictionary<string, List<NuGetVersion>>();
var noSuchPackage = new HashSet<string>();
// Legacy framework-satellite assemblies whose nuget package id differs from the assembly name.
var assemblyPackageAlias = new Dictionary<string, string> {
["System.Web.Mvc"] = "Microsoft.AspNet.Mvc",
["System.Web.Razor"] = "Microsoft.AspNet.Razor",
["System.Web.WebPages"] = "Microsoft.AspNet.WebPages",
["System.Web.WebPages.Razor"] = "Microsoft.AspNet.WebPages",
["System.Web.WebPages.Deployment"] = "Microsoft.AspNet.WebPages",
["System.Web.Helpers"] = "Microsoft.AspNet.WebPages",
["System.Web.Http"] = "Microsoft.AspNet.WebApi.Core",
["System.Web.Http.WebHost"] = "Microsoft.AspNet.WebApi.WebHost",
["System.Web.Http.SelfHost"] = "Microsoft.AspNet.WebApi.SelfHost",
["System.Net.Http.Formatting"] = "Microsoft.AspNet.WebApi.Client",
["Microsoft.Practices.Unity"] = "Unity",
["System.Data.SqlServerCe"] = "Microsoft.SqlServer.Compact",
["Microsoft.Practices.ServiceLocation"] = "CommonServiceLocator",
};
const string NetCoreRefPack = "Microsoft.NETCore.App.Ref";
// WPF/WinForms assemblies of .NET (Core) live in the WindowsDesktop ref pack, not on nuget.
var windowsDesktopPrefixes = new[] {
"PresentationCore", "PresentationFramework", "WindowsBase", "System.Xaml",
"System.Windows.", "System.Drawing", "ReachFramework", "System.Printing",
"UIAutomation", "Microsoft.VisualBasic.Forms",
};
// Render the aggregate report of a sweep and exit; nothing is decompiled in this mode.
if (args is ["--report", var ledgerPath, ..])
{
var outPath = args.Length > 2 ? args[2] : Path.ChangeExtension(ledgerPath, ".html");
RenderLedger(ledgerPath, outPath);
Console.WriteLine($"report: {Path.GetFullPath(outPath)}");
return 0;
}
var packages = args
.SelectMany(a => a.StartsWith('@') ? File.ReadAllLines(a[1..]) : new[] { a })
.Select(l => l.Trim())
.Where(l => l.Length > 0 && !l.StartsWith('#'))
.ToList();
if (packages.Count == 0)
{
Console.Error.WriteLine("usage: nugetfuzz <PackageId[@Version]>... | @packagelist.txt");
Console.Error.WriteLine(" nugetfuzz --report <ledger.jsonl> [out.html]");
return 1;
}
foreach (var spec in packages)
{
try
{
await ProcessPackage(spec);
}
catch (Exception ex) when (
ex is InvalidOperationException && ex.Message.Contains("not found")
|| ex is InvalidDataException)
{
// Deleted/delisted package or corrupt nupkg on nuget.org - not a decompiler issue.
Console.WriteLine($" skip {spec}: {ex.Message}");
}
catch (Exception ex)
{
Report(spec, "-", "-", ex);
}
}
Console.WriteLine();
Console.WriteLine($"=== {assemblyCount} assemblies, {typeCount} types decompiled ({charCount} chars), {refsResolved}/{refsTotal} refs resolved, {failures.Count} distinct failures ({failures.Values.Sum(f => f.Count)} total) ===");
foreach (var entry in failures.Values.OrderByDescending(f => f.Count))
Console.WriteLine($"{entry.Count,6}x {entry.Describe()}");
// A sweep runs this program once per package, so per-run findings are appended to a
// shared ledger; `--report <ledger>` renders the aggregate. Without a ledger the run
// reports only itself.
var ledger = Environment.GetEnvironmentVariable("NUGETFUZZ_LEDGER");
if (ledger != null)
{
AppendToLedger(ledger, failures.Values, assemblyCount, typeCount, refsResolved, refsTotal);
Console.WriteLine($"ledger: {Path.GetFullPath(ledger)}");
}
else
{
var htmlPath = Path.GetFullPath(Environment.GetEnvironmentVariable("NUGETFUZZ_HTML") ?? "nugetfuzz-report.html");
WriteHtmlReport(htmlPath, failures.Values.ToList(), assemblyCount, typeCount, refsResolved, refsTotal, dumpDir);
Console.WriteLine($"report: {htmlPath}");
}
return failures.Count == 0 ? 0 : 1;
async Task ProcessPackage(string spec)
{
var parts = spec.Split('@');
var id = parts[0];
var version = parts.Length > 1 ? NuGetVersion.Parse(parts[1]) : await ResolveVersion(id, null);
if (version == null)
{
Console.WriteLine($"=== {id}: no versions found, skipping ===");
return;
}
Console.WriteLine($"=== {id} {version} ===");
var dir = await GetPackage(id, version);
var matchTarget = installedTfm;
var pick = PickLib(dir, matchTarget);
if (pick == null)
{
matchTarget = net48;
pick = PickLib(dir, matchTarget);
}
if (pick == null)
{
Console.WriteLine(" no compatible lib assemblies, skipping");
return;
}
var (libFw, libDir) = pick.Value;
Console.WriteLine($" lib: {libFw.GetShortFolderName()}");
var searchDirs = await CollectDependencies(dir, matchTarget, id);
searchDirs.Insert(0, libDir);
// The installed runtime dir is a last-resort fallback only: Microsoft.NETCore.App
// ships stub facades (e.g. WindowsBase.dll without System.Windows.Point) that must
// not shadow the real assemblies from ref packs or dependency packages.
string? fallbackDir = null;
if (libFw.Framework == FrameworkConstants.FrameworkIdentifiers.Net)
searchDirs.AddRange(await GetNetFxRefDirs(libFw));
else
fallbackDir = Path.GetDirectoryName(typeof(object).Assembly.Location);
foreach (var dll in Directory.GetFiles(libDir, "*.dll").OrderBy(f => f))
await DecompileAssembly(id, dll, searchDirs, matchTarget, fallbackDir);
}
// Walks the dependency closure breadth-first, downloading each package once and
// collecting the lib dir that best matches the root package's target framework.
async Task<List<string>> CollectDependencies(string rootDir, NuGetFramework matchTarget, string rootId)
{
var searchDirs = new List<string>();
var visited = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { rootId };
var queue = new Queue<string>();
queue.Enqueue(rootDir);
while (queue.Count > 0)
{
var dir = queue.Dequeue();
var nuspecPath = Directory.GetFiles(dir, "*.nuspec").FirstOrDefault();
if (nuspecPath == null)
continue;
var groups = new NuspecReader(nuspecPath).GetDependencyGroups().ToList();
var nearestGroupFw = reducer.GetNearest(matchTarget, groups.Select(g => g.TargetFramework));
var group = groups.FirstOrDefault(g => g.TargetFramework.Equals(nearestGroupFw));
if (group == null)
continue;
foreach (var dep in group.Packages)
{
if (!visited.Add(dep.Id))
continue;
try
{
var depVersion = await ResolveVersion(dep.Id, dep.VersionRange);
if (depVersion == null)
continue;
var depDir = await GetPackage(dep.Id, depVersion);
var depPick = PickLib(depDir, matchTarget);
if (depPick != null)
searchDirs.Add(depPick.Value.dir);
queue.Enqueue(depDir);
}
catch (Exception ex)
{
Console.WriteLine($" ! dep {dep.Id}: {ex.Message}");
}
}
}
return searchDirs;
}
// Picks the lib/<tfm> directory nearest to the given target framework.
// Old-style packages with DLLs directly under lib/ are treated as classic .NET Framework.
(NuGetFramework fw, string dir)? PickLib(string pkgDir, NuGetFramework target)
{
var libRoot = Path.Combine(pkgDir, "lib");
if (!Directory.Exists(libRoot))
return null;
var map = new Dictionary<NuGetFramework, string>();
foreach (var d in Directory.GetDirectories(libRoot))
{
NuGetFramework fw;
try
{
fw = NuGetFramework.ParseFolder(Path.GetFileName(d));
}
catch
{
continue;
}
if (fw.IsSpecificFramework && Directory.GetFiles(d, "*.dll").Length > 0)
map[fw] = d;
}
var nearest = reducer.GetNearest(target, map.Keys);
if (nearest != null)
return (nearest, map[nearest]);
if (target.Framework == FrameworkConstants.FrameworkIdentifiers.Net
&& Directory.GetFiles(libRoot, "*.dll").Length > 0)
{
return (target, libRoot);
}
return null;
}
// Classic .NET Framework has no reference assemblies on this machine; fetch the
// matching Microsoft.NETFramework.ReferenceAssemblies package instead.
async Task<List<string>> GetNetFxRefDirs(NuGetFramework libFw)
{
var shortName = libFw.GetShortFolderName();
if (!Regex.IsMatch(shortName, "^net[0-9]+$"))
shortName = "net48";
// Reference-assembly packs exist only for these TFMs; map anything else (net30,
// net401, ...) to the smallest pack that is a superset of the requested framework.
string[] knownPacks = ["net20", "net35", "net40", "net45", "net451", "net452", "net46", "net461", "net462", "net47", "net471", "net472", "net48", "net481"];
if (!knownPacks.Contains(shortName))
{
static Version DigitsVersion(string tfm) => Version.Parse(string.Join('.', tfm[3..].ToCharArray()));
var requested = DigitsVersion(shortName);
shortName = knownPacks.FirstOrDefault(k => DigitsVersion(k) >= requested) ?? "net48";
}
var id = "Microsoft.NETFramework.ReferenceAssemblies." + shortName;
var version = await ResolveVersion(id, null)
?? throw new InvalidOperationException($"cannot resolve {id}");
var dir = await GetPackage(id, version);
var fxRoot = Directory.GetDirectories(Path.Combine(dir, "build", ".NETFramework")).Single();
var dirs = new List<string> { fxRoot };
var facades = Path.Combine(fxRoot, "Facades");
if (Directory.Exists(facades))
dirs.Add(facades);
return dirs;
}
async Task<NuGetVersion?> ResolveVersion(string id, VersionRange? range)
{
var versions = await GetVersions(id)
?? throw new InvalidOperationException($"package {id} not found");
if (range != null)
return range.FindBestMatch(versions) ?? versions.LastOrDefault();
return versions.LastOrDefault(v => !v.IsPrerelease) ?? versions.LastOrDefault();
}
async Task<List<NuGetVersion>?> GetVersions(string id)
{
var key = id.ToLowerInvariant();
if (noSuchPackage.Contains(key))
return null;
if (versionCache.TryGetValue(key, out var cached))
return cached;
try
{
var index = await http.GetFromJsonAsync<VersionIndex>(
$"https://api.nuget.org/v3-flatcontainer/{key}/index.json");
var versions = index!.versions.Select(NuGetVersion.Parse).ToList();
versionCache[key] = versions;
return versions;
}
catch (HttpRequestException)
{
noSuchPackage.Add(key);
return null;
}
}
// Some packages reference assemblies they never declare as dependencies (e.g.
// itextsharp 5.5.13.6 -> itext.commons). Best effort: for any assembly reference
// not satisfiable from the current search dirs, try the same-named nuget package,
// preferring the package version that matches the assembly version.
// Called by LoggingResolver whenever a reference resolves nowhere - main-module refs
// and transitive refs alike (e.g. a fetched Microsoft.AspNet.Mvc needing WebPages).
// Tries framework ref packs and same-named/aliased nuget packages; returns true when
// the dll is available in `dirs` afterwards.
async Task<bool> TryFetchMissingRef(IAssemblyReference reference, Version? coreVersion, NuGetFramework matchTarget, List<string> dirs)
{
bool Satisfied()
{
lock (dirs)
{
return dirs.Any(d => File.Exists(Path.Combine(d, reference.Name + ".dll")));
}
}
if (coreVersion != null
&& windowsDesktopPrefixes.Any(p => reference.Name.StartsWith(p, StringComparison.Ordinal)))
{
await AddRefPack("Microsoft.WindowsDesktop.App.Ref", coreVersion, dirs);
if (Satisfied())
return true;
}
if (coreVersion != null && reference.Name.StartsWith("Microsoft.AspNetCore", StringComparison.Ordinal))
{
await AddRefPack("Microsoft.AspNetCore.App.Ref", coreVersion, dirs);
if (Satisfied())
return true;
}
const string EntLib = "Microsoft.Practices.EnterpriseLibrary.";
var packageId = assemblyPackageAlias.GetValueOrDefault(reference.Name)
?? (reference.Name.StartsWith(EntLib, StringComparison.Ordinal)
? "EnterpriseLibrary." + reference.Name[EntLib.Length..]
: reference.Name);
var versions = await GetVersions(packageId);
if (versions == null || versions.Count == 0)
return false;
var version = versions.FirstOrDefault(v => !v.IsPrerelease && v.Version == reference.Version)
?? versions.LastOrDefault(v => !v.IsPrerelease && v.Major == reference.Version?.Major)
?? versions.LastOrDefault(v => !v.IsPrerelease) ?? versions[^1];
try
{
var pkgDir = await GetPackage(packageId, version);
var pick = PickLib(pkgDir, matchTarget);
if (pick != null)
{
Console.WriteLine($" + undeclared dependency {reference.Name} -> {packageId} {version} ({pick.Value.fw.GetShortFolderName()})");
lock (dirs)
{
dirs.Add(pick.Value.dir);
}
}
}
catch (Exception ex)
{
Console.WriteLine($" ! undeclared dependency {reference.Name}: {ex.Message}");
}
return Satisfied();
}
// Adds the ref/<tfm> directory of a framework ref pack (WindowsDesktop, AspNetCore)
// matching the module's .NET version. No-op if already added or unavailable.
async Task AddRefPack(string packId, Version coreVersion, List<string> searchDirs)
{
var versions = await GetVersions(packId);
if (versions == null)
return;
// Never fall back to "newest available": the ref packs only go back to 3.0, so a
// netcoreapp1.x/2.x assembly would silently bind against a current (or prerelease)
// BCL and decompile as a wall of "Unknown result type". Better to add nothing and
// say so than to resolve against the wrong framework.
var version = versions.LastOrDefault(v => !v.IsPrerelease && v.Major == coreVersion.Major && v.Minor == coreVersion.Minor)
?? versions.LastOrDefault(v => !v.IsPrerelease && v.Major <= coreVersion.Major);
if (version == null)
{
Console.WriteLine($" ! ref pack {packId}: nothing published for {coreVersion}, references may bind to the wrong framework");
return;
}
try
{
var dir = await GetPackage(packId, version);
var refRoot = Path.Combine(dir, "ref");
var refDir = Directory.Exists(refRoot) ? Directory.GetDirectories(refRoot).FirstOrDefault() : null;
if (refDir != null && !searchDirs.Contains(refDir))
{
Console.WriteLine($" + ref pack {packId} {version}");
// Microsoft.NETCore.App.Ref ships stub facades -- its WindowsBase.dll is 15 KB and
// has no DependencyObject -- which shadow the real assemblies and collapse whole
// type hierarchies to Unknown. Keep it behind every other ref pack.
int netCorePack = searchDirs.FindIndex(
d => d.Contains(NetCoreRefPack, StringComparison.OrdinalIgnoreCase));
if (netCorePack >= 0 && packId != NetCoreRefPack)
searchDirs.Insert(netCorePack, refDir);
else
searchDirs.Add(refDir);
}
}
catch (Exception ex)
{
Console.WriteLine($" ! ref pack {packId}: {ex.Message}");
}
}
async Task<string> GetPackage(string id, NuGetVersion version)
{
var idLower = id.ToLowerInvariant();
var v = version.ToNormalizedString().ToLowerInvariant();
// The machine-wide NuGet cache is checked first: every `dotnet restore` on this box
// already extracts packages there in the same layout, so anything a build has pulled
// in never gets downloaded a second time. Our own cache stays the write target -
// nothing here ever writes into the shared one.
var globalDir = Path.Combine(globalPackagesRoot, idLower, v);
if (Directory.Exists(globalDir))
return globalDir;
var dir = Path.Combine(cacheRoot, idLower, v);
if (!Directory.Exists(dir))
{
Console.WriteLine($" downloading {id} {v}");
var bytes = await http.GetByteArrayAsync(
$"https://api.nuget.org/v3-flatcontainer/{idLower}/{v}/{idLower}.{v}.nupkg");
var tmp = dir + ".tmp";
if (Directory.Exists(tmp))
Directory.Delete(tmp, true);
ZipFile.ExtractToDirectory(new MemoryStream(bytes), tmp);
Directory.Move(tmp, dir);
}
return dir;
}
async Task DecompileAssembly(string pkg, string dllPath, List<string> searchDirs, NuGetFramework matchTarget, string? fallbackDir)
{
var name = Path.GetFileName(dllPath);
PEFile module;
try
{
module = new PEFile(dllPath);
}
catch (Exception ex) when (ex is BadImageFormatException or MetadataFileNotSupportedException)
{
Console.WriteLine($" skip {name}: not a managed assembly");
return;
}
using (module)
{
// ".NETCoreApp,Version=v5.0" -> 5.0; null for .NET Framework / netstandard modules.
Version? coreVersion = null;
var tfmId = module.DetectTargetFrameworkId();
var versionIndex = tfmId.IndexOf("Version=v", StringComparison.Ordinal);
if (tfmId.StartsWith(".NETCoreApp", StringComparison.Ordinal) && versionIndex >= 0)
coreVersion = Version.Parse(tfmId[(versionIndex + 9)..]);
var resolver = new UniversalAssemblyResolver(dllPath, throwOnError: false, tfmId);
var orderedDirs = new List<string>(searchDirs);
// Bind to the ref packs of the framework the assembly was built for. Otherwise the
// references still resolve -- against whatever shared runtime happens to be
// installed -- and every type that moved or was removed since then decompiles as
// "Unknown result type", which is indistinguishable from a decompiler bug.
// The desktop and web packs have to be seeded here rather than left to
// TryFetchMissingRef, which only runs when a reference resolves nowhere: the
// facades described in AddRefPack satisfy those references, so it never fires.
if (coreVersion != null)
{
var refNames = module.AssemblyReferences.Select(r => r.Name).ToList();
if (refNames.Any(n => windowsDesktopPrefixes.Any(p => n.StartsWith(p, StringComparison.Ordinal))))
await AddRefPack("Microsoft.WindowsDesktop.App.Ref", coreVersion, orderedDirs);
if (refNames.Any(n => n.StartsWith("Microsoft.AspNetCore", StringComparison.Ordinal)))
await AddRefPack("Microsoft.AspNetCore.App.Ref", coreVersion, orderedDirs);
await AddRefPack(NetCoreRefPack, coreVersion, orderedDirs);
}
// Last-resort only, for the same reason the ref packs are ordered as they are.
if (fallbackDir != null)
orderedDirs.Add(fallbackDir);
foreach (var d in orderedDirs)
resolver.AddSearchDirectory(d);
var fetchGate = new object();
var logResolver = new LoggingResolver(resolver, orderedDirs, reference => {
lock (fetchGate)
{
return TryFetchMissingRef(reference, coreVersion, matchTarget, orderedDirs)
.GetAwaiter().GetResult();
}
});
CSharpDecompiler decompiler;
try
{
decompiler = new CSharpDecompiler(module, logResolver, new DecompilerSettings());
}
catch (Exception ex)
{
Report(pkg, name, "<typesystem>", ex);
return;
}
Console.WriteLine($" {name}");
assemblyCount++;
foreach (var type in decompiler.TypeSystem.MainModule.TopLevelTypeDefinitions.ToList())
{
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(60));
decompiler.CancellationToken = cts.Token;
try
{
var code = decompiler.DecompileTypeAsString(type.FullTypeName);
typeCount++;
charCount += code.Length;
// Compiler-generated types (<Module>, <PrivateImplementationDetails>,
// VB$AnonymousType_*, ...) are still decompiled to shake out edge cases,
// but empty output is normal for them ('<' and '$' match the decompiler's
// own generated-name detection in SRMExtensions.IsGeneratedName).
if (string.IsNullOrWhiteSpace(code) && !type.Name.StartsWith('<') && !type.Name.Contains('$'))
Report(pkg, name, type.FullTypeName.ToString(), new InvalidDataException("empty decompilation output"));
else if (dumpDir != null)
File.WriteAllText(Path.Combine(dumpDir, SanitizeFileName($"{pkg}.{type.FullTypeName}.cs")), code);
// ILFunction warnings (unknown result types, stack type mismatches, invalid IL)
// surface in the output as "//IL_xxxx: <message>" comments.
foreach (var warning in Regex.Matches(code, @"//IL_[0-9a-fA-F]+: (.*)")
.Select(m => m.Groups[1].Value.Trim()).Distinct())
{
Report(pkg, name, type.FullTypeName.ToString(), new DecompilerWarning(warning));
}
}
catch (OperationCanceledException)
{
Report(pkg, name, type.FullTypeName.ToString(), new TimeoutException("decompilation timed out (60s)"));
}
catch (Exception ex)
{
Report(pkg, name, type.FullTypeName.ToString(), ex);
}
}
var resolutions = logResolver.Resolutions;
var unresolved = resolutions.Where(kv => kv.Value == null).Select(kv => kv.Key).OrderBy(k => k).ToList();
refsTotal += resolutions.Count;
refsResolved += resolutions.Count - unresolved.Count;
Console.WriteLine($" refs: {resolutions.Count - unresolved.Count}/{resolutions.Count} resolved");
if (verbose)
{
foreach (var kv in resolutions.OrderBy(kv => kv.Key))
Console.WriteLine($" {kv.Key} -> {kv.Value ?? "NOT FOUND"}");
}
else
{
foreach (var u in unresolved)
Console.WriteLine($" ! unresolved: {u}");
}
}
}
void Report(string pkg, string asm, string type, Exception ex)
{
// Key on the innermost exception so the same defect hit via many members dedupes.
var inner = ex;
while (inner.InnerException != null)
inner = inner.InnerException;
var topFrame = (inner.StackTrace ?? "").Split('\n')
.Select(l => l.Trim())
.FirstOrDefault(l => l.Contains("ICSharpCode.Decompiler")) ?? "";
var kind = inner is AssertionFailedException ? "ASSERT"
: inner is TimeoutException ? "TIMEOUT"
: inner is DecompilerWarning ? "WARNING" : "EXCEPTION";
var key = $"{kind}|{inner.GetType().Name}|{inner.Message}|{topFrame}";
var location = $"{pkg} / {asm} / {type}";
if (failures.TryGetValue(key, out var existing))
{
failures[key] = existing with { Count = existing.Count + 1 };
Console.WriteLine($" [{kind}] (dup) {type}: {FirstLine(inner.Message)}");
}
else
{
failures[key] = new Finding(kind, inner.GetType().Name, FirstLine(inner.Message),
FirstLine(topFrame), location, ex.ToString(), 1);
Console.WriteLine($" [{kind}] {location}");
foreach (var line in ex.ToString().Split('\n').Take(30))
Console.WriteLine(" " + line.TrimEnd());
}
}
// One JSON line per finding plus one totals line, appended under a lock so concurrent
// package runs of a sweep can share the file. JSONL because a sweep is append-only and
// may be interrupted at any point: a truncated last line costs one finding, not the file.
static void AppendToLedger(string path, IEnumerable<Finding> findings, int assemblies, int types,
long refsResolved, long refsTotal)
{
var lines = new List<string>();
foreach (var f in findings)
{
// Carry the run's reference-resolution state on every finding: a warning produced
// while references were missing is an artefact of the missing references far more
// often than a decompiler defect ("might be due to ... missing references" is what
// the warning itself says), and the report separates the two on this basis.
lines.Add(JsonSerializer.Serialize(new LedgerEntry("finding", f.Kind, f.ExceptionType, f.Message,
f.Frame, f.FirstLocation, f.Detail, f.Count, 0, 0, refsResolved, refsTotal)));
}
lines.Add(JsonSerializer.Serialize(new LedgerEntry("totals", "", "", "", "", "", "", 0,
assemblies, types, refsResolved, refsTotal)));
var full = Path.GetFullPath(path);
Directory.CreateDirectory(Path.GetDirectoryName(full)!);
// Retry briefly: a sweep can have several package runs finishing at once.
for (int attempt = 0; ; attempt++)
{
try
{
using var stream = new FileStream(full, FileMode.Append, FileAccess.Write, FileShare.Read);
using var writer = new StreamWriter(stream);
foreach (var line in lines)
writer.WriteLine(line);
return;
}
catch (IOException) when (attempt < 20)
{
Thread.Sleep(50);
}
}
}
// Aggregates a sweep's ledger: findings with the same kind/type/message/frame collapse
// into one row carrying the summed hit count, so a defect hit by 400 packages reads as
// one entry, which is the whole point of surveying a sweep.
static void RenderLedger(string ledgerPath, string outPath)
{
var merged = new Dictionary<string, Finding>();
var degraded = new HashSet<string>(); // findings only ever seen with references missing
var clean = new HashSet<string>();
int assemblies = 0, types = 0;
long refsResolved = 0, refsTotal = 0;
int malformed = 0;
foreach (var line in File.ReadLines(ledgerPath))
{
LedgerEntry? entry;
try
{
entry = JsonSerializer.Deserialize<LedgerEntry>(line);
}
catch (JsonException)
{
malformed++; // truncated tail of an interrupted run
continue;
}
if (entry == null)
continue;
if (entry.Record == "totals")
{
assemblies += entry.Assemblies;
types += entry.Types;
refsResolved += entry.RefsResolved;
refsTotal += entry.RefsTotal;
continue;
}
var key = $"{entry.Kind}|{entry.ExceptionType}|{entry.Message}|{entry.Frame}";
merged[key] = merged.TryGetValue(key, out var existing)
? existing with { Count = existing.Count + entry.Count }
: new Finding(entry.Kind, entry.ExceptionType, entry.Message, entry.Frame,
entry.FirstLocation, entry.Detail, entry.Count);
// Ledger lines written before this attribution existed carry 0/0; treat those as
// unknown rather than clean, so they are never presented as confirmed defects.
(entry.RefsTotal > 0 && entry.RefsResolved == entry.RefsTotal ? clean : degraded).Add(key);
}
if (malformed > 0)
Console.WriteLine($" ({malformed} malformed ledger lines skipped)");
// A finding seen even once with every reference resolved is trustworthy; one that only
// ever appeared in degraded runs is suspect.
degraded.ExceptWith(clean);
WriteHtmlReport(outPath, merged.Values.ToList(), assemblies, types, refsResolved, refsTotal, null, degraded);
}
// Self-contained HTML view of a fuzz run: findings grouped by kind, most frequent
// first, each expandable to the full exception text. Opens straight off disk - a
// sweep over thousands of packages is far easier to triage here than in scrollback.
static void WriteHtmlReport(string path, List<Finding> findings, int assemblies, int types,
long refsResolved, long refsTotal, string? dumpDir, HashSet<string>? degradedKeys = null)
{
bool IsDegraded(Finding f) =>
degradedKeys?.Contains($"{f.Kind}|{f.ExceptionType}|{f.Message}|{f.Frame}") == true;
string Esc(string s) => s.Replace("&", "&amp;").Replace("<", "&lt;").Replace(">", "&gt;");
var html = new StringBuilder();
html.AppendLine("""
<!doctype html><html><head><meta charset="utf-8"><title>nugetfuzz report</title>
<style>
:root { color-scheme: light dark; --bg:#fff; --fg:#1a1a1a; --muted:#666; --line:#d8d8d8; --chip:#f0f0f0; }
@media (prefers-color-scheme: dark) { :root { --bg:#16181c; --fg:#e6e6e6; --muted:#9aa0a6; --line:#333; --chip:#24262b; } }
body { background:var(--bg); color:var(--fg); font:14px/1.5 system-ui,sans-serif; margin:0 auto; padding:24px; max-width:1100px; }
h1 { font-size:20px; margin:0 0 4px; } h2 { font-size:16px; margin:26px 0 8px; }
.meta { color:var(--muted); font-size:13px; }
details { border:1px solid var(--line); border-radius:6px; margin:6px 0; background:var(--chip); }
summary { cursor:pointer; padding:8px 10px; font-family:ui-monospace,monospace; font-size:13px; }
pre { margin:0; padding:10px; overflow-x:auto; background:var(--bg); font:12px/1.45 ui-monospace,monospace; }
.count { display:inline-block; min-width:3.5em; font-weight:600; }
.suspect { font-size:11px; padding:1px 6px; border-radius:10px; background:#8a6d1f22;
color:#a8791f; border:1px solid #a8791f55; margin-left:6px; }
.ASSERT { border-left:4px solid #d97706; } .EXCEPTION { border-left:4px solid #dc2626; }
.TIMEOUT { border-left:4px solid #7c3aed; } .WARNING { border-left:4px solid #2563eb; }
#filter { width:100%; padding:8px; margin:8px 0; border:1px solid var(--line); border-radius:6px;
background:var(--bg); color:var(--fg); font:13px ui-monospace,monospace; }
</style></head><body>
""");
html.AppendLine("<h1>nugetfuzz report</h1>");
html.AppendLine($"<div class=meta>{assemblies} assemblies, {types} types decompiled, "
+ $"{refsResolved}/{refsTotal} references resolved, {findings.Count} distinct findings "
+ $"({findings.Sum(f => f.Count)} total)"
+ (dumpDir != null ? $"<br>decompiled sources dumped to {Esc(dumpDir)}" : "") + "</div>");
html.AppendLine("<input id=filter placeholder='filter by message, type, package or frame'>");
foreach (var kind in new[] { "ASSERT", "EXCEPTION", "TIMEOUT", "WARNING" })
{
var group = findings.Where(f => f.Kind == kind).OrderByDescending(f => f.Count).ToList();
if (group.Count == 0)
continue;
html.AppendLine($"<h2>{kind} ({group.Count} distinct, {group.Sum(f => f.Count)} hits)</h2>");
foreach (var f in group)
{
// Only ever seen while references were missing: flagged, not hidden - the
// warning text itself blames missing references, so it is weak evidence.
var suspect = IsDegraded(f)
? " <span class=suspect title='only seen in runs with unresolved references'>refs incomplete</span>"
: "";
html.AppendLine($"<details class={kind}><summary><span class=count>{f.Count}x</span> "
+ $"{Esc(f.ExceptionType)}: {Esc(f.Message)}{suspect}</summary>");
html.AppendLine($"<pre>first: {Esc(f.FirstLocation)}\nframe: {Esc(f.Frame)}\n\n{Esc(f.Detail)}</pre></details>");
}
}
html.AppendLine("""
<script>
const box = document.getElementById('filter');
box.addEventListener('input', () => {
const needle = box.value.toLowerCase();
for (const d of document.querySelectorAll('details'))
d.style.display = d.textContent.toLowerCase().includes(needle) ? '' : 'none';
});
</script>
</body></html>
""");
File.WriteAllText(path, html.ToString());
}
static string SanitizeFileName(string s)
=> string.Concat(s.Split(Path.GetInvalidFileNameChars()));
static string FirstLine(string s)
{
var i = s.IndexOfAny(['\r', '\n']);
return i < 0 ? s : s[..i];
}
// One deduplicated defect: Count counts every location that hit it, Detail keeps the
// full exception text of the first one for triage.
record Finding(string Kind, string ExceptionType, string Message, string Frame,
string FirstLocation, string Detail, int Count)
{
public string Describe()
=> $"[{Kind}] {ExceptionType}: {Message} @ {Frame} (first: {FirstLocation})";
}
// One line of the sweep ledger: either a deduplicated finding or a per-run totals record.
record LedgerEntry(string Record, string Kind, string ExceptionType, string Message, string Frame,
string FirstLocation, string Detail, int Count, int Assemblies, int Types,
long RefsResolved, long RefsTotal);
record VersionIndex(string[] versions);
class AssertionFailedException(string message) : Exception(message);
class DecompilerWarning(string message) : Exception(message);
// Resolves assembly references from the given directories (in priority order) before
// falling back to the wrapped resolver, and records every resolution and its outcome.
// The wrapped UniversalAssemblyResolver consults the installed runtime BEFORE its search
// directories, which lets runtime stub facades shadow ref-pack assemblies - hence the
// directory probing happens here, in our order.
class LoggingResolver(IAssemblyResolver inner, List<string> dirs, Func<IAssemblyReference, bool> onMiss) : IAssemblyResolver
{
public readonly Dictionary<string, string?> Resolutions = new();
readonly Dictionary<string, MetadataFile?> loaded = new();
public MetadataFile? Resolve(IAssemblyReference reference)
{
var file = ResolveFromDirs(reference) ?? inner.Resolve(reference);
if (file == null && onMiss(reference))
{
lock (loaded)
{
loaded.Remove(reference.Name);
}
file = ResolveFromDirs(reference);
}
return Track(reference.FullName, file);
}
public MetadataFile? ResolveModule(MetadataFile mainModule, string moduleName)
=> Track($"{mainModule.Name}!{moduleName}", inner.ResolveModule(mainModule, moduleName));
public Task<MetadataFile?> ResolveAsync(IAssemblyReference reference)
=> Task.FromResult(Resolve(reference));
public Task<MetadataFile?> ResolveModuleAsync(MetadataFile mainModule, string moduleName)
=> Task.FromResult(ResolveModule(mainModule, moduleName));
MetadataFile? ResolveFromDirs(IAssemblyReference reference)
{
string[] snapshot;
lock (dirs)
{
snapshot = dirs.ToArray();
}
lock (loaded)
{
if (loaded.TryGetValue(reference.Name, out var cached))
return cached;
MetadataFile? result = null;
foreach (var dir in snapshot)
{
var path = Path.Combine(dir, reference.Name + ".dll");
if (!File.Exists(path))
continue;
try
{
result = new PEFile(path);
break;
}
catch (Exception)
{
// unreadable candidate; keep probing lower-priority dirs
}
}
loaded[reference.Name] = result;
return result;
}
}
MetadataFile? Track(string key, MetadataFile? file)
{
lock (Resolutions)
{
Resolutions[key] = file?.FileName;
}
return file;
}
}
class ThrowOnAssert : TraceListener
{
public override void Fail(string? message, string? detailMessage)
=> throw new AssertionFailedException($"{message} {detailMessage}".Trim());
public override void Write(string? message)
{
}
public override void WriteLine(string? message)
{
}
}
Loading…
Cancel
Save