[ILASM] Add support for deterministic builds and PDB checksums #109091

amanasifkhalid · 2024-10-21T22:37:03Z

Follow-up to #85344. Fixes #8293. Fixes #62484.

This PR simply pulls in the SHA-256 wrappers added in #109559 into the prototype in #85344.

…a256 in c.

…me out in CI

…sm-det

amanasifkhalid · 2025-01-06T18:43:08Z

The parent PR's PDB checksum implementation wasn't null-terminating the hash algorithm prefix ("SHA256", etc). With this fixed, dumpbin seems to be able to read the new debug directory entries correctly. For example, here's the debug directory dump for a binary built with /PDB /DET specified:

Debug Directories

        Time Type        Size      RVA  Pointer
    -------- ------- -------- -------- --------
    00000000 cv            42 00002160      360    Format: RSDS, {E4632B33-1CEC-8C2E-513D-F9C4DF5B2A1E}, 1, C:\wk\runtime2\IL-RT\trythrowexcept_d.pdb
    00000000 pdbhash       27 000021A2      3A2    SHA256: 33 2B 63 E4 EC 1C 2E 8C 51 3D F9 C4 DF 5B 2A 1E 2B FF 5A 00 2B A7 C0 AD 6E 7B 34 30 34 59 64 BA
    00000000 repro          0 00000000        0

amanasifkhalid · 2025-01-06T22:46:45Z

Determinism tests are passing in CI, too. cc @dotnet/jit-contrib, @jkotas could you PTAL? Thanks!

src/coreclr/ilasm/writer.cpp

jkotas · 2025-01-06T23:02:51Z

src/coreclr/dlls/mscorpe/pewriter.cpp

@@ -614,6 +614,12 @@ HRESULT PEWriter::Init(PESectionMan *pFrom, DWORD createFlags)
        m_ntHeaders->FileHeader.Characteristics |= VAL16(IMAGE_FILE_RELOCS_STRIPPED);
    }

+    if (createFlags & ICEE_CREATE_FILE_DET)
+    {
+        // We don't need a meaningful date/time stamp -- just a consistent one


jkotas · 2025-01-06T23:03:37Z

src/coreclr/ilasm/assem.cpp

+#if !defined(_WIN32) && !defined(__APPLE__)
+        if (!IsOpenSslAvailable())
+        {
+            fprintf(stderr, "\nWarning: OpenSSL not available. Disabling build determinism.\n");


I think this should be an error.

jkotas · 2025-01-06T23:06:12Z

src/tests/Common/CLRTest.Jit.targets

+            proc.kill()
+            sys.exit(1)
+
+        if hash != hash_file(inputAssemblyName):


Would it be better to compare the content of the whole file instead of just a hash?

That seems simpler; I'll move this over, too.

amanasifkhalid · 2025-01-07T19:14:58Z

Timestamps in the debug directory dumps are no longer zero, but are taken from the bytes after the GUID in the PDB checksum:

  Debug Directories

        Time Type        Size      RVA  Pointer
    -------- ------- -------- -------- --------
    005AFF2B cv            42 00002160      360    Format: RSDS, {E4632B33-1CEC-8C2E-513D-F9C4DF5B2A1E}, 1, C:\wk\runtime2\IL-RT\trythrowexcept_d.pdb
    005AFF2B pdbhash       27 000021A2      3A2    SHA256: 33 2B 63 E4 EC 1C 2E 8C 51 3D F9 C4 DF 5B 2A 1E 2B FF 5A 00 2B A7 C0 AD 6E 7B 34 30 34 59 64 BA
    00000000 repro          0 00000000        0

And the header timestamp matches:

FILE HEADER VALUES
             14C machine (x86)
               2 number of sections
          5AFF2B time date stamp
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             102 characteristics
                   Executable
                   32 bit word machine

amanasifkhalid · 2025-01-07T19:27:49Z

/azp run runtime-coreclr ilasm

azure-pipelines · 2025-01-07T19:28:01Z

Azure Pipelines successfully started running 1 pipeline(s).

jkotas · 2025-01-07T21:26:52Z

src/coreclr/ilasm/writer.cpp

+        // on the IMDInternalEmit interface.
+        //
+        // When the CLSID_CorMetaDataDispenser instance above has activated against a current
+        // clr.dll (which is the only supported configuration for the determinism feature), it is


ilasm in .NET Core has statically linked copy of metadata implementation. This comment that talks about clr.dll is not relevant here. Was it copied from .NET Framework port?

BTW: I do not think you would be able to use IMDInternalEmit interface for the .NET Framework port. It would create versioning problems. I think we will need a proper new interface defined for .NET Framework port.

Was it copied from .NET Framework port?

I believe so -- this comment is from the initial commit from the parent PR.

jkotas · 2025-01-07T21:31:39Z

src/coreclr/ilasm/writer.cpp

+
+            ULONG timestamp;
+            memcpy_s(&timestamp, sizeof(ULONG), pdbChecksum + sizeof(GUID), sizeof(ULONG));
+            m_pPortablePdbWriter->SetTimestamp(timestamp);


Do we need to generate deterministic GUID and the timestamp from the content hash even when we are not generating the PDBs?

That seems better than leaving the GUID/timestamp as zero. Are you suggesting we derive them from something like the hash we use for the MVID below, if the PDB stream isn't available?

Yes. I believe that generating .dll without .pdbs is fairly common.

It would be best to avoid depending on PDB stream to generate the GUID/timestamp in all cases, even when we are genering the PDBs.

amanasifkhalid · 2025-01-07T22:09:06Z

The determinism tests were failing because I was using different assembly names as inputs into ilasm, which would change the CodeView debug directory entry. This should be fixed

amanasifkhalid · 2025-01-07T22:09:11Z

/azp run runtime-coreclr ilasm

azure-pipelines · 2025-01-07T22:09:23Z

Azure Pipelines successfully started running 1 pipeline(s).

amanasifkhalid · 2025-01-08T18:50:47Z

/azp run runtime-coreclr ilasm

azure-pipelines · 2025-01-08T18:50:59Z

Azure Pipelines successfully started running 1 pipeline(s).

amanasifkhalid · 2025-01-08T19:43:25Z

/azp run runtime-coreclr ilasm

azure-pipelines · 2025-01-08T19:43:38Z

Azure Pipelines successfully started running 1 pipeline(s).

TIHan added 30 commits April 25, 2023 11:33

Added '/DET' flag to ILASM for deterministic compilations

1419f57

Using 'm_pInternalEmitForDeterministicMvid'

d3b329c

Removing use of BCRYPT APIs. Added TODOs.

0357bf1

Add use of BCYRPT for windows-only. Added TODO for non-win32.

54147cc

Preliminary work for deterministic PDB guid

5516e77

Fixed ordering problem

d6a16fa

Computing checksum and using result for the PDB guid

f2e4d2c

Refactoring apis

89df2d8

Added PdbChecksum and Deterministic Debug Directory entries

7374450

Minor whitespace cleanup

7a0dfcb

Merge remote-tracking branch 'upstream/main' into ilasm-det

615c6ce

mering with main

68482a4

Fix build

ef59bed

Added IILAsmPortablePdbWriter interface

028a8b6

Checking deterministic output for ILASM for all roundtrip tests

e1a10f8

Fixed getting interface. Testing determinism of pdb.

8ba6868

add comment

c023315

Remove use of goto

9e14b48

Added sha256 for non-win32

4038a1f

Using different impl for getting sha256 of a file in python. Using sh…

69fadf4

…a256 in c.

Trying to fix build

936f0a3

Fix build

d6b4258

disable testing determinism due to timeouts

9c1a645

Merge remote-tracking branch 'upstream/main' into ilasm-det

e8b9b7f

Temporarily disable emitting debug directories to see if the tests ti…

3331d9d

…me out in CI

Merge branch 'ilasm-det' of https://github.com/TIHan/runtime into ila…

7df2c80

…sm-det

Merge remote-tracking branch 'upstream/main' into ilasm-det

e78450f

Renable deterministic tests and debug directory creation

8e357b8

Added sha256.cpp

8867295

Remove sha256 impl for now

6dd077f

amanasifkhalid changed the title ~~[ILASM] Add support for deterministic compilation~~ [ILASM] Add support for deterministic builds and PDB checksums Jan 6, 2025

jkotas reviewed Jan 6, 2025

View reviewed changes

src/coreclr/ilasm/writer.cpp Show resolved Hide resolved

jkotas reviewed Jan 6, 2025

View reviewed changes

amanasifkhalid added 4 commits January 7, 2025 12:16

Error out if OpenSSL not available

2614e36

Compare files instead of hashes for determinism test

9247c7a

Computed timestamp for debug dirs from PDB checksum

92ef0a9

Extend PEWriter API surface to change file header timestamp

b7ad2c6

jkotas reviewed Jan 7, 2025

View reviewed changes

amanasifkhalid added 2 commits January 7, 2025 16:54

Fix ILAsm roundtrip tests

b15f810

Remove comment

db1cdfe

build-analysis bot mentioned this pull request Jan 7, 2025

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." dotnet/dnceng#1883

Open

3 tasks

amanasifkhalid added 3 commits January 8, 2025 13:22

Don't use PDB checksum for GUID/timestamp

2ce7321

Fix file renaming in determinism tests on Windows

266538d

Error message consistency

c957abf

Fix build for clang

f3da380

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ILASM] Add support for deterministic builds and PDB checksums #109091

[ILASM] Add support for deterministic builds and PDB checksums #109091

amanasifkhalid commented Oct 21, 2024 •

edited

Loading

amanasifkhalid commented Jan 6, 2025 •

edited

Loading

amanasifkhalid commented Jan 6, 2025

jkotas Jan 6, 2025

jkotas Jan 6, 2025

jkotas Jan 6, 2025

amanasifkhalid Jan 7, 2025

amanasifkhalid commented Jan 7, 2025

amanasifkhalid commented Jan 7, 2025

azure-pipelines bot commented Jan 7, 2025

jkotas Jan 7, 2025

jkotas Jan 7, 2025

amanasifkhalid Jan 7, 2025

jkotas Jan 7, 2025 •

edited

Loading

amanasifkhalid Jan 7, 2025

jkotas Jan 7, 2025

jkotas Jan 7, 2025

amanasifkhalid commented Jan 7, 2025

amanasifkhalid commented Jan 7, 2025

azure-pipelines bot commented Jan 7, 2025

amanasifkhalid commented Jan 8, 2025

azure-pipelines bot commented Jan 8, 2025

amanasifkhalid commented Jan 8, 2025

azure-pipelines bot commented Jan 8, 2025

[ILASM] Add support for deterministic builds and PDB checksums #109091

Are you sure you want to change the base?

[ILASM] Add support for deterministic builds and PDB checksums #109091

Conversation

amanasifkhalid commented Oct 21, 2024 • edited Loading

amanasifkhalid commented Jan 6, 2025 • edited Loading

amanasifkhalid commented Jan 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amanasifkhalid commented Jan 7, 2025

amanasifkhalid commented Jan 7, 2025

azure-pipelines bot commented Jan 7, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkotas Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amanasifkhalid commented Jan 7, 2025

amanasifkhalid commented Jan 7, 2025

azure-pipelines bot commented Jan 7, 2025

amanasifkhalid commented Jan 8, 2025

azure-pipelines bot commented Jan 8, 2025

amanasifkhalid commented Jan 8, 2025

azure-pipelines bot commented Jan 8, 2025

amanasifkhalid commented Oct 21, 2024 •

edited

Loading

amanasifkhalid commented Jan 6, 2025 •

edited

Loading

jkotas Jan 7, 2025 •

edited

Loading