"Percentage complete" reporting does not take into account skipped files with --overwrite=ifSourceNewer

[michafaw]

(Reposted from a comment in #2415)

Which version of the AzCopy was used?

10.26.0

Which platform are you using? (ex: Windows, Mac, Linux)

Windows 11

What command did you run?

azcopy cp source destination --recursive=true --overwrite=ifSourceNewer --as-subdir=false --output-type=json --skip-version-check
(source is a local file path, destination is an Azure Blob Container)

What problem was encountered?

The reported progress percentage is inaccurate if existing files at the destination cause some copy operations to be skipped due to --overwrite=ifSourceNewer. This is counter to the guidance given here by @nakulkar-msft in 2023, but does line up with the (possibly out of date) calculation info provided here in 2020. In other words, the reported progress represents only the bytes that have been transferred, not those that have been transferred+skipped as expected.

• From observation the current calculation for percent is still: bytes transferred / total bytes
• Expected (bytes transferred + bytes skipped) / total bytes OR bytes transferred / total bytes needing to be transferred

How can we reproduce the problem in the simplest way?

Workflow 1: Run the command above, with the destination empty. All progress reporting will be accurate.✅ Without making changes to the source or destination files, run the command again. The "live" percentage will stay at 0.0% as it progresses, ending at 0.0% instead of 100.0%.❌ This can be seen at the end of the log file:

2024/09/20 19:20:53 0.0 %, 0 Done, 0 Failed, 0 Pending, 128 Skipped, 128 Total, 
2024/09/20 19:20:53 Closing Log

Workflow 2: Run the command above, with the destination empty. All progress reporting will be accurate.✅ Add files to the source directory and run the command again. The "live" percentage will increase as it copies the new files, but will not increase for skipped files. The final percent will only represent the new files, such as 43.7%.❌

2024/09/20 19:24:02 43.7 %, 23 Done, 0 Failed, 0 Pending, 128 Skipped, 151 Total, 
2024/09/20 19:24:02 Closing Log

Have you found a mitigation/solution?

No. It's a reporting issue that causes incorrect/misleading UI in my customer's product, but the actual work being done is still accurate (thanks!).