Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error when closing file #1199

Open
fj-morales opened this issue Dec 11, 2024 · 2 comments
Open

Error when closing file #1199

fj-morales opened this issue Dec 11, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@fj-morales
Copy link

fj-morales commented Dec 11, 2024

Mountpoint for Amazon S3 version

mount-s3 1.13.0

AWS Region

NA

Describe the running environment

A Centos7 vm connecting to a non-AWS S3 provider, based on OpenStack and CEPH

Mountpoint options

export AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxx
export AWS_ENDPOINT_URL=https://mys3.data

mount-s3   --region=Default --force-path-style --allow-delete --allow-overwrite --upload-checksums=off  --dir-mode=0777 --file-mode=0666 --allow-other  mybucket   /my_local_dir -d -f

What happened?

Attempted to copy a file to the bucket throws i/o error when closing the file. The file is actually well-written (verified by sha256sum checksum).

cp -f testfile_100M.bin /my_local_dir/
cp: overwrite ‘/my_local_dir/testfile_100M.bin’? y
cp: failed to close ‘/my_local_dir/testfile_100M.bin’: Input/output error

Relevant log output

2024-12-11T16:42:13.040462Z  WARN getxattr{req=6 ino=2 name="security.capability"}: mountpoint_s3::fuse: getxattr failed: operation not supported by Mountpoint
2024-12-11T16:42:13.040554Z  WARN removexattr{req=7 ino=2 name="security.ima"}: mountpoint_s3::fuse: removexattr failed: operation not supported by Mountpoint
2024-12-11T16:42:13.318212Z  WARN put_object_single{id=8 bucket="mybucket" key="testfile_100M.bin"}: mountpoint_s3_client::s3_crt_client: duration=73.368095ms request_id=tx00000af56083bb51794bc-006759c0e5-92f3e-default error=ServiceError(PreconditionFailed) meta request failed
2024-12-11T16:42:13.327727Z  WARN flush{req=1609 ino=2 fh=1 pid=12097 name="testfile_100M.bin"}: mountpoint_s3::fuse: flush failed: put failed: put request failed: Service error: At least one of the pre-conditions you specified did not hold
@fj-morales fj-morales added the bug Something isn't working label Dec 11, 2024
@muddyfish
Copy link
Contributor

Hello, thank you for making this report. From the logs, it appears that your S3 implementation is returning a 412 precondition failed response to Mountpoint. Running with --debug --debug-crt should help verify that.

@fj-morales
Copy link
Author

fj-morales commented Dec 12, 2024

Hi, here's the log with the extra information from --debug --debug-crt. I'm not exactly sure what I'm searching for:
log.txt

github-merge-queue bot pushed a commit that referenced this issue Jan 7, 2025
We recently saw an error in #1199 where "Header not found" was emitted,
but its really unclear what header was missing.

2024-12-12T18:33:59.379478Z WARN flush{req=1609 ino=2 fh=1 pid=29257
name="testfile_100M.bin"}: mountpoint_s3::fuse: flush failed: put
failed: put request failed: Client error: Internal S3 client error:
Header not found

This change updates the `HeadersError::HeaderNotFound` enum variant to
contain a copy of the header name, such that error messages can emit it
for debugging purposes.

It may make more sense to have all the header names we use statically
defined somewhere, such that we could include a static reference to the
header and avoid allocating for an error message. However, we don't
expect there to be any performance regression introduced by this change.
This move to static values could be made later.

### Does this change impact existing behavior?

Header not found and invalid header value errors will now include the
header name when printing the error message.

The enum variants change meaning any code using the enum may be
impacted.

### Does this change need a changelog entry?

Not for Mountpoint itself. I have added a change log entry to
`mountpoint-s3-crt` since it is a breaking API change.

---

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license and I agree to the terms of
the [Developer Certificate of Origin
(DCO)](https://developercertificate.org/).

---------

Signed-off-by: Daniel Carl Jones <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants