Gcsfuse v2.9.0

Streaming Writes:

Streaming writes is a new write path that uploads data directly to Google Cloud Storage (GCS) as it's written. The previous, and currently default write path temporarily stages the entire write in a local directory, uploading to GCS on close/fsync. This reduces both latency and disk space usage, making it particularly beneficial for large, sequential writes such as checkpoints. To enable streaming writes, use the following:

Command-line flag: --enable-streaming-writes
Configuration file: write: enable-streaming-writes:true

This will become the default write path in the future.

Memory Usage: Each file opened for streaming writes will consume approximately 64MB of RAM during the upload process. This memory is released when the file handle is closed. This should be considered when planning resource allocation for applications using streaming writes.

Important Considerations and Caveats:

1. New files, Sequential Writes: Streaming writes are designed for sequential writes to a new, single file only. Modifying existing files, or doing out-of-order writes (whether from the same file handle or concurrent writes from multiple file handles) will cause GCSFuse to automatically revert to the existing behavior of staging writes to a temporary file on disk. An informational log message will be emitted when this fallback occurs.

2. File System Semantics Change:

   • FSync Operation does not finalize the Object: When streaming writes are enabled, the fsync operation will not finalize the object on GCS. Instead, the object will be finalized only when the file is closed. This is a key difference from the previous behavior and should be considered when using streaming writes. Relying on fsync for data durability with streaming writes enabled is not recommended. Data is only guaranteed to be on GCS after the file is closed.
   • Read Operations During Write: Today the application can read the data when the writes are in progress for that file. With buffered writes, the application will not be able to read until the object is finalized i.e, fclose() is called. Applications should ensure that they do not attempt to read from a file while it is being written to using streaming writes.

3. Write Stalls and Chunk Uploads: Streaming writes do not currently implement chunk-level timeouts or retries. Write operations may stall, and chunk uploads that encounter errors will eventually fail after the default 32-second deadline.

Bug Fixes and Improvements

• GCSFuse would crash if a file-path with basename "\n" is looked up. It's now fixed. GCSFuse now correctly returns ENOENT (File Not Found) in such cases. (#2939)
• GCSFuse now returns EEXIST error code instead of EIO when attempting to create a directory that already exists on HNS buckets. (#2935)

What's Changed

• Close github issues if Cx doesn't response for 2 weeks by @raj-prince in #2896
• Skipping KLC test for kernel version b/w v6.9.x and v6.12.x by @raj-prince in #2901
• Skipping Kernel List Cache E2E Test on the Unsupported Kernel Version by @raj-prince in #2904
• Add stale_handle integration tests to run_tests_mounted_directory.sh by @vipnydav in #2903
• Refactor list_large_dir tests to use storage client to delete directories instead of gcloud. by @vipnydav in #2905
• Merge gcsfuse-prelaunch to gcsfuse/master by @gargnitingoogle in #2906
• Update Semantics doc about changes in GCSFuse behavior after Write Failure Improvements by @vipnydav in #2877
• Increasing the wait time for cache population by @vadlakondaswetha in #2907
• Skip monitoring tests from cd by @kislaykishore in #2921
• [Move Object] Support on ZB by @Tulsishah in #2912
• Semantic doc update for Kernel List Cache Bug by @raj-prince in #2909
• Enable dynamic creation of storage clients based on bucket type and specified protocol by @abhishek10004 in #2922
• Update troubleshooting doc for crash unlock of unlocked mutex by @raj-prince in #2895
• Partially Revert "Disable tpc and emulator tests in e2e" by @gargnitingoogle in #2926
• Empty_gcs_file tests under streaming_writes package by @vadlakondaswetha in #2910
• Update direct dependencies by @kislaykishore in #2924
• Skip test when testing with installed package by @kislaykishore in #2928
• Onboard local file integration tests to stretchr/testify and export CommonLocalFileTestSuite by @meet2mky in #2902
• Fixing reader growth logic & logging errors in MRD::Read by @abhishek10004 in #2932
• added stat calls in truncate tests for streaming writes by @vadlakondaswetha in #2923
• Fix "Already Exist" error code for folder creation by @Tulsishah in #2935
• Stopping use of timer based cleanup logic for MRD by @abhishek10004 in #2931
• Fix crash when the file name being statted is just "\n" by @kislaykishore in #2933
• Fixing the truncate test for empty GCS file in streaming writes flow by @vadlakondaswetha in #2941
• Handle write edge case when first write is out of order write by @ashmeenkaur in #2929
• Fix TPC by @gargnitingoogle in #2940
• Streaming writes config validation changes by @ashmeenkaur in #2947
• Finalize object for any error in buffered writes flow by @ashmeenkaur in #2948
• [Readability fix] Be explicit that nil child is being returned. by @kislaykishore in #2943
• Add streaming writes config to useragent by @ashmeenkaur in #2949
• Make test readable by @Tulsishah in #2944
• Remove chunk retry timeout for buffered writes writer by @ashmeenkaur in #2954
• Streaming writes config defaults by @ashmeenkaur in #2955
• Add Symlink/ReadFile tests for streaming writes scenario and run CommonLocalFileTestSuite from Streaming writes package. by @meet2mky in #2938
• fix re-flush for streaming writes by @ashmeenkaur in #2956
• Fixing MinObject related issues in MRDWrapper by @abhishek10004 in #2936
• Improving error conversion logic by @abhishek10004 in #2953
• Close upload failure channel in case of error during finalize by @ashmeenkaur in #2957
• Doing flush for pending writes on rename file by @vadlakondaswetha in #2958
• Add streaming_writes integration tests to run_tests_mounted_directory.sh by @ashmeenkaur in #2966
• upgrade to golang 1.23.5 by @ashmeenkaur in #2965

New Contributors

• @meet2mky made their first contribution in #2902

Full Changelog: v2.8.0...v2.9.0

Gcsfuse v2.8.0

Bug Fixes & Improvements

Write Precondition Checks

Prior to v2.8.0, GCSfuse ignored precondition requests when the same object was being written to from different mounts simultaneously. Now, if a file being written to has already been modified or deleted by another mount, GCSfuse will return an ESTALE error. The new behavior provides users with a clear indication of potential conflicts between different mounts.

gRPC

Added retries for authenticated requests.

Negative Stat Cache

The default time to live (TTL) for stat-cache negative entries is reduced from 60 seconds to 5 seconds with this release. This is done to ensure improved coherency in multi-node AI/ML tasks.

Dependency Upgrades / CVE fixes

Upgraded dependencies for better stability PR#2800 PR#2801 PR#2847

What's Changed

• rename test function so common tests are also run by @ashmeenkaur in #2846
• Bump go.opentelemetry.io/otel/exporters/stdout/stdouttrace from 1.32.0 to 1.33.0 by @dependabot in #2801
• Bump go.opentelemetry.io/otel/exporters/prometheus from 0.54.0 to 0.55.0 by @dependabot in #2800
• Bump google.golang.org/protobuf from 1.36.0 to 1.36.1 by @dependabot in #2847
• Fix mounted directory script by @Tulsishah in #2849
• Add troubleshooting guide about implicit dir by @Tulsishah in #2848
• Fix typo by @Tulsishah in #2851
• Bump Dependabot PR by @Tulsishah in #2850
• Fix flaky parallel dirOps e2e tests by @Tulsishah in #2852
• Fix kernel cache ttl test failure for GKE by @Tulsishah in #2853
• Make OTel metrics docstring and names same as that of OC. by @kislaykishore in #2857
• Add Integration Tests for Write Failure Improvements by @vipnydav in #2845
• streaming writes flush and sync flow by @ashmeenkaur in #2856
• Make OTel the default for metrics by @kislaykishore in #2859
• [Move Object] Flag for MoveObject API by @Tulsishah in #2870
• Avoiding nested panic while lookUpLocalFileInode by @raj-prince in #2869
• Refactor: ValidateStaleNFSFileHandleError for ARM64 Compatibility by @vipnydav in #2872
• Set the precondition-errors flag to default true as feature is complete. by @vipnydav in #2867
• Disable write back cache when streaming writes are enabled by @ashmeenkaur in #2858
• [Move Object] fs layer changes by @Tulsishah in #2871
• Handle global max blocks semaphore properly by @ashmeenkaur in #2860
• Reduce number of iterations for flake-detector by @kislaykishore in #2874
• fix: Retry gRPC UNAUTHENTICATED errors like 401s. by @cjc25 in #2781
• Composite tests for negative entry in stat cache after object/directory deletion by @codechanges in #2868
• Remove experimental prefix from streaming writes by @ashmeenkaur in #2880
• Revert "Enable OTel metrics by default (#2859)" by @kislaykishore in #2882
• [Move Object] Rename benchmark e2e perf test by @Tulsishah in #2878
• Refactor write failure composite tests by @vipnydav in #2854
• Create global max blocks semaphore only once at file system level. by @ashmeenkaur in #2883
• review comment from PR: #2860 by @ashmeenkaur in #2884
• Remove unnecessary comment about lock by @Tulsishah in #2886
• Negative cache TTL Fix by @ankitaluthra1 in #2837
• Negative cache integration tests by @ankitaluthra1 in #2887
• Upgrading direct go dependencies by @raj-prince in #2888
• Turn on the flag for Move object by @Tulsishah in #2889
• Update wrapped error while returning FileClobberedError after calling clobbered method by @vipnydav in #2885
• Added new integration test cases for streaming writes flow by @vadlakondaswetha in #2842
• Write Failure Improvements - Do not deviate from local file system when file is unlinked from the same mount by @vipnydav in #2890
• Downgrade go-sdk from v1.50.0 to v1.49.0 by @raj-prince in #2891
• Revert "Turn on the flag for Move object" by @Tulsishah in #2892
• Semantic doc for unsupported object with suffix \n by @raj-prince in #2893

New Contributors

• @cjc25 made their first contribution in #2781

Full Changelog: v2.7.0...v2.8.0

Gcsfuse v2.7.0

Write Stability improvements: Reduces write tail latencies by increasing the number of retries in case of a write operation stall. The retries are configured to kick in if the stall is greater than 10 seconds. This feature will enhance the application's ability to successfully complete write operations, especially for large files such as checkpoint writes.

Bug Fixes & Improvements:

1. gRPC: Fixed a rare pre-condition error in gRPC client protocol. The pre-condition error arises when moving a directory to an empty directory. PR#2838
2. Negative metadata caches entries upon the deletion of an object/folder. PR#2822

Dependency Upgrades / CVE fixes:

1. Fixing CVE in the crypto library. PR #2809
2. Upgraded dependencies for better stability
   i. go-lang version from 1.23.3 to 1.23.4
   ii. Other dependency upgrade
   PR#2762 , PR#2756, PR#2757, PR#2758, PR#2759, PR#2760, PR#2761, PR#2731

Gcsfuse v2.5.2

Dependency Upgrades / CVE fixes:
This release is built on top of v2.5.1 with the additional security/CVE fixes for indirect dependency x/crypto library.

Gcsfuse v2.6.0

gRPC:

• A new gRPC client version provides significant read performance improvements at lower CPU utilization.
• Allowlist access is no longer needed to use gRPC. To use gRPC, pass the cli flag --client-protocol=grpc OR add gcs-connection:client-protocol: grpc in the mount-config file.
• Known issues:
  • gRPC: File system write operation hangs while repeatedly writing over a single > 16MiB file - Issue#2784
  • gRPC: File system returns Input/output error while repeatedly writing over a single < 16MiB file - Issue#2783

List Payload Reduction:

• The Cloud Storage ListObjects API now retrieves only essential GCS object attributes required for serving list requests, resulting in significantly less response payloads and reduced latency. PR#2627

Bug Fixes & Improvements:

• All flags are now supported in the persistent mounting via /etc/fstab. PR#2478
• SIGTERM to unmount: Sending SIGTERM signal to GCSFuse process will now unmount the filesystem.
• Enhanced error handling for background processes now captures runtime crash logs. Users can specify a log file path to store these logs (the file will have a .stderr extension).

Gcsfuse v2.5.1

Bug Fixes:
Improved error handling: GCSFuse will now retry requests to hierarchical namespace bucket APIs that encounter deadline exceeded errors. This improves stability and prevents unnecessary failures, especially in high latency environments.

Gcsfuse v2.5.0

• Hierarchical Namespace Enabled Buckets: With this release users can also mount buckets with hierarchical namespace enabled. HNS-enabled buckets offer several advantages over standard buckets when used with cloud storage fuse:
  • Renaming directories is now supported natively with Hierarchical Namespace Buckets, where Rename operation is fast and atomic to benefit workloads such as AI/ML checkpointing.
  • Users don't need to specify the --implicit-dirs command-line option when using hierarchical namespace enabled buckets. HNS buckets inherently understand directories, so gcsfuse does not need to simulate directories using placeholder objects ( 0-byte objects ending with '/' ).
  • Kernel’s Lookup operations are also improved in HNS buckets. The Objects.list api is replaced with a more efficient Folder:get api.
• Config CLI Parity:
  • All params in config-file will be available as CLI flags and vice versa going forward. Documentation links:
    • CLI Flags
    • Config File
  • Deprecated and hidden flags won't show up in the help-doc any more.
• Write Enhancements:
  • Information of newly created files is added to the type cache. This helps reduce an additional GCS list call if the bucket has been mounted with –implicit-dirs flag. (PR#2303)
• Deprecated - The flags --debug_gcs and --debug_fuse flags are now deprecated. Set --log-severity=TRACE to achieve the same behavior.

Gcsfuse v2.4.1

• Bug Fixes & Improvements:
  • Fix for os.RemoveAll(directory) failure with kernel list cache and remote file addition: PR#2163.
  • Fixed an issue in listing during creation of files in the same directory (Issue#2220, PR#2237).
  • Additional performance improvements when using Parallel downloads (PR#2287).
• Metrics:
  • Categorize the various (approximately 150) fs_errors into a handful of categories. This helps reduce the upper bound of cardinality of these error labels which avoids it from getting dropped by the Google Cloud Metrics pipeline (PR#2321, PR#2382, PR#2390).
  • Change unit of ops and file-cache latency metrics to microseconds to improve the resolution of these metrics (PR#2380).
• Dependency Upgrades / CVE fixes:
  • Upgraded dependencies for better stability (PR#2301, PR#2355, PR#2364)
  • Upgrade Golang version to 1.23.0 (PR#2365)

Gcsfuse v2.1.1

This release is built on top of v2.1.0 with the additional security fixes of Golang upgrade to 1.22.4

Gcsfuse v2.4.0

• Parallel download:
  • Accelerates reads of large files, by using the file cache directory as a prefetch buffer using multiple workers to download large files in parallel.
  • This feature is useful for single threaded read scenarios that load large (>1GiB) files, such as model serving use cases and checkpoint restores.
  • This feature is disabled by default. To enable this feature:
    • Enabling the file cache feature (GKE instructions) is a prerequisite for using the parallel download feature, which uses the cache directory as a prefetch buffer. Although a cache is typically associated with repeat reads, with parallel downloads even first reads of large files are accelerated.
      • The file being read must fit within the file cache directory’s available capacity, which can be controlled by max-size-mb. A value of “-1” allows it to use the cache volume’s entire capacity, or you can give it a value in Megabytes.
      • If the same file will be read multiple times, increase the ttl-secs value. A value of "-1" bypasses TTL expiration and serves the file from the cache if it's available.
    • Set file-cache:enable-parallel-downloads:true in the config file to enable parallel downloads. The default is false.
    • Additional optional parameters:
      • file-cache:parallel-downloads-per-file: The number of maximum workers to spawn per file to download the object from GCS into the file-cache. Default is 16.
      • file-cache:max-parallel-downloads: The number of maximum workers that can be spawned at any given time across all the download jobs of files. The default is set to 2x the number of CPU cores on the machine. A value of -1 means no limit.
      • file-cache:download-chunk-size-mb: The size of each read request in MiB that each goroutine makes to GCS when downloading the object into file-cache. Default is 50. A parallel download will only trigger if the file being read is => this value specified
    • Note: If your application does high read parallelism (>8 threads), a slight performance degradation may be observed if using this feature. High read parallelism is typically seen in training workloads so should not be used for training workloads, and is therefore recommended only for model serving and checkpoint restores, which are typically single threaded large file reads.
  • Addresses #1300
• Kernel-List-Cache
  • List responses, that happen as a part of a readdir operation, are cached in the kernel page cache. This can significantly speed up AI/ML training runs, which do full directory listing first, by serving repeat ListObjects calls locally from the kernel page cache. Due to potential coherency/consistency issues, it is recommended to use on read only volumes, specifically for serving and training.
  • This feature is disabled by default. To enable this feature:
    • Control cache invalidation via the --kernel-list-cache-ttl-secs cli flag or file-system:kernel-list-cache-ttl-secs config flag, where a value of:
      • 0 means disabled. This is the default value.
      • valid positive - represents the ttl (in seconds) to keep the directory list response in the kernel page-cache.
      • -1 to bypass a TTL expiration and serve the list response from the cache whenever it's available.
  • Addresses #184
• CLI-flags Take Precedence over Config, behavior change for logging-flags: Going forward command-line flags will always take precedence for all settings. Change in behavior for logging flags (--log-file & --log-format), where config was taking precedence but not cli-flags will take. This will affect only if the same settings are set in both CLI and config. - #2077

Dependency Upgrades / CVE fixes:

• Upgraded dependencies for better stability (PR #2209, #2196, #2195, #2086)