Merge pull request #152 from polac24/xcode14-fallback

Xcode 14.0 support

.github/ISSUE_TEMPLATE/BUG_REPORT.md

@@ -0,0 +1,65 @@
+---
+name: ⚠️ Bug Report
+about: Something isn't working as expected
+
+---
+
+<!--
+PLEASE HELP US PROCESS GITHUB ISSUES FASTER BY PROVIDING THE FOLLOWING INFORMATION.
+-->
+
+**My integration setup**
+
+[ ] CocoaPods cocoapods-xcremotecache plugin
+[ ] Automatic integration using `xcprepare integrate ...`
+[ ] Manual integration
+[ ] Carthage
+
+**Expected/desired behavior**
+<!-- Describe what the desired behavior would be. -->
+
+**Minimal reproduction of the problem with instructions**
+<!-- Please provide the *STEPS TO REPRODUCE*. -->
+
+**Producer Logs**
+<!-- Capture logs from 10 minutes: `log show --predicate 'sender BEGINSWITH "xc"' --style compact --info --debug -last 10m` -->
+
+<details>
+  <pre> [REPLACE THIS WITH YOUR INFORMATION] </pre>
+</details>
+
+**Consumer Logs**
+<!-- Capture logs from 10 minutes: `log show --predicate 'sender BEGINSWITH "xc"' --style compact --info --debug -last 10m` -->
+
+<details>
+  <pre> [REPLACE THIS WITH YOUR INFORMATION] </pre>
+</details>
+
+**Pods/Carthage file**
+<!-- Delete if you don't use CocoaPods or Carthage -->
+
+<details>
+  <pre> [REPLACE THIS WITH YOUR INFORMATION] </pre>
+</details>
+
+**Environment**
+
+* **XCRemoteCache:** X.Y.Z
+* **cocoapods-xcremotecache:** X.Y.Z <!-- check with `gem list cocoapods-xcremotecache` >
+* **HTTP cache server:** ... <!-- e.g. demo docker, nginx, AWS etc. >
+* **Xcode:** X.Y.Z
+
+**Post build stats**
+<!-- 
+To capture build statistics: 
+* call `xcprepare stats --reset` (or `XCRC/xcprepare stats --reset` for CocoaPods)
+* Build a project in Xcode
+* `xcprepare stats` (or `XCRC/xcprepare stats` for CocoaPods) 
+-->
+
+<details>
+  <pre> [REPLACE THIS WITH YOUR INFORMATION] </pre>
+</details>
+
+**Others**
+<!-- Anything else relevant?  Operating system version, , ... -->

.github/ISSUE_TEMPLATE/DOCUMENTATION_ISSUE.md

@@ -0,0 +1,16 @@
+---
+name: 📕 Documentation Issue
+about: Suggestion for a change in a documentation
+
+---
+
+<!--
+PLEASE HELP US PROCESS GITHUB ISSUES FASTER BY PROVIDING THE FOLLOWING INFORMATION.
+-->
+
+**A suggestion**
+<!-- Describe how could the documentation be improved. -->
+
+**Which file, section, line**
+<!-- Provide a section it relates (if exist). -->
+

.github/ISSUE_TEMPLATE/FEATURE_REQUEST.md

@@ -0,0 +1,20 @@
+---
+name: 🙏 Future Request
+about: Suggestion for an improvement, either behaviour or implementation
+
+---
+
+<!--
+PLEASE HELP US PROCESS GITHUB ISSUES FASTER BY PROVIDING THE FOLLOWING INFORMATION.
+-->
+
+
+**Expected/desired behavior**
+<!-- Describe what the desired behavior would be. -->
+
+**Relevant integration setup**
+
+[ ] CocoaPods cocoapods-xcremotecache plugin
+[ ] Automatic integration using `xcprepare integrate ...`
+[ ] Manual integration
+[ ] Carthage

.github/workflows/ci.yaml

@@ -9,11 +9,13 @@ jobs:
       - uses: actions/checkout@v1
       - name: SwiftLint
         uses: norio-nomura/action-swiftlint@3.1.0
+        with:
+          args: --strict
 
   macOS:
-    runs-on: macOS-latest
+    runs-on: macos-12
     env:
-      XCODE_VERSION: ${{ '12.4' }}
+      XCODE_VERSION: ${{ '13.3.1' }}
     steps:
       - name: Select Xcode
         run: "sudo xcode-select -s /Applications/Xcode_$XCODE_VERSION.app"
@@ -23,3 +25,5 @@ jobs:
         run: rake build[release]
       - name: Test
         run: rake test
+      - name: E2ETests
+        run: rake e2e_only

.github/workflows/docs.yaml

@@ -0,0 +1,24 @@
+name: Docs
+
+on:
+  push:
+    branches:
+      - master
+
+jobs:
+  docs:
+    runs-on: macos-12
+    env:
+      XCODE_VERSION: ${{ '13.3.1' }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: Select Xcode
+        run: "sudo xcode-select -s /Applications/Xcode_$XCODE_VERSION.app"
+      - name: "Generate documentation"
+        run: "swift package --allow-writing-to-directory ./docs  generate-documentation --target XCRemoteCache --disable-indexing --transform-for-static-hosting --output-path ./docs  --hosting-base-path XCRemoteCache/"
+      - name: Deploy GH-pages
+        uses: peaceiris/actions-gh-pages@v3
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./docs
+          keep_files: false

.github/workflows/release.yaml

@@ -6,9 +6,9 @@ on:
 jobs:
   macOS:
     name: Add macOS binaries to release
-    runs-on: macOS-latest
+    runs-on: macos-12
     env:
-      XCODE_VERSION: ${{ '12.4' }}
+      XCODE_VERSION: ${{ '13.3.1' }}
     steps:
       - name: Select Xcode
         run: "sudo xcode-select -s /Applications/Xcode_$XCODE_VERSION.app"
@@ -44,3 +44,29 @@ jobs:
           file_glob: true
           tag: ${{ github.ref }}
           overwrite: true
+  cocoapods:
+    name: Publish CocoaPods plugin
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    defaults:
+      run:
+        working-directory: cocoapods-plugin
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: '3.0'
+      - run: bundle install
+      - name: Publish to RubyGems
+        run: |
+          mkdir -p $HOME/.gem
+          touch $HOME/.gem/credentials
+          chmod 0600 $HOME/.gem/credentials
+          printf -- "---\n:rubygems_api_key: ${GEM_HOST_API_KEY}\n" > $HOME/.gem/credentials
+          gem build *.gemspec
+          CURRENT_VERSION=$(gem list cocoapods-xcremotecache  --remote  -q | sed 's/[^0-9\.]//g')
+          [ -f cocoapods-xcremotecache-$CURRENT_VERSION.gem ] && echo "Version $CURRENT_VERSION already exists" || gem push *.gem
+        env:
+          GEM_HOST_API_KEY: "${{secrets.RUBYGEMS_AUTH_TOKEN}}"

.gitignore

@@ -1,9 +1,13 @@
 .DS_Store
 /.build
 /Packages
-/*.xcodeproj
+*.xcodeproj/
+*.xcworkspace/
 DerivedData
 /.swiftpm/
 releases
 tmp/
 .idea/
+xcuserdata
+*.gem
+Pods/

.swiftlint.yml

@@ -5,7 +5,6 @@ disabled_rules:
   - superfluous_disable_command # Disabled since we disable some rules pre-emptively to avoid issues in the future
   - todo                        # Temporarily disabled. We have too many right now hiding real issues :(
   - nesting                     # Does not make sense anymore since Swift 4 uses nested `CodingKeys` enums for example
-  - trailing_dot_in_comments    # Triggers warnings for generated file headers
 
 opt_in_rules:
   - anyobject_protocol
@@ -64,6 +63,8 @@ excluded:
   - docs/
   - fastlane/
   - DerivedData/
+  - e2eTests/XCRemoteCacheSample/Pods
+  - e2eTests/StandaloneSampleApp
 
 attributes:
   always_on_same_line:
@@ -88,6 +89,25 @@ file_header:
                      \/\/  Created by .*? on .*\.
                      \/\/  Copyright © \d{4} .*\. All rights reserved\.
                      \/\/
+  required_pattern: |
+                     \/\/ Copyright \(c\) \d{4} Spotify AB\.
+                     \/\/
+                     \/\/ Licensed to the Apache Software Foundation \(ASF\) under one
+                     \/\/ or more contributor license agreements\.  See the NOTICE file
+                     \/\/ distributed with this work for additional information
+                     \/\/ regarding copyright ownership\.  The ASF licenses this file
+                     \/\/ to you under the Apache License, Version 2.0 \(the
+                     \/\/ "License"\); you may not use this file except in compliance
+                     \/\/ with the License\.  You may obtain a copy of the License at
+                     \/\/
+                     \/\/ http:\/\/www.apache.org\/licenses\/LICENSE-2\.0
+                     \/\/
+                     \/\/ Unless required by applicable law or agreed to in writing,
+                     \/\/ software distributed under the License is distributed on an
+                     \/\/ \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+                     \/\/ KIND, either express or implied\.  See the License for the
+                     \/\/ specific language governing permissions and limitations
+                     \/\/ under the License\.
 force_cast: warning
 force_try: warning
 implicit_getter: warning
@@ -124,6 +144,6 @@ custom_rules:
     severity: warning
   trailing_dot_in_comments:
     name: "Trailing dot in comments"
-    regex: '^[ ]*///?[^\n]*\.\n'
+    regex: '^(?!\/\/\ Copyright\ \(c\)\ \d{4}\ Spotify AB\.|\/\/\ under\ the\ License\.)[ ]*///?[^\n]*\.\n'
     message: "There shouldn't be trailing dot in comments"
     severity: warning

Package.resolved

@@ -3,16 +3,16 @@
     "pins": [
       {
         "package": "AEXML",
-        "repositoryURL": "https://github.com/tadija/AEXML",
+        "repositoryURL": "https://github.com/tadija/AEXML.git",
         "state": {
           "branch": null,
-          "revision": "8623e73b193386909566a9ca20203e33a09af142",
-          "version": "4.5.0"
+          "revision": "38f7d00b23ecd891e1ee656fa6aeebd6ba04ecc3",
+          "version": "4.6.1"
         }
       },
       {
         "package": "PathKit",
-        "repositoryURL": "https://github.com/kylef/PathKit",
+        "repositoryURL": "https://github.com/kylef/PathKit.git",
         "state": {
           "branch": null,
           "revision": "73f8e9dca9b7a3078cb79128217dc8f2e585a511",
@@ -37,13 +37,22 @@
           "version": "0.0.5"
         }
       },
+      {
+        "package": "SwiftDocCPlugin",
+        "repositoryURL": "https://github.com/apple/swift-docc-plugin",
+        "state": {
+          "branch": null,
+          "revision": "3303b164430d9a7055ba484c8ead67a52f7b74f6",
+          "version": "1.0.0"
+        }
+      },
       {
         "package": "XcodeProj",
         "repositoryURL": "https://github.com/tuist/XcodeProj.git",
         "state": {
           "branch": null,
-          "revision": "0b18c3e7a10c241323397a80cb445051f4494971",
-          "version": "8.0.0"
+          "revision": "c75c3acc25460195cfd203a04dde165395bf00e0",
+          "version": "8.7.1"
         }
       },
       {
@@ -51,8 +60,8 @@
         "repositoryURL": "https://github.com/jpsim/Yams.git",
         "state": {
           "branch": null,
-          "revision": "53741ba55ecca5c7149d8c9f810913ec80845c69",
-          "version": "3.0.0"
+          "revision": "00c403debcd0a007b854bb35e598466207a2d58c",
+          "version": "5.0.0"
         }
       },
       {
@@ -60,8 +69,8 @@
         "repositoryURL": "https://github.com/marmelroy/Zip.git",
         "state": {
           "branch": null,
-          "revision": "80b1c3005ee25b4c7ce46c4029ac3347e8d5e37e",
-          "version": "2.0.0"
+          "revision": "67fa55813b9e7b3b9acee9c0ae501def28746d76",
+          "version": "2.1.2"
         }
       }
     ]

Package.swift

@@ -1,4 +1,5 @@
-// swift-tools-version:5.1
+// swift-tools-version:5.3
+// swiftlint:disable:previous file_header
 // The swift-tools-version declares the minimum version of Swift required to build this package
 
 import PackageDescription
@@ -12,10 +13,11 @@ let package = Package(
         .executable(name: "xcprebuild", targets: ["xcprebuild"]),
     ],
     dependencies: [
-        .package(url: "https://github.com/marmelroy/Zip.git", from: "2.0.0"),
-        .package(url: "https://github.com/jpsim/Yams.git", from: "3.0.0"),
+        .package(url: "https://github.com/marmelroy/Zip.git", from: "2.1.2"),
+        .package(url: "https://github.com/jpsim/Yams.git", from: "5.0.0"),
         .package(url: "https://github.com/apple/swift-argument-parser", from: "0.0.1"),
-        .package(url: "https://github.com/tuist/XcodeProj.git", from: "8.0.0"),
+        .package(url: "https://github.com/tuist/XcodeProj.git", from: "8.7.1"),
+        .package(url: "https://github.com/apple/swift-docc-plugin", from: "1.0.0"),
     ],
     targets: [
         .target(
@@ -49,14 +51,19 @@ let package = Package(
             name: "xcld",
             dependencies: ["XCRemoteCache"]
         ),
+        .target(
+            name: "xcldplusplus",
+            dependencies: ["XCRemoteCache"]
+        ),
         .target(
             // Wrapper target that builds all binaries but does nothing in runtime
             name: "Aggregator",
-            dependencies: ["xcprebuild", "xcswiftc", "xclibtool", "xcpostbuild", "xcprepare", "xcld"]
+            dependencies: ["xcprebuild", "xcswiftc", "xclibtool", "xcpostbuild", "xcprepare", "xcld", "xcldplusplus"]
         ),
         .testTarget(
             name: "XCRemoteCacheTests",
-            dependencies: ["XCRemoteCache"]
+            dependencies: ["XCRemoteCache"],
+            resources: [.copy("TestData")]
         ),
     ]
 )

README.md

@@ -1,5 +1,6 @@
 <p align="center">
-    <img src="docs/img/logo.png" width="75%">
+    <img src="docs/img/logo.png#gh-light-mode-only" width="75%">
+    <img src="docs/img/logo-dark.png#gh-dark-mode-only" width="75%">
 </p>
 
 _XCRemoteCache is a remote cache tool for Xcode projects. It reuses target artifacts generated on a remote machine, served from a simple REST server._
@@ -7,6 +8,7 @@ _XCRemoteCache is a remote cache tool for Xcode projects. It reuses target artif
 [![Build Status](https://github.com/spotify/XCRemoteCache/workflows/CI/badge.svg)](https://github.com/spotify/XCRemoteCache/workflows/CI/badge.svg)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
 [![Slack](https://slackin.spotify.com/badge.svg)](https://slackin.spotify.com)
+[![Docs](https://spotify.github.io/XCRemoteCache/documentation/xcremotecache/)](https://github.com/spotify/XCRemoteCache/workflows/Docs/badge.svg)
 
 - [How and Why?](#how-and-why)
   * [Accurate target input files](#accurate-target-input-files)
@@ -37,10 +39,14 @@ _XCRemoteCache is a remote cache tool for Xcode projects. It reuses target artif
   * [Amazon S3 and Google Cloud Storage](#amazon-s3-and-google-cloud-storage)
 - [CocoaPods plugin](#cocoapods-plugin)
 - [Requirements](#requirements)
+- [Apple silicon support](#apple-silicon-support)
+  * [Artifacts per architecture (Recommended)](#artifacts-per-architecture-recommended)
+  * [Fat artifacts](#fat-artifacts)
 - [Limitations](#limitations)
 - [FAQ](#faq)
 - [Development](#development)
 - [Release](#release)
+  * [Releasing CocoaPods plugin](#releasing-cocoapods-plugin)
   * [Building release package](#building-release-package)
 - [Contributing](#contributing)
 - [Code of conduct](#code-of-conduct)
@@ -190,6 +196,9 @@ Configure Xcode targets that **should use** XCRemoteCache:
 * `SWIFT_EXEC` - location of `xcprepare` (e.g. `xcremotecache/xcswiftc`)
 * `LIBTOOL` - location of `xclibtool` (e.g. `xcremotecache/xclibtool`)
 * `LD` - location of `xcld` (e.g. `xcremotecache/xcld`)
+* `LDPLUSPLUS` - location of `xcldplusplus` (e.g. `xcremotecache/xcldplusplus`)
+* `XCRC_PLATFORM_PREFERRED_ARCH` - `$(LINK_FILE_LIST_$(CURRENT_VARIANT)_$(PLATFORM_PREFERRED_ARCH):dir:standardizepath:file:default=arm64)`
+* `SWIFT_USE_INTEGRATED_DRIVER` - `NO` (required in Xcode 14.0+)
 
 <details>
   <summary>Screenshot</summary>
@@ -209,8 +218,8 @@ Configure Xcode targets that **should use** XCRemoteCache:
 * command: `"$SCRIPT_INPUT_FILE_0"`
 * input files: location of `xcpostbuild` command (e.g. `xcremotecache/xcpostbuild`)
 * output files: 
-  * `$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(PLATFORM_PREFERRED_ARCH).swiftmodule.md5`
-  * `$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)$(LLVM_TARGET_TRIPLE_SUFFIX).swiftmodule.md5`
+  * `$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(XCRC_PLATFORM_PREFERRED_ARCH).swiftmodule.md5`
+  * `$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(XCRC_PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)$(LLVM_TARGET_TRIPLE_SUFFIX).swiftmodule.md5`
 * discovery dependency file: `$(TARGET_TEMP_DIR)/postbuild.d`
 
 <details>
@@ -258,7 +267,7 @@ $ xcremotecache/xcprepare mark --configuration Debug --platform iphonesimulator
 
 That command creates an empty file on a remote server which informs that for given sha, configuration, platform, Xcode versions etc. all artifacts are available.
 
-_Note that for the `producer` mode, the prebuild build phase and `xccc`, `xcld`, `xclibtool` wrappers become no-op, so it is recommended to not add them for the `producer` mode._
+_Note that for the `producer` mode, the prebuild build phase and `xccc`, `xcld`, `xcldplusplus`, `xclibtool` wrappers become no-op, so it is recommended to not add them for the `producer` mode._
 
 ## A full list of configuration parameters:
 
@@ -282,7 +291,7 @@ _Note that for the `producer` mode, the prebuild build phase and `xccc`, `xcld`,
 | `cache_commit_history` | Number of historical git commits to look for cache artifacts | `10` | ⬜️ |
 | `source_root` | Source root of the Xcode project | `""` | ⬜️ |
 | `fingerprint_override_extension` | Fingerprint override extension (sample override `Module.swiftmodule/x86_64.swiftmodule.md5`) | `md5` | ⬜️ |
-| `extra_configuration_file` | Configuration file that overrides project configuration | `user.rcinfo` | ⬜️ |
+| `extra_configuration_file` | Configuration file that overrides project configuration (this property can be overriden multiple times in different files to chain extra configuration files) | `user.rcinfo` | ⬜️ |
 | `publishing_sha` | Custom commit sha to publish artifact (producer only) | `nil` | ⬜️ |
 | `artifact_maximum_age` | Maximum age in days HTTP response should be locally cached before being evicted | `30` | ⬜️ |
 | `custom_fingerprint_envs` | Extra ENV keys that should be convoluted into the environment fingerprint | `[]` | ⬜️ |
@@ -299,11 +308,17 @@ _Note that for the `producer` mode, the prebuild build phase and `xccc`, `xcld`,
 | `product_files_extensions_with_content_override ` | List of all extensions that should carry over source fingerprints. Extensions of all product files that contain non-deterministic content (absolute paths, timestamp, etc) should be included. | `["swiftmodule"]` | ⬜️ |
 | `thinning_enabled ` | If true, support for thin projects is enabled | `false` | ⬜️ |
 | `thinning_target_module_name ` | Module name of a target that works as a helper for thinned targets | `"ThinningRemoteCacheModule"` | ⬜️ |
+| `prettify_meta_files` | A Boolean value that opts-in pretty JSON formatting for meta files | `false` | ⬜️ |
 | `aws_secret_key` | Secret key for AWS V4 Signature Authorization. If this is set to a non-empty String - an Authentication Header will be added based on this and the other `aws_*` parameters.| `""` | ⬜️ |
 | `aws_access_key` | Access key for AWS V4 Signature Authorization. | `""` | ⬜️ |
+| `aws_security_token` | Temporary security token provided by the AWS Security Token Service. | `nil` | ⬜️ |
 | `aws_region` | Region for AWS V4 Signature Authorization. E.g. `eu`.  | `""` | ⬜️ |
 | `aws_service` | Service for AWS V4 Signature Authorization. E.g. `storage`. | `""` | ⬜️ |
-
+| `out_of_band_mappings` | A dictionary of files path remapping that should be applied to make it absolute path agnostic on a list of dependencies. Useful if a project refers files out of repo root, either compilation files or precompiled dependencies. Keys represent generic replacement and values are substrings that should be replaced. Example: for mapping `["COOL_LIBRARY": "/CoolLibrary"]` `/CoolLibrary/main.swift`will be represented as `$(COOL_LIBRARY)/main.swift`). Warning: remapping order is not-deterministic so avoid remappings with multiple matchings. | `[:]` | ⬜️ |
+| `disable_certificate_verification` | A Boolean value that opts-in SSL certificate validation is disabled | `false` | ⬜️ |
+| `disable_vfs_overlay` | A feature flag to disable virtual file system overlay support (temporary) | `false` | ⬜️ |
+| `custom_rewrite_envs` | A list of extra ENVs that should be used as placeholders in the dependency list. ENV rewrite process is optimistic - does nothing if an ENV is not defined in the pre/postbuild process. | `[]` | ⬜️ |
+| `irrelevant_dependencies_paths` | Regexes of files that should not be included in a list of dependencies. Warning! Add entries here with caution - excluding dependencies that are relevant might lead to a target overcaching. The regex can match either partially or fully the filepath, e.g. `\\.modulemap$` will exclude all `.modulemap` files. | `[]` | ⬜️ |
 
 ## Backend cache server
 
@@ -341,6 +356,8 @@ XCRemoteCache supports Amazon S3 and Google Cloud Storage buckets to be used as
 
 To set it up use the configuration parameters `aws_secret_key`, `aws_access_key`, `aws_region`, and `aws_service` in the `.rcinfo` file. Specify the URL to the bucket in cache-addresses field in the same file.
 
+XCRemoteCache also supports [AWS Temporary Access Keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#temporary-access-keys). Use additional `aws_security_token` parameter combined with `aws_secret_key`, `aws_access_key` to set it up. [This page](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html) describes how to receive a security token. 
+
 Example
 ```yaml
 ...
@@ -359,6 +376,31 @@ Retention Policy: Buckets usually have a retention policy option which ensures o
 
 Head over to our [cocoapods-plugin](cocoapods-plugin/README.md) docs to see how to integrate XCRemoteCache in your CocoaPods project.
 
+## Apple silicon support
+
+### Artifacts per architecture (Recommended)
+
+_If all of your machines (both producer and all consumers have the same architecture, either Intel or Apple Silicon), you don't have to do anything._
+
+XCRemoteCache supports building artifacts for Apple silicon consumers. Is it recommended to build separately for `x86_64` and `arm64` architectures to have single-architecture artifacts that do not require downloading irrelevant binaries. Here are required steps if you want to support both Intel and Apple silicon consumers.
+
+* Building for a simulator on a producer: run a first build for `x86_64`, clean a build and build again for `arm64`, e.g.:
+```
+xcodebuild ARCHS=x86_64 ONLY_ACTIVE_ARCH=NO build ...
+xcodebuild clean
+xcodebuild ARCHS=arm64 ONLY_ACTIVE_ARCH=NO build ...
+```
+
+### Fat artifacts
+
+If you prefer to generate far artifacts (with both Intel and Apple silicon binaries), you can disable "Build Archive Architecture Only" on a producer side, e.g.
+
+```
+xcodebuild ONLY_ACTIVE_ARCH=NO build ...
+```
+
+Note: This setup is not recommended and may not be supported in future XCRemoteCache releases.
+
 ## Requirements
 
 * The repo under `git` version control
@@ -369,12 +411,14 @@ Head over to our [cocoapods-plugin](cocoapods-plugin/README.md) docs to see how
 * Recommended: multi-targets Xcode project
 * Recommended: do not use fast-forward PR strategy (use merge or squash instead)
 * Recommended: avoid `DWARF with dSYM File` "Debug Information Format" build setting. Use `DWARF` instead
+* Recommended: avoid having a symbolic link in the source root (e.g. placing a project in `/tmp`) 
 
 ## Limitations
 
 * Swift Package Manager (SPM) dependencies are not supported. _Because SPM does not allow customizing Build Settings, XCRemoteCache cannot specify `clang` and `swiftc` wrappers that control if the local compilation should be skipped (cache hit) or not (cache miss)_
 * Filenames with `_vers.c` suffix are reserved and cannot be used as a source file
 * All compilation files should be referenced via the git repo root. Referencing `/AbsolutePath/someOther.swift` or `../../someOther.swift` that resolve to the location outside of the git repo root is prohibited.
+* The new Swift driver (introduced by default in Xcode 14.0) is not supported and has to be disabled when using XCRemoteCache
 
 ## FAQ
 
@@ -389,6 +433,12 @@ Follow the [Development](docs/Development.md) guide. It has all the information
 To release a version, in [Releases](https://github.com/spotify/XCRemoteCache/releases) draft a new release with `v0.3.0{-rc0}` tag format. 
 Packages with binaries will be automatically uploaded to the GitHub [Releases](https://github.com/spotify/XCRemoteCache/releases) page.
 
+### Releasing CocoaPods plugin
+
+Bump a gem version defined in [gem_version.rb](cocoapods-plugin/lib/cocoapods-xcremotecache/gem_version.rb) and create a new release described above.
+
+A plugin is automatically uploaded to [RubyGems](https://rubygems.org/gems/cocoapods-xcremotecache) if a given version doesn't exist yet.
+
 ### Building release package
 
 To build a release zip package for a single platform (e.g. `x86_64-apple-macosx`, `arm64-apple-macosx`), call:

Rakefile

@@ -1,4 +1,5 @@
 # encoding: utf-8
+require_relative 'tasks/e2e'
 
 ################################
 # Rake configuration
@@ -9,7 +10,7 @@ DERIVED_DATA_DIR = File.join('.build').freeze
 RELEASES_ROOT_DIR = File.join('releases').freeze
 
 EXECUTABLE_NAME = 'XCRemoteCache'
-EXECUTABLE_NAMES = ['xclibtool', 'xcpostbuild', 'xcprebuild', 'xcprepare', 'xcswiftc', 'xcld'] 
+EXECUTABLE_NAMES = ['xclibtool', 'xcpostbuild', 'xcprebuild', 'xcprepare', 'xcswiftc', 'xcld', 'xcldplusplus'] 
 PROJECT_NAME = 'XCRemoteCache'
 
 SWIFTLINT_ENABLED = true
@@ -77,6 +78,12 @@ task :test do
   spm_test()
 end
 
+desc 'build and run E2E tests'
+task :e2e => [:build, :e2e_only]
+
+desc 'run E2E tests without building the XCRemoteCache binary'
+task :e2e_only => ['e2e:run']
+
 ################################
 # Helper functions
 ################################

Sources/XCRemoteCache/Artifacts/ArtifactSwiftProductsBuilder.swift

@@ -51,8 +51,6 @@ protocol ArtifactSwiftProductsBuilder {
 /// # {workingDir}/xccache/produced/include/#{moduleName} (if `moduleName` is defined)
 class ArtifactSwiftProductsBuilderImpl: ArtifactSwiftProductsBuilder {
 
-    /// List of all required swiftmodule related extensions that should be copied to the artifact
-    private static let swiftmoduleExtensionsToInclude = ["swiftmodule", "swiftdoc", "swiftsourceinfo"]
     private let workingDir: URL
     private let moduleName: String?
     private let fileManager: FileManager

Sources/XCRemoteCache/Artifacts/SwiftmoduleFileExtension.swift

@@ -30,6 +30,7 @@ enum SwiftmoduleFileExtension: String {
     case swiftmodule
     case swiftdoc
     case swiftsourceinfo
+    case swiftinterface
 }
 
 extension SwiftmoduleFileExtension {
@@ -38,5 +39,6 @@ extension SwiftmoduleFileExtension {
         .swiftmodule: .required,
         .swiftdoc: .required,
         .swiftsourceinfo: .optional,
+        .swiftinterface: .optional,
     ]
 }

Sources/XCRemoteCache/Commands/Libtool/XCLibtoolCreateUniversalBinary.swift

@@ -52,7 +52,7 @@ class XCLibtoolCreateUniversalBinary: XCLibtoolLogic {
         let config: XCRemoteCacheConfig
         do {
             let srcRoot: URL = URL(fileURLWithPath: fileManager.currentDirectoryPath)
-            config = try XCRemoteCacheConfigReader(srcRootPath: srcRoot.path, fileManager: fileManager)
+            config = try XCRemoteCacheConfigReader(srcRootPath: srcRoot.path, fileReader: fileManager)
                 .readConfiguration()
         } catch {
             errorLog("Libtool initialization failed with error: \(error). Fallbacking to libtool")

Sources/XCRemoteCache/Commands/Plugins/Thinning/ThinningCreatorPlugin.swift

@@ -38,7 +38,7 @@ class ThinningCreatorPlugin: ArtifactCreatorPlugin {
 
     /// Default Initializer
     /// - Parameter targetTempDir: Location of current target-specific temp dir (TARGET_TEMP_DIR)
-    /// - Parameter modeMarkerPath: path of maker file that informs if a given target can reuse remote artifacts.
+    /// - Parameter modeMarkerPath: path of maker file that informs if a given target can reuse remote artifacts
     /// - Parameter dirScanner: scanner to access disk and read files and directories hierarchy
     init(targetTempDir: URL, modeMarkerPath: String, dirScanner: DirScanner) {
         self.targetTempDir = targetTempDir
@@ -94,7 +94,7 @@ class ThinningCreatorPlugin: ArtifactCreatorPlugin {
         // ProducerFast mode:
         // If a target reused already existing artifact, it still has `$(TARGET_TEMP_DIR)/rc.enabled` marker file
         // and the reused zip is placed in:
-        // `$(TARGET_TEMP_DIR)/xccache/{{fileKey}}.zip` location.
+        // `$(TARGET_TEMP_DIR)/xccache/{{fileKey}}.zip` location
 
         let targetEnabledMarker = tempDir.appendingPathComponent(modeMarkerPath)
         let targetReusedArtifactRootDir = tempDir.appendingPathComponent("xccache")

Sources/XCRemoteCache/Commands/Plugins/Thinning/ThinningPrebuildPlugin.swift

@@ -25,7 +25,7 @@ class ThinningConsumerPlugin {
 
     deinit {
         // initialised but never run plugin suggests that standard target fallbacks to the local development
-        // and DerivedData still misses build artifacts.
+        // and DerivedData still misses build artifacts
         guard wasRun else {
             let errorMessage = """
             \(type(of: self)) plugin has never been run, thinning cannot be supported. Verify you \

Sources/XCRemoteCache/Commands/Postbuild/Postbuild.swift

@@ -153,7 +153,7 @@ class Postbuild {
         // Replace all local paths to the generic ones (e.g. $SRCROOT)
         let remappers = [remapper] + creatorPlugins.compactMap(\.customPathsRemapper)
         let remapper = DependenciesRemapperComposite(remappers)
-        let abstractFingerprintFiles = remapper.replace(localPaths: dependencies.map(\.path))
+        let abstractFingerprintFiles = try remapper.replace(localPaths: dependencies.map(\.path))
         // TODO: use `inputs` read by dependenciesReader
         var meta = MainArtifactMeta(
             dependencies: abstractFingerprintFiles,

Sources/XCRemoteCache/Commands/Postbuild/PostbuildContext.swift

@@ -32,12 +32,15 @@ enum MachOType: String, Codable {
 enum PostbuildContextError: Error {
     /// URL address is not a valid URL
     case invalidAddress(String)
+    /// ARCHS env does not contain any architecture to build
+    case missingArchitecture
 }
 
 public struct PostbuildContext {
     var mode: Mode
     var targetName: String
     var targetTempDir: URL
+    var derivedFilesDir: URL
     /// Location where all compilation outputs (.o) are placed
     var compilationTempDir: URL
     var configuration: String
@@ -64,6 +67,9 @@ public struct PostbuildContext {
     var machOType: MachOType
     var wasDsymGenerated: Bool
     var dSYMPath: URL
+    // building architecture. Used to find all dependencies from *.d files
+    // Warning: if two architectures are built (e.g. for disabled "Build Archive
+    // Architecture Only"), a first architecture one is picked
     let arch: String
     let builtProductsDir: URL
     /// Location to the product bundle. Can be nil for libraries
@@ -71,19 +77,31 @@ public struct PostbuildContext {
     let derivedSourcesDir: URL
     /// List of all targets to downloaded from the thinning aggregation target
     var thinnedTargets: [String]
-    /// Action type: build, indexbuild etc.
+    /// Action type: build, indexbuild etc
     var action: BuildActionType
     let modeMarkerPath: String
+    /// location of the json file that define virtual files system overlay
+    /// (mappings of the virtual location file -> local file path)
+    let overlayHeadersPath: URL
+    /// Regexes of files that should not be included in the dependency list
+    let irrelevantDependenciesPaths: [String]
 }
 
 extension PostbuildContext {
+    // swiftlint:disable:next function_body_length
     init(_ config: XCRemoteCacheConfig, env: [String: String]) throws {
         mode = config.mode
         let targetNameValue: String = try env.readEnv(key: "TARGET_NAME")
         targetName = targetNameValue
         targetTempDir = try env.readEnv(key: "TARGET_TEMP_DIR")
-        arch = try env.readEnv(key: "PLATFORM_PREFERRED_ARCH")
-        compilationTempDir = try env.readEnv(key: "OBJECT_FILE_DIR_normal").appendingPathComponent(arch)
+        derivedFilesDir = try env.readEnv(key: "DERIVED_FILE_DIR")
+        let archs: [String] = try env.readEnv(key: "ARCHS").split(separator: " ").map(String.init)
+        guard let firstArch = archs.first, !firstArch.isEmpty else {
+            throw PostbuildContextError.missingArchitecture
+        }
+        arch = firstArch
+        let variant: String = try env.readEnv(key: "CURRENT_VARIANT")
+        compilationTempDir = try env.readEnv(key: "OBJECT_FILE_DIR_\(variant)").appendingPathComponent(arch)
         configuration = try env.readEnv(key: "CONFIGURATION")
         platform = try env.readEnv(key: "PLATFORM_NAME")
         xcodeBuildNumber = try env.readEnv(key: "XCODE_PRODUCT_BUILD_VERSION")
@@ -107,7 +125,7 @@ extension PostbuildContext {
         dSYMPath = try env.readEnv(key: "DWARF_DSYM_FOLDER_PATH")
             .appendingPathComponent(env.readEnv(key: "DWARF_DSYM_FILE_NAME"))
         builtProductsDir = try env.readEnv(key: "BUILT_PRODUCTS_DIR")
-        if let contentsFolderPath = env.readEnv(key: "CONTENTS_FOLDER_PATH") {
+        if let contentsFolderPath: String = env.readEnv(key: "CONTENTS_FOLDER_PATH") {
             bundleDir = productsDir.appendingPathComponent(contentsFolderPath)
         } else {
             bundleDir = nil
@@ -117,5 +135,8 @@ extension PostbuildContext {
         thinnedTargets = thinFocusedTargetsString.split(separator: ",").map(String.init)
         action = (try? BuildActionType(rawValue: env.readEnv(key: "ACTION"))) ?? .unknown
         modeMarkerPath = config.modeMarkerPath
+        /// Note: The file has yaml extension, even it is in the json format
+        overlayHeadersPath = targetTempDir.appendingPathComponent("all-product-headers.yaml")
+        irrelevantDependenciesPaths = config.irrelevantDependenciesPaths
     }
 }

Sources/XCRemoteCache/Commands/Postbuild/XCPostbuild.swift

@@ -19,6 +19,7 @@
 
 import Foundation
 
+// swiftlint:disable type_body_length
 /// Checks current mode from a configuration and based on that:
 /// * triggers build completion
 /// * triggers uploading artifacts to the server for a 'producer' mode
@@ -33,8 +34,9 @@ public class XCPostbuild {
         let context: PostbuildContext
         let cacheHitLogger: CacheHitLogger
         do {
-            config = try XCRemoteCacheConfigReader(env: env, fileManager: fileManager).readConfiguration()
+            config = try XCRemoteCacheConfigReader(env: env, fileReader: fileManager).readConfiguration()
             context = try PostbuildContext(config, env: env)
+            updateProcessTag(context.targetName)
             let counterFactory: FileStatsCoordinator.CountersFactory = { file, count in
                 ExclusiveFileCounter(ExclusiveFile(file, mode: .override), countersCount: count)
             }
@@ -66,9 +68,10 @@ public class XCPostbuild {
             // Initialize dependencies
             let primaryGitBranch = GitBranch(repoLocation: config.primaryRepo, branch: config.primaryBranch)
             let gitClient = GitClientImpl(repoRoot: config.repoRoot, primary: primaryGitBranch, shell: shellGetStdout)
-            let pathRemapper = try StringDependenciesRemapper.buildFromEnvs(
-                keys: DependenciesMapping.rewrittenEnvs,
-                envs: env
+            let envsRemapper = try PathDependenciesRemapperFactory().build(
+                orderKeys: DependenciesMapping.rewrittenEnvs + config.customRewriteEnvs,
+                envs: env,
+                customMappings: config.outOfBandMappings
             )
             let envFingerprint = try EnvironmentFingerprintGenerator(
                 configuration: config,
@@ -85,7 +88,7 @@ public class XCPostbuild {
                 algorithm: MD5Algorithm()
             )
             let organizer = ZipArtifactOrganizer(targetTempDir: context.targetTempDir, fileManager: fileManager)
-            let metaWriter = JsonMetaWriter(fileWriter: fileManager)
+            let metaWriter = JsonMetaWriter(fileWriter: fileManager, pretty: config.prettifyMetaFiles)
             let artifactCreator = BuildArtifactCreator(
                 buildDir: context.productsDir,
                 tempDir: context.targetTempDir,
@@ -111,6 +114,7 @@ public class XCPostbuild {
                 awsV4Signature = AWSV4Signature(
                     secretKey: config.AWSSecretKey,
                     accessKey: config.AWSAccessKey,
+                    securityToken: config.AWSSecurityToken,
                     region: config.AWSRegion,
                     service: config.AWSService,
                     date: Date(timeIntervalSinceNow: 0)
@@ -144,12 +148,32 @@ public class XCPostbuild {
                 fileDependeciesReaderFactory: fileReaderFactory,
                 dirScanner: fileManager
             )
+            var remappers: [DependenciesRemapper] = []
+            if !config.disableVFSOverlay {
+                // As the PostbuildContext assumes file location and filename (`all-product-headers.yaml`)
+                // do not fail in case of a missing headers overlay file. In the future, all overlay files could be
+                // captured from the swiftc invocation similarly is stored in the `history.compile`
+                // for the consumer mode
+                let overlayReader = JsonOverlayReader(
+                    context.overlayHeadersPath,
+                    mode: .bestEffort,
+                    fileReader: fileManager
+                )
+                let overlayRemapper = OverlayDependenciesRemapper(
+                    overlayReader: overlayReader
+                )
+                remappers.append(overlayRemapper)
+            }
+            remappers.append(envsRemapper)
+            let pathRemapper = DependenciesRemapperComposite(remappers)
             let dependencyProcessor = DependencyProcessorImpl(
                 xcode: context.xcodeDir,
                 product: context.productsDir,
                 source: context.srcRoot,
                 intermediate: context.targetTempDir,
-                bundle: context.bundleDir
+                derivedFiles: context.derivedFilesDir,
+                bundle: context.bundleDir,
+                skippedRegexes: context.irrelevantDependenciesPaths
             )
             // Override fingerprints for all produced '.swiftmodule' files
             let fingerprintOverrideManager = FingerprintOverrideManagerImpl(
@@ -300,3 +324,4 @@ public class XCPostbuild {
         }
     }
 }
+// swiftlint:enable type_body_length

Sources/XCRemoteCache/Commands/Prebuild/Prebuild.swift

@@ -63,7 +63,9 @@ class Prebuild {
         do {
             let metaData = try networkClient.fetch(.meta(commit: commit))
             let meta = try metaReader.read(data: metaData)
-            let localDependencies = remapper.replace(genericPaths: meta.dependencies).map(URL.init(fileURLWithPath:))
+            let localDependencies = try remapper.replace(
+                genericPaths: meta.dependencies
+            ).map(URL.init(fileURLWithPath:))
             let localFingerprint = try generateFingerprint(for: localDependencies)
             if localFingerprint.raw != meta.rawFingerprint {
                 if context.forceCached {

Sources/XCRemoteCache/Commands/Prebuild/PrebuildContext.swift

@@ -43,6 +43,9 @@ public struct PrebuildContext {
     let targetName: String
     /// List of all targets to downloaded from the thinning aggregation target
     var thinnedTargets: [String]?
+    /// location of the json file that define virtual files system overlay
+    /// (mappings of the virtual location file -> local file path)
+    let overlayHeadersPath: URL
 }
 
 extension PrebuildContext {
@@ -64,5 +67,7 @@ extension PrebuildContext {
         self.targetName = targetName
         let thinFocusedTargetsString: String? = env.readEnv(key: "SPT_XCREMOTE_CACHE_THINNED_TARGETS")
         thinnedTargets = thinFocusedTargetsString?.split(separator: ",").map(String.init)
+        /// Note: The file has yaml extension, even it is in the json format
+        overlayHeadersPath = targetTempDir.appendingPathComponent("all-product-headers.yaml")
     }
 }

Sources/XCRemoteCache/Commands/Prebuild/XCPrebuild.swift

@@ -29,8 +29,9 @@ public class XCPrebuild {
         let config: XCRemoteCacheConfig
         let context: PrebuildContext
         do {
-            config = try XCRemoteCacheConfigReader(env: env, fileManager: fileManager).readConfiguration()
+            config = try XCRemoteCacheConfigReader(env: env, fileReader: fileManager).readConfiguration()
             context = try PrebuildContext(config, env: env)
+            updateProcessTag(context.targetName)
         } catch {
             // Fatal error:
             exit(1, "FATAL: Prebuild initialization failed with error: \(error)")
@@ -77,6 +78,10 @@ public class XCPrebuild {
             exit(0)
         }
 
+        let compilationHistoryOrganizer = CompilationHistoryFileOrganizer(
+            context.compilationHistoryFile,
+            fileManager: fileManager
+        )
         do {
             let envFingerprint = try EnvironmentFingerprintGenerator(
                 configuration: config,
@@ -95,6 +100,7 @@ public class XCPrebuild {
                 awsV4Signature = AWSV4Signature(
                     secretKey: config.AWSSecretKey,
                     accessKey: config.AWSAccessKey,
+                    securityToken: config.AWSSecurityToken,
                     region: config.AWSRegion,
                     service: config.AWSService,
                     date: Date(timeIntervalSinceNow: 0)
@@ -115,10 +121,27 @@ public class XCPrebuild {
             )
             let client: NetworkClient = config.disableHttpCache ? networkClient : cacheNetworkClient
             let remoteNetworkClient = RemoteNetworkClientImpl(client, urlBuilder)
-            let pathRemapper = try StringDependenciesRemapper.buildFromEnvs(
-                keys: DependenciesMapping.rewrittenEnvs,
-                envs: env
+            let envsRemapper = try PathDependenciesRemapperFactory().build(
+                orderKeys: DependenciesMapping.rewrittenEnvs + config.customRewriteEnvs,
+                envs: env,
+                customMappings: config.outOfBandMappings
             )
+            var remappers: [DependenciesRemapper] = []
+            if !config.disableVFSOverlay {
+                // As PrebuildContext assumes file location and its filename (`all-product-headers.yaml`)
+                // do not fail in case of a missing headers overlay file
+                let overlayReader = JsonOverlayReader(
+                    context.overlayHeadersPath,
+                    mode: .bestEffort,
+                    fileReader: fileManager
+                )
+                let overlayRemapper = OverlayDependenciesRemapper(
+                    overlayReader: overlayReader
+                )
+                remappers.append(overlayRemapper)
+            }
+            remappers.append(envsRemapper)
+            let pathRemapper = DependenciesRemapperComposite(remappers)
             let filesFingerprintGenerator = FingerprintAccumulatorImpl(
                 algorithm: MD5Algorithm(),
                 fileManager: fileManager
@@ -129,10 +152,6 @@ public class XCPrebuild {
                 algorithm: MD5Algorithm()
             )
             let organizer = ZipArtifactOrganizer(targetTempDir: context.targetTempDir, fileManager: fileManager)
-            let compilationHistoryOrganizer = CompilationHistoryFileOrganizer(
-                context.compilationHistoryFile,
-                fileManager: fileManager
-            )
             let metaReader = JsonMetaReader(fileAccessor: fileManager)
             var consumerPlugins: [ArtifactConsumerPrebuildPlugin] = []
 
@@ -170,7 +189,6 @@ public class XCPrebuild {
             case .compatible(localDependencies: let dependencies):
                 // TODO: pass `allowedInputFiles` observed in the build time
                 try modeController.enable(allowedInputFiles: dependencies, dependencies: dependencies)
-                compilationHistoryOrganizer.reset()
             }
         } catch {
             disableRemoteCache(
@@ -178,6 +196,7 @@ public class XCPrebuild {
                 errorMessage: "Prebuild step failed with error: \(error)"
             )
         }
+        compilationHistoryOrganizer.reset()
     }
 
     private func disableRemoteCache(modeController: PhaseCacheModeController, errorMessage: String?) {

Sources/XCRemoteCache/Commands/Prepare/CCWrapperBuilder.swift

@@ -72,7 +72,17 @@ class TemplateBasedCCWrapperBuilder: CCWrapperBuilder {
         )
         infoLog("ClangWrapperBuilder compiles file at \(compilationFile).")
         // -O3: optimize for faster execution
-        let args = [clangCommand, "-O3", compilationFile.path, "-o", destination.path]
+        let args = [
+            clangCommand,
+            "-arch",
+            "arm64",
+            "-arch",
+            "x86_64",
+            "-O3",
+            compilationFile.path,
+            "-o",
+            destination.path,
+        ]
         let compilationOutput = try shell("xcrun", args, URL(fileURLWithPath: "").path, nil)
         infoLog("Clang compilation output: \(compilationOutput)")
     }
@@ -385,6 +395,8 @@ class TemplateBasedCCWrapperBuilder: CCWrapperBuilder {
             const char *dependency_arg_name = "-MF";
             const char *output_arg_name = "-o";
             const char *serialize_diagnostics_arg_name = "--serialize-diagnostics";
+            const char *language_mode_arg_name = "-x";
+            const char *precompile_header_arg_value = "objective-c-header";
             const char *clang_cmd = "\(clangCommand)";
             const char *markerFile = "\(markerFilename)";
             const char *compilationHistoryFile = "\(compilationHistoryFilename)";
@@ -399,6 +411,7 @@ class TemplateBasedCCWrapperBuilder: CCWrapperBuilder {
             const char *output_file= NULL;
             const char *input_file = NULL;
             const char *diagnostics_file = NULL;
+            const char *language_mode = NULL;
 
             for (int i = 1; i < argc; i++) {
                 if (strcmp(argv[i], dependency_arg_name) == 0 && i < (argc - 1) ) {
@@ -419,6 +432,12 @@ class TemplateBasedCCWrapperBuilder: CCWrapperBuilder {
                     i += 1;
                     clang_args[i] = argv[i];
                     diagnostics_file = argv[i];
+                } if (strcmp(argv[i], language_mode_arg_name) == 0 && i < (argc - 1) ) {
+                    // called with "-x path" pattern and not the last argument
+                    clang_args[i] = argv[i];
+                    i += 1;
+                    clang_args[i] = argv[i];
+                    language_mode = argv[i];
                 } else if (
                     isSuffixed(argv[i],".m") ||
                     isSuffixed(argv[i],".mm") ||
@@ -426,10 +445,13 @@ class TemplateBasedCCWrapperBuilder: CCWrapperBuilder {
                     isSuffixed(argv[i],".cc") ||
                     isSuffixed(argv[i],".cpp") ||
                     isSuffixed(argv[i],".c++") ||
-                    isSuffixed(argv[i],".cxx")
+                    isSuffixed(argv[i],".cxx") ||
+                    isSuffixed(argv[i],".S") ||
+                    isSuffixed(argv[i],".s")
                 ) {
                     // a full list of extensions is taken from https://clang.llvm.org/docs/ClangFormatStyleOptions.html
                     // support for .m,.mm,.c,.cc,.cpp,.c++,.cxx input files
+                    // .s and .S are assembly files
                     clang_args[i] = argv[i];
                     input_file = argv[i];
                 } else {
@@ -438,6 +460,14 @@ class TemplateBasedCCWrapperBuilder: CCWrapperBuilder {
                 }
             }
 
+           // null-terminating the args array needed for local compilation fallback
+           clang_args[argc] = NULL;
+
+           // Verify mode. Even a target is cached, pch mode is not supported. Fallback to the local compilation
+           if (language_mode != NULL && strcmp(language_mode, precompile_header_arg_value) == 0) {
+               return execvp(clang_cmd, (char *const*) clang_args);
+           }
+
            // Verify all input arguments
            if (dependency_file == NULL) {
                fprintf(stderr, "error: missing %s input\\n", dependency_arg_name);
@@ -508,8 +538,6 @@ class TemplateBasedCCWrapperBuilder: CCWrapperBuilder {
                }
            }
 
-           // null-terminating the args array
-           clang_args[argc] = NULL;
            #pragma GCC diagnostic push
            #pragma GCC diagnostic ignored "-Wincompatible-pointer-types-discards-qualifiers"
            /// execvp takes $PATH to consideration

Sources/XCRemoteCache/Commands/Prepare/Integrate/BuildSettingsIntegrateAppender.swift

@@ -33,15 +33,18 @@ protocol BuildSettingsIntegrateAppender {
 class XcodeProjBuildSettingsIntegrateAppender: BuildSettingsIntegrateAppender {
     private let mode: Mode
     private let repoRoot: URL
+    private let fakeSrcRoot: URL
 
-    init(mode: Mode, repoRoot: URL) {
+    init(mode: Mode, repoRoot: URL, fakeSrcRoot: URL) {
         self.mode = mode
         self.repoRoot = repoRoot
+        self.fakeSrcRoot = fakeSrcRoot
     }
 
     func appendToBuildSettings(buildSettings: BuildSettings, wrappers: XCRCBinariesPaths) -> BuildSettings {
         var result = buildSettings
         result["SWIFT_EXEC"] = wrappers.swiftc.path
+        result["SWIFT_USE_INTEGRATED_DRIVER"] = "NO"
         // When generating artifacts, no need to shell-out all compilation commands to our wrappers
         if case .consumer = mode {
             result["CC"] = wrappers.cc.path
@@ -61,7 +64,11 @@ class XcodeProjBuildSettingsIntegrateAppender: BuildSettingsIntegrateAppender {
         result["OTHER_SWIFT_FLAGS"] = swiftFlags.settingValue
         result["OTHER_CFLAGS"] = clangFlags.settingValue
 
-        result["XCRC_FAKE_SRCROOT"] = "/\(String(repeating: "x", count: 10))"
+        result["XCRC_FAKE_SRCROOT"] = "\(fakeSrcRoot.path)"
+        result["XCRC_PLATFORM_PREFERRED_ARCH"] =
+        """
+        $(LINK_FILE_LIST_$(CURRENT_VARIANT)_$(PLATFORM_PREFERRED_ARCH):dir:standardizepath:file:default=arm64)
+        """
         return result
     }
 }

Sources/XCRemoteCache/Commands/Prepare/Integrate/IncludeOracle.swift

@@ -35,7 +35,7 @@ struct IncludeExcludeOracle: IncludeOracle {
 
 
     func shouldInclude(identifier: OracleIdentifierType) -> Bool {
-        // exclude array has precedence.
+        // exclude array has precedence
         if excludes.contains(identifier) {
             return false
         }

Sources/XCRemoteCache/Commands/Prepare/Integrate/IntegrateContext.swift

@@ -53,6 +53,7 @@ extension IntegrateContext {
             swiftc: binariesDir.appendingPathComponent("xcswiftc"),
             libtool: binariesDir.appendingPathComponent("xclibtool"),
             ld: binariesDir.appendingPathComponent("xcld"),
+            ldplusplus: binariesDir.appendingPathComponent("xcldplusplus"),
             prebuild: binariesDir.appendingPathComponent("xcprebuild"),
             postbuild: binariesDir.appendingPathComponent("xcpostbuild")
         )

Sources/XCRemoteCache/Commands/Prepare/Integrate/XCIntegrate.swift

@@ -74,7 +74,7 @@ public class XCIntegrate {
             let binariesDir = commandURL.deletingLastPathComponent()
 
             let srcRoot: URL = URL(fileURLWithPath: projectPath).deletingLastPathComponent()
-            let config = try XCRemoteCacheConfigReader(srcRootPath: srcRoot.path, fileManager: fileManager)
+            let config = try XCRemoteCacheConfigReader(srcRootPath: srcRoot.path, fileReader: fileManager)
                 .readConfiguration()
 
             let context = try IntegrateContext(
@@ -97,7 +97,8 @@ public class XCIntegrate {
             )
             let buildSettingsAppender = XcodeProjBuildSettingsIntegrateAppender(
                 mode: context.mode,
-                repoRoot: context.repoRoot
+                repoRoot: context.repoRoot,
+                fakeSrcRoot: context.fakeSrcRoot
             )
             let lldbPatcher: LLDBInitPatcher
             switch lldbMode {

Sources/XCRemoteCache/Commands/Prepare/Integrate/XCRCBinariesPaths.swift

@@ -26,6 +26,7 @@ struct XCRCBinariesPaths {
     let swiftc: URL
     let libtool: URL
     let ld: URL
+    let ldplusplus: URL
     let prebuild: URL
     let postbuild: URL
 }

Sources/XCRemoteCache/Commands/Prepare/Integrate/XcodeProjIntegrate.swift

@@ -100,11 +100,11 @@ struct XcodeProjIntegrate: Integrate {
             outputPaths: [
                 """
                 $(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/\
-                $(PLATFORM_PREFERRED_ARCH).swiftmodule.md5
+                $(XCRC_PLATFORM_PREFERRED_ARCH).swiftmodule.md5
                 """,
                 """
                 $(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/\
-                $(PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)\
+                $(XCRC_PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)\
                 $(LLVM_TARGET_TRIPLE_SUFFIX).swiftmodule.md5
                 """,
             ],
@@ -227,13 +227,15 @@ struct XcodeProjIntegrate: Integrate {
         let previousRCPhases = target.buildPhases.filter(isRCPhase)
         target.buildPhases.removeAll(where: previousRCPhases.contains)
 
-        if target.buildPhases.map(\.buildPhase).contains(.sources) {
+        if let sourceIndex = target.buildPhases.map(\.buildPhase).firstIndex(of: .sources) {
             // add (pre|post)build phases only when a target has some compilation steps
             // otherwise they make no sense (nothing to store in an artifact)
-            pbxproj.add(object: prebuildPhase)
-            target.buildPhases.insert(prebuildPhase, at: 0)
+            // add postbuild right after compilation as custom build steps may depend on files generated by
+            // the xcpostbuild (e.g. to copy -Swift.h.md5)
             pbxproj.add(object: postbuildPhase)
-            target.buildPhases.append(postbuildPhase)
+            target.buildPhases.insert(postbuildPhase, at: sourceIndex + 1)
+            pbxproj.add(object: prebuildPhase)
+            target.buildPhases.insert(prebuildPhase, at: sourceIndex)
         }
     }
 

Sources/XCRemoteCache/Commands/Prepare/XCConfig.swift

@@ -18,7 +18,6 @@
 // under the License.
 
 import Foundation
-import Yams
 
 /// Print current configuration to the console
 public class XCConfig {
@@ -33,7 +32,7 @@ public class XCConfig {
         let fileManager = FileManager.default
         let config: XCRemoteCacheConfig
         do {
-            config = try XCRemoteCacheConfigReader(env: env, fileManager: fileManager).readConfiguration()
+            config = try XCRemoteCacheConfigReader(env: env, fileReader: fileManager).readConfiguration()
         } catch {
             exit(1, "FATAL: Prepare initialization failed with error: \(error)")
         }

Sources/XCRemoteCache/Commands/Prepare/XCPrepare.swift

@@ -18,7 +18,6 @@
 // under the License.
 
 import Foundation
-import Yams
 
 /// Switch between Online/Offline modes
 public enum XCPrepareMode {
@@ -62,7 +61,7 @@ public class XCPrepare {
         var context: PrepareContext
         let xcodeVersion: String
         do {
-            config = try XCRemoteCacheConfigReader(env: env, fileManager: fileManager).readConfiguration()
+            config = try XCRemoteCacheConfigReader(env: env, fileReader: fileManager).readConfiguration()
             context = try PrepareContext(config, offline: offline)
             xcodeVersion = try customXcodeBuildNumber ?? XcodeProbeImpl(shell: shellGetStdout).read().buildVersion
         } catch {
@@ -79,6 +78,7 @@ public class XCPrepare {
                 awsV4Signature = AWSV4Signature(
                     secretKey: config.AWSSecretKey,
                     accessKey: config.AWSAccessKey,
+                    securityToken: config.AWSSecurityToken,
                     region: config.AWSRegion,
                     service: config.AWSService,
                     date: Date(timeIntervalSinceNow: 0)

Sources/XCRemoteCache/Commands/Prepare/XCPrepareMark.swift

@@ -46,7 +46,7 @@ public class XCPrepareMark {
         let context: PrepareMarkContext
         let xcodeVersion: String
         do {
-            config = try XCRemoteCacheConfigReader(env: env, fileManager: fileManager).readConfiguration()
+            config = try XCRemoteCacheConfigReader(env: env, fileReader: fileManager).readConfiguration()
             context = try PrepareMarkContext(config)
             xcodeVersion = try xcode ?? XcodeProbeImpl(shell: shellGetStdout).read().buildVersion
         } catch {
@@ -60,6 +60,7 @@ public class XCPrepareMark {
                 awsV4Signature = AWSV4Signature(
                     secretKey: config.AWSSecretKey,
                     accessKey: config.AWSAccessKey,
+                    securityToken: config.AWSSecurityToken,
                     region: config.AWSRegion,
                     service: config.AWSService,
                     date: Date(timeIntervalSinceNow: 0)

Sources/XCRemoteCache/Commands/Prepare/XCStats.swift

@@ -18,7 +18,6 @@
 // under the License.
 
 import Foundation
-import Yams
 
 /// Manages XCRemoteCache statistics: rests, print to the standard output etc
 public class XCStats {
@@ -37,7 +36,7 @@ public class XCStats {
         let config: XCRemoteCacheConfig
         let context: XCStatsContext
         do {
-            config = try XCRemoteCacheConfigReader(env: env, fileManager: fileManager).readConfiguration()
+            config = try XCRemoteCacheConfigReader(env: env, fileReader: fileManager).readConfiguration()
             try context = XCStatsContext(config, fileManager: fileManager)
         } catch {
             exit(1, "FATAL: Prepare initialization failed with error: \(error)")

Sources/XCRemoteCache/Commands/ProductBinaryCreator/XCCreateBinary.swift

@@ -70,7 +70,7 @@ public class XCCreateBinary {
         let config: XCRemoteCacheConfig
         do {
             let srcRoot: URL = URL(fileURLWithPath: fileManager.currentDirectoryPath)
-            config = try XCRemoteCacheConfigReader(srcRootPath: srcRoot.path, fileManager: fileManager)
+            config = try XCRemoteCacheConfigReader(srcRootPath: srcRoot.path, fileReader: fileManager)
                 .readConfiguration()
         } catch {
             errorLog("\(stepDescription) initialization failed with error: \(error). Fallbacking to \(fallbackCommand)")

Sources/XCRemoteCache/Commands/Swiftc/SwiftcOrchestrator.swift

@@ -110,8 +110,8 @@ class SwiftcOrchestrator {
                     try invocationStorage.store(args: invocationArgs)
                 }
             } catch {
-                // The critical section is protected by a lock. Some other process already called compilation history.
-                // We only need to call our current step then.
+                // The critical section is protected by a lock. Some other process already called compilation history
+                // We only need to call our current step then
                 fallbackToDefault(command: swiftcCommand)
             }
         case .consumer:

Sources/XCRemoteCache/Commands/Swiftc/SwiftcProductsGenerator.swift

@@ -20,7 +20,7 @@
 import Foundation
 
 enum DiskSwiftcProductsGeneratorError: Error {
-    /// When a generator was asked to generate unknown swiftmodule extension file.
+    /// When a generator was asked to generate unknown swiftmodule extension file
     /// Probably a programmer error: asking to generate excessive extensions, not listed in
     /// `SwiftmoduleFileExtension.SwiftmoduleExtensions`
     case unknownSwiftmoduleFile

Sources/XCRemoteCache/Commands/Swiftc/XCSwiftc.swift

@@ -70,7 +70,7 @@ public class XCSwiftc {
         let context: SwiftcContext
         do {
             let srcRoot: URL = URL(fileURLWithPath: fileManager.currentDirectoryPath)
-            config = try XCRemoteCacheConfigReader(srcRootPath: srcRoot.path, fileManager: fileManager)
+            config = try XCRemoteCacheConfigReader(srcRootPath: srcRoot.path, fileReader: fileManager)
                 .readConfiguration()
             context = try SwiftcContext(config: config, input: inputArgs)
         } catch {

Sources/XCRemoteCache/Config/XCRemoteCacheConfig.swift

@@ -95,10 +95,10 @@ public struct XCRemoteCacheConfig: Encodable {
     ///  Disable cache for http requests to fecth metadata and download artifacts
     var disableHttpCache: Bool = false
     /// Path, relative to $TARGET_TEMP_DIR which gathers all compilation commands that should be e
-    /// xecuted if a target switches to local compilation.
+    /// xecuted if a target switches to local compilation
     /// Example: A new `.swift` file invalidates remote arXcodeProjIntegrate.swifttifact and triggers local compilation
     /// When that happens, all previously skipped clang build steps
-    /// need to be eventually called locally - this file lists all these commands.
+    /// need to be eventually called locally - this file lists all these commands
     var compilationHistoryFile: String = "history.compile"
     /// Timeout for remote response data interval (in seconds). If an interval between data chunks is
     /// longer than a timeout, a request fails
@@ -112,14 +112,37 @@ public struct XCRemoteCacheConfig: Encodable {
     var thinningEnabled: Bool = false
     /// Module name of a target that works as a helper for thinned targets
     var thinningTargetModuleName: String = "ThinningRemoteCacheModule"
+    /// Opt-in pretty json formatting for meta files
+    var prettifyMetaFiles: Bool = false
     /// Secret key for AWS V4 Signature, if this is set the Authentication Header will be added
     var AWSSecretKey: String = ""
     /// Access key for AWS V4 Signature
     var AWSAccessKey: String = ""
+    /// Temporary security token provided by the AWS Security Token Service
+    var AWSSecurityToken: String?
     /// Region for AWS V4 Signature (e.g. `eu`)
     var AWSRegion: String = ""
     /// Service for AWS V4 Signature (e.g. `storage`)
     var AWSService: String = ""
+    /// A dictionary of files path remapping that should be applied to make it absolute path agnostic on a list of
+    /// dependencies. Useful if a project refers files out of repo root, either compilation files or precompiled
+    /// dependencies. Keys represent generic replacement and values are substrings that should be replaced
+    /// Example: for mapping `["COOL_LIBRARY": "/CoolLibrary"]`
+    /// `/CoolLibrary/main.swift`will be represented as `$(COOL_LIBRARY)/main.swift`)
+    /// Warning: remapping order is not-deterministic so avoid remappings with multiple matchings
+    var outOfBandMappings: [String: String] = [:]
+    /// If true, SSL certificate validation is disabled
+    var disableCertificateVerification: Bool = false
+    /// A feature flag to disable virtual file system overlay support (temporary)
+    var disableVFSOverlay: Bool = false
+    /// A list of extra ENVs that should be used as placeholders in the dependency list
+    /// ENV rewrite process is optimistic - does nothing if an ENV is not defined in the pre/postbuild process
+    var customRewriteEnvs: [String] = []
+    /// Regexes of files that should not be included in a list of dependencies. Warning! Add entries here
+    /// with caution - excluding dependencies that are relevant might lead to a target overcaching
+    /// Note: The regex can match either partially or fully the filepath, e.g. `\\.modulemap$` will exclude
+    /// all `.modulemap` files
+    var irrelevantDependenciesPaths: [String] = []
 }
 
 extension XCRemoteCacheConfig {
@@ -165,10 +188,17 @@ extension XCRemoteCacheConfig {
             scheme.productFilesExtensionsWithContentOverride ?? productFilesExtensionsWithContentOverride
         merge.thinningEnabled = scheme.thinningEnabled ?? thinningEnabled
         merge.thinningTargetModuleName = scheme.thinningTargetModuleName ?? thinningTargetModuleName
+        merge.prettifyMetaFiles = scheme.prettifyMetaFiles ?? prettifyMetaFiles
         merge.AWSAccessKey = scheme.AWSAccessKey ?? AWSAccessKey
         merge.AWSSecretKey = scheme.AWSSecretKey ?? AWSSecretKey
+        merge.AWSSecurityToken = scheme.AWSSecurityToken ?? AWSSecurityToken
         merge.AWSRegion = scheme.AWSRegion ?? AWSRegion
         merge.AWSService = scheme.AWSService ?? AWSService
+        merge.outOfBandMappings = scheme.outOfBandMappings ?? outOfBandMappings
+        merge.disableCertificateVerification = scheme.disableCertificateVerification ?? disableCertificateVerification
+        merge.disableVFSOverlay = scheme.disableVFSOverlay ?? disableVFSOverlay
+        merge.customRewriteEnvs = scheme.customRewriteEnvs ?? customRewriteEnvs
+        merge.irrelevantDependenciesPaths = scheme.irrelevantDependenciesPaths ?? irrelevantDependenciesPaths
         return merge
     }
 
@@ -223,10 +253,17 @@ struct ConfigFileScheme: Decodable {
     let productFilesExtensionsWithContentOverride: [String]?
     let thinningEnabled: Bool?
     let thinningTargetModuleName: String?
+    let prettifyMetaFiles: Bool?
     let AWSSecretKey: String?
     let AWSAccessKey: String?
+    let AWSSecurityToken: String?
     let AWSRegion: String?
     let AWSService: String?
+    let outOfBandMappings: [String: String]?
+    let disableCertificateVerification: Bool?
+    let disableVFSOverlay: Bool?
+    let customRewriteEnvs: [String]?
+    let irrelevantDependenciesPaths: [String]?
 
     // Yams library doesn't support encoding strategy, see https://github.com/jpsim/Yams/issues/84
     enum CodingKeys: String, CodingKey {
@@ -264,10 +301,17 @@ struct ConfigFileScheme: Decodable {
         case productFilesExtensionsWithContentOverride = "product_files_extensions_with_content_override"
         case thinningEnabled = "thinning_enabled"
         case thinningTargetModuleName = "thinning_target_module_name"
+        case prettifyMetaFiles = "prettify_meta_files"
         case AWSSecretKey = "aws_secret_key"
         case AWSAccessKey = "aws_access_key"
+        case AWSSecurityToken = "aws_security_token"
         case AWSRegion = "aws_region"
         case AWSService = "aws_service"
+        case outOfBandMappings = "out_of_band_mappings"
+        case disableCertificateVerification = "disable_certificate_verification"
+        case disableVFSOverlay = "disable_vfs_overlay"
+        case customRewriteEnvs = "custom_rewrite_envs"
+        case irrelevantDependenciesPaths = "irrelevant_dependencies_paths"
     }
 }
 
@@ -280,31 +324,41 @@ class XCRemoteCacheConfigReader {
     /// Name of the configuration file, required in $(SRCROOT) location
     private static let configurationFile = ".rcinfo"
     private let srcRoot: String
-    private let fileManager: FileManager
+    private let fileReader: FileReader
     private lazy var yamlDecorer = YAMLDecoder(encoding: .utf8)
 
-    init(env: [String: String], fileManager: FileManager) throws {
+    init(env: [String: String], fileReader: FileReader) throws {
         let explicitSrcRoot: String? = env.readEnv(key: "SRCROOT")
-        srcRoot = explicitSrcRoot ?? fileManager.currentDirectoryPath
-        self.fileManager = fileManager
+        srcRoot = explicitSrcRoot ?? FileManager.default.currentDirectoryPath
+        self.fileReader = fileReader
     }
 
-    init(srcRootPath srcRoot: String, fileManager: FileManager) {
+    init(srcRootPath srcRoot: String, fileReader: FileReader) {
         self.srcRoot = srcRoot
-        self.fileManager = fileManager
+        self.fileReader = fileReader
     }
 
+    // Reads the final configuration by loading all extra configs
+    // until reaching a config that doesn't override `extraConfigurationFile`
     func readConfiguration() throws -> XCRemoteCacheConfig {
         let rootURL = URL(fileURLWithPath: srcRoot)
         let configURL = URL(fileURLWithPath: Self.configurationFile, relativeTo: rootURL)
         let userConfigs = try readUserConfig(configURL)
         var config = XCRemoteCacheConfig(sourceRoot: srcRoot).merged(with: userConfigs)
-        let extraConfURL = URL(fileURLWithPath: config.extraConfigurationFile, relativeTo: rootURL)
-        do {
-            let extraConfig = try readUserConfig(extraConfURL)
-            config = config.merged(with: extraConfig)
-        } catch {
-            infoLog("Extra config override failed with \(error). Skipping extra configuration")
+        var extraConfURL = URL(fileURLWithPath: config.extraConfigurationFile, relativeTo: rootURL)
+        var visitedFiles = Set([configURL])
+        while !visitedFiles.contains(extraConfURL) {
+            do {
+                let extraConfig = try readUserConfig(extraConfURL)
+                debugLog("Reading extra configuration from \(extraConfURL)")
+                config = config.merged(with: extraConfig)
+                visitedFiles.insert(extraConfURL)
+                // Advance extra configuration
+                extraConfURL = URL(fileURLWithPath: config.extraConfigurationFile, relativeTo: rootURL)
+            } catch {
+                infoLog("Extra config override failed with \(error). Skipping extra configuration")
+                break
+            }
         }
 
         return try config.verifyAndApplyDefaults()
@@ -312,7 +366,7 @@ class XCRemoteCacheConfigReader {
 
     /// Reads user configuration from a file
     private func readUserConfig(_ file: URL) throws -> ConfigFileScheme {
-        let configurationContent = fileManager.contents(atPath: file.path)
+        let configurationContent = try fileReader.contents(atPath: file.path)
         guard let configurationData = configurationContent else {
             throw XCRemoteCacheConfigReaderError.missingConfigurationFile(file)
         }

Sources/XCRemoteCache/Dependencies/DependenciesReader.swift

@@ -51,34 +51,22 @@ public class FileDependenciesReader: DependenciesReader {
     public func findDependencies() throws -> [String] {
         let yaml = try readRaw()
 
-        struct ParseState {
-            var buffer: String = ""
-            var prevChar: Character?
-            var result: [String] = []
-            func with(buffer: String? = nil, prevChar: Character? = nil, result: [String]? = nil) -> ParseState {
-                var new = self
-                new.buffer = buffer ?? new.buffer
-                new.prevChar = prevChar ?? new.prevChar
-                new.result = result ?? new.result
-                return new
-            }
-        }
-
         let dependencies = yaml.reduce(Set<String>()) { prev, arg1 -> Set<String> in
             let (key, value) = arg1
             switch key {
             case "dependencies":
                 // 'clang' output formatting
-                return Set(splitDependencyFileList(value))
+                return Set(parseDependencyFileList(value))
             case let s where s.hasSuffix(".o") || s.hasSuffix(".bc"):
                 // 'swiftc' output formatting
-                // take dependencies from any .o or .bc file.
+                // take dependencies from any .o or .bc file
                 // Note: For WMO, all .{o|bc} files have the same dependencies
-                return Set(splitDependencyFileList(value))
+                return Set(parseDependencyFileList(value))
             default:
                 return prev
             }
         }
+
         return Array(dependencies)
     }
 
@@ -92,56 +80,92 @@ public class FileDependenciesReader: DependenciesReader {
         return yaml.mapValues { $0.components(separatedBy: .whitespaces) }
     }
 
-    private func readRaw() throws -> [String: String] {
+    func readRaw() throws -> [String: String] {
+        let fileData = try getFileData()
+        let fileString = try getFileStringFromData(fileData: fileData)
+        let yaml = try getYaml(fileString: fileString)
+        return yaml
+    }
+
+    func getFileData() throws -> Data {
         guard let fileData = fileManager.contents(atPath: file.path) else {
             throw DependenciesReaderError.readingError
         }
+        return fileData
+    }
+
+    func getFileStringFromData(fileData: Data) throws -> String {
         guard let fileString = String(data: fileData, encoding: .utf8) else {
             throw DependenciesReaderError.invalidFile
         }
-        // .d matches the .yaml format
+        return fileString
+    }
+
+    func getYaml(fileString: String) throws -> [String: String] {
         guard let yaml = try Yams.load(yaml: fileString) as? [String: String] else {
             throw DependenciesReaderError.invalidFile
         }
         return yaml
     }
 
-    /// Splits space or new line separated files into a set of files
+    /// Parses the String to get the list of files
+    /// It iterates over the String using its UTF8View since it is more performant (String type operates
+    /// in a higher abstraction level and supports features that have a negative impact in the performance)
     /// It supports escaping whitespace charaters, prefixed with "\\"
     /// - Parameter string: string of whitespace charaters separated file paths
     /// - Returns: Array of all file paths
-    private func splitDependencyFileList(_ string: String) -> [String] {
-        struct ParseState {
-            var buffer: String = ""
-            var prevChar: Character?
-            var result: [String] = []
-            func with(buffer: String? = nil, prevChar: Character? = nil, result: [String]? = nil) -> ParseState {
-                var new = self
-                new.buffer = buffer ?? new.buffer
-                new.prevChar = prevChar ?? new.prevChar
-                new.result = result ?? new.result
-                return new
-            }
+    func parseDependencyFileList(_ string: String) -> [String] {
+        var result: [String] = []
+        var prevChar: UTF8.CodeUnit?
+
+        // These index are used to move over the UTF8View of the string
+        // The goal is to optimize the memory used, since UTF8View uses
+        // the same memory as the original String without copying it
+        var startIndex = string.utf8.startIndex
+        var endIndex = startIndex
+
+        // This buffer is only used to save the part of the path that has been already parsed when finding a backslash
+        var buffer: String = ""
+
+        for c in string.utf8 {
+          switch c {
+          case UTF8.CodeUnit(ascii: "\n") where prevChar == UTF8.CodeUnit(ascii: "\\"):
+              startIndex = string.utf8.index(after: startIndex)
+              endIndex = startIndex
+          case UTF8.CodeUnit(ascii: " ") where startIndex == endIndex && buffer.isEmpty:
+              startIndex = string.utf8.index(after: startIndex)
+              endIndex = startIndex
+          case UTF8.CodeUnit(ascii: " ") where prevChar != UTF8.CodeUnit(ascii: "\\"):
+              // If a space is found and it is not escaped, then that's the end of the file path
+              buffer += String(Substring(string.utf8[startIndex ..< endIndex]))
+              result.append(buffer)
+              buffer = ""
+              prevChar = nil
+              startIndex = string.utf8.index(after: endIndex)
+              endIndex = startIndex
+          case UTF8.CodeUnit(ascii: "\\"):
+              // If a backslash is found it is not included in the file path
+              // The current parsed range of the UTF8View is saved in the buffer as a String
+              buffer += String(Substring(string.utf8[startIndex ..< endIndex]))
+              // The backslash is assigned as the previous char
+              prevChar = c
+              // The indexes are moved to the next char so we continue parsing the String
+              startIndex = string.utf8.index(after: endIndex)
+              endIndex = startIndex
+          default:
+              // As long as it is possible the indexes are used to track the range of the string that
+              // will be included in the file path (until it ends or until a backslash is found)
+              endIndex = string.utf8.index(after: endIndex)
+              // The char is assigned as the previous char
+              prevChar = c
+          }
         }
-        let parseResult = string.reduce(ParseState()) { total, char in
-            switch char {
-            case "\n" where total.prevChar == "\\":
-                return total
-            case " " where total.buffer.isEmpty:
-                return total
-            case " " where total.prevChar == "\\":
-                return total.with(buffer: "\(total.buffer) ")
-            case " ":
-                return total.with(buffer: "", prevChar: nil, result: total.result + [total.buffer])
-            case "\\":
-                return total.with(prevChar: "\\")
-            default:
-                return total.with(buffer: "\(total.buffer)\(char)", prevChar: char, result: total.result)
-            }
+
+        if startIndex != endIndex {
+            buffer += String(Substring(string.utf8[startIndex ..< endIndex]))
+            result.append(buffer)
         }
-        if !parseResult.buffer.isEmpty {
-            return parseResult.result + [parseResult.buffer]
-        }
-        return parseResult.result
+
+        return result
     }
 }

Sources/XCRemoteCache/Dependencies/DependenciesRemapper.swift

@@ -22,9 +22,9 @@ import Foundation
 /// Replaces paths formats between generic (placeholders-based) and local
 protocol DependenciesRemapper {
     /// Replaces all generic paths (with placeholders) to a local paths
-    func replace(genericPaths: [String]) -> [String]
+    func replace(genericPaths: [String]) throws -> [String]
     /// Replaces all local paths to the generic dependencies paths
-    func replace(localPaths: [String]) -> [String]
+    func replace(localPaths: [String]) throws -> [String]
 }
 
 class DependenciesRemapperComposite: DependenciesRemapper {
@@ -34,15 +34,15 @@ class DependenciesRemapperComposite: DependenciesRemapper {
         self.remappers = remappers
     }
 
-    func replace(genericPaths: [String]) -> [String] {
-        remappers.reduce(genericPaths) { prev, mapper in
-            mapper.replace(genericPaths: prev)
+    func replace(genericPaths: [String]) throws -> [String] {
+        try remappers.reversed().reduce(genericPaths) { prev, mapper in
+            try mapper.replace(genericPaths: prev)
         }
     }
 
-    func replace(localPaths: [String]) -> [String] {
-        remappers.reduce(localPaths) { prev, mapper in
-            mapper.replace(localPaths: prev)
+    func replace(localPaths: [String]) throws -> [String] {
+        try remappers.reduce(localPaths) { prev, mapper in
+            try mapper.replace(localPaths: prev)
         }
     }
 }
@@ -59,16 +59,16 @@ final class StringDependenciesRemapper: DependenciesRemapper {
         self.mappings = mappings
     }
 
-    func replace(genericPaths: [String]) -> [String] {
+    func replace(genericPaths: [String]) throws -> [String] {
         return genericPaths.map { path in
-            let localPath = mappings.reduce(path) { prevPath, mapping in
+            let localPath = mappings.reversed().reduce(path) { prevPath, mapping in
                 prevPath.replacingOccurrences(of: mapping.generic, with: mapping.local)
             }
             return localPath
         }
     }
 
-    func replace(localPaths: [String]) -> [String] {
+    func replace(localPaths: [String]) throws -> [String] {
         return localPaths.map { path in
             let result = mappings.reduce(path) { prevPath, mapping in
                 prevPath.replacingOccurrences(of: mapping.local, with: mapping.generic)
@@ -77,14 +77,3 @@ final class StringDependenciesRemapper: DependenciesRemapper {
         }
     }
 }
-
-
-extension StringDependenciesRemapper {
-    static func buildFromEnvs(keys: [String], envs: [String: String]) throws -> Self {
-        let mappings: [Mapping] = try keys.map { key in
-            let localValue: String = try envs.readEnv(key: key)
-            return Mapping(generic: "$(\(key))", local: localValue)
-        }
-        return Self(mappings: mappings)
-    }
-}

Sources/XCRemoteCache/Dependencies/DependenciesWriter.swift

@@ -50,7 +50,7 @@ public class FileDependenciesWriter: DependenciesWriter {
         var content = ""
         for (file, deps) in dependencies {
             content.append(file + ": ")
-            content.append(deps.joined(separator: " "))
+            content.append(deps.map { $0.replacingOccurrences(of: " ", with: "\\ ") }.joined(separator: " "))
             content.append("\n")
         }
         try content.write(to: file, atomically: true, encoding: .utf8)

Sources/XCRemoteCache/Dependencies/DependencyProcessor.swift

@@ -27,8 +27,11 @@ public struct Dependency: Equatable {
         case source
         case fingerprint
         case intermediate
+        case derivedFile
         // Product of the target itself
         case ownProduct
+        // User-excluded path
+        case userExcluded
         case unknown
     }
 
@@ -55,14 +58,18 @@ class DependencyProcessorImpl: DependencyProcessor {
     private let productPath: String
     private let sourcePath: String
     private let intermediatePath: String
+    private let derivedFilesPath: String
     private let bundlePath: String?
+    private let skippedRegexes: [String]
 
-    init(xcode: URL, product: URL, source: URL, intermediate: URL, bundle: URL?) {
-        xcodePath = xcode.path
-        productPath = product.path
-        sourcePath = source.path
-        intermediatePath = intermediate.path
-        bundlePath = bundle?.path
+    init(xcode: URL, product: URL, source: URL, intermediate: URL, derivedFiles: URL, bundle: URL?, skippedRegexes: [String]) {
+        xcodePath = xcode.path.dirPath()
+        productPath = product.path.dirPath()
+        sourcePath = source.path.dirPath()
+        intermediatePath = intermediate.path.dirPath()
+        derivedFilesPath = derivedFiles.path.dirPath()
+        bundlePath = bundle?.path.dirPath()
+        self.skippedRegexes = skippedRegexes
     }
 
     func process(_ files: [URL]) -> [Dependency] {
@@ -72,11 +79,15 @@ class DependencyProcessorImpl: DependencyProcessor {
 
     private func classify(_ files: [URL]) -> [Dependency] {
         return files.map { file -> Dependency in
-            let filePath = file.path
-            if filePath.hasPrefix(xcodePath) {
+            let filePath = file.resolvingSymlinksInPath().path
+            if skippedRegexes.contains(where: { filePath.range(of: $0, options: .regularExpression) != nil }) {
+                return Dependency(url: file, type: .userExcluded)
+            } else if filePath.hasPrefix(xcodePath) {
                 return Dependency(url: file, type: .xcode)
             } else if filePath.hasPrefix(intermediatePath) {
                 return Dependency(url: file, type: .intermediate)
+            } else if filePath.hasPrefix(derivedFilesPath) {
+                return Dependency(url: file, type: .derivedFile)
             } else if let bundle = bundlePath, filePath.hasPrefix(bundle) {
                 // If a target produces a bundle, explicitly classify all
                 // of products to distinguish from other targets products
@@ -107,7 +118,18 @@ class DependencyProcessorImpl: DependencyProcessor {
         // - All files in `*/Interemediates/*` - this file are created on-fly for a given target
         // - Some files may depend on its own product (e.g. .m may #include *-Swift.h) - we know products will match
         //   because in case of a hit, these will be taken from the artifact
-        let irrelevantDependenciesType: [Dependency.Kind] = [.xcode, .intermediate, .ownProduct]
+        // - Customized DERIVED_FILE_DIR may change a directory of
+        //   derived files, which by default is under `*/Interemediates`
+        // - User-specified (in .rcinfo) files to exclude
+        let irrelevantDependenciesType: [Dependency.Kind] = [
+            .xcode, .intermediate, .ownProduct, .derivedFile, .userExcluded,
+        ]
         return !irrelevantDependenciesType.contains(dependency.type)
     }
 }
+
+fileprivate extension String {
+    func dirPath() -> String {
+        hasSuffix("/") ? self : appending("/")
+    }
+}

Sources/XCRemoteCache/Dependencies/OverlayDependenciesRemapper.swift

@@ -0,0 +1,67 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import Foundation
+
+/// File paths remapper according the virtual file system mappings
+/// - Warning: this class is not thread safe
+class OverlayDependenciesRemapper: DependenciesRemapper {
+    private let overlayReader: OverlayReader
+    private var mappings: [OverlayMapping]?
+
+    init(overlayReader: OverlayReader) {
+        self.overlayReader = overlayReader
+    }
+
+    /// Lazily Reads mappings from a file
+    /// - Warning: this function is not thread safe
+    private func getMappings() throws -> [OverlayMapping] {
+        guard let mappings = mappings else {
+            let mappings = try overlayReader.provideMappings()
+            self.mappings = mappings
+            return mappings
+        }
+        return mappings
+    }
+
+    private func mapPath(
+        _ path: String,
+        source: KeyPath<OverlayMapping, URL>,
+        destination: KeyPath<OverlayMapping, URL>
+    ) throws -> String {
+        guard let mapping = try getMappings().first(where: { $0[keyPath: source].path == path }) else {
+            // TODO: support partial mappings, where a directory path can be replaced with some other directory
+            // no direct mapping found
+            return path
+        }
+        return mapping[keyPath: destination].path
+    }
+
+    func replace(genericPaths: [String]) throws -> [String] {
+        try genericPaths.map {
+            try mapPath($0, source: \.virtual, destination: \.local)
+        }
+    }
+
+    func replace(localPaths: [String]) throws -> [String] {
+        try localPaths.map {
+            try mapPath($0, source: \.local, destination: \.virtual)
+        }
+    }
+}

Sources/XCRemoteCache/Dependencies/OverlayReader.swift

@@ -0,0 +1,133 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import Foundation
+
+/// Maps overlay's virtual URL with an actual (local) location
+struct OverlayMapping: Hashable {
+    let virtual: URL
+    let local: URL
+}
+
+enum JsonOverlayReaderError: Error {
+    /// The source file is missing
+    case missingSourceFile(URL)
+    /// The file exists but its content is invalid
+    case invalidSourceContent(URL)
+    /// the overlay format is not supported - either contains a nested directory or a single file 
+    case unsupportedFormat
+}
+/// Provides virtual file system overlay mappings
+protocol OverlayReader {
+    func provideMappings() throws -> [OverlayMapping]
+}
+
+class JsonOverlayReader: OverlayReader {
+
+    enum Mode {
+        /// Interrupts the operation if the representation file is missing
+        case strict
+        /// Assume empty overlay mapping if the file doesn't exist
+        case bestEffort
+    }
+
+    private struct Overlay: Decodable {
+        enum OverlayType: String, Decodable {
+            case file
+            case directory
+        }
+
+        struct Content: Decodable {
+            let externalContents: String
+            let name: String
+            let type: OverlayType
+
+            enum CodingKeys: String, CodingKey {
+                case externalContents = "external-contents"
+                case name
+                case type
+            }
+        }
+
+        struct RootContent: Decodable {
+            let contents: [Content]
+            let name: String
+            let type: OverlayType
+        }
+        let roots: [RootContent]
+    }
+
+    private lazy var jsonDecoder = JSONDecoder()
+    private let json: URL
+    private let mode: Mode
+    private let fileReader: FileReader
+
+
+    init(_ json: URL, mode: Mode, fileReader: FileReader) {
+        self.json = json
+        self.mode = mode
+        self.fileReader = fileReader
+    }
+
+    func provideMappings() throws -> [OverlayMapping] {
+        guard let jsonContent = try fileReader.contents(atPath: json.path) else {
+            switch mode {
+            case .strict:
+                throw JsonOverlayReaderError.missingSourceFile(json)
+            case .bestEffort:
+                debugLog("overlay mapping file \(json) doesn't exist. Skipping overlay for the best-effort mode.")
+                return []
+            }
+        }
+        do {
+        let overlay: Overlay = try jsonDecoder.decode(Overlay.self, from: jsonContent)
+            let mappings: [OverlayMapping] = try overlay.roots.reduce([]) { prev, root in
+                switch root.type {
+                case .directory:
+                    // iterate all contents
+                    let dir = URL(fileURLWithPath: root.name)
+                    let mappings: [OverlayMapping] = try root.contents.map { content in
+                        switch content.type {
+                        case .file:
+                            let virtual = dir.appendingPathComponent(content.name)
+                            let local = URL(fileURLWithPath: content.externalContents)
+                            return .init(virtual: virtual, local: local)
+                        case .directory:
+                            throw JsonOverlayReaderError.unsupportedFormat
+                        }
+
+                    }
+                    return prev + mappings
+                case .file:
+                    throw JsonOverlayReaderError.unsupportedFormat
+                }
+            }
+            return mappings
+        } catch {
+            switch mode {
+            case .strict:
+                throw error
+            case .bestEffort:
+                errorLog("Overlay reader has failed with an error \(error). Best-effort mode - skipping an overlay.")
+                return []
+            }
+        }
+    }
+
+}

Sources/XCRemoteCache/Dependencies/PathDependenciesRemapperFactory.swift

@@ -0,0 +1,47 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import Foundation
+
+enum PathDependenciesRemapperFactoryError: Error {
+    /// Remapping keys are duplicated and can lead to undetermined results
+    case mappingKeyDuplication
+}
+
+class PathDependenciesRemapperFactory {
+    func build(
+        orderKeys: [String],
+        envs: [String: String],
+        customMappings: [String: String]
+    ) throws -> StringDependenciesRemapper {
+        let mappingMap = try envs.merging(customMappings) { _, _ in
+            throw PathDependenciesRemapperFactoryError.mappingKeyDuplication
+        }
+        let mappingOrderKeys = orderKeys + customMappings.keys
+        let mappings: [StringDependenciesRemapper.Mapping] = mappingOrderKeys.compactMap { key in
+            guard let localURL: URL = mappingMap.readEnv(key: key) else {
+                debugLog("\(key) ENV to map a dependency is not defined")
+                return nil
+            }
+            infoLog("Found url to remapp: \(localURL). Remapping: \(localURL.standardized.path)")
+            return StringDependenciesRemapper.Mapping(generic: "$(\(key))", local: localURL.standardized.path)
+        }
+        return StringDependenciesRemapper(mappings: mappings)
+    }
+}

Sources/XCRemoteCache/Dependencies/TargetDepdenciesReader.swift

@@ -42,7 +42,7 @@ class TargetDependenciesReader: DependenciesReader {
         let allURLs = try dirScanner.items(at: directory)
         let mergedDependencies = try allURLs.reduce(Set<String>()) { (prev: Set<String>, file) in
             // include only these .d files that either have corresponding .o file (incremental) or end
-            // with '-master' (whole-module).
+            // with '-master' (whole-module)
             // Otherwise .d is probably just a leftover from previous builds
             let correspondingOutputURL = file.deletingPathExtension().appendingPathExtension("o")
             let isDependencyFile = file.pathExtension == "d"

Sources/XCRemoteCache/Fingerprint/EnvironmentFingerprint.swift

@@ -19,10 +19,10 @@
 
 /// Generates a fingerprint string of the environment (compilation context)
 class EnvironmentFingerprintGenerator {
-    /// Default ENV variables constituing the environment fingerprint
+    /// Default ENV variables constituting the environment fingerprint
     private static let defaultEnvFingerprintKeys = [
         "GCC_PREPROCESSOR_DEFINITIONS",
-        "CLANG_PROFILE_DATA_DIRECTORY",
+        "CLANG_COVERAGE_MAPPING",
         "TARGET_NAME",
         "CONFIGURATION",
         "PLATFORM_NAME",
@@ -31,6 +31,7 @@ class EnvironmentFingerprintGenerator {
         "DYLIB_COMPATIBILITY_VERSION",
         "DYLIB_CURRENT_VERSION",
         "PRODUCT_MODULE_NAME",
+        "ARCHS",
     ]
     private let version: String
     private let customFingerprintEnvs: [String]
@@ -51,7 +52,9 @@ class EnvironmentFingerprintGenerator {
         }
         try fill(envKeys: Self.defaultEnvFingerprintKeys + customFingerprintEnvs)
         try generator.append(version)
-        return try generator.generate()
+        let result = try generator.generate()
+        generatedFingerprint = result
+        return result
     }
 
     /// Creates a fingerprint of the environemtn, by hashing all ENVs specified in keys

Sources/XCRemoteCache/Logger/Logger.swift

@@ -22,34 +22,36 @@ import Foundation
 import os.log
 
 
+private var processTag: String = ""
+
 public func exit(_ exitCode: Int32, _ message: String) -> Never {
-    os_log("%{public}@", log: OSLog.default, type: .error, message)
+    os_log("%{public}@%{public}@", log: OSLog.default, type: .error, processTag, message)
     printError(errorMessage: message)
     exit(exitCode)
 }
 
 func defaultLog(_ message: String) {
-    os_log("%{public}@", log: OSLog.default, type: .default, message)
+    os_log("%{public}@%{public}@", log: OSLog.default, type: .default, processTag, message)
 }
 
 func errorLog(_ message: String) {
-    os_log("%{public}@", log: OSLog.default, type: .error, message)
+    os_log("%{public}@%{public}@", log: OSLog.default, type: .error, processTag, message)
 }
 
 func infoLog(_ message: String) {
-    os_log("%{public}@", log: OSLog.default, type: .info, message)
+    os_log("%{public}@%{public}@", log: OSLog.default, type: .info, processTag, message)
 }
 
 func debugLog(_ message: String) {
-    os_log("%{public}@", log: OSLog.default, type: .debug, message)
+    os_log("%{public}@%{public}@", log: OSLog.default, type: .debug, processTag, message)
 }
 
 func printError(errorMessage: String) {
-    fputs("error: \(errorMessage)\n", stderr)
+    fputs("error: \(processTag)\(errorMessage)\n", stderr)
 }
 
 func printWarning(_ message: String) {
-    print("warning: \(message)")
+    print("warning: \(processTag)\(message)")
 }
 
 /// Prints a message to the user. It shows in Xcode (if applies) or console output
@@ -57,3 +59,7 @@ func printWarning(_ message: String) {
 func printToUser(_ message: String) {
     print("[RC] \(message)")
 }
+
+func updateProcessTag(_ tag: String) {
+    processTag = "(\(tag)) "
+}

Sources/XCRemoteCache/Models/MetaWriter.swift

@@ -20,18 +20,23 @@
 import Foundation
 
 protocol MetaWriter {
-    func write<T>(_ meta: T, locationDir : URL) throws -> URL where T : Meta
+    func write<T>(_ meta: T, locationDir: URL) throws -> URL where T: Meta
 }
 
 class JsonMetaWriter: MetaWriter {
-    private let metaEncoder = JSONEncoder()
+    private let metaEncoder: JSONEncoder
     private let fileWriter: FileWriter
 
-    init(fileWriter: FileWriter) {
+    init(fileWriter: FileWriter, pretty: Bool) {
         self.fileWriter = fileWriter
+        let encoder = JSONEncoder()
+        if pretty {
+            encoder.outputFormatting = .prettyPrinted
+        }
+        self.metaEncoder = encoder
     }
 
-    func write<T>(_ meta: T, locationDir : URL) throws -> URL where T : Meta {
+    func write<T>(_ meta: T, locationDir: URL) throws -> URL where T: Meta {
         let metaURL = locationDir.appendingPathComponent(meta.fileKey).appendingPathExtension("json")
         let metaData = try metaEncoder.encode(meta)
         try fileWriter.write(toPath: metaURL.path, contents: metaData)

Sources/XCRemoteCache/Network/Authentication/AWSV4Signature.swift

@@ -23,17 +23,21 @@ struct AWSV4Signature {
 
     let secretKey: String
     let accessKey: String
+    let securityToken: String?
     let region: String
     let service: String
     let date: Date
 
-
     func addSignatureHeaderTo(request: inout URLRequest) {
 
         request.setValue(request.url?.host, forHTTPHeaderField: "host")
         request.setValue(StringToSign.ISO8601BasicFormatter.string(from: date), forHTTPHeaderField: "x-amz-date")
         request.setValue((request.httpBody ?? Data()).sha256(), forHTTPHeaderField: "x-amz-content-sha256")
 
+        if let securityToken = securityToken {
+            request.setValue(securityToken, forHTTPHeaderField: "x-amz-security-token")
+        }
+
         let canonicalRequest = CanonicalRequest(request: request)
         let stringToSign = StringToSign(
             region: region,

Sources/XCRemoteCache/Network/IgnoringCertificatesTrustManager.swift

@@ -0,0 +1,32 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import Foundation
+
+final class IgnoringCertificatesTrustManager: NSObject, URLSessionDelegate {
+    func urlSession(_ session: URLSession, didReceive challenge: URLAuthenticationChallenge, completionHandler: @escaping (URLSession.AuthChallengeDisposition, URLCredential?) -> Void) {
+        guard let serverTrust = challenge.protectionSpace.serverTrust else {
+            completionHandler(.performDefaultHandling, nil)
+            return
+        }
+
+        let urlCredential = URLCredential(trust: serverTrust)
+        completionHandler(.useCredential, urlCredential)
+    }
+}

Sources/XCRemoteCache/Network/URLSessionFactory.swift

@@ -36,6 +36,17 @@ class DefaultURLSessionFactory: URLSessionFactory {
         let configuration = URLSessionConfiguration.default
         configuration.httpAdditionalHeaders = config.requestCustomHeaders
         configuration.timeoutIntervalForRequest = config.timeoutResponseDataChunksInterval
-        return URLSession(configuration: configuration)
+        configuration.urlCache?.memoryCapacity = 0
+        configuration.urlCache?.diskCapacity = 0
+        switch config.disableCertificateVerification {
+        case true:
+            return URLSession(
+                configuration: configuration,
+                delegate: IgnoringCertificatesTrustManager(),
+                delegateQueue: nil
+            )
+        case false:
+            return URLSession(configuration: configuration)
+        }
     }
 }

Sources/XCRemoteCache/Stats/ExclusiveFileAccessor.swift

@@ -76,8 +76,8 @@ class ExclusiveFile: ExclusiveFileAccessor {
         guard flock(fd, LOCK_EX) == 0 else {
             throw FileAccessorError.lockingFailure
         }
-        // While having a lock, make sure the file still exists.
-        // It might delete it while we were waiting for a lock.
+        // While having a lock, make sure the file still exists
+        // It might delete it while we were waiting for a lock
         guard access(fileURL.path, F_OK) == 0 else {
             throw FileAccessorError.lockingFailure
         }

Sources/XCRemoteCache/Utils/ENVReader.swift

@@ -24,8 +24,15 @@ enum EnvironmentError: Error {
 }
 
 extension Dictionary where Key == String, Value == String {
-    func readEnv(key: String) throws -> URL {
+    func readEnv(key: String) -> URL? {
         guard let value = self[key].map(URL.init(fileURLWithPath:)) else {
+            return nil
+        }
+        return value
+    }
+
+    func readEnv(key: String) throws -> URL {
+        guard let value: URL = readEnv(key: key) else {
             throw EnvironmentError.missingEnv(key)
         }
         return value

Sources/xcld/XCLDMain.swift

@@ -21,7 +21,7 @@ import Foundation
 import XCRemoteCache
 
 /// Wrapper for a `LD` program that copies the dynamic executable from a cached-downloaded location
-/// Fallbacks to a standard `clang` when the Ramote cache is not applicable (e.g. modified sources)
+/// Fallbacks to a standard `clang` when the Remote cache is not applicable (e.g. modified sources)
 public class XCLDMain {
     public func main() {
         let args = ProcessInfo().arguments
@@ -48,7 +48,16 @@ public class XCLDMain {
             i += 1
         }
         guard let outputInput = output, let filelistInput = filelist, let dependencyInfoInput = dependencyInfo else {
-            exit(1, "Missing 'output' argument. Args: \(args)")
+            let ldCommand = "clang"
+            print("Fallbacking to compilation using \(ldCommand).")
+
+            let args = ProcessInfo().arguments
+            let paramList = [ldCommand] + args.dropFirst()
+            let cargs = paramList.map { strdup($0) } + [nil]
+            execvp(ldCommand, cargs)
+
+            /// C-function `execv` returns only when the command fails
+            exit(1)
         }
 
 

Sources/xcldplusplus/XCLDPlusPlus.swift

@@ -0,0 +1,75 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import Foundation
+import XCRemoteCache
+
+/// Wrapper for a `LDPLUSPLUS` program that copies the dynamic executable from a cached-downloaded location
+/// Fallbacks to a standard `clang++` when the Remote cache is not applicable (e.g. modified sources)
+public class XCLDPlusPlusMain {
+    public func main() {
+        let args = ProcessInfo().arguments
+        var output: String?
+        var filelist: String?
+        var dependencyInfo: String?
+        var i = 0
+        while i < args.count {
+            switch args[i] {
+            case "-o":
+                output = args[i + 1]
+                i += 1
+            case "-filelist":
+                filelist = args[i + 1]
+                i += 1
+            case "-dependency_info":
+                // Skip following `-Xlinker` argument. Sample call:
+                // `clang -dynamiclib  ... -Xlinker -dependency_info -Xlinker /path/Target_dependency_info.dat`
+                dependencyInfo = args[i + 2]
+                i += 2
+            default:
+                break
+            }
+            i += 1
+        }
+        guard let outputInput = output, let filelistInput = filelist, let dependencyInfoInput = dependencyInfo else {
+            let ldCommand = "clang++"
+            print("Fallbacking to compilation using \(ldCommand).")
+
+            let args = ProcessInfo().arguments
+            let paramList = [ldCommand] + args.dropFirst()
+            let cargs = paramList.map { strdup($0) } + [nil]
+            execvp(ldCommand, cargs)
+
+            /// C-function `execv` returns only when the command fails
+            exit(1)
+        }
+
+
+        // TODO: consider using `clang_command` from .rcinfo
+        /// concrete clang path should be taken from the current toolchain
+        let fallbackCommand = "clang++"
+        XCCreateBinary(
+            output: outputInput,
+            filelist: filelistInput,
+            dependencyInfo: dependencyInfoInput,
+            fallbackCommand: fallbackCommand,
+            stepDescription: "xcldplusplus"
+        ).run()
+    }
+}

Sources/xcldplusplus/main.swift

@@ -0,0 +1,22 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import XCRemoteCache
+
+XCLDPlusPlusMain().main()

Sources/xcswiftc/XCSwiftcMain.swift

@@ -60,8 +60,16 @@ public class XCSwiftcMain {
             let targetInputInput = target,
             let swiftFileListInput = swiftFileList
             else {
-                print("Missing argument. Args: \(args)")
-                exit(1)
+            let swiftcCommand = "swiftc"
+            print("Fallbacking to compilation using \(swiftcCommand).")
+
+            let args = ProcessInfo().arguments
+            let paramList = [swiftcCommand] + args.dropFirst()
+            let cargs = paramList.map { strdup($0) } + [nil]
+            execvp(swiftcCommand, cargs)
+
+            /// C-function `execv` returns only when the command fails
+            exit(1)
         }
         let swiftcArgsInput = SwiftcArgInput(
             objcHeaderOutput: objcHeaderOutputInput,

Tests/XCRemoteCacheTests/Artifacts/ArtifactSwiftProductsBuilderImplTests.swift

@@ -27,6 +27,7 @@ class ArtifactSwiftProductsBuilderImplTests: FileXCTestCase {
     private var swiftmoduleFile: URL!
     private var swiftmoduleDocFile: URL!
     private var swiftmoduleSourceInfoFile: URL!
+    private var swiftmoduleInterfaceFile: URL!
     private var workingDir: URL!
     private var builder: ArtifactSwiftProductsBuilderImpl!
 
@@ -37,6 +38,7 @@ class ArtifactSwiftProductsBuilderImplTests: FileXCTestCase {
         swiftmoduleFile = moduleDir.appendingPathComponent("MyModule.swiftmodule")
         swiftmoduleDocFile = moduleDir.appendingPathComponent("MyModule.swiftdoc")
         swiftmoduleSourceInfoFile = moduleDir.appendingPathComponent("MyModule.swiftsourceinfo")
+        swiftmoduleInterfaceFile = moduleDir.appendingPathComponent("MyModule.swiftinterface")
         workingDir = rootDir.appendingPathComponent("working")
         builder = ArtifactSwiftProductsBuilderImpl(
             workingDir: workingDir,
@@ -69,7 +71,7 @@ class ArtifactSwiftProductsBuilderImplTests: FileXCTestCase {
         )
     }
 
-    func testIncludesAllSwiftmoduleFiles() throws {
+    func testIncludesAllBasicSwiftmoduleFiles() throws {
         try fileManager.spt_createEmptyFile(swiftmoduleFile)
         try fileManager.spt_createEmptyFile(swiftmoduleDocFile)
         try fileManager.spt_createEmptyFile(swiftmoduleSourceInfoFile)
@@ -91,6 +93,32 @@ class ArtifactSwiftProductsBuilderImplTests: FileXCTestCase {
         XCTAssertTrue(fileManager.fileExists(atPath: expectedBuildedSwiftSourceInfoFile.path))
     }
 
+    func testIncludesAllEvolutionEnabledSwiftmoduleFiles() throws {
+        try fileManager.spt_createEmptyFile(swiftmoduleFile)
+        try fileManager.spt_createEmptyFile(swiftmoduleDocFile)
+        try fileManager.spt_createEmptyFile(swiftmoduleSourceInfoFile)
+        try fileManager.spt_createEmptyFile(swiftmoduleInterfaceFile)
+        let builderSwiftmoduleDir =
+            builder
+                .buildingArtifactSwiftModulesLocation()
+                .appendingPathComponent("arm64")
+        let expectedBuildedSwiftmoduleFile =
+            builderSwiftmoduleDir.appendingPathComponent("MyModule.swiftmodule")
+        let expectedBuildedSwiftmoduledocFile =
+            builderSwiftmoduleDir.appendingPathComponent("MyModule.swiftdoc")
+        let expectedBuildedSwiftSourceInfoFile =
+            builderSwiftmoduleDir.appendingPathComponent("MyModule.swiftsourceinfo")
+        let expectedBuildedSwiftInterfaceFile =
+            builderSwiftmoduleDir.appendingPathComponent("MyModule.swiftinterface")
+
+        try builder.includeModuleDefinitionsToTheArtifact(arch: "arm64", moduleURL: swiftmoduleFile)
+
+        XCTAssertTrue(fileManager.fileExists(atPath: expectedBuildedSwiftmoduleFile.path))
+        XCTAssertTrue(fileManager.fileExists(atPath: expectedBuildedSwiftmoduledocFile.path))
+        XCTAssertTrue(fileManager.fileExists(atPath: expectedBuildedSwiftSourceInfoFile.path))
+        XCTAssertTrue(fileManager.fileExists(atPath: expectedBuildedSwiftInterfaceFile.path))
+    }
+
     func testFailsIncludingWhenMissingRequiredSwiftmoduleFiles() throws {
         XCTAssertThrowsError(
             try builder.includeModuleDefinitionsToTheArtifact(

Tests/XCRemoteCacheTests/Artifacts/BuildArtifactCreatorTests.swift

@@ -31,6 +31,7 @@ class BuildArtifactCreatorTests: FileXCTestCase {
     private var swiftmoduleURL: URL!
     private var swiftdocURL: URL!
     private var swiftSourceInfoURL: URL!
+    private var swiftInterfaceURL: URL!
     private var executablePath: String!
     private var executableURL: URL!
     private var creator: BuildArtifactCreator!
@@ -50,6 +51,8 @@ class BuildArtifactCreatorTests: FileXCTestCase {
             .appendingPathComponent("Target.swiftdoc")
         swiftSourceInfoURL = workDirectory.appendingPathComponent("Objects-normal")
             .appendingPathComponent("Target.swiftsourceinfo")
+        swiftInterfaceURL = workDirectory.appendingPathComponent("Objects-normal")
+            .appendingPathComponent("Target.swiftinterface")
         executablePath = "libTarget.a"
         executableURL = buildDir.appendingPathComponent(executablePath)
         dSYM = executableURL.deletingPathExtension().appendingPathExtension(".dSYM")
@@ -63,7 +66,7 @@ class BuildArtifactCreatorTests: FileXCTestCase {
             moduleName: "Target",
             modulesFolderPath: "",
             dSYMPath: dSYM,
-            metaWriter: JsonMetaWriter(fileWriter: fileManager),
+            metaWriter: JsonMetaWriter(fileWriter: fileManager, pretty: false),
             fileManager: fileManager
         )
     }
@@ -114,6 +117,28 @@ class BuildArtifactCreatorTests: FileXCTestCase {
         ])
     }
 
+    func testPackagesEvolutionEnabledSwiftmoduleFiles() throws {
+        try fileManager.spt_createEmptyFile(swiftmoduleURL)
+        try fileManager.spt_createEmptyFile(swiftdocURL)
+        try fileManager.spt_createEmptyFile(swiftSourceInfoURL)
+        try fileManager.spt_createEmptyFile(swiftInterfaceURL)
+
+        try creator.includeModuleDefinitionsToTheArtifact(arch: "arch", moduleURL: swiftmoduleURL)
+        let artifact = try creator.createArtifact(artifactKey: "key", meta: sampleMeta)
+
+        let unzippedURL = workDirectory.appendingPathComponent(UUID().uuidString)
+        try Zip.unzipFile(artifact.package, destination: unzippedURL, overwrite: true, password: nil, progress: nil)
+        let allFiles = try fileManager.spt_allFilesRecusively(unzippedURL)
+        XCTAssertEqual(Set(allFiles), [
+            unzippedURL.appendingPathComponent("libTarget.a"),
+            unzippedURL.appendingPathComponent("fileKey.json"),
+            unzippedURL.appendingPathComponent("swiftmodule/arch/Target.swiftmodule"),
+            unzippedURL.appendingPathComponent("swiftmodule/arch/Target.swiftdoc"),
+            unzippedURL.appendingPathComponent("swiftmodule/arch/Target.swiftsourceinfo"),
+            unzippedURL.appendingPathComponent("swiftmodule/arch/Target.swiftinterface"),
+        ])
+    }
+
     func testFailsPackageWhenSwiftmoduleRelatedFilesAreMissing() throws {
         // Creating only `Target.swiftmodule`, without `.swiftdoc`
         try fileManager.spt_createEmptyFile(swiftmoduleURL)

Tests/XCRemoteCacheTests/Artifacts/ZipArtifactCreatorTests.swift

@@ -37,7 +37,7 @@ class ZipArtifactCreatorTests: FileXCTestCase {
         workingDir = try prepareTempDir().appendingPathComponent("creator")
         creator = ZipArtifactCreator(
             workingDir: workingDir,
-            metaWriter: JsonMetaWriter(fileWriter: fileManager),
+            metaWriter: JsonMetaWriter(fileWriter: fileManager, pretty: false),
             fileManager: fileManager
         )
     }

Tests/XCRemoteCacheTests/Commands/Plugins/Thinning/ThinningCreatorPluginTests.swift

@@ -131,7 +131,7 @@ class ThinningCreatorPluginTests: FileXCTestCase {
 
         XCTAssertEqual(extraKeys, [
             "thinning_Generated": "000",
-            "thinning_Reused": "999"
+            "thinning_Reused": "999",
         ])
     }
 }

Tests/XCRemoteCacheTests/Commands/PostbuildContextTests.swift

@@ -26,8 +26,9 @@ class PostbuildContextTests: FileXCTestCase {
     private static let SampleEnvs = [
         "TARGET_NAME": "TARGET_NAME",
         "TARGET_TEMP_DIR": "TARGET_TEMP_DIR",
-        "PLATFORM_PREFERRED_ARCH": "PLATFORM_PREFERRED_ARCH",
-        "OBJECT_FILE_DIR_normal": "OBJECT_FILE_DIR_normal" ,
+        "DERIVED_FILE_DIR": "DERIVED_FILE_DIR",
+        "ARCHS": "x86_64",
+        "OBJECT_FILE_DIR_normal": "/OBJECT_FILE_DIR_normal" ,
         "CONFIGURATION": "CONFIGURATION",
         "PLATFORM_NAME": "PLATFORM_NAME",
         "XCODE_PRODUCT_BUILD_VERSION": "XCODE_PRODUCT_BUILD_VERSION",
@@ -42,6 +43,7 @@ class PostbuildContextTests: FileXCTestCase {
         "DWARF_DSYM_FILE_NAME": "DWARF_DSYM_FILE_NAME",
         "BUILT_PRODUCTS_DIR": "BUILT_PRODUCTS_DIR",
         "DERIVED_SOURCES_DIR": "DERIVED_SOURCES_DIR",
+        "CURRENT_VARIANT": "normal",
     ]
 
     override func setUpWithError() throws {
@@ -87,4 +89,45 @@ class PostbuildContextTests: FileXCTestCase {
 
         XCTAssertEqual(context.action, .index)
     }
+
+    func testReadsSingleArch() throws {
+        var envs = Self.SampleEnvs
+        envs["ARCHS"] = "x86_64"
+        let context = try PostbuildContext(config, env: envs)
+
+        XCTAssertEqual(context.arch, "x86_64")
+    }
+
+    func testReadsFirstArch() throws {
+        var envs = Self.SampleEnvs
+        envs["ARCHS"] = "x86_64 arm64"
+        let context = try PostbuildContext(config, env: envs)
+
+        XCTAssertEqual(context.arch, "x86_64")
+    }
+
+    func testFailsForEmptyArch() throws {
+        var envs = Self.SampleEnvs
+        envs["ARCHS"] = ""
+
+        XCTAssertThrowsError(try PostbuildContext(config, env: envs))
+    }
+
+    func testFailsForMissingArch() throws {
+        var envs = Self.SampleEnvs
+        envs["ARCHS"] = nil
+
+        XCTAssertThrowsError(try PostbuildContext(config, env: envs))
+    }
+
+    func testFindTempDirForCustomVariant() throws {
+        var envs = Self.SampleEnvs
+        envs["ARCHS"] = "x86_64"
+        envs["CURRENT_VARIANT"] = "custom"
+        envs["OBJECT_FILE_DIR_custom"] = "/OBJECT_FILE_DIR_custom"
+
+        let context = try PostbuildContext(config, env: envs)
+
+        XCTAssertEqual(context.compilationTempDir, "/OBJECT_FILE_DIR_custom/x86_64")
+    }
 }

Tests/XCRemoteCacheTests/Commands/PostbuildTests.swift

@@ -20,12 +20,14 @@
 @testable import XCRemoteCache
 import XCTest
 
+// swiftlint:disable file_length
 // swiftlint:disable:next type_body_length
 class PostbuildTests: FileXCTestCase {
     private var postbuildContext = PostbuildContext(
         mode: .producer,
         targetName: "",
         targetTempDir: "",
+        derivedFilesDir: "",
         compilationTempDir: "",
         configuration: "",
         platform: "",
@@ -51,7 +53,9 @@ class PostbuildTests: FileXCTestCase {
         derivedSourcesDir: "",
         thinnedTargets: [],
         action: .build,
-        modeMarkerPath: ""
+        modeMarkerPath: "",
+        overlayHeadersPath: "",
+        irrelevantDependenciesPaths: []
     )
     private var network = RemoteNetworkClientImpl(
         NetworkClientFake(fileManager: .default),
@@ -76,7 +80,9 @@ class PostbuildTests: FileXCTestCase {
         product: "/Product",
         source: "/Source",
         intermediate: "/Intermediate",
-        bundle: nil
+        derivedFiles: "/DerivedFiles",
+        bundle: nil,
+        skippedRegexes: []
     )
     private var overrideManager = FingerprintOverrideManagerImpl(
         overridingFileExtensions: ["swiftmodule"],
@@ -85,7 +91,7 @@ class PostbuildTests: FileXCTestCase {
     )
     private var modeController = CacheModeControllerFake()
     private var metaReader = JsonMetaReader(fileAccessor: FileManager.default)
-    private var metaWriter = JsonMetaWriter(fileWriter: FileManager.default)
+    private var metaWriter = JsonMetaWriter(fileWriter: FileManager.default, pretty: false)
     private static let SampleMeta = MainArtifactSampleMeta.defaults
     private var sampleMetaFile: URL!
 
@@ -638,4 +644,3 @@ class PostbuildTests: FileXCTestCase {
         XCTAssertEqual(downloadedMeta, expectedMeta)
     }
 }
-// swiftlint:disable:next file_length

Tests/XCRemoteCacheTests/Commands/PrebuildTests.swift

@@ -62,7 +62,8 @@ class PrebuildTests: FileXCTestCase {
             forceCached: false,
             compilationHistoryFile: compilationHistory,
             turnOffRemoteCacheOnFirstTimeout: true,
-            targetName: ""
+            targetName: "",
+            overlayHeadersPath: ""
         )
         contextCached = PrebuildContext(
             targetTempDir: sampleURL,
@@ -74,7 +75,8 @@ class PrebuildTests: FileXCTestCase {
             forceCached: true,
             compilationHistoryFile: compilationHistory,
             turnOffRemoteCacheOnFirstTimeout: true,
-            targetName: ""
+            targetName: "",
+            overlayHeadersPath: ""
         )
         organizer = ArtifactOrganizerFake(artifactRoot: artifactsRoot, unzippedExtension: "unzip")
         globalCacheSwitcher = InMemoryGlobalCacheSwitcher()
@@ -238,7 +240,8 @@ class PrebuildTests: FileXCTestCase {
             forceCached: false,
             compilationHistoryFile: compilationHistory,
             turnOffRemoteCacheOnFirstTimeout: true,
-            targetName: ""
+            targetName: "",
+            overlayHeadersPath: ""
         )
 
         let prebuild = Prebuild(
@@ -268,7 +271,8 @@ class PrebuildTests: FileXCTestCase {
             forceCached: false,
             compilationHistoryFile: compilationHistory,
             turnOffRemoteCacheOnFirstTimeout: true,
-            targetName: ""
+            targetName: "",
+            overlayHeadersPath: ""
         )
         metaContent = try generateMeta(fingerprint: generator.generate(), filekey: "1")
         let downloadedArtifactPackage = artifactsRoot.appendingPathComponent("1")
@@ -330,7 +334,8 @@ class PrebuildTests: FileXCTestCase {
             forceCached: false,
             compilationHistoryFile: compilationHistory,
             turnOffRemoteCacheOnFirstTimeout: false,
-            targetName: ""
+            targetName: "",
+            overlayHeadersPath: ""
         )
         try globalCacheSwitcher.enable(sha: "1")
         let prebuild = Prebuild(

Tests/XCRemoteCacheTests/Commands/Prepare/Integrate/XcodeProjBuildSettingsIntegrateAppenderTests.swift

@@ -0,0 +1,63 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+@testable import XCRemoteCache
+import XCTest
+
+class XcodeProjBuildSettingsIntegrateAppenderTests: XCTestCase {
+    private let rootURL: URL = "/root"
+    private let binariesDir: URL = "/binaries"
+    private var buildSettings: BuildSettings!
+    private var binaries: XCRCBinariesPaths!
+
+    override func setUp() {
+        super.setUp()
+        buildSettings = BuildSettings()
+        binaries = XCRCBinariesPaths(
+            prepare: binariesDir.appendingPathComponent("xcprepare"),
+            cc: binariesDir.appendingPathComponent("xccc"),
+            swiftc: binariesDir.appendingPathComponent("xcswiftc"),
+            libtool: binariesDir.appendingPathComponent("xclibtool"),
+            ld: binariesDir.appendingPathComponent("xcld"),
+            ldplusplus: binariesDir.appendingPathComponent("xcldplusplus"),
+            prebuild: binariesDir.appendingPathComponent("xcprebuild"),
+            postbuild: binariesDir.appendingPathComponent("xcpostbuild")
+        )
+    }
+
+    func testProducerSettingFakeSrcRoot() throws {
+        let mode: Mode = .producer
+        let fakeRootURL: URL = "/xxxxxxxxxxP"
+        let appender = XcodeProjBuildSettingsIntegrateAppender(mode: mode, repoRoot: rootURL, fakeSrcRoot: fakeRootURL)
+        let result = appender.appendToBuildSettings(buildSettings: buildSettings, wrappers: binaries)
+        let resultURL = try XCTUnwrap(result["XCRC_FAKE_SRCROOT"] as? String)
+
+        XCTAssertEqual(resultURL, fakeRootURL.path)
+    }
+
+    func testConsumerSettingFakeSrcRoot() throws {
+        let mode: Mode = .consumer
+        let fakeRootURL: URL = "/xxxxxxxxxxC"
+        let appender = XcodeProjBuildSettingsIntegrateAppender(mode: mode, repoRoot: rootURL, fakeSrcRoot: fakeRootURL)
+        let result = appender.appendToBuildSettings(buildSettings: buildSettings, wrappers: binaries)
+        let resultURL: String = try XCTUnwrap(result["XCRC_FAKE_SRCROOT"] as? String)
+
+        XCTAssertEqual(resultURL, fakeRootURL.path)
+    }
+}

Tests/XCRemoteCacheTests/Commands/SwiftcTests.swift

@@ -280,6 +280,9 @@ class SwiftcTests: FileXCTestCase {
         let artifactSwiftSourceInfo = URL(
             fileURLWithPath: "/cachedArtifact/swiftmodule/archTest/Target.swiftsourceinfo"
         )
+        let artifactSwiftInterfaceInfo = URL(
+            fileURLWithPath: "/cachedArtifact/swiftmodule/archTest/Target.swiftinterface"
+        )
 
         artifactOrganizer = ArtifactOrganizerFake(artifactRoot: artifactRoot)
         let swiftc = Swiftc(
@@ -303,12 +306,14 @@ class SwiftcTests: FileXCTestCase {
         let swiftModuleURL = swiftModuleFiles.0[.swiftmodule]
         let swiftDocURL = swiftModuleFiles.0[.swiftdoc]
         let swiftSourceInfoURL = swiftModuleFiles.0[.swiftsourceinfo]
+        let swiftInterfaceURL = swiftModuleFiles.0[.swiftinterface]
         let swiftHeaderURL = swiftModuleFiles.1
 
         XCTAssertEqual(swiftModuleURL, artifactSwiftmodule)
         XCTAssertEqual(swiftDocURL, artifactSwiftdoc)
         XCTAssertEqual(swiftSourceInfoURL, artifactSwiftSourceInfo)
         XCTAssertEqual(swiftHeaderURL, artifactObjCHeader)
+        XCTAssertEqual(swiftInterfaceURL, artifactSwiftInterfaceInfo)
     }
 
 

Tests/XCRemoteCacheTests/Commands/XCCCTests.swift

@@ -28,7 +28,7 @@ class TemplateBasedCCWrapperBuilderTests: FileXCTestCase {
     private static let history = "history.compile"
     private static let prebuild = "prebuild.d"
     private static let commitSha = "321"
-    private static let timeout = 5.0
+    private static let timeout = 10.0
 
     static let xccc: URL = {
         let fileManager = FileManager.default
@@ -47,7 +47,8 @@ class TemplateBasedCCWrapperBuilderTests: FileXCTestCase {
         let app = appDir.appendingPathComponent("xccc")
         try? fileManager.removeItem(at: appDir)
         try? FileManager.default.createDirectory(at: appDir, withIntermediateDirectories: true, attributes: nil)
-        try? builder.compile(to: app, commitSha: commitSha)
+        // swiftlint:disable:next force_try
+        try! builder.compile(to: app, commitSha: commitSha)
         return app
     }()
 
@@ -332,11 +333,31 @@ class TemplateBasedCCWrapperBuilderTests: FileXCTestCase {
         XCTAssertNotEqual(newFileOutputData, Data())
     }
 
+    func testPCHCompilationFallbacks() throws {
+        // Marker is empty to mimic the new file scenario
+        let pchFile = directory.appendingPathComponent("input.pch")
+        createValidPCHFile(pchFile)
+        arguments = ["-x", "objective-c-header", "-MF", dependencyFile.path, "-o", outputFile.path, pchFile.path]
+
+        try shellExec(Self.xccc.path, args: arguments, inDir: directory.path)
+
+        XCTAssertTrue(fileManager.fileExists(atPath: outputFile.path))
+    }
+
     /// Creates a simple C code in the location
     private func createValidCFile(_ location: URL) {
         fileManager.createFile(atPath: location.path, contents: "int main(){}".data(using: .utf8), attributes: nil)
     }
 
+    /// Creates a simple PCH code in the location
+    private func createValidPCHFile(_ location: URL) {
+        fileManager.createFile(
+            atPath: location.path,
+            contents: "#import <Availability.h>".data(using: .utf8),
+            attributes: nil
+        )
+    }
+
     /// Creates a C code that requires extra CUSTOM_STR clang macro to compile
     private func createInvalidCFile(_ location: URL) {
         fileManager.createFile(

Tests/XCRemoteCacheTests/Config/XCRemoteCacheConfigReaderTests.swift

@@ -0,0 +1,88 @@
+// Copyright (c) 2022 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+@testable import XCRemoteCache
+import XCTest
+
+class XCRemoteCacheConfigReaderTests: XCTestCase {
+
+    private var fileReader: FileAccessorFake!
+    private var reader: XCRemoteCacheConfigReader!
+
+    override func setUp() {
+        super.setUp()
+        fileReader = FileAccessorFake(mode: .normal)
+        reader = XCRemoteCacheConfigReader(srcRootPath: "/", fileReader: fileReader)
+    }
+
+    func testReadsFromExtraConfig() throws {
+        try fileReader.write(toPath: "/.rcinfo", contents: "cache_addresses: [test]")
+
+        let config = try reader.readConfiguration()
+
+        XCTAssertEqual(config.cacheAddresses, ["test"])
+    }
+
+    func testOverridesExtraConfigFromExtraFile() throws {
+        try fileReader.write(toPath: "/.rcinfo", contents: "cache_addresses: [test]")
+        try fileReader.write(toPath: "/user.rcinfo", contents: "cache_addresses: [user]")
+
+        let config = try reader.readConfiguration()
+
+        XCTAssertEqual(config.cacheAddresses, ["user"])
+    }
+
+    func testReadsExtraConfigMultipleTimes() throws {
+        try fileReader.write(toPath: "/.rcinfo", contents: "cache_addresses: [test]")
+        try fileReader.write(toPath: "/user.rcinfo", contents: """
+        cache_addresses: [user]
+        extra_configuration_file: user2.rcinfo
+        """)
+        try fileReader.write(toPath: "/user2.rcinfo", contents: "cache_addresses: [user2]")
+
+        let config = try reader.readConfiguration()
+
+        XCTAssertEqual(config.cacheAddresses, ["user2"])
+    }
+
+    func testBreaksImportingExtraConfigIfReachingALoop() throws {
+        try fileReader.write(toPath: "/.rcinfo", contents: "cache_addresses: [test]")
+        try fileReader.write(toPath: "/user.rcinfo", contents: """
+        cache_addresses: [user]
+        extra_configuration_file: .rcinfo
+        """)
+
+        let config = try reader.readConfiguration()
+
+        XCTAssertEqual(config.cacheAddresses, ["user"])
+    }
+
+    func testBreaksImportingExtraConfigIfFileDoesntExist() throws {
+        try fileReader.write(toPath: "/.rcinfo", contents: "cache_addresses: [test]")
+        try fileReader.write(toPath: "/user.rcinfo", contents: """
+        cache_addresses: [user]
+        extra_configuration_file: nonexisting.rcinfo
+        """)
+
+        let config = try reader.readConfiguration()
+
+        XCTAssertEqual(config.cacheAddresses, ["user"])
+        XCTAssertEqual(config.extraConfigurationFile, "nonexisting.rcinfo")
+    }
+}

Tests/XCRemoteCacheTests/Dependencies/DependenciesReaderPerformanceTest.swift

@@ -0,0 +1,139 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+@testable import XCRemoteCache
+
+import XCTest
+
+class DependenciesReaderPerformanceTest: XCTestCase {
+
+    private static let resourcesSubdirectory = "TestData/Dependencies/DependenciesReaderPerformanceTest"
+
+    private func pathForTestData(name: String) throws -> URL {
+        return try XCTUnwrap(Bundle.module.url(
+            forResource: name,
+            withExtension: "d",
+            subdirectory: DependenciesReaderPerformanceTest.resourcesSubdirectory
+        ))
+    }
+
+    func testFindDependenciesPerformance() throws {
+        let file = try pathForTestData(name: "dependencies")
+        let reader = FileDependenciesReader(file, accessor: FileManager.default)
+
+        self.measure { // 0.005
+            do {
+                _ = try reader.findDependencies()
+            } catch {
+                print("Error reading dependencies")
+            }
+        }
+    }
+
+    func testReadRawFilePerformance() throws {
+        let file = try pathForTestData(name: "dependencies")
+        let reader = FileDependenciesReader(file, accessor: FileManager.default)
+
+        self.measure { // 0.002
+            do {
+                _ = try reader.readRaw()
+            } catch {
+                print("Error reading dependencies")
+            }
+        }
+    }
+
+    func testGetFileDataPerformance() throws {
+        let file = try pathForTestData(name: "dependencies")
+        let reader = FileDependenciesReader(file, accessor: FileManager.default)
+
+        self.measure { // 0.00008
+            do {
+                _ = try reader.getFileData()
+            } catch {
+                print("Error reading dependencies")
+            }
+        }
+    }
+
+    func testGetFileStringPerformance() throws {
+        let file = try pathForTestData(name: "dependencies")
+        let reader = FileDependenciesReader(file, accessor: FileManager.default)
+        let fileData = try reader.getFileData()
+
+        self.measure { // 0.00002
+            do {
+                _ = try reader.getFileStringFromData(fileData: fileData)
+            } catch {
+                print("Error reading dependencies")
+            }
+        }
+    }
+
+    func testGetYamlPerformance() throws { // 0.222
+        let file = try pathForTestData(name: "dependencies")
+        let reader = FileDependenciesReader(file, accessor: FileManager.default)
+        let fileData = try reader.getFileData()
+        let fileString = try reader.getFileStringFromData(fileData: fileData)
+
+        self.measure { // 0.0022
+            do {
+                _ = try reader.getYaml(fileString: fileString)
+            } catch {
+                print("Error reading dependencies")
+            }
+        }
+    }
+
+    func testParseDependencyFileListUsingUTF8View() throws {
+        let file = try pathForTestData(name: "dependencies")
+        let reader = FileDependenciesReader(file, accessor: FileManager.default)
+        let fileData = try reader.getFileData()
+        let fileString = try reader.getFileStringFromData(fileData: fileData)
+        let yaml = try reader.getYaml(fileString: fileString)
+
+        guard let dependencies = yaml["dependencies"] else {
+            XCTAssertTrue(false)
+            return
+        }
+
+        self.measure { // 0.004
+            let deps = reader.parseDependencyFileList(dependencies)
+            XCTAssertTrue(deps.count == 1000)
+        }
+    }
+
+    func testDeprecatedParseDependenciesFilesListOfAnObjectUsingUTF8View() throws {
+        let file = try pathForTestData(name: "dependencies")
+        let reader = FileDependenciesReader(file, accessor: FileManager.default)
+        let fileData = try reader.getFileData()
+        let fileString = try reader.getFileStringFromData(fileData: fileData)
+        let yaml = try reader.getYaml(fileString: fileString)
+
+        guard let dependencies = yaml["/This/Is/A/Path/To/Some/Object/objectfile.o"] else {
+            XCTAssertTrue(false)
+            return
+        }
+
+        self.measure { // 0.00048
+            let deps = reader.parseDependencyFileList(dependencies)
+            XCTAssertTrue(deps.count == 100)
+        }
+    }
+}

Tests/XCRemoteCacheTests/Dependencies/DependenciesRemapperCompositeTests.swift

@@ -29,53 +29,120 @@ class DependenciesRemapperCompositeTests: XCTestCase {
         StringDependenciesRemapper.Mapping(generic: "$(PWD)", local: "/pwd"),
     ]
 
-    func testNoRemappersIsTransparent() {
+    func testNoRemappersIsTransparent() throws {
         let remapper = DependenciesRemapperComposite([])
 
-        let genericPath = remapper.replace(localPaths: ["/tmp/root/some.swift"])
+        let genericPath = try remapper.replace(localPaths: ["/tmp/root/some.swift"])
 
         XCTAssertEqual(genericPath, ["/tmp/root/some.swift"])
     }
 
-    func testOneRemapperReplacesLocalPaths() {
+    func testOneRemapperReplacesLocalPaths() throws {
         let remapper = DependenciesRemapperComposite([
             StringDependenciesRemapper(mappings: mappings1),
         ])
 
-        let genericPath = remapper.replace(localPaths: ["/tmp/root/some.swift"])
+        let genericPath = try remapper.replace(localPaths: ["/tmp/root/some.swift"])
 
         XCTAssertEqual(genericPath, ["$(SRC_ROOT)/some.swift"])
     }
 
-    func testOneRemapperReplacesGenericPaths() {
+    func testOneRemapperReplacesGenericPaths() throws {
         let remapper = DependenciesRemapperComposite([
             StringDependenciesRemapper(mappings: mappings1),
         ])
 
-        let localPath = remapper.replace(genericPaths: ["$(SRC_ROOT)/some.swift"])
+        let localPath = try remapper.replace(genericPaths: ["$(SRC_ROOT)/some.swift"])
 
         XCTAssertEqual(localPath, ["/tmp/root/some.swift"])
     }
 
-    func testTwoRemappersReplacesLocalPaths() {
+    func testTwoRemappersReplacesLocalPaths() throws {
         let remapper = DependenciesRemapperComposite([
             StringDependenciesRemapper(mappings: mappings1),
             StringDependenciesRemapper(mappings: mappings2),
         ])
 
-        let genericPath = remapper.replace(localPaths: ["/tmp/root/some.swift", "/pwd/other.swift"])
+        let genericPath = try remapper.replace(localPaths: ["/tmp/root/some.swift", "/pwd/other.swift"])
 
         XCTAssertEqual(genericPath, ["$(SRC_ROOT)/some.swift", "$(PWD)/other.swift"])
     }
 
-    func testOneRemappersReplacesGenericPaths() {
+    func testOneRemappersReplacesGenericPaths() throws {
         let remapper = DependenciesRemapperComposite([
             StringDependenciesRemapper(mappings: mappings1),
             StringDependenciesRemapper(mappings: mappings2),
         ])
 
-        let localPath = remapper.replace(genericPaths: ["$(SRC_ROOT)/some.swift", "$(PWD)/other.swift"])
+        let localPath = try remapper.replace(genericPaths: ["$(SRC_ROOT)/some.swift", "$(PWD)/other.swift"])
 
         XCTAssertEqual(localPath, ["/tmp/root/some.swift", "/pwd/other.swift"])
     }
+
+    func testRemapsMultipleMatchingMappers() throws {
+        let remapper = DependenciesRemapperComposite([
+            StringDependenciesRemapper(mappings: [
+                StringDependenciesRemapper.Mapping(
+                    generic: "$(ROOT)",
+                    local: "/root"
+                ),
+            ]),
+            StringDependenciesRemapper(mappings: [
+                StringDependenciesRemapper.Mapping(
+                    generic: "$(SPECIFIC)",
+                    local: "$(ROOT)/specific"
+                ),
+            ]),
+        ])
+        let localPaths = ["/root/specific/file"]
+
+        let genericPaths = try remapper.replace(localPaths: localPaths)
+
+        XCTAssertEqual(genericPaths, ["$(SPECIFIC)/file"])
+    }
+
+    func testRemapsBackToLocalWithRevertedRemappersOrder() throws {
+        let remapper = DependenciesRemapperComposite([
+            StringDependenciesRemapper(mappings: [
+                StringDependenciesRemapper.Mapping(
+                    generic: "$(ROOT)",
+                    local: "/root"
+                ),
+            ]),
+            StringDependenciesRemapper(mappings: [
+                StringDependenciesRemapper.Mapping(
+                    generic: "$(SPECIFIC)",
+                    local: "$(ROOT)/specific"
+                ),
+            ]),
+        ])
+        let genericPaths = ["$(SPECIFIC)/file"]
+
+        let localPaths = try remapper.replace(genericPaths: genericPaths)
+
+        XCTAssertEqual(localPaths, ["/root/specific/file"])
+    }
+
+    func testRemappingTwoMappingsBackAndForthIsIdentical() throws {
+        let remapper = DependenciesRemapperComposite([
+            StringDependenciesRemapper(mappings: [
+                StringDependenciesRemapper.Mapping(
+                    generic: "$(ROOT)",
+                    local: "/root"
+                ),
+            ]),
+            StringDependenciesRemapper(mappings: [
+                StringDependenciesRemapper.Mapping(
+                    generic: "$(SPECIFIC)",
+                    local: "$(ROOT)/specific"
+                ),
+            ]),
+        ])
+        let localPaths = ["/root/specific/file"]
+
+        let genericPaths = try remapper.replace(localPaths: localPaths)
+        let remappedLocalPaths = try remapper.replace(genericPaths: genericPaths)
+
+        XCTAssertEqual(localPaths, remappedLocalPaths)
+    }
 }

Tests/XCRemoteCacheTests/Dependencies/DependenciesWriterTests.swift

@@ -0,0 +1,47 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+@testable import XCRemoteCache
+import XCTest
+
+class FileDependenciesWriterTests: XCTestCase {
+
+    private func generateTempFileURL(name: String = #function) throws -> URL {
+        let directory = NSTemporaryDirectory()
+        return try NSURL.fileURL(withPathComponents: [directory, name]).unwrap()
+    }
+
+    func testWriteDependencyWithSpace() throws {
+        let url = try generateTempFileURL()
+        let writer = FileDependenciesWriter(url, accessor: FileManager.default)
+
+        try writer.writeGeneric(dependencies: [
+            "/SomePath/Pods/Target Support Files/lottie-ios/lottie-ios-dummy.m",
+        ])
+
+        let expectedContent = """
+        dependencies: /SomePath/Pods/Target\\ Support\\ Files/lottie-ios/lottie-ios-dummy.m
+
+        """
+
+        let content = String(data: try Data(contentsOf: url), encoding: .utf8)
+
+        XCTAssertEqual(content, expectedContent)
+    }
+}

Tests/XCRemoteCacheTests/Dependencies/DependencyProcessorImplTests.swift

@@ -20,24 +20,28 @@
 @testable import XCRemoteCache
 import XCTest
 
-class DependencyProcessorImplTests: XCTestCase {
+class DependencyProcessorImplTests: FileXCTestCase {
 
     let processor = DependencyProcessorImpl(
         xcode: "/Xcode",
         product: "/Product",
         source: "/Source",
         intermediate: "/Intermediate",
-        bundle: "/Bundle"
+        derivedFiles: "/DerivedFiles",
+        bundle: "/Bundle",
+        skippedRegexes: []
     )
 
-    func testIntermediateFileIsSkippedForProductAndSourceSubdirectory() {
+    func testIntermediateFileIsskippedRegexesForProductAndSourceSubdirectory() {
         let intermediateFile: URL = "/Intermediate/some"
         let processor = DependencyProcessorImpl(
             xcode: "/Xcode",
             product: "/",
             source: "/",
             intermediate: "/Intermediate",
-            bundle: nil
+            derivedFiles: "/DerivedFiles",
+            bundle: nil,
+            skippedRegexes: []
         )
 
         XCTAssertEqual(
@@ -46,14 +50,16 @@ class DependencyProcessorImplTests: XCTestCase {
         )
     }
 
-    func testBundleFileIsSkippedForProductAndSourceSubdirectory() {
+    func testBundleFileIsskippedRegexesForProductAndSourceSubdirectory() {
         let bundleFile: URL = "/Bundle/some"
         let processor = DependencyProcessorImpl(
             xcode: "/Xcode",
             product: "/",
             source: "/",
             intermediate: "/Intermediate",
-            bundle: "/Bundle"
+            derivedFiles: "/DerivedFiles",
+            bundle: "/Bundle",
+            skippedRegexes: []
         )
 
         XCTAssertEqual(
@@ -62,6 +68,22 @@ class DependencyProcessorImplTests: XCTestCase {
         )
     }
 
+    func testFiltersOutGeneratedSwiftHeaders() throws {
+        let dependencies = processor.process([
+            "/DerivedFiles/ModuleName-Swift.h",
+        ])
+
+        XCTAssertEqual(dependencies, [])
+    }
+
+    func testFiltersOutDerivedFile() throws {
+        let dependencies = processor.process([
+            "/DerivedFiles/output.h",
+        ])
+
+        XCTAssertEqual(dependencies, [])
+    }
+
     func testFiltersOutProductModulemap() throws {
         let dependencies = processor.process([
             "/Product/some.modulemap",
@@ -70,6 +92,14 @@ class DependencyProcessorImplTests: XCTestCase {
         XCTAssertEqual(dependencies, [])
     }
 
+    func testDoesNotFilterOutOtherProductModulemap() throws {
+        let dependencies = processor.process([
+            "/ProductOther/some.modulemap",
+        ])
+
+        XCTAssertEqual(dependencies, [.init(url: "/ProductOther/some.modulemap", type: .unknown)])
+    }
+
     func testDoesNotFilterOutNonProductModulemap() throws {
         let dependencies = processor.process([
             "/Source/some.modulemap",
@@ -109,4 +139,152 @@ class DependencyProcessorImplTests: XCTestCase {
 
         XCTAssertEqual(dependencies, [.init(url: "/xxx/some", type: .unknown)])
     }
+
+    func testFiltersOutIntermediateBySymlink() throws {
+        let sampleDir = try prepareTempDir()
+
+        let intermediateDirReal = sampleDir.appendingPathComponent("Intermediate")
+        let symlink = sampleDir.appendingPathComponent("symlink")
+        let someFilename = "some"
+
+        let processor = DependencyProcessorImpl(
+            xcode: "/Xcode",
+            product: "/Product",
+            source: "/Source",
+            intermediate: intermediateDirReal,
+            derivedFiles: "/DerivedFiles",
+            bundle: "/Bundle",
+            skippedRegexes: []
+        )
+
+        let intermediateFileSymlink = createSymlink(
+            filename: someFilename,
+            sourceDir: symlink,
+            destinationDir: intermediateDirReal
+        )
+
+        let dependencies = processor.process([
+            intermediateFileSymlink,
+        ])
+
+        XCTAssertEqual(dependencies, [])
+    }
+
+    func testDoesNotFilterOutSourceBySymlink() throws {
+        let sampleDir = try prepareTempDir()
+
+        let sourceDirReal = sampleDir.appendingPathComponent("Source")
+        let symlink = sampleDir.appendingPathComponent("symlink")
+        let someFilename = "some"
+
+        let processor = DependencyProcessorImpl(
+            xcode: "/Xcode",
+            product: "/Product",
+            source: sourceDirReal,
+            intermediate: "/Intermediate",
+            derivedFiles: "/DerivedFiles",
+            bundle: "/Bundle",
+            skippedRegexes: []
+        )
+
+        let sourceFileSymlink = createSymlink(
+            filename: someFilename,
+            sourceDir: symlink,
+            destinationDir: sourceDirReal
+        )
+
+        let dependencies = processor.process([
+            sourceFileSymlink,
+        ])
+
+        XCTAssertEqual(dependencies, [.init(url: sourceFileSymlink, type: .source)])
+    }
+
+    /**
+     * Creates Symlink from sourceDir to destinationDir and creates empty file inside it
+     * return URL with symbolic link from sourceDir to destinationDir
+     */
+    fileprivate func createSymlink(filename: String, sourceDir: URL, destinationDir: URL) -> URL {
+        let fileMng = FileManager.default
+
+        XCTAssertNoThrow(try fileMng.spt_forceSymbolicLink(
+            at: sourceDir,
+            withDestinationURL: destinationDir
+        ))
+        XCTAssertNoThrow(try fileMng.spt_createEmptyFile(destinationDir.appendingPathComponent(filename)))
+
+        return sourceDir.appendingPathComponent(filename)
+    }
+
+    func testSkipsCustomizedDerivedDirFileUnderSources() {
+        let derivedFile: URL = "/DerivedFiles/Module-Swift.h"
+        let processor = DependencyProcessorImpl(
+            xcode: "/Xcode",
+            product: "/",
+            source: "/",
+            intermediate: "/Intermediate",
+            derivedFiles: "/DerivedFiles",
+            bundle: nil,
+            skippedRegexes: []
+        )
+
+        XCTAssertEqual(
+            processor.process([derivedFile]),
+            []
+        )
+    }
+
+    func testSkippsFilesWithFullMatch() {
+        let source: URL = "/someFile.m"
+        let processor = DependencyProcessorImpl(
+            xcode: "/Xcode",
+            product: "/",
+            source: "/",
+            intermediate: "/Intermediate",
+            derivedFiles: "/DerivedFiles",
+            bundle: nil,
+            skippedRegexes: ["/someFile\\.m"]
+        )
+
+        XCTAssertEqual(
+            processor.process([source]),
+            []
+        )
+    }
+
+    func testSkippsFilesWithPartialMatch() {
+        let derivedModulemap: URL = "/module.modulemap"
+        let processor = DependencyProcessorImpl(
+            xcode: "/Xcode",
+            product: "/product",
+            source: "/",
+            intermediate: "/Intermediate",
+            derivedFiles: "/DerivedFiles",
+            bundle: nil,
+            skippedRegexes: ["\\.modulemap$"]
+        )
+
+        XCTAssertEqual(
+            processor.process([derivedModulemap]),
+            []
+        )
+    }
+
+    func testDoesntSkipFileIfInvalidRegex() {
+        let source: URL = "/someFile.m"
+        let processor = DependencyProcessorImpl(
+            xcode: "/Xcode",
+            product: "/product",
+            source: "/",
+            intermediate: "/Intermediate",
+            derivedFiles: "/DerivedFiles",
+            bundle: nil,
+            skippedRegexes: ["\\"]
+        )
+
+        XCTAssertEqual(
+            processor.process([source]),
+            [.init(url: source, type: .source)]
+        )
+    }
 }

Tests/XCRemoteCacheTests/Dependencies/OverlayDependenciesRemapperTests.swift

@@ -0,0 +1,70 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+@testable import XCRemoteCache
+import XCTest
+
+class OverlayDependenciesRemapperTests: XCTestCase {
+    private let overlayReader = OverlayReaderFake(
+        mappings: [.init(virtual: "/file.h", local: "/Intermediate/Some/file.h")]
+    )
+
+    func testMappingFromLocalToGeneric() throws {
+        let remapper = OverlayDependenciesRemapper(overlayReader: overlayReader)
+
+        let dependencies = try remapper.replace(localPaths: ["/Intermediate/Some/file.h"])
+        XCTAssertEqual(dependencies, ["/file.h"])
+    }
+
+    func testMappingFromGenericToLocal() throws {
+        let remapper = OverlayDependenciesRemapper(overlayReader: overlayReader)
+
+
+        let dependencies = try remapper.replace(genericPaths: ["/file.h"])
+        XCTAssertEqual(dependencies, ["/Intermediate/Some/file.h"])
+    }
+
+    func testGenericDependenciesAreNotMerged() throws {
+        let remapper = OverlayDependenciesRemapper(overlayReader: overlayReader)
+
+
+        let dependencies = try remapper.replace(localPaths: ["/Intermediate/Some/file.h", "/file.h"])
+        XCTAssertEqual(dependencies, ["/file.h", "/file.h"])
+    }
+
+    func testLocalDependenciesAreNotMerged() throws {
+        let remapper = OverlayDependenciesRemapper(overlayReader: overlayReader)
+
+
+        let dependencies = try remapper.replace(genericPaths: ["/Intermediate/Some/file.h", "/file.h"])
+        XCTAssertEqual(dependencies, ["/Intermediate/Some/file.h", "/Intermediate/Some/file.h"])
+    }
+
+    func testMappingsAreReadOnce() throws {
+        let remapper = OverlayDependenciesRemapper(overlayReader: overlayReader)
+
+        let firstDependencies = try remapper.replace(localPaths: ["/Intermediate/Some/file.h"])
+        // Update mappings in-fly to verify the previous value is cached in a remapper
+        overlayReader.mappings = []
+        let secondDependencies = try remapper.replace(localPaths: ["/Intermediate/Some/file.h"])
+
+        XCTAssertEqual(firstDependencies, ["/file.h"])
+        XCTAssertEqual(secondDependencies, ["/file.h"])
+    }
+}

Tests/XCRemoteCacheTests/Dependencies/OverlayReaderTests.swift

@@ -0,0 +1,94 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+@testable import XCRemoteCache
+import XCTest
+
+class JsonOverlayReaderTests: FileXCTestCase {
+    private static let resourcesSubdirectory = "TestData/Dependencies/JsonOverlayReaderTests"
+
+    func testParsingWithSuccess() throws {
+        let file = try pathForTestData(name: "overlayReaderDefault")
+        let reader = JsonOverlayReader(file, mode: .strict, fileReader: FileManager.default)
+        let mappings = try reader.provideMappings()
+
+        let expectedMappings = [
+            OverlayMapping(
+                virtual: "/DerivedDataProducts/Target1.framework/Headers/Target1.h",
+                local: "/Path/Target1/Target1.h"
+            ),
+            OverlayMapping(
+                virtual: "/DerivedDataProducts/Target2.framework/Modules/module.modulemap",
+                local: "/DerivedDataIntermediate/Target2.build/module.modulemap"
+            ),
+        ]
+        XCTAssertEqual(Set(mappings), Set(expectedMappings))
+    }
+
+    func testFailsWithMissingFileForStrictMode() throws {
+        let file: URL = "nonExiting"
+        let reader = JsonOverlayReader(file, mode: .strict, fileReader: FileManager.default)
+
+        XCTAssertThrowsError(try reader.provideMappings())
+    }
+
+    func testReturnsEmpptyMappingForMissingFileForBestEffortMode() throws {
+        let file: URL = "nonExiting"
+        let reader = JsonOverlayReader(file, mode: .bestEffort, fileReader: FileManager.default)
+
+        let mappings = try reader.provideMappings()
+
+        XCTAssertEqual(mappings, [])
+    }
+
+    func testParsingEmptyOverlay() throws {
+        let file = try pathForTestData(name: "overlayReaderEmpty")
+        let reader = JsonOverlayReader(file, mode: .strict, fileReader: FileManager.default)
+        let mappings = try reader.provideMappings()
+
+        XCTAssertEqual(mappings, [])
+    }
+
+    func testInvalidJsonDoesntThrowForBestEffortMode() throws {
+        let workingDir = try prepareTempDir()
+        let file = workingDir.appendingPathExtension("overlay.json")
+        try fileManager.spt_createEmptyFile(file)
+        let reader = JsonOverlayReader(file, mode: .bestEffort, fileReader: fileManager)
+        let mappings = try reader.provideMappings()
+
+        XCTAssertEqual(mappings, [])
+    }
+
+    func testInvalidJsonThrowsForStrictMode() throws {
+        let workingDir = try prepareTempDir()
+        let file = workingDir.appendingPathExtension("overlay.json")
+        try fileManager.spt_createEmptyFile(file)
+        let reader = JsonOverlayReader(file, mode: .strict, fileReader: fileManager)
+
+        XCTAssertThrowsError(try reader.provideMappings())
+    }
+
+    private func pathForTestData(name: String) throws -> URL {
+        return try XCTUnwrap(Bundle.module.url(
+            forResource: name,
+            withExtension: "json",
+            subdirectory: JsonOverlayReaderTests.resourcesSubdirectory
+        ))
+    }
+}

Tests/XCRemoteCacheTests/Dependencies/PathDependenciesRemapperFactoryTests.swift

@@ -0,0 +1,94 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+@testable import XCRemoteCache
+import XCTest
+
+class PathDependenciesRemapperFactoryTests: XCTestCase {
+    private var factory: PathDependenciesRemapperFactory!
+
+    override func setUp() {
+        super.setUp()
+        factory = PathDependenciesRemapperFactory()
+    }
+
+    func testMappingsFromEnvMaps() throws {
+        let remapper = try factory.build(
+            orderKeys: ["SRC_ROOT"],
+            envs: ["SRC_ROOT": "/tmp/root"],
+            customMappings: [:]
+        )
+
+        let localPaths = try remapper.replace(genericPaths: ["$(SRC_ROOT)/some.swift"])
+        XCTAssertEqual(localPaths, ["/tmp/root/some.swift"])
+    }
+
+    func testMappingsGenericWhenMappingHasParentDir() throws {
+        let remapper = try factory.build(
+            orderKeys: ["SRC_ROOT"],
+            envs: ["SRC_ROOT": "/tmp/root/extra/.."],
+            customMappings: [:]
+        )
+
+        let localPaths = try remapper.replace(genericPaths: ["$(SRC_ROOT)/some.swift"])
+        XCTAssertEqual(localPaths, ["/tmp/root/some.swift"])
+    }
+
+    func testMappingsLocalWhenMappingHasParentDir() throws {
+        let remapper = try factory.build(
+            orderKeys: ["SRC_ROOT"],
+            envs: ["SRC_ROOT": "/tmp/root/excessive/.."],
+            customMappings: [:]
+        )
+
+        let localPaths = try remapper.replace(localPaths: ["/tmp/root/some.swift"])
+        XCTAssertEqual(localPaths, ["$(SRC_ROOT)/some.swift"])
+    }
+
+    func testMissingEnvIsSkipped() throws {
+        XCTAssertNoThrow(
+            try factory.build(
+                orderKeys: ["SRC_ROOT"],
+                envs: ["NO_SRC_ROOT": ""],
+                customMappings: [:]
+            )
+        )
+    }
+
+    func testBuildingRemapperWithMergedCustomMappings() throws {
+        let remapper = try factory.build(
+            orderKeys: ["PWD"],
+            envs: ["PWD": "/some"],
+            customMappings: ["TMP": "/tmp"]
+        )
+
+        let genericPaths = try remapper.replace(localPaths: ["/some/repoFile.swift", "/tmp/externalFile.swift"])
+        XCTAssertEqual(genericPaths, ["$(PWD)/repoFile.swift", "$(TMP)/externalFile.swift"])
+    }
+
+    func testFailsBuildingRemapperWithConflictedMappings() throws {
+        XCTAssertThrowsError(
+            try factory.build(
+            orderKeys: ["PWD"],
+            envs: ["PWD": "/some"],
+            customMappings: ["PWD": "/other"]
+            )
+        )
+    }
+}

Tests/XCRemoteCacheTests/Dependencies/StringDependenciesRemapperTests.swift

@@ -30,34 +30,34 @@ class StringDependenciesRemapperTests: XCTestCase {
         remapper = StringDependenciesRemapper(mappings: mappings)
     }
 
-    func testMappingSingleGenericPathReplacesWithLocalPath() {
-        let localPaths = remapper.replace(genericPaths: ["$(SRC_ROOT)/some.swift"])
+    func testMappingSingleGenericPathReplacesWithLocalPath() throws {
+        let localPaths = try remapper.replace(genericPaths: ["$(SRC_ROOT)/some.swift"])
 
         XCTAssertEqual(localPaths, ["/tmp/root/some.swift"])
     }
 
-    func testRewritingSingleLocalPathReplacesWithGenericPath() {
-        let genericPaths = remapper.replace(localPaths: ["/tmp/root/some.swift"])
+    func testRewritingSingleLocalPathReplacesWithGenericPath() throws {
+        let genericPaths = try remapper.replace(localPaths: ["/tmp/root/some.swift"])
 
         XCTAssertEqual(genericPaths, ["$(SRC_ROOT)/some.swift"])
     }
 
-    func testRewritingLocalToGenericAndLocalIsIdentical() {
+    func testRewritingLocalToGenericAndLocalIsIdentical() throws {
         let inputLocalPaths = ["/tmp/root/some.swift"]
 
-        let genericPaths = remapper.replace(localPaths: inputLocalPaths)
-        let localPaths = remapper.replace(genericPaths: genericPaths)
+        let genericPaths = try remapper.replace(localPaths: inputLocalPaths)
+        let localPaths = try remapper.replace(genericPaths: genericPaths)
 
         XCTAssertEqual(localPaths, inputLocalPaths)
     }
 
-    func testRewritingUnrelatedDirReturnsInputPath() {
-        let genericPaths = remapper.replace(localPaths: ["/other/some.swift"])
+    func testRewritingUnrelatedDirReturnsInputPath() throws {
+        let genericPaths = try remapper.replace(localPaths: ["/other/some.swift"])
 
         XCTAssertEqual(genericPaths, ["/other/some.swift"])
     }
 
-    func testMultipleMatchesTakeTheFirstMapping() {
+    func testMultipleMatchesTakeTheFirstMapping() throws {
         let mappings: [StringDependenciesRemapper.Mapping] = [
             .init(generic: "$(SRC_ROOT)", local: "/tmp/root"),
             .init(generic: "$(PWD)", local: "/tmp"),
@@ -65,22 +65,34 @@ class StringDependenciesRemapperTests: XCTestCase {
         remapper = StringDependenciesRemapper(mappings: mappings)
 
 
-        let genericPaths = remapper.replace(localPaths: ["/tmp/root/some.swift", "/tmp/extra.swift"])
+        let genericPaths = try remapper.replace(localPaths: ["/tmp/root/some.swift", "/tmp/extra.swift"])
 
         XCTAssertEqual(genericPaths, ["$(SRC_ROOT)/some.swift", "$(PWD)/extra.swift"])
     }
 
-    func testMappingsFromEnvMaps() throws {
-        remapper = try StringDependenciesRemapper.buildFromEnvs(keys: ["SRC_ROOT"], envs: ["SRC_ROOT": "/tmp/root"])
+    func testMappingsLocalPathsIsDoneInOrder() throws {
+        let mappings: [StringDependenciesRemapper.Mapping] = [
+            .init(generic: "$(TMP)", local: "/tmp"),
+            .init(generic: "$(ROOT)", local: "$(TMP)/root"),
+        ]
+        remapper = StringDependenciesRemapper(mappings: mappings)
 
-        let localPaths = remapper.replace(genericPaths: ["$(SRC_ROOT)/some.swift"])
+
+        let genericPaths = try remapper.replace(localPaths: ["/tmp/root/some.swift"])
+
+        XCTAssertEqual(genericPaths, ["$(ROOT)/some.swift"])
+    }
+
+    func testMappingsGenericPathsIsDoneInReversedOrder() throws {
+        let mappings: [StringDependenciesRemapper.Mapping] = [
+            .init(generic: "$(TMP)", local: "/tmp"),
+            .init(generic: "$(ROOT)", local: "$(TMP)/root"),
+        ]
+        remapper = StringDependenciesRemapper(mappings: mappings)
+
+
+        let localPaths = try remapper.replace(genericPaths: ["$(ROOT)/some.swift"])
 
         XCTAssertEqual(localPaths, ["/tmp/root/some.swift"])
     }
-
-    func testInvalidMappingsFromEnvFAils() throws {
-        XCTAssertThrowsError(
-            try StringDependenciesRemapper.buildFromEnvs(keys: ["SRC_ROOT"], envs: ["NO_SRC_ROOT": ""])
-        )
-    }
 }

Tests/XCRemoteCacheTests/Fingerprint/EnvironmentFingerprintGeneratorTests.swift

@@ -24,7 +24,7 @@ class EnvironmentFingerprintGeneratorTests: XCTestCase {
 
     private static let defaultENV = [
         "GCC_PREPROCESSOR_DEFINITIONS": "GCC",
-        "CLANG_PROFILE_DATA_DIRECTORY": "CLANG",
+        "CLANG_COVERAGE_MAPPING": "YES",
         "TARGET_NAME": "TARGET",
         "CONFIGURATION": "CONG",
         "PLATFORM_NAME": "PLAT",
@@ -33,18 +33,23 @@ class EnvironmentFingerprintGeneratorTests: XCTestCase {
         "DYLIB_COMPATIBILITY_VERSION": "2",
         "DYLIB_CURRENT_VERSION": "3",
         "PRODUCT_MODULE_NAME": "4",
+        "ARCHS": "AR",
     ]
     /// Corresponds to EnvironmentFingerprintGenerator.version
     private static let currentVersion = "5"
 
     private var config: XCRemoteCacheConfig!
-    private var generator: FingerprintAccumulator!
+    private var generator: FingerprintAccumulator! {
+        return generatorFake
+    }
+
+    private var generatorFake: FingerprintAccumulatorFake!
     private var fingerprintGenerator: EnvironmentFingerprintGenerator!
 
     override func setUp() {
         super.setUp()
         config = XCRemoteCacheConfig(sourceRoot: "")
-        generator = FingerprintAccumulatorFake()
+        generatorFake = FingerprintAccumulatorFake()
         fingerprintGenerator = EnvironmentFingerprintGenerator(
             configuration: config,
             env: Self.defaultENV,
@@ -55,7 +60,7 @@ class EnvironmentFingerprintGeneratorTests: XCTestCase {
     func testConsidersDefaultEnvs() throws {
         let fingerprint = try fingerprintGenerator.generateFingerprint()
 
-        XCTAssertEqual(fingerprint, "GCC,CLANG,TARGET,CONG,PLAT,XC,1,2,3,4,\(Self.currentVersion)")
+        XCTAssertEqual(fingerprint, "GCC,YES,TARGET,CONG,PLAT,XC,1,2,3,4,AR,\(Self.currentVersion)")
     }
 
     func testFingerprintIncludesVersionAsLastComponent() throws {
@@ -73,7 +78,7 @@ class EnvironmentFingerprintGeneratorTests: XCTestCase {
 
         let fingerprint = try fingerprintGenerator.generateFingerprint()
 
-        XCTAssertEqual(fingerprint, ",,,,,,,,,,\(Self.currentVersion)")
+        XCTAssertEqual(fingerprint, ",,,,,,,,,,,\(Self.currentVersion)")
     }
 
     func testConsidersCustomEnvs() throws {
@@ -89,6 +94,13 @@ class EnvironmentFingerprintGeneratorTests: XCTestCase {
 
         let fingerprint = try fingerprintGenerator.generateFingerprint()
 
-        XCTAssertEqual(fingerprint, "GCC,CLANG,TARGET,CONG,PLAT,XC,1,2,3,4,CUSTOM_VALUE,\(Self.currentVersion)")
+        XCTAssertEqual(fingerprint, "GCC,YES,TARGET,CONG,PLAT,XC,1,2,3,4,AR,CUSTOM_VALUE,\(Self.currentVersion)")
+    }
+
+    func testFingerprintIsGeneratedOnce() throws {
+        let fingerprint1 = try fingerprintGenerator.generateFingerprint()
+        let fingerprint2 = try fingerprintGenerator.generateFingerprint()
+        XCTAssertEqual(fingerprint1, fingerprint2)
+        XCTAssertEqual(generatorFake.generateCallsCount, 1)
     }
 }

Tests/XCRemoteCacheTests/Models/JsonMetaWriterTests.swift

@@ -24,7 +24,7 @@ class JsonMetaWriterTests: XCTestCase {
 
     func testWritesToFileWithFilekeyFilename() throws {
         let fileAccessor = FileAccessorFake(mode: .normal)
-        let writer = JsonMetaWriter(fileWriter: fileAccessor)
+        let writer = JsonMetaWriter(fileWriter: fileAccessor, pretty: false)
         let workingDir: URL = "/"
         let meta = MainArtifactSampleMeta.defaults
 
@@ -35,7 +35,20 @@ class JsonMetaWriterTests: XCTestCase {
 
     func testWritesMetaInValidFormat() throws {
         let fileAccessor = FileAccessorFake(mode: .normal)
-        let writer = JsonMetaWriter(fileWriter: fileAccessor)
+        let writer = JsonMetaWriter(fileWriter: fileAccessor, pretty: false)
+        let reader = JsonMetaReader(fileAccessor: fileAccessor)
+        let workingDir: URL = "/"
+        let meta = MainArtifactSampleMeta.defaults
+
+        let url = try writer.write(MainArtifactSampleMeta.defaults, locationDir: workingDir)
+
+        let readMeta = try reader.read(localFile: url)
+        XCTAssertEqual(readMeta, meta)
+    }
+
+    func testWritesPrettyMetaInValidFormat() throws {
+        let fileAccessor = FileAccessorFake(mode: .normal)
+        let writer = JsonMetaWriter(fileWriter: fileAccessor, pretty: true)
         let reader = JsonMetaReader(fileAccessor: fileAccessor)
         let workingDir: URL = "/"
         let meta = MainArtifactSampleMeta.defaults

Tests/XCRemoteCacheTests/Network/Authentication/AWSV4SignatureTest.swift

@@ -35,10 +35,12 @@ class AWSV4SignatureTest: XCTestCase {
 
         let key = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"
         let accessKey = "AKIDEXAMPLE"
+        let securityToken = "IQoJb3JpZ2luX2VjENv//////////wEaCXVzLWVhc3Q+bsHwqnovXtl/1JVe61XHMnAw3AIXwOAOxqMvhw=="
 
         AWSV4Signature(
             secretKey: key,
             accessKey: accessKey,
+            securityToken: securityToken,
             region: "us-east-1",
             service: "iam",
             date: Date(timeIntervalSince1970: 1_440_938_160)
@@ -49,8 +51,8 @@ class AWSV4SignatureTest: XCTestCase {
     func testAuthHeaderContainsCorrectSignature() throws {
         XCTAssertEqual(
             "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20150830/us-east-1/iam/aws4_request, " +
-                "SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, " +
-                "Signature=dd479fa8a80364edf2119ec24bebde66712ee9c9cb2b0d92eb3ab9ccdc0c3947",
+                "SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, " +
+                "Signature=d26ab974bba1b248f041ea1120064e1fa672d6f06cac2cff42b38acea87b76e5",
             request.allHTTPHeaderFields?["Authorization"]
         )
     }

Tests/XCRemoteCacheTests/Network/NetworkClientImplTests.swift

@@ -203,6 +203,7 @@ class NetworkClientImplTests: XCTestCase {
         let signature = AWSV4Signature(
             secretKey: "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY",
             accessKey: "AKIDEXAMPLE",
+            securityToken: "IQoJb3JpZ2luX2VjENv//////////wEaCXVzLWVhc3Q+bsHwqnovXtl/1JVe61XHMnAw3AIXwOAOxqMvhw==",
             region: "us-east-1",
             service: "iam",
             date: Date(timeIntervalSince1970: 1_440_938_160)
@@ -213,8 +214,8 @@ class NetworkClientImplTests: XCTestCase {
 
         XCTAssertEqual(
             "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20150830/us-east-1/iam/aws4_request, " +
-                "SignedHeaders=host;x-amz-content-sha256;x-amz-date, " +
-                "Signature=acdb223475463b6ce711b063f199387a7a12bd638e4dccaaeb5efcbf159b6454",
+                "SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, " +
+                "Signature=e5578464567fb97fd26e871702e4ec4ff7d61cb87eb72a40d22b80e12da30c34",
             try requests[0].allHTTPHeaderFields?["Authorization"].unwrap()
         )
     }

Tests/XCRemoteCacheTests/TestData/Dependencies/JsonOverlayReaderTests/overlayReaderDefault.json

@@ -0,0 +1,31 @@
+{
+  "case-sensitive": "false",
+  "roots":
+    [
+      {
+        "contents":
+          [
+            {
+              "external-contents": "/Path/Target1/Target1.h",
+              "name": "Target1.h",
+              "type": "file",
+            },
+          ],
+        "name": "/DerivedDataProducts/Target1.framework/Headers",
+        "type": "directory",
+      },
+      {
+        "contents":
+          [
+            {
+              "external-contents": "/DerivedDataIntermediate/Target2.build/module.modulemap",
+              "name": "module.modulemap",
+              "type": "file",
+            },
+          ],
+        "name": "/DerivedDataProducts/Target2.framework/Modules",
+        "type": "directory",
+      },
+    ],
+  "version": 0,
+}

Tests/XCRemoteCacheTests/TestData/Dependencies/JsonOverlayReaderTests/overlayReaderEmpty.json

@@ -0,0 +1 @@
+{"case-sensitive":"false","roots":[],"version":0}

Tests/XCRemoteCacheTests/TestDoubles/FingerprintAccumulatorFake.swift

@@ -34,5 +34,11 @@ class FingerprintAccumulatorFake: FingerprintAccumulator {
         appendedStrings.append("FILE{\(file.path)}")
     }
 
-    func generate() throws -> RawFingerprint { return appendedStrings.joined(separator: ",") }
+    private(set) var generateCallsCount = 0
+    func generate() throws -> RawFingerprint {
+        defer {
+            generateCallsCount += 1
+        }
+        return appendedStrings.joined(separator: ",")
+    }
 }

Tests/XCRemoteCacheTests/TestDoubles/OverlayReaderFake.swift

@@ -0,0 +1,33 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import Foundation
+@testable import XCRemoteCache
+
+class OverlayReaderFake: OverlayReader {
+    var mappings: [OverlayMapping]
+
+    init(mappings: [OverlayMapping]) {
+        self.mappings = mappings
+    }
+
+    func provideMappings() throws -> [OverlayMapping] {
+        return mappings
+    }
+}

cocoapods-plugin/README.md

@@ -52,6 +52,9 @@ An object that is passed to the `xcremotecache` can contain all properties suppo
 | `modify_lldb_init` | Controls if the pod integration should modify `~/.lldbinit` | `true` | ⬜️ |
 | `xccc_file` | The path where should be placed the `xccc` binary (in the pod installation phase) | `{podfile_dir}/.rc/xccc` | ⬜️ |
 | `remote_commit_file` | The path of the file with the remote commit sha (in the pod installation phase) | `{podfile_dir}/.rc/arc.rc`| ⬜️ |
+| `prettify_meta_files` | A Boolean value that opts-in pretty JSON formatting for meta files | `false` | ⬜️ |
+| `fake_src_root` | An arbitrary source location shared between producers and consumers. Should be unique for a project. | `/xxxxxxxxxx` | ⬜️ |
+| `disable_certificate_verification` | A Boolean value that opts-in SSL certificate validation is disabled | `false` | ⬜️ |
 
 ## Uninstalling
 
@@ -60,7 +63,3 @@ To fully uninstall the plugin, call:
 ```bash
 gem uninstall cocoapods-xcremotecache
 ```
-
-## Limitations
-
-* When `generate_multiple_pod_projects` mode is enabled, only first-party targets are cached by XCRemoteCache (all dependencies are compiled locally).

cocoapods-plugin/lib/cocoapods-xcremotecache/command/hooks.rb

@@ -17,17 +17,19 @@ require 'cocoapods/resolver'
 require 'open-uri'
 require 'yaml'
 require 'json'
+require 'pathname'
 
 
 module CocoapodsXCRemoteCacheModifier
   # Registers for CocoaPods plugin hooks
   module Hooks
     BIN_DIR = '.rc'
-    FAKE_SRCROOT = "/#{'x' * 10 }"
     LLDB_INIT_COMMENT="#RemoteCacheCustomSourceMap"
     LLDB_INIT_PATH = "#{ENV['HOME']}/.lldbinit"
     FAT_ARCHIVE_NAME_INFIX = 'arm64-x86_64'
+    XCRC_COOCAPODS_ROOT_KEY = 'XCRC_COOCAPODS_ROOT'
 
+    # List of plugins' user properties that should not be copied to .rcinfo 
     CUSTOM_CONFIGURATION_KEYS = [
       'enabled', 
       'xcrc_location',
@@ -36,7 +38,8 @@ module CocoapodsXCRemoteCacheModifier
       'final_target',
       'check_build_configuration', 
       'check_platform', 
-      'modify_lldb_init'
+      'modify_lldb_init',
+      'fake_src_root',
     ]
 
     class XCRemoteCache
@@ -58,8 +61,16 @@ module CocoapodsXCRemoteCacheModifier
           'xccc_file' => "#{BIN_DIR}/xccc",
           'remote_commit_file' => "#{BIN_DIR}/arc.rc",
           'exclude_targets' => [],
+          'prettify_meta_files' => false,
+          'fake_src_root' => "/#{'x' * 10 }",
+          'disable_certificate_verification' => false,
+          'custom_rewrite_envs' => []
         }
         @@configuration.merge! default_values.select { |k, v| !@@configuration.key?(k) }
+        # Always include XCRC_COOCAPODS_ROOT_KEY in custom_rewrite_envs
+        unless @@configuration['custom_rewrite_envs'].include?(XCRC_COOCAPODS_ROOT_KEY)
+          @@configuration['custom_rewrite_envs'] << XCRC_COOCAPODS_ROOT_KEY
+        end
       end
 
       def self.validate_configuration()
@@ -76,8 +87,8 @@ module CocoapodsXCRemoteCacheModifier
         end
 
         mode = @@configuration['mode']
-        unless mode == 'consumer' || mode == 'producer'
-          throw "Incorrect 'mode' value. Allowed values are ['consumer', 'producer'], but you provided '#{mode}'. A typo?"
+        unless mode == 'consumer' || mode == 'producer' || mode == 'producer-fast'
+          throw "Incorrect 'mode' value. Allowed values are ['consumer', 'producer', 'producer-fast'], but you provided '#{mode}'. A typo?"
         end
 
         unless mode == 'consumer' || @@configuration.key?('final_target')
@@ -97,23 +108,31 @@ module CocoapodsXCRemoteCacheModifier
       # @param repo_distance [Integer] distance from the git repo root to the target's $SRCROOT 
       # @param xc_location [String] path to the dir with all XCRemoteCache binaries, relative to the repo root
       # @param xc_cc_path [String] path to the XCRemoteCache clang wrapper, relative to the repo root
-      # @param mode [String] mode name ('consumer', 'producer' etc.)
+      # @param mode [String] mode name ('consumer', 'producer', 'producer-fast' etc.)
       # @param exclude_build_configurations [String[]] list of targets that should have disabled remote cache
       # @param final_target [String] name of target that should trigger marking
-      def self.enable_xcremotecache(target, repo_distance, xc_location, xc_cc_path, mode, exclude_build_configurations, final_target)
+      def self.enable_xcremotecache(target, repo_distance, xc_location, xc_cc_path, mode, exclude_build_configurations, final_target, fake_src_root)
         srcroot_relative_xc_location = parent_dir(xc_location, repo_distance)
+        # location of the entrite CocoaPods project, relative to SRCROOT
+        srcroot_relative_project_location = parent_dir('', repo_distance)
 
         target.build_configurations.each do |config|
           # apply only for relevant Configurations
           next if exclude_build_configurations.include?(config.name)
           if mode == 'consumer'
             config.build_settings['CC'] = ["$SRCROOT/#{parent_dir(xc_cc_path, repo_distance)}"]
+          elsif mode == 'producer' || mode == 'producer-fast'
+            config.build_settings.delete('CC') if config.build_settings.key?('CC')
           end
           config.build_settings['SWIFT_EXEC'] = ["$SRCROOT/#{srcroot_relative_xc_location}/xcswiftc"]
           config.build_settings['LIBTOOL'] = ["$SRCROOT/#{srcroot_relative_xc_location}/xclibtool"]
           config.build_settings['LD'] = ["$SRCROOT/#{srcroot_relative_xc_location}/xcld"]
+          config.build_settings['LDPLUSPLUS'] = ["$SRCROOT/#{srcroot_relative_xc_location}/xcldplusplus"]
+          config.build_settings['SWIFT_USE_INTEGRATED_DRIVER'] = ['NO']
 
-          config.build_settings['XCREMOTE_CACHE_FAKE_SRCROOT'] = FAKE_SRCROOT
+          config.build_settings['XCREMOTE_CACHE_FAKE_SRCROOT'] = fake_src_root
+          config.build_settings['XCRC_PLATFORM_PREFERRED_ARCH'] = ["$(LINK_FILE_LIST_$(CURRENT_VARIANT)_$(PLATFORM_PREFERRED_ARCH):dir:standardizepath:file:default=arm64)"]
+          config.build_settings[XCRC_COOCAPODS_ROOT_KEY] = ["$SRCROOT/#{srcroot_relative_project_location}"]
           debug_prefix_map_replacement = '$(SRCROOT' + ':dir:standardizepath' * repo_distance + ')'
           add_cflags!(config.build_settings, '-fdebug-prefix-map', "#{debug_prefix_map_replacement}=$(XCREMOTE_CACHE_FAKE_SRCROOT)")
           add_swiftflags!(config.build_settings, '-debug-prefix-map', "#{debug_prefix_map_replacement}=$(XCREMOTE_CACHE_FAKE_SRCROOT)")
@@ -126,7 +145,8 @@ module CocoapodsXCRemoteCacheModifier
               phase.name != nil && phase.name.start_with?("[XCRC] Prebuild")
             end
           end
-          prebuild_script = existing_prebuild_script || target.new_shell_script_build_phase("[XCRC] Prebuild")
+
+          prebuild_script = existing_prebuild_script || target.new_shell_script_build_phase("[XCRC] Prebuild #{target.name}")
           prebuild_script.shell_script = "\"$SCRIPT_INPUT_FILE_0\""
           prebuild_script.input_paths = ["$SRCROOT/#{srcroot_relative_xc_location}/xcprebuild"]
           prebuild_script.output_paths = [
@@ -135,9 +155,12 @@ module CocoapodsXCRemoteCacheModifier
           ]
           prebuild_script.dependency_file = "$(TARGET_TEMP_DIR)/prebuild.d"
 
-          # Move prebuild (last element) to the first position (to make it real 'prebuild')
-          target.build_phases.rotate!(-1) if existing_prebuild_script.nil?
-        elsif mode == 'producer'
+          # Move prebuild (last element) to the position before compile sources phase (to make it real 'prebuild')
+          if !existing_prebuild_script 
+            compile_phase_index = target.build_phases.index(target.source_build_phase)
+            target.build_phases.insert(compile_phase_index, target.build_phases.delete(prebuild_script))
+          end
+        elsif mode == 'producer' || mode == 'producer-fast'
           # Delete existing prebuild build phase (to support switching between modes)
           target.build_phases.delete_if do |phase|
             if phase.respond_to?(:name)
@@ -152,24 +175,29 @@ module CocoapodsXCRemoteCacheModifier
             phase.name != nil && phase.name.start_with?("[XCRC] Postbuild")
           end
         end
-        postbuild_script = existing_postbuild_script || target.new_shell_script_build_phase("[XCRC] Postbuild")
+        postbuild_script = existing_postbuild_script || target.new_shell_script_build_phase("[XCRC] Postbuild #{target.name}")
         postbuild_script.shell_script = "\"$SCRIPT_INPUT_FILE_0\""
         postbuild_script.input_paths = ["$SRCROOT/#{srcroot_relative_xc_location}/xcpostbuild"]
         postbuild_script.output_paths = [
-          "$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(PLATFORM_PREFERRED_ARCH).swiftmodule.md5",
-          "$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)$(LLVM_TARGET_TRIPLE_SUFFIX).swiftmodule.md5"
+          "$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(XCRC_PLATFORM_PREFERRED_ARCH).swiftmodule.md5",
+          "$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(XCRC_PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)$(LLVM_TARGET_TRIPLE_SUFFIX).swiftmodule.md5"
         ]
         postbuild_script.dependency_file = "$(TARGET_TEMP_DIR)/postbuild.d"
+        # Move postbuild (last element) to the position after compile sources phase (to make it real 'postbuild')
+        if !existing_postbuild_script 
+          compile_phase_index = target.build_phases.index(target.source_build_phase)
+          target.build_phases.insert(compile_phase_index + 1, target.build_phases.delete(postbuild_script))
+        end
 
         # Mark a sha as ready for a given platform and configuration when building the final_target
-        if mode == 'producer' && target.name == final_target
+        if (mode == 'producer' || mode == 'producer-fast') && target.name == final_target
           existing_mark_script = target.build_phases.detect do |phase|
             if phase.respond_to?(:name)
               phase.name != nil && phase.name.start_with?("[XCRC] Mark")
             end
           end
           mark_script = existing_mark_script || target.new_shell_script_build_phase("[XCRC] Mark")
-          mark_script.shell_script = "\"$SCRIPT_INPUT_FILE_0\" mark --configuration $CONFIGURATION --platform $PLATFORM_NAME"
+          mark_script.shell_script = "\"$SCRIPT_INPUT_FILE_0\" mark --configuration \"$CONFIGURATION\" --platform $PLATFORM_NAME"
           mark_script.input_paths = ["$SRCROOT/#{srcroot_relative_xc_location}/xcprepare"]
         else
           # Delete existing mark build phase (to support switching between modes or changing the final target)
@@ -187,8 +215,12 @@ module CocoapodsXCRemoteCacheModifier
           config.build_settings.delete('SWIFT_EXEC') if config.build_settings.key?('SWIFT_EXEC')
           config.build_settings.delete('LIBTOOL') if config.build_settings.key?('LIBTOOL')
           config.build_settings.delete('LD') if config.build_settings.key?('LD')
-          # Add Fake src root for ObjC & Swift
+          config.build_settings.delete('LDPLUSPLUS') if config.build_settings.key?('LDPLUSPLUS')
+          config.build_settings.delete('SWIFT_USE_INTEGRATED_DRIVER') if config.build_settings.key?('SWIFT_USE_INTEGRATED_DRIVER')
+          # Remove Fake src root for ObjC & Swift
           config.build_settings.delete('XCREMOTE_CACHE_FAKE_SRCROOT')
+          config.build_settings.delete('XCRC_PLATFORM_PREFERRED_ARCH')
+          config.build_settings.delete(XCRC_COOCAPODS_ROOT_KEY)
           remove_cflags!(config.build_settings, '-fdebug-prefix-map')
           remove_swiftflags!(config.build_settings, '-debug-prefix-map')
         end
@@ -208,7 +240,7 @@ module CocoapodsXCRemoteCacheModifier
       end
 
       def self.download_xcrc_if_needed(local_location)
-        required_binaries = ['xcld', 'xclibtool', 'xcpostbuild', 'xcprebuild', 'xcprepare', 'xcswiftc']
+        required_binaries = ['xcld', 'xcldplusplus', 'xclibtool', 'xcpostbuild', 'xcprebuild', 'xcprepare', 'xcswiftc']
         binaries_exist = required_binaries.reduce(true) do |exists, filename|
           file_path = File.join(local_location, filename) 
           exists = exists && File.exist?(file_path)
@@ -250,14 +282,14 @@ module CocoapodsXCRemoteCacheModifier
       end
 
       def self.add_cflags!(options, key, value)
-        return if options.fetch('OTHER_CFLAGS',[]).include?(' ' + value)
-        options['OTHER_CFLAGS'] = remove_cflags!(options, key) << " #{key}=#{value}" 
+        return if options.fetch('OTHER_CFLAGS',[]).include?(value)
+        options['OTHER_CFLAGS'] = remove_cflags!(options, key) << "#{key}=#{value}" 
       end
 
       def self.remove_cflags!(options, key)
         cflags_arr = options.fetch('OTHER_CFLAGS', ['$(inherited)'])
         cflags_arr = [cflags_arr] if cflags_arr.kind_of? String
-        options['OTHER_CFLAGS'] = cflags_arr.delete_if {|flag| flag.start_with?(" #{key}=") }
+        options['OTHER_CFLAGS'] = cflags_arr.delete_if {|flag| flag.include?("#{key}=") }
         options['OTHER_CFLAGS']
       end
 
@@ -272,14 +304,29 @@ module CocoapodsXCRemoteCacheModifier
       end
 
       # Uninstall the XCRemoteCache
-      def self.disable_xcremotecache(user_project)
+      def self.disable_xcremotecache(user_project, pods_project = nil)
         user_project.targets.each do |target|
           disable_xcremotecache_for_target(target)
         end
         user_project.save()
 
+        unless pods_project.nil?
+          pods_project.native_targets.each do |target|
+            disable_xcremotecache_for_target(target)
+          end
+          pods_proj_directory = pods_project.project_dir
+          pods_project.root_object.project_references.each do |subproj_ref|
+            generated_project = Xcodeproj::Project.open("#{pods_proj_directory}/#{subproj_ref[:project_ref].path}")
+            generated_project.native_targets.each do |target|
+              disable_xcremotecache_for_target(target)
+            end
+            generated_project.save()
+          end
+          pods_project.save()
+        end
+
         # Remove .lldbinit rewrite
-        save_lldbinit_rewrite(nil) unless !@@configuration['modify_lldb_init']
+        save_lldbinit_rewrite(nil,nil) unless !@@configuration['modify_lldb_init']
       end
 
       # Returns the content (array of lines) of the lldbinit with stripped XCRemoteCache rewrite
@@ -301,17 +348,17 @@ module CocoapodsXCRemoteCacheModifier
       end
 
       # Append source rewrite command to the lldbinit content
-      def self.add_lldbinit_rewrite(lines_content, user_proj_directory)
+      def self.add_lldbinit_rewrite(lines_content, user_proj_directory,fake_src_root)
         all_lines = lines_content.clone
         all_lines << LLDB_INIT_COMMENT
-        all_lines << "settings set target.source-map #{FAKE_SRCROOT} #{user_proj_directory}"    
+        all_lines << "settings set target.source-map #{fake_src_root} #{user_proj_directory}"    
         all_lines << ""
         all_lines
       end
 
-      def self.save_lldbinit_rewrite(user_proj_directory)
+      def self.save_lldbinit_rewrite(user_proj_directory,fake_src_root)
         lldbinit_lines = clean_lldbinit_content(LLDB_INIT_PATH)
-        lldbinit_lines = add_lldbinit_rewrite(lldbinit_lines, user_proj_directory) unless user_proj_directory.nil?
+        lldbinit_lines = add_lldbinit_rewrite(lldbinit_lines, user_proj_directory,fake_src_root) unless user_proj_directory.nil?
         File.write(LLDB_INIT_PATH, lldbinit_lines.join("\n"), mode: "w")
       end
 
@@ -334,7 +381,6 @@ module CocoapodsXCRemoteCacheModifier
           end
 
           validate_configuration()
-
           mode = @@configuration['mode']
           xccc_location = @@configuration['xccc_file']
           remote_commit_file = @@configuration['remote_commit_file']
@@ -344,6 +390,7 @@ module CocoapodsXCRemoteCacheModifier
           final_target = @@configuration['final_target']
           check_build_configuration = @@configuration['check_build_configuration']
           check_platform = @@configuration['check_platform']
+          fake_src_root = @@configuration['fake_src_root']
 
           xccc_location_absolute = "#{user_proj_directory}/#{xccc_location}"
           xcrc_location_absolute = "#{user_proj_directory}/#{xcrc_location}"
@@ -353,7 +400,8 @@ module CocoapodsXCRemoteCacheModifier
           download_xcrc_if_needed(xcrc_location_absolute)
 
           # Save .rcinfo
-          save_rcinfo(generate_rcinfo(), user_proj_directory)
+          root_rcinfo = generate_rcinfo()
+          save_rcinfo(root_rcinfo, user_proj_directory)
 
           # Create directory for xccc & arc.rc location
           Dir.mkdir(BIN_DIR) unless File.exist?(BIN_DIR)
@@ -368,18 +416,42 @@ module CocoapodsXCRemoteCacheModifier
           # Always integrate XCRemoteCache to all Pods, in case it will be needed later
           unless installer_context.pods_project.nil?
             # Attach XCRemoteCache to Pods targets
-            installer_context.pods_project.targets.each do |target|
+            # Enable only for native targets which can have compilation steps
+            installer_context.pods_project.native_targets.each do |target|
+                next if target.source_build_phase.files_references.empty?
                 next if target.name.start_with?("Pods-")
                 next if target.name.end_with?("Tests")
                 next if exclude_targets.include?(target.name)
-                enable_xcremotecache(target, 1, xcrc_location, xccc_location, mode, exclude_build_configurations, final_target)
+                enable_xcremotecache(target, 1, xcrc_location, xccc_location, mode, exclude_build_configurations, final_target,fake_src_root)
             end
 
             # Create .rcinfo into `Pods` directory as that .xcodeproj reads configuration from .xcodeproj location
             pods_proj_directory = installer_context.sandbox_root
 
-            # Manual .rcinfo generation (in YAML format)
-            save_rcinfo({'extra_configuration_file' => "#{user_proj_directory}/.rcinfo"}, pods_proj_directory)
+            # Attach XCRemoteCache to Generated Pods projects
+            installer_context.pods_project.root_object.project_references.each do |subproj_ref|
+                generated_project = Xcodeproj::Project.open("#{pods_proj_directory}/#{subproj_ref[:project_ref].path}")
+                generated_project.native_targets.each do |target|
+                    next if target.source_build_phase.files_references.empty?
+                    next if target.name.end_with?("Tests")
+                    next if exclude_targets.include?(target.name)
+                    enable_xcremotecache(target, 1, xcrc_location, xccc_location, mode, exclude_build_configurations, final_target,fake_src_root)
+                end
+                generated_project.save()
+            end
+
+            # Manual Pods/.rcinfo generation
+            
+            # all paths in .rcinfo are relative to the root so paths used in Pods.xcodeproj need to be aligned
+            pods_path = Pathname.new(pods_proj_directory)
+            root_path = Pathname.new(user_proj_directory)
+            root_path_to_pods = root_path.relative_path_from(pods_path)
+            
+            pods_rcinfo = root_rcinfo.merge({
+              'remote_commit_file' => "#{root_path_to_pods}/#{remote_commit_file}",
+              'xccc_file' => "#{root_path_to_pods}/#{xccc_location}"
+            })
+            save_rcinfo(pods_rcinfo, pods_proj_directory)
 
             installer_context.pods_project.save()
           end
@@ -389,12 +461,12 @@ module CocoapodsXCRemoteCacheModifier
             prepare_result = YAML.load`#{xcrc_location_absolute}/xcprepare --configuration #{check_build_configuration} --platform #{check_platform}`
             unless prepare_result['result'] || mode != 'consumer'
               # Uninstall the XCRemoteCache for the consumer mode
-              disable_xcremotecache(user_project)
+              disable_xcremotecache(user_project, installer_context.pods_project)
               Pod::UI.puts "[XCRC] XCRemoteCache disabled - no artifacts available"
               next
             end
           rescue => error
-            disable_xcremotecache(user_project)
+            disable_xcremotecache(user_project, installer_context.pods_project)
             Pod::UI.puts "[XCRC] XCRemoteCache failed with an error: #{error}."
             next
           end
@@ -403,15 +475,15 @@ module CocoapodsXCRemoteCacheModifier
           # Attach XCRC to the app targets
           user_project.targets.each do |target|
               next if exclude_targets.include?(target.name)
-              enable_xcremotecache(target, 0, xcrc_location, xccc_location, mode, exclude_build_configurations, final_target)
+              enable_xcremotecache(target, 0, xcrc_location, xccc_location, mode, exclude_build_configurations, final_target,fake_src_root)
           end
 
           # Set Target sourcemap
           if @@configuration['modify_lldb_init']
-            save_lldbinit_rewrite(user_proj_directory)
+            save_lldbinit_rewrite(user_proj_directory,fake_src_root)
           else
             Pod::UI.puts "[XCRC] lldbinit modification is disabled. Debugging may behave weirdly"
-            Pod::UI.puts "[XCRC] put \"settings set target.source-map #{FAKE_SRCROOT} \#{your_project_directory}\" to your \".lldbinit\" "
+            Pod::UI.puts "[XCRC] put \"settings set target.source-map #{fake_src_root} \#{your_project_directory}\" to your \".lldbinit\" "
           end
 
           user_project.save()
@@ -419,7 +491,7 @@ module CocoapodsXCRemoteCacheModifier
         rescue Exception => e
           Pod::UI.puts "[XCRC] XCRemoteCache disabled with error: #{e}"
           puts e.full_message(highlight: true, order: :top)
-          disable_xcremotecache(user_project)
+          disable_xcremotecache(user_project, installer_context.pods_project)
         end
       end
     end

cocoapods-plugin/lib/cocoapods-xcremotecache/gem_version.rb

@@ -13,5 +13,5 @@
 # limitations under the License.
 
 module CocoapodsXcremotecache
-  VERSION = "0.0.3"
+  VERSION = "0.0.13"
 end

docs/Development.md

@@ -1,5 +1,7 @@
 ## Development - How to build
 
+Generated [docs](https://spotify.github.io/XCRemoteCache/documentation/xcremotecache/).
+
 ### Building the library:
 
 `CONFIG=Debug rake build`
@@ -23,9 +25,48 @@ If you prefer to edit in Xcode, run `swift package generate-xcodeproj`. Do **not
 
 The generated Xcode project contains schemes for each output application (like `xcswiftc`, `xcprebuild` etc.) so to build a single app, just select the appropriate scheme and build (⌘+B). If you want to build all applications at once, select `Aggregator` scheme that automatically builds all apps. `Aggregator` target in `Package.swift` is defined only for development convenience, it shouldn't be ever used as a dependency.   
 
+#### Debugging the app in a real project
+
+Debugging XCRemoteCache in a real project is simple:
+* Open XCRemoteCache's `Package.swift` in Xcode and select a scheme that corresponds to the process you want to debug, e.g. `xcpostbuild`
+
+![Select scheme](./img/debug-scheme.png)
+
+* Build the app with Product->Build (or ⌘+B)
+* Open the scheme's settings (⌘+⇧+<) and for the "Run" action, select "Wait for the executable to be launched"
+
+![Configure scheme](./img/debug-scheme-wait.png)
+
+* Find the produced binary in your DerivedData. The default location would be `~/Library/Developer/Xcode/DerivedData/XCRemoteCache-{hash}/Build/Products/Debug/xcpostbuild`
+* In the project you want to debug, pick the target you want to inspect and expand the build phase that calls the process
+* Replace the Input Files path so it points the locally built binary placed in DerivedData
+
+![Update phase](./img/debug-phase-update.png)
+
+* Now, you can run the **XCRemoteCache** project (not the project you want to debug). Because Xcode will not initiate a new process, it will just wait until someone else triggers it.
+
+![LLDB waiting](./img/debug-wait.png)
+
+* Finally, build the project to want to debug and expect your XCRemoteCache breakpoints to hit
+
+![Breakpoint hit](./img/debug-breakpoint.png)
+
+> Tip: If your breakpoints don't hit, try adding a dummy `sleep(1)` in the process entry point (e.g. at the top of `XCPostbuild.main` function)  
+
 #### Running tests in Xcode
 
-All unit tests are placed in `XCRemoteCacheTests`. To run them from Xcode, just pick any application scheme and run tests (⌘+U).   
+All unit tests are placed in `XCRemoteCacheTests`. To run them from Xcode, just pick any application scheme and run tests (⌘+U).
+
+#### Running E2E tests
+
+E2E tests build a CocoaPods plugin, locally build both `producer` and `consumer` modes and verify 100% hit rate. All `Podfile` templates are placed in [e2eTests/tests](../e2eTests/tests). As a backend server, nginx server with a sample [nginx.conf](../e2eTests/nginx/nginx.conf) configuration is used. The sample server exposes local location `/tmp/cache` under http://localhost:8080.
+
+To run tests locally, install `nginx` (e.g. `brew install nginx`) and call: 
+
+```bash
+rake 'build[release]'
+rake e2e_only
+```
 
 ## Project organization
 
@@ -71,3 +112,5 @@ To enable thinning target on the consumer side:
 * Prefer using fakes instead of spies or mocks. Place testing doubles in [Tests/XCRemoteCacheTests/TestDoubles](../Tests/XCRemoteCacheTests/TestDoubles) so other tests can reuse them
 
 * If you test a scenario that accesses a file on a disk, consider using the `DiskUsageSizeProviderTests` class that isolates a working directory and eliminates potential file leaks between testcases
+
+* For dependency injection arguments, avoid passing default values (e.g. `init(dep: SomeDependency = SomeDependency())` and require passing explicit values (e.g. `init(dep: SomeDependency))`. It will be clear from a call site which dependencies is used and suggests adding a testcase when a new dependency is added.

docs/FAQ.md

@@ -50,3 +50,18 @@ log show --predicate 'sender == "xcprepare"' --style compact --info --debug -las
 log show --predicate 'sender BEGINSWITH "xc"' --style compact --info --debug -last 10m
 ```
 </details>
+
+### Troubleshooting cache misses
+
+Here is a non-exhaustive list of steps that may help with troubleshooting poor cache hit rate.
+
+1. ***Producer&Consumer:*** Review XCRemoteCache [Requirements](../#Requirements) and [Limitations](../#limitations)
+1. ***Producer&Consumer:*** Make sure a producer build uses the same architecture(s) as a consumer. You can inspect `ARCHS` Build Setting in Xcode's Script Phase output logs. Navigate to the report navigator (⌘+9) and expand XCRemoteCache's `prebuild` step output using the "collapsed menu icon" (aka hamburger menu)
+1. ***Producer:*** Verify that all Xcode targets have a Build Phase called `postbuild`
+1. ***Producer:*** If you are using optional XCRemoteCache auto-marking feature (`--final-producer-target` or `final_target`) verify an extra Build Phase called `mark` is added to the specified target
+1. ***Producer:*** After a full build, review logs according to [docs](#how-can-i-find-xcremotecache-logs)
+1. ***Consumer:*** Verify that all Xcode targets have extra XCRemoteCache Build Phase called `prebuild` and `postbuild`
+1. ***Consumer:*** After a full build, review according to [docs](#how-can-i-find-xcremotecache-logs). Find a ***first:*** target that reports a cache miss with a message like `Prebuild step failed with error: ...`. If a target reports faces a cache miss, it may have a knock-on effect where a lot of its consumers (dependant targets) need to be built locally too
+1. ***Consumer:*** ***After a full build, review all meta files placed in `~/Library/Caches/XCRemoteCache/{your_host_path}/meta/*.json` and make sure no absolute paths are used in its `dependencies`. All paths should start a placeholder, like `$(SRCROOT)` or `$(BUILD_DIR)`***
+1. ***Consumer:*** If you are integrating XCRemoteCache and rebuild artifacts for the same sha, previously downloaded artifacts placed in a local cache may still be used on a consumer side. You can either manually delete your local cache at `~/Library/Caches/XCRemoteCache/` before any consumer build or disable a local cache with `artifact_maximum_age: 0` property in `.rcinfo`
+1. ***Consumer:*** To find an actual cache hit, before building in Xcode reset statistics with `xcprepare stats --reset` and once it is done, call `xcprepare stats` to find a cache hit rate

e2eTests/StandaloneSampleApp/.rcinfo

@@ -0,0 +1,8 @@
+---
+cache_addresses: 
+  - 'http://localhost:8080/cache/pods'
+primary_repo: '.'
+primary_branch: 'e2e-test-branch'
+mode: 'consumer'
+final_target': XCRemoteCacheSample'
+artifact_maximum_age: 0 # do not use local cache in ~/Library/Caches/XCRemoteCache

e2eTests/StandaloneSampleApp/MixedTarget/MixedTarget-Bridging-Header.h

@@ -0,0 +1,5 @@
+//
+//  Use this file to import your target's public headers that you would like to expose to Swift.
+//
+
+#import "SomeObjC.h"