Merge pull request #46 from spotify/adjust-xccc-for-apple-silicon

adjust `xccc` to support `arm64`

.github/workflows/ci.yaml

@@ -13,7 +13,7 @@ jobs:
   macOS:
     runs-on: macOS-latest
     env:
-      XCODE_VERSION: ${{ '12.4' }}
+      XCODE_VERSION: ${{ '13.1' }}
     steps:
       - name: Select Xcode
         run: "sudo xcode-select -s /Applications/Xcode_$XCODE_VERSION.app"

.github/workflows/release.yaml

@@ -8,7 +8,7 @@ jobs:
     name: Add macOS binaries to release
     runs-on: macOS-latest
     env:
-      XCODE_VERSION: ${{ '12.4' }}
+      XCODE_VERSION: ${{ '13.1' }}
     steps:
       - name: Select Xcode
         run: "sudo xcode-select -s /Applications/Xcode_$XCODE_VERSION.app"

README.md

@@ -1,5 +1,6 @@
 <p align="center">
-    <img src="docs/img/logo.png" width="75%">
+    <img src="docs/img/logo.png#gh-light-mode-only" width="75%">
+    <img src="docs/img/logo-dark.png#gh-dark-mode-only" width="75%">
 </p>
 
 _XCRemoteCache is a remote cache tool for Xcode projects. It reuses target artifacts generated on a remote machine, served from a simple REST server._
@@ -37,6 +38,9 @@ _XCRemoteCache is a remote cache tool for Xcode projects. It reuses target artif
   * [Amazon S3 and Google Cloud Storage](#amazon-s3-and-google-cloud-storage)
 - [CocoaPods plugin](#cocoapods-plugin)
 - [Requirements](#requirements)
+- [Apple silicon support](#apple-silicon-support)
+  * [Artifacts per architecture (Recommended)](#artifacts-per-architecture-recommended)
+  * [Fat artifacts](#fat-artifacts)
 - [Limitations](#limitations)
 - [FAQ](#faq)
 - [Development](#development)
@@ -190,6 +194,7 @@ Configure Xcode targets that **should use** XCRemoteCache:
 * `SWIFT_EXEC` - location of `xcprepare` (e.g. `xcremotecache/xcswiftc`)
 * `LIBTOOL` - location of `xclibtool` (e.g. `xcremotecache/xclibtool`)
 * `LD` - location of `xcld` (e.g. `xcremotecache/xcld`)
+* `XCRC_PLATFORM_PREFERRED_ARCH` - `$(LINK_FILE_LIST_$(CURRENT_VARIANT)_$(PLATFORM_PREFERRED_ARCH):dir:standardizepath:file:default=arm64)`
 
 <details>
   <summary>Screenshot</summary>
@@ -209,8 +214,8 @@ Configure Xcode targets that **should use** XCRemoteCache:
 * command: `"$SCRIPT_INPUT_FILE_0"`
 * input files: location of `xcpostbuild` command (e.g. `xcremotecache/xcpostbuild`)
 * output files: 
-  * `$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(PLATFORM_PREFERRED_ARCH).swiftmodule.md5`
-  * `$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)$(LLVM_TARGET_TRIPLE_SUFFIX).swiftmodule.md5`
+  * `$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(XCRC_PLATFORM_PREFERRED_ARCH).swiftmodule.md5`
+  * `$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(XCRC_PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)$(LLVM_TARGET_TRIPLE_SUFFIX).swiftmodule.md5`
 * discovery dependency file: `$(TARGET_TEMP_DIR)/postbuild.d`
 
 <details>
@@ -299,10 +304,12 @@ _Note that for the `producer` mode, the prebuild build phase and `xccc`, `xcld`,
 | `product_files_extensions_with_content_override ` | List of all extensions that should carry over source fingerprints. Extensions of all product files that contain non-deterministic content (absolute paths, timestamp, etc) should be included. | `["swiftmodule"]` | ⬜️ |
 | `thinning_enabled ` | If true, support for thin projects is enabled | `false` | ⬜️ |
 | `thinning_target_module_name ` | Module name of a target that works as a helper for thinned targets | `"ThinningRemoteCacheModule"` | ⬜️ |
+| `prettify_meta_files` | A Boolean value that opts-in pretty JSON formatting for meta files | `false` | ⬜️ |
 | `aws_secret_key` | Secret key for AWS V4 Signature Authorization. If this is set to a non-empty String - an Authentication Header will be added based on this and the other `aws_*` parameters.| `""` | ⬜️ |
 | `aws_access_key` | Access key for AWS V4 Signature Authorization. | `""` | ⬜️ |
 | `aws_region` | Region for AWS V4 Signature Authorization. E.g. `eu`.  | `""` | ⬜️ |
 | `aws_service` | Service for AWS V4 Signature Authorization. E.g. `storage`. | `""` | ⬜️ |
+| `out_of_band_mappings` | A dictionary of files path remapping that should be applied to make it absolute path agnostic on a list of dependencies. Useful if a project refers files out of repo root, either compilation files or precompiled dependencies. Keys represent generic replacement and values are substrings that should be replaced. Example: for mapping `["COOL_LIBRARY": "/CoolLibrary"]` `/CoolLibrary/main.swift`will be represented as `$(COOL_LIBRARY)/main.swift`). Warning: remapping order is not-deterministic so avoid remappings with multiple matchings. | `[:]` | ⬜️ |
 
 
 ## Backend cache server
@@ -359,6 +366,31 @@ Retention Policy: Buckets usually have a retention policy option which ensures o
 
 Head over to our [cocoapods-plugin](cocoapods-plugin/README.md) docs to see how to integrate XCRemoteCache in your CocoaPods project.
 
+## Apple silicon support
+
+### Artifacts per architecture (Recommended)
+
+_If all of your machines (both producer and all consumers have the same architecture, either Intel or Apple Silicon), you don't have to do anything._
+
+XCRemoteCache supports building artifacts for Apple silicon consumers. Is it recommended to build separately for `x86_64` and `arm64` architectures to have single-architecture artifacts that do not require downloading irrelevant binaries. Here are required steps if you want to support both Intel and Apple silicon consumers.
+
+* Building for a simulator on a producer: run a first build for `x86_64`, clean a build and build again for `arm64`, e.g.:
+```
+xcodebuild ARCHS=x86_64 ONLY_ACTIVE_ARCH=NO build ...
+xcodebuild clean
+xcodebuild ARCHS=arm64 ONLY_ACTIVE_ARCH=NO build ...
+```
+
+### Fat artifacts
+
+If you prefer to generate far artifacts (with both Intel and Apple silicon binaries), you can disable "Build Archive Architecture Only" on a producer side, e.g.
+
+```
+xcodebuild ONLY_ACTIVE_ARCH=NO build ...
+```
+
+Note: This setup is not recommended and may not be supported in future XCRemoteCache releases.
+
 ## Requirements
 
 * The repo under `git` version control

Sources/XCRemoteCache/Commands/Postbuild/PostbuildContext.swift

@@ -32,6 +32,8 @@ enum MachOType: String, Codable {
 enum PostbuildContextError: Error {
     /// URL address is not a valid URL
     case invalidAddress(String)
+    /// ARCHS env does not contain any architecture to build
+    case missingArchitecture
 }
 
 public struct PostbuildContext {
@@ -64,6 +66,9 @@ public struct PostbuildContext {
     var machOType: MachOType
     var wasDsymGenerated: Bool
     var dSYMPath: URL
+    // building architecture. Used to find all dependencies from *.d files
+    // Warning: if two architectures are built (e.g. for disabled "Build Archive
+    // Architecture Only"), a first architecture one is picked
     let arch: String
     let builtProductsDir: URL
     /// Location to the product bundle. Can be nil for libraries
@@ -82,8 +87,13 @@ extension PostbuildContext {
         let targetNameValue: String = try env.readEnv(key: "TARGET_NAME")
         targetName = targetNameValue
         targetTempDir = try env.readEnv(key: "TARGET_TEMP_DIR")
-        arch = try env.readEnv(key: "PLATFORM_PREFERRED_ARCH")
-        compilationTempDir = try env.readEnv(key: "OBJECT_FILE_DIR_normal").appendingPathComponent(arch)
+        let archs: [String] = try env.readEnv(key: "ARCHS").split(separator: " ").map(String.init)
+        guard let firstArch = archs.first, !firstArch.isEmpty else {
+            throw PostbuildContextError.missingArchitecture
+        }
+        arch = firstArch
+        let variant: String = try env.readEnv(key: "CURRENT_VARIANT")
+        compilationTempDir = try env.readEnv(key: "OBJECT_FILE_DIR_\(variant)").appendingPathComponent(arch)
         configuration = try env.readEnv(key: "CONFIGURATION")
         platform = try env.readEnv(key: "PLATFORM_NAME")
         xcodeBuildNumber = try env.readEnv(key: "XCODE_PRODUCT_BUILD_VERSION")

Sources/XCRemoteCache/Commands/Postbuild/XCPostbuild.swift

@@ -66,9 +66,10 @@ public class XCPostbuild {
             // Initialize dependencies
             let primaryGitBranch = GitBranch(repoLocation: config.primaryRepo, branch: config.primaryBranch)
             let gitClient = GitClientImpl(repoRoot: config.repoRoot, primary: primaryGitBranch, shell: shellGetStdout)
-            let pathRemapper = try StringDependenciesRemapper.buildFromEnvs(
-                keys: DependenciesMapping.rewrittenEnvs,
-                envs: env
+            let pathRemapper = try StringDependenciesRemapperFactory().build(
+                orderKeys: DependenciesMapping.rewrittenEnvs,
+                envs: env,
+                customMappings: config.outOfBandMappings
             )
             let envFingerprint = try EnvironmentFingerprintGenerator(
                 configuration: config,
@@ -85,7 +86,7 @@ public class XCPostbuild {
                 algorithm: MD5Algorithm()
             )
             let organizer = ZipArtifactOrganizer(targetTempDir: context.targetTempDir, fileManager: fileManager)
-            let metaWriter = JsonMetaWriter(fileWriter: fileManager)
+            let metaWriter = JsonMetaWriter(fileWriter: fileManager, pretty: config.prettifyMetaFiles)
             let artifactCreator = BuildArtifactCreator(
                 buildDir: context.productsDir,
                 tempDir: context.targetTempDir,

Sources/XCRemoteCache/Commands/Prebuild/XCPrebuild.swift

@@ -115,9 +115,10 @@ public class XCPrebuild {
             )
             let client: NetworkClient = config.disableHttpCache ? networkClient : cacheNetworkClient
             let remoteNetworkClient = RemoteNetworkClientImpl(client, urlBuilder)
-            let pathRemapper = try StringDependenciesRemapper.buildFromEnvs(
-                keys: DependenciesMapping.rewrittenEnvs,
-                envs: env
+            let pathRemapper = try StringDependenciesRemapperFactory().build(
+                orderKeys: DependenciesMapping.rewrittenEnvs,
+                envs: env,
+                customMappings: config.outOfBandMappings
             )
             let filesFingerprintGenerator = FingerprintAccumulatorImpl(
                 algorithm: MD5Algorithm(),

Sources/XCRemoteCache/Commands/Prepare/CCWrapperBuilder.swift

@@ -72,7 +72,7 @@ class TemplateBasedCCWrapperBuilder: CCWrapperBuilder {
         )
         infoLog("ClangWrapperBuilder compiles file at \(compilationFile).")
         // -O3: optimize for faster execution
-        let args = [clangCommand, "-O3", compilationFile.path, "-o", destination.path]
+        let args = [clangCommand, "-arch", "arm64", "-arch", "x86_64", "-O3", compilationFile.path, "-o", destination.path]
         let compilationOutput = try shell("xcrun", args, URL(fileURLWithPath: "").path, nil)
         infoLog("Clang compilation output: \(compilationOutput)")
     }

Sources/XCRemoteCache/Commands/Prepare/Integrate/BuildSettingsIntegrateAppender.swift

@@ -62,6 +62,7 @@ class XcodeProjBuildSettingsIntegrateAppender: BuildSettingsIntegrateAppender {
         result["OTHER_CFLAGS"] = clangFlags.settingValue
 
         result["XCRC_FAKE_SRCROOT"] = "/\(String(repeating: "x", count: 10))"
+        result["XCRC_PLATFORM_PREFERRED_ARCH"] = "$(LINK_FILE_LIST_$(CURRENT_VARIANT)_$(PLATFORM_PREFERRED_ARCH):dir:standardizepath:file:default=arm64)"
         return result
     }
 }

Sources/XCRemoteCache/Commands/Prepare/Integrate/XcodeProjIntegrate.swift

@@ -100,11 +100,11 @@ struct XcodeProjIntegrate: Integrate {
             outputPaths: [
                 """
                 $(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/\
-                $(PLATFORM_PREFERRED_ARCH).swiftmodule.md5
+                $(XCRC_PLATFORM_PREFERRED_ARCH).swiftmodule.md5
                 """,
                 """
                 $(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/\
-                $(PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)\
+                $(XCRC_PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)\
                 $(LLVM_TARGET_TRIPLE_SUFFIX).swiftmodule.md5
                 """,
             ],

Sources/XCRemoteCache/Config/XCRemoteCacheConfig.swift

@@ -112,6 +112,8 @@ public struct XCRemoteCacheConfig: Encodable {
     var thinningEnabled: Bool = false
     /// Module name of a target that works as a helper for thinned targets
     var thinningTargetModuleName: String = "ThinningRemoteCacheModule"
+    /// Opt-in pretty json formatting for meta files
+    var prettifyMetaFiles: Bool = false
     /// Secret key for AWS V4 Signature, if this is set the Authentication Header will be added
     var AWSSecretKey: String = ""
     /// Access key for AWS V4 Signature
@@ -120,6 +122,13 @@ public struct XCRemoteCacheConfig: Encodable {
     var AWSRegion: String = ""
     /// Service for AWS V4 Signature (e.g. `storage`)
     var AWSService: String = ""
+    /// A dictionary of files path remapping that should be applied to make it absolute path agnostic on a list of dependencies.
+    /// Useful if a project refers files out of repo root, either compilation files or precompiled dependencies.
+    /// Keys represent generic replacement and values are substrings that should be replaced.
+    /// Example: for mapping `["COOL_LIBRARY": "/CoolLibrary"]`
+    /// `/CoolLibrary/main.swift`will be represented as `$(COOL_LIBRARY)/main.swift`).
+    /// Warning: remapping order is not-deterministic so avoid remappings with multiple matchings.
+    var outOfBandMappings: [String: String] = [:]
 }
 
 extension XCRemoteCacheConfig {
@@ -165,10 +174,12 @@ extension XCRemoteCacheConfig {
             scheme.productFilesExtensionsWithContentOverride ?? productFilesExtensionsWithContentOverride
         merge.thinningEnabled = scheme.thinningEnabled ?? thinningEnabled
         merge.thinningTargetModuleName = scheme.thinningTargetModuleName ?? thinningTargetModuleName
+        merge.prettifyMetaFiles = scheme.prettifyMetaFiles ?? prettifyMetaFiles
         merge.AWSAccessKey = scheme.AWSAccessKey ?? AWSAccessKey
         merge.AWSSecretKey = scheme.AWSSecretKey ?? AWSSecretKey
         merge.AWSRegion = scheme.AWSRegion ?? AWSRegion
         merge.AWSService = scheme.AWSService ?? AWSService
+        merge.outOfBandMappings = scheme.outOfBandMappings ?? outOfBandMappings
         return merge
     }
 
@@ -223,10 +234,12 @@ struct ConfigFileScheme: Decodable {
     let productFilesExtensionsWithContentOverride: [String]?
     let thinningEnabled: Bool?
     let thinningTargetModuleName: String?
+    let prettifyMetaFiles: Bool?
     let AWSSecretKey: String?
     let AWSAccessKey: String?
     let AWSRegion: String?
     let AWSService: String?
+    let outOfBandMappings: [String: String]?
 
     // Yams library doesn't support encoding strategy, see https://github.com/jpsim/Yams/issues/84
     enum CodingKeys: String, CodingKey {
@@ -264,10 +277,12 @@ struct ConfigFileScheme: Decodable {
         case productFilesExtensionsWithContentOverride = "product_files_extensions_with_content_override"
         case thinningEnabled = "thinning_enabled"
         case thinningTargetModuleName = "thinning_target_module_name"
+        case prettifyMetaFiles = "prettify_meta_files"
         case AWSSecretKey = "aws_secret_key"
         case AWSAccessKey = "aws_access_key"
         case AWSRegion = "aws_region"
         case AWSService = "aws_service"
+        case outOfBandMappings = "out_of_band_mappings"
     }
 }
 

Sources/XCRemoteCache/Dependencies/DependenciesRemapper.swift

@@ -35,7 +35,7 @@ class DependenciesRemapperComposite: DependenciesRemapper {
     }
 
     func replace(genericPaths: [String]) -> [String] {
-        remappers.reduce(genericPaths) { prev, mapper in
+        remappers.reversed().reduce(genericPaths) { prev, mapper in
             mapper.replace(genericPaths: prev)
         }
     }
@@ -61,7 +61,7 @@ final class StringDependenciesRemapper: DependenciesRemapper {
 
     func replace(genericPaths: [String]) -> [String] {
         return genericPaths.map { path in
-            let localPath = mappings.reduce(path) { prevPath, mapping in
+            let localPath = mappings.reversed().reduce(path) { prevPath, mapping in
                 prevPath.replacingOccurrences(of: mapping.generic, with: mapping.local)
             }
             return localPath
@@ -77,14 +77,3 @@ final class StringDependenciesRemapper: DependenciesRemapper {
         }
     }
 }
-
-
-extension StringDependenciesRemapper {
-    static func buildFromEnvs(keys: [String], envs: [String: String]) throws -> Self {
-        let mappings: [Mapping] = try keys.map { key in
-            let localValue: String = try envs.readEnv(key: key)
-            return Mapping(generic: "$(\(key))", local: localValue)
-        }
-        return Self(mappings: mappings)
-    }
-}

Sources/XCRemoteCache/Dependencies/StringDependenciesRemapperFactory.swift

@@ -0,0 +1,43 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import Foundation
+
+enum StringDependenciesRemapperFactoryError: Error {
+    /// Remapping keys are duplicated and can lead to undetermined results
+    case mappingKeyDuplication
+}
+
+class StringDependenciesRemapperFactory {
+    func build(
+        orderKeys: [String],
+        envs: [String: String],
+        customMappings: [String: String]
+    ) throws -> StringDependenciesRemapper {
+        let mappingMap = try envs.merging(customMappings) { envValue, outOfBandMapping in
+            throw StringDependenciesRemapperFactoryError.mappingKeyDuplication
+        }
+        let mappingOrderKeys =  orderKeys + customMappings.keys
+        let mappings: [StringDependenciesRemapper.Mapping] = try mappingOrderKeys.map { key in
+            let localValue: String = try mappingMap.readEnv(key: key)
+            return StringDependenciesRemapper.Mapping(generic: "$(\(key))", local: localValue)
+        }
+        return StringDependenciesRemapper(mappings: mappings)
+    }
+}

Sources/XCRemoteCache/Fingerprint/EnvironmentFingerprint.swift

@@ -19,10 +19,10 @@
 
 /// Generates a fingerprint string of the environment (compilation context)
 class EnvironmentFingerprintGenerator {
-    /// Default ENV variables constituing the environment fingerprint
+    /// Default ENV variables constituting the environment fingerprint
     private static let defaultEnvFingerprintKeys = [
         "GCC_PREPROCESSOR_DEFINITIONS",
-        "CLANG_PROFILE_DATA_DIRECTORY",
+        "CLANG_COVERAGE_MAPPING",
         "TARGET_NAME",
         "CONFIGURATION",
         "PLATFORM_NAME",
@@ -31,6 +31,7 @@ class EnvironmentFingerprintGenerator {
         "DYLIB_COMPATIBILITY_VERSION",
         "DYLIB_CURRENT_VERSION",
         "PRODUCT_MODULE_NAME",
+        "ARCHS"
     ]
     private let version: String
     private let customFingerprintEnvs: [String]

Sources/XCRemoteCache/Models/MetaWriter.swift

@@ -24,11 +24,16 @@ protocol MetaWriter {
 }
 
 class JsonMetaWriter: MetaWriter {
-    private let metaEncoder = JSONEncoder()
+    private let metaEncoder: JSONEncoder
     private let fileWriter: FileWriter
 
-    init(fileWriter: FileWriter) {
+    init(fileWriter: FileWriter, pretty: Bool) {
         self.fileWriter = fileWriter
+        let encoder = JSONEncoder()
+        if pretty {
+            encoder.outputFormatting = .prettyPrinted
+        }
+        self.metaEncoder = encoder
     }
 
     func write<T>(_ meta: T, locationDir : URL) throws -> URL where T : Meta {

Tests/XCRemoteCacheTests/Artifacts/BuildArtifactCreatorTests.swift

@@ -63,7 +63,7 @@ class BuildArtifactCreatorTests: FileXCTestCase {
             moduleName: "Target",
             modulesFolderPath: "",
             dSYMPath: dSYM,
-            metaWriter: JsonMetaWriter(fileWriter: fileManager),
+            metaWriter: JsonMetaWriter(fileWriter: fileManager, pretty: false),
             fileManager: fileManager
         )
     }

Tests/XCRemoteCacheTests/Artifacts/ZipArtifactCreatorTests.swift

@@ -37,7 +37,7 @@ class ZipArtifactCreatorTests: FileXCTestCase {
         workingDir = try prepareTempDir().appendingPathComponent("creator")
         creator = ZipArtifactCreator(
             workingDir: workingDir,
-            metaWriter: JsonMetaWriter(fileWriter: fileManager),
+            metaWriter: JsonMetaWriter(fileWriter: fileManager, pretty: false),
             fileManager: fileManager
         )
     }

Tests/XCRemoteCacheTests/Commands/PostbuildContextTests.swift

@@ -26,8 +26,8 @@ class PostbuildContextTests: FileXCTestCase {
     private static let SampleEnvs = [
         "TARGET_NAME": "TARGET_NAME",
         "TARGET_TEMP_DIR": "TARGET_TEMP_DIR",
-        "PLATFORM_PREFERRED_ARCH": "PLATFORM_PREFERRED_ARCH",
-        "OBJECT_FILE_DIR_normal": "OBJECT_FILE_DIR_normal" ,
+        "ARCHS": "x86_64",
+        "OBJECT_FILE_DIR_normal": "/OBJECT_FILE_DIR_normal" ,
         "CONFIGURATION": "CONFIGURATION",
         "PLATFORM_NAME": "PLATFORM_NAME",
         "XCODE_PRODUCT_BUILD_VERSION": "XCODE_PRODUCT_BUILD_VERSION",
@@ -42,6 +42,7 @@ class PostbuildContextTests: FileXCTestCase {
         "DWARF_DSYM_FILE_NAME": "DWARF_DSYM_FILE_NAME",
         "BUILT_PRODUCTS_DIR": "BUILT_PRODUCTS_DIR",
         "DERIVED_SOURCES_DIR": "DERIVED_SOURCES_DIR",
+        "CURRENT_VARIANT": "normal"
     ]
 
     override func setUpWithError() throws {
@@ -87,4 +88,45 @@ class PostbuildContextTests: FileXCTestCase {
 
         XCTAssertEqual(context.action, .index)
     }
+
+    func testReadsSingleArch() throws {
+        var envs = Self.SampleEnvs
+        envs["ARCHS"] = "x86_64"
+        let context = try PostbuildContext(config, env: envs)
+
+        XCTAssertEqual(context.arch, "x86_64")
+    }
+
+    func testReadsFirstArch() throws {
+        var envs = Self.SampleEnvs
+        envs["ARCHS"] = "x86_64 arm64"
+        let context = try PostbuildContext(config, env: envs)
+
+        XCTAssertEqual(context.arch, "x86_64")
+    }
+
+    func testFailsForEmptyArch() throws {
+        var envs = Self.SampleEnvs
+        envs["ARCHS"] = ""
+
+        XCTAssertThrowsError(try PostbuildContext(config, env: envs))
+    }
+
+    func testFailsForMissingArch() throws {
+        var envs = Self.SampleEnvs
+        envs["ARCHS"] = nil
+
+        XCTAssertThrowsError(try PostbuildContext(config, env: envs))
+    }
+
+    func testFindTempDirForCustomVariant() throws {
+        var envs = Self.SampleEnvs
+        envs["ARCHS"] = "x86_64"
+        envs["CURRENT_VARIANT"] = "custom"
+        envs["OBJECT_FILE_DIR_custom"] = "/OBJECT_FILE_DIR_custom"
+
+        let context = try PostbuildContext(config, env: envs)
+
+        XCTAssertEqual(context.compilationTempDir, "/OBJECT_FILE_DIR_custom/x86_64")
+    }
 }

Tests/XCRemoteCacheTests/Commands/PostbuildTests.swift

@@ -85,7 +85,7 @@ class PostbuildTests: FileXCTestCase {
     )
     private var modeController = CacheModeControllerFake()
     private var metaReader = JsonMetaReader(fileAccessor: FileManager.default)
-    private var metaWriter = JsonMetaWriter(fileWriter: FileManager.default)
+    private var metaWriter = JsonMetaWriter(fileWriter: FileManager.default, pretty: false)
     private static let SampleMeta = MainArtifactSampleMeta.defaults
     private var sampleMetaFile: URL!
 

Tests/XCRemoteCacheTests/Commands/XCCCTests.swift

@@ -28,7 +28,7 @@ class TemplateBasedCCWrapperBuilderTests: FileXCTestCase {
     private static let history = "history.compile"
     private static let prebuild = "prebuild.d"
     private static let commitSha = "321"
-    private static let timeout = 5.0
+    private static let timeout = 10.0
 
     static let xccc: URL = {
         let fileManager = FileManager.default

Tests/XCRemoteCacheTests/Dependencies/DependenciesRemapperCompositeTests.swift

@@ -78,4 +78,41 @@ class DependenciesRemapperCompositeTests: XCTestCase {
 
         XCTAssertEqual(localPath, ["/tmp/root/some.swift", "/pwd/other.swift"])
     }
+
+    func testRemapsMultipleMatchingMappers() throws {
+        let remapper = DependenciesRemapperComposite([
+            StringDependenciesRemapper(mappings: [StringDependenciesRemapper.Mapping(generic: "$(ROOT)", local: "/root")]),
+            StringDependenciesRemapper(mappings: [StringDependenciesRemapper.Mapping(generic: "$(SPECIFIC)", local: "$(ROOT)/specific")])
+        ])
+        let localPaths = ["/root/specific/file"]
+
+        let genericPaths = remapper.replace(localPaths: localPaths)
+
+        XCTAssertEqual(genericPaths, ["$(SPECIFIC)/file"])
+    }
+
+    func testRemapsBackToLocalWithRevertedRemappersOrder() throws {
+        let remapper = DependenciesRemapperComposite([
+            StringDependenciesRemapper(mappings: [StringDependenciesRemapper.Mapping(generic: "$(ROOT)", local: "/root")]),
+            StringDependenciesRemapper(mappings: [StringDependenciesRemapper.Mapping(generic: "$(SPECIFIC)", local: "$(ROOT)/specific")])
+        ])
+        let genericPaths = ["$(SPECIFIC)/file"]
+
+        let localPaths = remapper.replace(genericPaths: genericPaths)
+
+        XCTAssertEqual(localPaths, ["/root/specific/file"])
+    }
+
+    func testRemappingTwoMappingsBackAndForthIsIdentical() throws {
+        let remapper = DependenciesRemapperComposite([
+            StringDependenciesRemapper(mappings: [StringDependenciesRemapper.Mapping(generic: "$(ROOT)", local: "/root")]),
+            StringDependenciesRemapper(mappings: [StringDependenciesRemapper.Mapping(generic: "$(SPECIFIC)", local: "$(ROOT)/specific")])
+        ])
+        let localPaths = ["/root/specific/file"]
+
+        let genericPaths = remapper.replace(localPaths: localPaths)
+        let remappedLocalPaths = remapper.replace(genericPaths: genericPaths)
+
+        XCTAssertEqual(localPaths, remappedLocalPaths)
+    }
 }

Tests/XCRemoteCacheTests/Dependencies/StringDependenciesRemapperFactoryTests.swift

@@ -0,0 +1,71 @@
+// Copyright (c) 2021 Spotify AB.
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+@testable import XCRemoteCache
+import XCTest
+
+class StringDependenciesRemapperFactoryTests: XCTestCase {
+    private var factory: StringDependenciesRemapperFactory!
+
+    override func setUp() {
+        factory = StringDependenciesRemapperFactory()
+    }
+
+    func testMappingsFromEnvMaps() throws {
+        let remapper = try factory.build(
+            orderKeys: ["SRC_ROOT"],
+            envs: ["SRC_ROOT": "/tmp/root"],
+            customMappings: [:]
+        )
+
+        let localPaths = remapper.replace(genericPaths: ["$(SRC_ROOT)/some.swift"])
+        XCTAssertEqual(localPaths, ["/tmp/root/some.swift"])
+    }
+
+    func testInvalidMappingsFromEnvFails() throws {
+        XCTAssertThrowsError(
+            try factory.build(
+                orderKeys: ["SRC_ROOT"],
+                envs: ["NO_SRC_ROOT": ""],
+                customMappings: [:]
+            )
+        )
+    }
+
+    func testBuildingRemapperWithMergedCustomMappings() throws {
+        let remapper = try factory.build(
+            orderKeys: ["PWD"],
+            envs: ["PWD": "/some"],
+            customMappings: ["TMP": "/tmp"]
+        )
+
+        let genericPaths = remapper.replace(localPaths: ["/some/repoFile.swift", "/tmp/externalFile.swift"])
+        XCTAssertEqual(genericPaths, ["$(PWD)/repoFile.swift", "$(TMP)/externalFile.swift"])
+    }
+
+    func testFailsBuildingRemapperWithConflictedMappings() throws {
+        XCTAssertThrowsError(
+            try factory.build(
+            orderKeys: ["PWD"],
+            envs: ["PWD": "/some"],
+            customMappings: ["PWD": "/other"]
+            )
+        )
+    }
+}

Tests/XCRemoteCacheTests/Dependencies/StringDependenciesRemapperTests.swift

@@ -70,17 +70,29 @@ class StringDependenciesRemapperTests: XCTestCase {
         XCTAssertEqual(genericPaths, ["$(SRC_ROOT)/some.swift", "$(PWD)/extra.swift"])
     }
 
-    func testMappingsFromEnvMaps() throws {
-        remapper = try StringDependenciesRemapper.buildFromEnvs(keys: ["SRC_ROOT"], envs: ["SRC_ROOT": "/tmp/root"])
+    func testMappingsLocalPathsIsDoneInOrder() {
+        let mappings: [StringDependenciesRemapper.Mapping] = [
+            .init(generic: "$(TMP)", local: "/tmp"),
+            .init(generic: "$(ROOT)", local: "$(TMP)/root"),
+        ]
+        remapper = StringDependenciesRemapper(mappings: mappings)
 
-        let localPaths = remapper.replace(genericPaths: ["$(SRC_ROOT)/some.swift"])
+
+        let genericPaths = remapper.replace(localPaths: ["/tmp/root/some.swift"])
+
+        XCTAssertEqual(genericPaths, ["$(ROOT)/some.swift"])
+    }
+
+    func testMappingsGenericPathsIsDoneInReversedOrder() {
+        let mappings: [StringDependenciesRemapper.Mapping] = [
+            .init(generic: "$(TMP)", local: "/tmp"),
+            .init(generic: "$(ROOT)", local: "$(TMP)/root"),
+        ]
+        remapper = StringDependenciesRemapper(mappings: mappings)
+
+
+        let localPaths = remapper.replace(genericPaths: ["$(ROOT)/some.swift"])
 
         XCTAssertEqual(localPaths, ["/tmp/root/some.swift"])
     }
-
-    func testInvalidMappingsFromEnvFAils() throws {
-        XCTAssertThrowsError(
-            try StringDependenciesRemapper.buildFromEnvs(keys: ["SRC_ROOT"], envs: ["NO_SRC_ROOT": ""])
-        )
-    }
 }

Tests/XCRemoteCacheTests/Fingerprint/EnvironmentFingerprintGeneratorTests.swift

@@ -24,7 +24,7 @@ class EnvironmentFingerprintGeneratorTests: XCTestCase {
 
     private static let defaultENV = [
         "GCC_PREPROCESSOR_DEFINITIONS": "GCC",
-        "CLANG_PROFILE_DATA_DIRECTORY": "CLANG",
+        "CLANG_COVERAGE_MAPPING": "YES",
         "TARGET_NAME": "TARGET",
         "CONFIGURATION": "CONG",
         "PLATFORM_NAME": "PLAT",
@@ -33,6 +33,7 @@ class EnvironmentFingerprintGeneratorTests: XCTestCase {
         "DYLIB_COMPATIBILITY_VERSION": "2",
         "DYLIB_CURRENT_VERSION": "3",
         "PRODUCT_MODULE_NAME": "4",
+        "ARCHS": "AR"
     ]
     /// Corresponds to EnvironmentFingerprintGenerator.version
     private static let currentVersion = "5"
@@ -55,7 +56,7 @@ class EnvironmentFingerprintGeneratorTests: XCTestCase {
     func testConsidersDefaultEnvs() throws {
         let fingerprint = try fingerprintGenerator.generateFingerprint()
 
-        XCTAssertEqual(fingerprint, "GCC,CLANG,TARGET,CONG,PLAT,XC,1,2,3,4,\(Self.currentVersion)")
+        XCTAssertEqual(fingerprint, "GCC,YES,TARGET,CONG,PLAT,XC,1,2,3,4,AR,\(Self.currentVersion)")
     }
 
     func testFingerprintIncludesVersionAsLastComponent() throws {
@@ -73,7 +74,7 @@ class EnvironmentFingerprintGeneratorTests: XCTestCase {
 
         let fingerprint = try fingerprintGenerator.generateFingerprint()
 
-        XCTAssertEqual(fingerprint, ",,,,,,,,,,\(Self.currentVersion)")
+        XCTAssertEqual(fingerprint, ",,,,,,,,,,,\(Self.currentVersion)")
     }
 
     func testConsidersCustomEnvs() throws {
@@ -89,6 +90,6 @@ class EnvironmentFingerprintGeneratorTests: XCTestCase {
 
         let fingerprint = try fingerprintGenerator.generateFingerprint()
 
-        XCTAssertEqual(fingerprint, "GCC,CLANG,TARGET,CONG,PLAT,XC,1,2,3,4,CUSTOM_VALUE,\(Self.currentVersion)")
+        XCTAssertEqual(fingerprint, "GCC,YES,TARGET,CONG,PLAT,XC,1,2,3,4,AR,CUSTOM_VALUE,\(Self.currentVersion)")
     }
 }

Tests/XCRemoteCacheTests/Models/JsonMetaWriterTests.swift

@@ -24,7 +24,7 @@ class JsonMetaWriterTests: XCTestCase {
 
     func testWritesToFileWithFilekeyFilename() throws {
         let fileAccessor = FileAccessorFake(mode: .normal)
-        let writer = JsonMetaWriter(fileWriter: fileAccessor)
+        let writer = JsonMetaWriter(fileWriter: fileAccessor, pretty: false)
         let workingDir: URL = "/"
         let meta = MainArtifactSampleMeta.defaults
 
@@ -35,7 +35,20 @@ class JsonMetaWriterTests: XCTestCase {
 
     func testWritesMetaInValidFormat() throws {
         let fileAccessor = FileAccessorFake(mode: .normal)
-        let writer = JsonMetaWriter(fileWriter: fileAccessor)
+        let writer = JsonMetaWriter(fileWriter: fileAccessor, pretty: false)
+        let reader = JsonMetaReader(fileAccessor: fileAccessor)
+        let workingDir: URL = "/"
+        let meta = MainArtifactSampleMeta.defaults
+
+        let url = try writer.write(MainArtifactSampleMeta.defaults, locationDir: workingDir)
+
+        let readMeta = try reader.read(localFile: url)
+        XCTAssertEqual(readMeta, meta)
+    }
+
+    func testWritesPrettyMetaInValidFormat() throws {
+        let fileAccessor = FileAccessorFake(mode: .normal)
+        let writer = JsonMetaWriter(fileWriter: fileAccessor, pretty: true)
         let reader = JsonMetaReader(fileAccessor: fileAccessor)
         let workingDir: URL = "/"
         let meta = MainArtifactSampleMeta.defaults

cocoapods-plugin/README.md

@@ -52,6 +52,7 @@ An object that is passed to the `xcremotecache` can contain all properties suppo
 | `modify_lldb_init` | Controls if the pod integration should modify `~/.lldbinit` | `true` | ⬜️ |
 | `xccc_file` | The path where should be placed the `xccc` binary (in the pod installation phase) | `{podfile_dir}/.rc/xccc` | ⬜️ |
 | `remote_commit_file` | The path of the file with the remote commit sha (in the pod installation phase) | `{podfile_dir}/.rc/arc.rc`| ⬜️ |
+| `prettify_meta_files` | A Boolean value that opts-in pretty JSON formatting for meta files | `false` | ⬜️ |
 
 ## Uninstalling
 

cocoapods-plugin/lib/cocoapods-xcremotecache/command/hooks.rb

@@ -17,6 +17,7 @@ require 'cocoapods/resolver'
 require 'open-uri'
 require 'yaml'
 require 'json'
+require 'pathname'
 
 
 module CocoapodsXCRemoteCacheModifier
@@ -58,6 +59,7 @@ module CocoapodsXCRemoteCacheModifier
           'xccc_file' => "#{BIN_DIR}/xccc",
           'remote_commit_file' => "#{BIN_DIR}/arc.rc",
           'exclude_targets' => [],
+          'prettify_meta_files' => false
         }
         @@configuration.merge! default_values.select { |k, v| !@@configuration.key?(k) }
       end
@@ -114,6 +116,7 @@ module CocoapodsXCRemoteCacheModifier
           config.build_settings['LD'] = ["$SRCROOT/#{srcroot_relative_xc_location}/xcld"]
 
           config.build_settings['XCREMOTE_CACHE_FAKE_SRCROOT'] = FAKE_SRCROOT
+          config.build_settings['XCRC_PLATFORM_PREFERRED_ARCH'] = ["$(LINK_FILE_LIST_$(CURRENT_VARIANT)_$(PLATFORM_PREFERRED_ARCH):dir:standardizepath:file:default=arm64)"]
           debug_prefix_map_replacement = '$(SRCROOT' + ':dir:standardizepath' * repo_distance + ')'
           add_cflags!(config.build_settings, '-fdebug-prefix-map', "#{debug_prefix_map_replacement}=$(XCREMOTE_CACHE_FAKE_SRCROOT)")
           add_swiftflags!(config.build_settings, '-debug-prefix-map', "#{debug_prefix_map_replacement}=$(XCREMOTE_CACHE_FAKE_SRCROOT)")
@@ -156,8 +159,8 @@ module CocoapodsXCRemoteCacheModifier
         postbuild_script.shell_script = "\"$SCRIPT_INPUT_FILE_0\""
         postbuild_script.input_paths = ["$SRCROOT/#{srcroot_relative_xc_location}/xcpostbuild"]
         postbuild_script.output_paths = [
-          "$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(PLATFORM_PREFERRED_ARCH).swiftmodule.md5",
-          "$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)$(LLVM_TARGET_TRIPLE_SUFFIX).swiftmodule.md5"
+          "$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(XCRC_PLATFORM_PREFERRED_ARCH).swiftmodule.md5",
+          "$(TARGET_BUILD_DIR)/$(MODULES_FOLDER_PATH)/$(PRODUCT_MODULE_NAME).swiftmodule/$(XCRC_PLATFORM_PREFERRED_ARCH)-$(LLVM_TARGET_TRIPLE_VENDOR)-$(SWIFT_PLATFORM_TARGET_PREFIX)$(LLVM_TARGET_TRIPLE_SUFFIX).swiftmodule.md5"
         ]
         postbuild_script.dependency_file = "$(TARGET_TEMP_DIR)/postbuild.d"
 
@@ -353,7 +356,8 @@ module CocoapodsXCRemoteCacheModifier
           download_xcrc_if_needed(xcrc_location_absolute)
 
           # Save .rcinfo
-          save_rcinfo(generate_rcinfo(), user_proj_directory)
+          root_rcinfo = generate_rcinfo()
+          save_rcinfo(root_rcinfo, user_proj_directory)
 
           # Create directory for xccc & arc.rc location
           Dir.mkdir(BIN_DIR) unless File.exist?(BIN_DIR)
@@ -378,8 +382,18 @@ module CocoapodsXCRemoteCacheModifier
             # Create .rcinfo into `Pods` directory as that .xcodeproj reads configuration from .xcodeproj location
             pods_proj_directory = installer_context.sandbox_root
 
-            # Manual .rcinfo generation (in YAML format)
-            save_rcinfo({'extra_configuration_file' => "#{user_proj_directory}/.rcinfo"}, pods_proj_directory)
+            # Manual Pods/.rcinfo generation
+            
+            # all paths in .rcinfo are relative to the root so paths used in Pods.xcodeproj need to be aligned
+            pods_path = Pathname.new(pods_proj_directory)
+            root_path = Pathname.new(user_proj_directory)
+            root_path_to_pods = root_path.relative_path_from(pods_path)
+            
+            pods_rcinfo = root_rcinfo.merge({
+              'remote_commit_file' => "#{root_path_to_pods}/#{remote_commit_file}",
+              'xccc_file' => "#{root_path_to_pods}/#{xccc_location}"
+            })
+            save_rcinfo(pods_rcinfo, pods_proj_directory)
 
             installer_context.pods_project.save()
           end

cocoapods-plugin/lib/cocoapods-xcremotecache/gem_version.rb

@@ -13,5 +13,5 @@
 # limitations under the License.
 
 module CocoapodsXcremotecache
-  VERSION = "0.0.3"
+  VERSION = "0.0.4"
 end

docs/Development.md

@@ -71,3 +71,5 @@ To enable thinning target on the consumer side:
 * Prefer using fakes instead of spies or mocks. Place testing doubles in [Tests/XCRemoteCacheTests/TestDoubles](../Tests/XCRemoteCacheTests/TestDoubles) so other tests can reuse them
 
 * If you test a scenario that accesses a file on a disk, consider using the `DiskUsageSizeProviderTests` class that isolates a working directory and eliminates potential file leaks between testcases
+
+* For dependency injection arguments, avoid passing default values (e.g. `init(dep: SomeDependency = SomeDependency())` and require passing explicit values (e.g. `init(dep: SomeDependency))`. It will be clear from a call site which dependencies is used and suggests adding a testcase when a new dependency is added.