Contents

Read These First

Introduction to conan-ue4cli

Contents

Important note: the conan-ue4cli documentation is currently under active development and some sections are still missing. This notice will be removed once the documentation has been completed and finalised.

Introduction

The conan-ue4cli Python package is a plugin for ue4cli that provides functionality for integrating third-party libraries into Unreal Engine projects and plugins using the Conan C++ package management system. The article Cross-platform library integration in Unreal Engine 4 introduced conan-ue4cli for the first time and discusses the underlying factors that motivated the development of conan-ue4cli, along with the rationale underpinning its key design choices. The relevant information from that article is presented below, reworked to reflect the changes that conan-ue4cli has undergone since it was first created.

Motivation

The ability to integrate third-party libraries into Unreal Engine 4 projects and plugins allows developers to augment the existing capabilities of the Engine with the functionality of any arbitrary codebase. Since the release of Unreal Engine 4.24, the Third-Party Libraries page of the official Unreal Engine documentation has served as the definitive guide to integrating third-party libraries into the Unreal Engine, and is accompanied by the “Third Party Plugin” template in the Unreal Editor which generates boilerplate code for integrating third-party libraries into an Unreal plugin. The documentation and template code make use of a type of module known as an External Module, which contains no source code outside of its C# module rules file and simply acts as a mechanism to consume pre-compiled binaries for one or more third-party libraries. This type of module is used extensively throughout the source code of the Unreal Engine itself to provide access to the many third-party libraries that come bundled with the Engine. External Modules provide a solid mechanism for consuming pre-built binaries, but they do not address the far more complex task of building those binaries in the first place.

Building third-party libraries in the correct manner to ensure binary compatibility with the Unreal Engine is not a simple task. Across all platforms, nuances related to compiler toolchain invocations, linker behaviour, and Application Binary Interfaces (ABIs) can result in obstacles that require significant developer effort to diagnose and address. Epic Games does not provide a formalised framework for building compatible libraries, and exploring the Unreal Engine source tree reveals a plethora of hand-written scripts designed to address these issues on a library-by-library basis. The development and maintenance of such scripts no doubt involves a significant amount of engineering effort, as suggested by the relatively slow cadence of updates to the Unreal Engine’s bundled third-party libraries. Replicating this effort is not unrealistic for large and experienced development teams, but for smaller teams and inexperienced users such as academic researchers who are looking to extend the Unreal Engine to encompass new and exploratory use cases, such an undertaking is often all but impossible. There is a clear need for a workflow that formalises and automates as much of the library compilation and integration process as possible. Such a workflow should integrate smoothly with the Unreal Engine’s build system, scale to arbitrary numbers of dependencies, and abstract away as much of the underlying detail involved in addressing key issues that developers may face during library integration as possible.

Key issues

Symbol interposition

Symbol interposition is a linker feature that allows executables to handle the existence of multiple instances of a given symbol within a process’s address space. When attempting to resolve a reference to a symbol for which multiple instances are available, a linker’s default behaviour will typically be to select the first instance that it finds. Under macOS and Linux, the dlsym() function supports the RTLD_DEFAULT and RTLD_NEXT flags, which provide functionality to control this symbol resolution behaviour. Equivalent functionality is not provided by the Windows API because DLLs must explicitly export public symbols and applications must either explicitly link against the corresponding import libraries or else retrieve symbols programmatically.

The Unreal Engine bundles a number of pre-built third-party libraries in the Engine/Source/ThirdParty subdirectory of the Engine’s source tree. The prebuilt binaries for the majority of these dependencies are static libraries. As a consequence of symbol interposition, linking against additional third-party libraries which declare symbols with the same name as those that exist within a Unreal-bundled library can cause unintended collisions. As an example, consider a scenario in which we are linking against a static build of the OpenCV computer vision library. If OpenCV has been built against a different version of libpng than the version which is bundled with the Unreal Engine, then we will encounter a symbol collision when attempting to read a PNG image using the OpenCV API. The linker will select the symbols from the Unreal-bundled version of libpng instead of those from the OpenCV version, and the library version mismatch will result in an error.

Symbol interposition issues can occur under the following circumstances:

  • When linking against shared libraries under macOS or Linux; and
  • When linking against static libraries under Windows, macOS or Linux.

The most effective way to prevent symbol interposition issues is to ensure that all third-party libraries are built against the Unreal-bundled versions of their own dependencies (where applicable.) Although this does require building each library from source, the use of a package management system and associated repository can restrict the occurrence of such builds to once per combination of target platform and Engine version.

STL and libc++ issues under Linux

The default implementation of the C++ standard library that ships with most Linux distributions is libstdc++, the GNU C++ Library. libstdc++ is licensed under the GNU General Public License, a strong copyleft license. To allow non-GPL licensed software to link against libstdc++, its license includes the GCC Runtime Library Exception, which provides additional clauses stating that any independent code that links against the runtime library is exempt from the terms of the GPL, so long as a set of eligibility criteria are met. However, the extent to which it is permissible to distribute libstdc++ itself (including its accompanying header files) alongside non-GPL software appears to be a matter of contention.

The Unreal Engine does not utilise libstdc++. Linux builds of the Engine and all of its bundled third-party libraries are built against libc++, the implementation of the C++ standard library from the LLVM project. libc++ is dual licensed under the MIT license and the BSD-like UIUC License, which are permissive, non-copyleft licenses. libc++ and its headers are bundled with the Engine in the ThirdParty directory of the Engine’s source tree. This choice of runtime library is ostensibly due to legal concerns regarding the ability to distribute libstdc++ with the Engine 12.

libc++ is not ABI compatible with libstdc++, except for a small handful of low-level features. As a consequence, any code that has been compiled with libc++ and utilises features of the C++ standard library (such as the container classes from the STL) cannot interoperate with other code that uses these features and has been compiled against libstdc++ (and vice versa.) The practical upshot of this incompatibility is that any third-party libraries which have been built against libstdc++ can only communicate with Unreal Engine C++ code through the use of pure C interfaces. In order to achieve full C++ interoperability, it is necessary to compile third-party libraries against libc++ (preferably the version of libc++ that is bundled with the Engine, for maximum compatibility.) This is effectively an extension of the solution to symbol interposition (building libraries from source against the Unreal-bundled versions of their dependencies), whereby we treat libc++ as a dependency of all libraries when building under Linux.

Solution rationale

As stated in the Motivation section, there is a clear need for a workflow that formalises and automates as much of the process of integrating third-party libraries into the Unreal Engine as possible. The conan-ue4cli Python package is designed to provide this workflow and address all of the issues discussed in the sections above. The Conan C++ package management system is used to provide a formalised framework for building C++ libraries from source using established best practices, and Conan profiles and packages generated by conan-ue4cli are used to abstract away the details of building libraries in a manner that makes them compatible with the Unreal Engine.

The Conan package manager was selected for three reasons:

  • It is cross-platform and supports all of the desktop platforms that are supported by the Unreal Engine.

  • It is agnostic of build system and is designed to be extensible to arbitrary build systems. This is crucial because the ability to produce JSON output data using the built-in json generator and then consume that information from an External Module inside an Unreal project or plugin is the key to seamless integration with the Unreal Engine’s build system.

  • It supports both public repositories on Bintray and self-hosted repositories through JFrog Artifactory Community Edition for C/C++, making it suitable for use with both publicly-available open source projects and private, in-house projects.

Workflow overview

conan-ue4cli extends Conan for use with the Unreal Engine by providing functionality to facilitate the following workflow:

  1. Conan packages are generated to wrap all Unreal-bundled third-party libraries, as well as the compiler toolchain itself when targeting Linux platforms.

  2. Conan profiles are generated to ensure user packages are built with the correct configuration and against the wrapper packages for the Unreal-bundled versions of any dependency libraries.

  3. Packages are built using the generated profiles.

  4. Boilerplate code is generated for External Modules that consume the built Conan packages.

  5. Optionally, precomputed dependency data is generated for one or more target platforms so Unreal projects or plugins that consume third-party libraries can be shared with other developers who do not have conan-ue4cli installed.

The entire conan-ue4cli workflow is discussed in detail in the Workflow section of the documentation.

Getting started

  1. Since conan-ue4cli acts as a bridge between the Conan C++ package management system and the Unreal Engine’s build system, there are a number of concepts related to both of these systems that developers must be familiar with before they can start making use of conan-ue4cli. It is critical that you start by reading the Key concepts page and familiarising yourself with all of the relevant background material.

  2. Next, head to the Installation and setup page for details on how to install and configure conan-ue4cli.

  3. Once everything is configured correctly, work through each of the other workflow stages in turn: Creating Conan packages, Consuming Conan packages and finally Distributing projects and plugins.

  4. For information on each of the individual commands provided by conan-ue4cli, check out the Commands section.

References

  1. Linker error with C++ std lib on Linux - UE4 AnswerHub ↩

  2. Compile Unreal with gcc and system STL for linux - UE4 AnswerHub ↩