* update nanopb from 0.3.9.5 to 0.3.9.6
This fixes issues building firestore SDK with Python3.
Interestingly, latest version 0.4.2 still errors during the build
but 0.3.9.6 works. The git commit for this fix in nanopb
https://github.com/nanopb/nanopb/commit/63f8e5e4946a5f8f0211b454bd37187871c3d023
* Update generated sources
* updating just nanopb dependency version
nanopb updated from 0.3.9.5 to 0.3.9.6
(pod version 1.30905.0 to 1.30906.0)
* Add missing source specifier
* Generated files for protos using nanopb 0.3.9.6
* removed unnecessary whitespaces in files generated by protoc
* using more qualified relative paths for header includes
* Changelog
Co-authored-by: Marek Gilbert <mcg@google.com>
Co-authored-by: Paul Beusterien <paulbeusterien@google.com>
This unblocks nanopb changes proposed in #4264.
This upgrades everything in the gRPC constellation of dependencies to match gRPC at 1.28:
* gRPC to 1.28.0 (note that gRPC 1.28.1, while released, is a fix for just the python binding and there's no gRPC-C++ CocoaPod at that version)
* Abseil to 20200225 (as of gRPC 1.27 this is now a gRPC dependency too; there's a later patch but gRPC doesn't use it yet)
* Protobuf C++ to 3.11.4 (only used for testing in Firestore)
* C-Ares to 1.15.0
* BoringSSL to match (no change between gRPC 1.27.0 and 1.28.0)
Also, this disables CFStream-based transport on Apple platforms. This works around b/133182964, wherein CFStream will occasionally fail to raise a has-bytes-available events, causing Firestore to appear to hang.
* Use early exit in objc_framework CMakeLists.txt
* Clean up objc_framework CMakeLists.txt
* Use `version` consistently
* Use `firebase_ios_version` for the repo-level version
* Remove INCLUDE directories that are no longer needed
* Sort dependencies
* Simplify Example/App/CMakeLists.txt
* Add `OBJC_FLAGS` directly, avoiding a separate `sources` variable.
* Add resources directly
* Rename test host app to firebase_firestore_example_app
* Add a firebase_ios prefix to functions and variables
This reduces the chance of collision with other projects.
* Simplify compiler_setup
Now that compiler_setup.cmake no longer has side-effects, fold
compiler_id.cmake and archive_options.cmake into it and move
the include of compiler_setup up to the top of the main build.
* Add support for building for iOS
* Make building benchmarks optional
* Disable protoc-based source generators when cross-compiling
* Make downloading googletest optional
* Add DISABLE_STRICT_WARNINGS mode to cc_rules
This makes it possible to build non-Firestore components in the CMake
build to a lower standard than that to which we hold ourselves without
sacrificing the utility of the various build rules that we have.
* Upgrade gRPC to 1.24.3
* Upgrade C++ Protobuf to 3.9.2
Disable -Wunreachable-code in protobuf-generated code
* Regenerate C++ protos with protobuf 3.9.2
* Generate objc protos with Protobuf 3.9.2
* Require CMake 3.5.1
This is the version required by the latest versions of gRPC and Google
Benchmark. See discussion https://github.com/google/benchmark/pull/613.
Remove version check around find_package(XCTest)
Also, add a dependency on objc protos that was missing but somehow
implicitly being added before.
* Upgrade Google Benchmark to 1.5.0
The as-of-yet-unreleased Clang 10 adds some new warnings which make the CMake build fail (due to `-Werror`):
* BoringSSL build now fails because some of its C code contains fallthroughs in switch statements that aren't properly annotated. This wasn't an issue before because previous versions of Clang ignored `-Wimplicit-fallthrough` in C mode;
* serializer test contains an expression that produces the following warning:
```
/src/firebase-ios-sdk/Firestore/core/test/firebase/firestore/remote/serializer_test.cc:446:29:
error: implicit conversion from 'std::__1::numeric_limits<long>::type' (aka 'long') to 'double'
changes value from 9223372036854775807 to 9223372036854775808 [-Werror,
-Wimplicit-int-float-conversion]
```
Fixes:
* upgrade the version of BoringSSL used to the latest master. There's no particular reason to choose this commit, other than it happens to be the latest. The actual fixes we need are [this](https://github.com/google/boringssl/commit/05cd93068b0a553afc48f69acbceae10c6a17593) and [this](https://github.com/google/boringssl/commit/fbebe833b180a1a0685803c59bba38916826dd2b);
* apply `static_cast` to the serializer test. Also, because the cast produces a different value, add a couple of additional values to test.
The breakage was discovered thanks to the fuzz build (which already uses Clang 10 from trunk).
gRPC packages 0.3.7-dev from 2016 in its own third_party directory and
builds it into libraries produced by CMake. Firestore depends on 0.3.9.2
via external dependency.
Firestore checks in its nanopb-generated sources for compatibility with
CocoaPods, where running the generator at build time on remote users'
machines is perilous. This means that Firestore's generated code must
match the version of nanopb available in CocoaPods.
Unfortunately, generated code from 0.3.7-dev and 0.3.9.2 are not
compatible with each other so we can't just downgrade the Firestore
CMake build to match gRPC and use its copy.
This change excludes gRPC's copy of nanopb until we get this fixed
upstream.
Several minor changes related to `Datastore`:
* `WatchStream` and `WriteStream` now use `GrpcConnection` instead of `Datastore` to create the underlying `GrpcStream`. This is in line with consolidating management of gRPC objects into `GrpcConnection` and removes dependence of streams on `Datastore`;
* Add `CreateWatchStream`/`CreateWriteStream` functions to `Datastore`;
* For functions creating gRPC wrappers (`GrpcStream`), pass token as `auth::Token` instead of `string_view`. This prevents confusing the argument order between the token and the RPC name, and allows moving the logic for retrieving a token for an unauthenticated user into `GrpcConnection`.
* Most significantly, move conversion from `grpc::Status` to `util::Status` from `Datastore` into the `util::Status` class itself. The main motivation is to avoid any dependence of `GrpcStream`, a pure C++ class, on `Datastore`, an Objective-C++ class. The downside is diverging from `FSTDatastore`. I think it's fine, because a) other platforms don't have an equivalent of `Datastore::ConvertStatus`; b) this is in line with how Windows/Apple Foundation/POSIX errors are handled by `util::Status` already;
* in CMake build, bump the used version of gRPC to 1.14.2. This is to be able to include gRPC headers using `grpcpp` path, not deprecated `grpc++`.
* Rename nanopb/.../*.pb.c to nanopb/.../*.nanopb.c
This ensures that all generated nanopb files use the same (sub)
extension of nanopb. This matches up with nanopb's --extension
parameter (which will be enabled later in this PR.)
* Remove references to NANOPB_PROTOC_BIN
Which wasn't being used, and wasn't pointing to a valid protoc.
* Add ability to generate protos from cmake build.
Currently, it's an entirely manual process via 'generate_protos' target.
This target is not automatically run during a build.
See Firestore/Protos/build-protos.sh for expected usage.
* cleanup/update license in build-protos.sh
* Build grpc proto sources (objc only)
* Use 32bit nanopb fields (instead of 16)
To match xcode build, but also to allow google/api/annotations.proto to
compile. (A recent change to nanopb now enforces this.) This isn't
strictly necessary, since we don't make use of that proto, so we
could've simply removed it instead (and in fact, the objc protos do
exactly that) but matching the xcode build certainly seems worthwhile.
* Rebuild all proto sources
* Skip whitespace checks on protoc generated files
* Detect OSS Fuzz thorough `LIB_FUZZING_ENGINE` env var.
* Avoid building libFuzzer if not fuzzing.
* Load the fuzzing library either from the library provided by OSS Fuzz or from libFuzzer.a that was manually built from sources.
* Added `cc_binary` rule to allow executables.
* Added `libFuzzer` as an external project that Firestore depends on.
* LibFuzzer is checked out from its SVN repository.
* LibFuzzer is built using its `build.sh` script to produce the library `libFuzzer.a`.
* Added a `FindLibFuzzer.cmake` script that looks for `libFuzzer.a`.
* Added a new cmake option `FUZZING` to enable fuzzing and adding the appropriate compiler flags.
* Added `fuzzing_options.cmake` rules to check for compiler type and add the fuzzing flags accordingly.
* Added a `firebase_firestore_fuzzing_serializer` target that depends on `libFuzzer` and `firebase_firestore_remote` to fuzz test the serializer.
* Add compiler flags for MSVC on VS2017
* Force gtest to use the dynamic runtime on Windows.
* Avoid setting flags that MSVC does not understand
* Add compiler_id: cheap shortcuts for conditional features on a compiler
* Fix googletest build under VS2017
* Force disable Abseil's tests, which don't compile under VS2017.
* Make Firestore CMake a self-contained build
Now shell out to a sub-CMake to prepare depenencies without needing an
explciit superbuild structure. This makes the Visual Studio and CLion
projects driven from the CMake configuration actually usable.
* Make LevelDB an optional dependency.
* Move third-party dependencies to the top-level
* Make Firestore CMake a self-contained build
Now shell out to a sub-CMake to prepare depenencies without needing an
explciit superbuild structure. This makes the Visual Studio and CLion
projects driven from the CMake configuration actually usable.
* Move third-party dependencies to the top-level
* Rename utils.cmake to cc_rules.cmake
This makes it less of a dumping ground
* Fix sign mismatch in FIRApp
* Implement a podspec_framework CMake function
... that generates a CMake framework library target from a podspec.
* Remove manual CMake scripts for xcodebuild
* Build zlib with grpc's add_subdirectory
* Build grpc and its children with add_subdirectory.
* Build c-ares with grpc's add_subdirectory
* Convert protobuf to a download-only ExternalProject
* Temporarily break protobuf within grpc
This works because libgrpc has no actual dependency on protobuf
* Update boringssl to master@{2018-07-10}
* Add a download-only boringssl ExternalProject
Match boringssl version to grpc submodule version
Note: not yet in use
* Split grpc into separate download and build tasks
This allows other external targets to download into the gRPC source tree
before grpc is actually built.
* Convert grpc to an http download
* Build googletest with add_subdirectory
Also:
* Revert library-name modifications in the abseil-cpp build
* Move abseil-cpp add_subdirectory build into place
* Build nanopb with add_subdirectory
* Force nanopb first to work around ODR violations
This is a temporary work-around for b/111328563.
* Add a download-only boringssl ExternalProject
Match boringssl version to grpc submodule version
* Split grpc into separate download and build tasks
This allows other external targets to download into the gRPC source tree
before grpc is actually built.
* Convert grpc to an http download
* Pull out sanitizer options into a separate file
* Reorder packages
* Remove the extra "external/$target" directory from PREFIX
Instead of BINARY_DIR/external/foo/src/foo the contents will be in
BINARY_DIR/src/foo.
* Make libprotobuf-nanopb targets line up with nanopb's CMakeLists.txt
* Fix CMake style
* Add cmake dependencies where required
... instead of relying on a topological sort in the top-level
CMakeLists.txt.
* Add early exits to external project builds
* Add an external build for c-ares
* Use installed c-ares package configuration
* Adjust protobuf configuration to use the standard Protobuf_DIR variable.
* Have grpc depend on c-ares
* Drop UPDATE_COMMAND where it doesn't matter
* Install protobuf
* Use the built-in FindProtobuf.cmake
* Add additional build configuration flags
* Preserve generator configuration in the protobuf sub-build
Without this, the build fails on Win64 because the default
configuration builds a Win32 libprotobuf and the Win64 build rejects
it.
* Wire the installed protobuf into the gRPC build.
* Install nanopb
* Ignore vim backup files
* Prepare for installed subprojects
This adds FIREBASE_BINARY_DIR to point to the out-of-source build
directory (which previously was passed around as FIREBASE_INSTALL_DIR).
This repurposes FIREBASE_INSTALL_DIR to point to
${FIREBASE_BINARY_DIR}/opt, and installation root for subprojects that
have an install step.
* Allow download directory to be specified
* Build protobuf with CMake for compatibility with Windows.
Fixes b/74067357
* Update to nanopb 0.3.9.1
This release includes functional CMake support for building the protoc
plugin.
* Fix license
* Build nanopb out-of-source
This is possible now that the build is CMake-based.
* Find protoc on windows
* Define CMAKE_ARGS in a more straightforward way
* Adjust libprotobuf locations to account for CMake build
Exception: grpc. Due to it's use of git submodules, it's not completely trivial to convert it (though probably wouldn't be too much more work.)
This helps address our build being throttled. (Maybe. This assumes github allows more fetching of tgz's than clones.) It should also speed up our build times.
The downloaded tarballs are placed into ${PROJECT_BINARY_DIR}/downloads. This allows for eventual caching in travis.
* Build (grpc's) nanopb with -DPB_FIELD_16BIT
We require (at least) 16 bit fields. (By default, nanopb uses 8 bit
fields, ie allowing up to 256 field tags.)
Also note that this patch adds this to grpc's nanopb, rather than to our
nanopb. We'll need to eventually either:
a) we instruct grpc to use our nanopb
b) we rely on grpc's nanopb instead of using our own.
(^ marked as a TODO for now.)
* Add some dependant protos
Imported from protobuf. Nanopb requires these to be present (though
anything using libprotobuf does not, as these are already built into
that.)
* Add generated nanopb protos based off of newly added proto definitions
* Build the nanopb protos
* Serialize and deserialize null
* Fix nanopb (in cmake build)
Look for binaries in the src dir (since that's where we build now.) This
error would be masked if a previous build had completed prior to
switching nanopb to build out of src.
Also, don't patch the protoc path multiple times. This could be
triggered by (eg) 'make && make clean && make'.
* Add resource_path.{h,cc} to the cmake build
* Fix signed/unsigned int comparison warnings
* Ensure FieldValue tag_ is initialized during cp/mv ctor.
Otherwise, the assignment operator attempts to deallocate based on the
(uninitialized) tag_ variable, posssibly leading to segfaults.
* Fix tests that throw exceptions.
The (previous) tests checked to ensure that an abort() occurs, but if
ABSL_HAVE_EXCEPTIONS is defined on non-macos (which is currently the
default) then the assertions will throw a std::logic_error rather than
abort()ing.
On macos, an exception is thrown too, but the exception doesn't derrive
from std::exception, so ASSERT_DEATH_* doesn't catch it (hence why
ASSERT_DEATH_* actually works.)
To resolve this, I've switched to ASSERT_ANY_THROW.
We check them out from a git tag, so this *should* be a noop. However,
cmake seems to want to rebuild these dependencies every time you run
make as it assumes the dependency *might* have been updated. (In
practice, this isn't completely awful, as make notices the files haven't
changed, so files don't actually get recompiled. But the configure step
is still re-run and all the files still need to be rescanned.)
Skipping the update step speeds up the build considerably.
On my linux box, running:
cmake .. && make -j && time make -j
takes ~8.5s prior to this CL and ~6.5 afterwards. (6s is used by the
test suite.) The upcoming protobuf addition would otherwise have made
this much worse. (It takes a long time to ./configure.)
We originally had this as it was thought that cmake would spawn n*m jobs
(where n is the number of jobs it was instructed to create, ie. make
-j8, and m is the number of external projects.) However, it isn't
supposed to do that, and doesn't appear to be doing that right now. So
we'll remove this.
If the problem re-appears, we'll add these back in. (Symptom was being
unable to spawn /bin/sh.)
* Clean up quoting and other minor issues
* Reorganize CMake build output
Make it clearer which parts of the output pertain to external projects.
* Use a consistent ordering of ExternalProject arguments
* Prevent the top-level build from running in parallel
This prevents spurious failures when running make -j.
* Actually parse arguments in the xcodebuild function
* Use ExternalProject features when available
* submodule limits from CMake 3.0
* shallow clones from CMake 3.6
* git progress output from CMake 3.8
* Only build the parts of leveldb we need
Skip building the tools and other libraries
* Avoid installing ExternalProjects
Consume build output directly so that we can build just the targets we
need. Installing causes all targets to be built.
This doesn't matter as much for these targets but the gRPC build
includes a ton of stuff we don't need so it's worth adopting this as a
general strategy.
* Define an external build for grpc
* Test that grpc can link successfully.
* Add a FindGRPC CMake module
* Actually comment ExternalProjext_GitSource
* Port StringPrintf from //base.
Prefer this to approaches based on variadic templates. While the variadic
template mechanisms are strictly safer, they result in binary bloat we
can't afford.
This is essentially the same StringPrintf previously open sourced as a
part of protobuf, though updated for C++11 which saves a copy and a
temporary buffer on the heap.
* Add abseil as a subdirectory of Firestore
This saves having to redefine all the libraries that abseil defines as
imported libraries.
* Rename firebase_firesture_util_log_* targets
Cut the log out of the name to reflect that these will get more
components besides just logging.
* Don't bother specifying a download directory to CMake ExternalProject
* Teach CMake to build pure Xcode projects as dependencies
This allows downstream code (like log_apple.mm) to consume this for
testing within the CMake build without requiring a CMake-native build
for these components.
This makes integrating these components into the cmake build essentially
free from the point of view of the consumed component.
* Get the CMake build semi-working on Linux again
Many prebuilt versions of cmake on Linux lack the ability to download
over https so use git to get googletest.
Don't attempt to build FirebaseCore on Linux; there's no xcodebuild.
Note the build is still ultimately broken because we don't yet have an
alternative to arc4random on Linux but at least this is no more broken
than it was before.
* Use Debug as the default build type if unspecified
* Add a leveldb ExternalProject, FindLevelDB module, and use it in Firestore
* Accept an externally built leveldb via LEVELDB_ROOT
* Clone leveldb via https: instead of git:
* Remove extra DOWNLOAD_DIR which doesn't work with GIT sources
