Commit Graph
24 Commits
Author SHA1 Message Date
Visu 94ab466f50 Make the URLs used by Google Data Transport available in a Public API. (#7167) 2020-12-15 12:35:50 -08:00
Maksym Malyhin bde8e07844 GDT: move some headers from public to internal (#6643)
* Move extra public headers to Internal and fix imports

* Cleanup

* Use relative paths in internal headers

* Fix imports in internal headers

* ./scripts/style.sh

* Fix GDTTestApp

* Fix test app

* Use real uploader for watchOS test app

* comma
2020-10-05 14:00:49 -04:00
Maksym Malyhin 57101b28a9 GDTTransformer: fix background tasks handling (#6457)
* GDTCORTransformer: background task tests

* GDTCORTransformer: background tasks handling fix

* Copyright

* Changelog

* Comments

* rand() -> arc4random()

* GDTCORTransformer tests for default initializer

* Fix default initializer

* ./scripts/style.sh

* Pass weak application reference to the blocks
2020-09-15 11:02:48 -04:00
Maksym Malyhin a59febae5a GDT: storage size limit (#6365)
* GDTCORFlatFileStorageTest: test storage size limit

* Size limit slow implementation.

* GDTCORFlatFileStorage: prefetch file size

* GDTCORDirectorySizeCalculator introduced

* GDTCORDirectorySizeCalculator integration with GDTCORFlatFileStorage

* Storage size calculation tests

* Cleanup

* Cleanup

* API docs

* Fix type

* Tests

* Tests - don't hardcode event size

* Changelog

* Fix tests on slower CI.

* API docs

* Tests comments

* GDTCORFlatFileStorageTest failure debug

* GDTCORFlatFileStorageTest debug: make sure all generated events stored.

* GDTCORFlatFileStorageTest debug

* GDTCORFlatFileStorageTest debug: dynamically generate events

* Cleanup

* Renaming GDTCORDirectorySizeCalculator->GDTCORDirectorySizeTracker

* Size tracker lazy initialization

* Remove old code

* Size tracker methods renaming

* Cleanup

* Additional tests
2020-09-11 10:42:30 -04:00
Paul Beusterien 85e3354346 GDT - new public header structure (#6177) 2020-07-31 08:12:23 -07:00
Maksym Malyhin ec018dcbd7 [GDTCORUploadCoordinator startTimer]: don't restart timer if already started (#6032)
* [GDTCORUploadCoordinator startTimer]: don't restart timer if already started

* style

* `GDTCORUploadCoordinator.timer` - nullable

* GDTCORUploadCoordinator test utils thread safety.
2020-07-13 17:35:20 -04:00
Maksym Malyhin 527ce9049f Use UUID as GDTCOREvent.eventID (#5907)
* Remove unused header from FIRCoreDiagnosticsTest

* Use UUID as GDTCOREvent

* Fix analyzer issue

* run ./scripts/style.sh

* Code review comments
2020-06-25 09:46:01 -04:00
Paul Beusterien 5a5914da7e GDT Repo-relative imports and SPM (#5909) 2020-06-24 17:50:15 -07:00
Michael Haney a00703a543 Completely reimplement storage in GDT (#5877)
* Revert "Revert GDTCORStorage changes for now (#5778)"

This reverts commit 93eb6d8c48.

* Store events and implement unit tests (#5790)

Resolves many TODOs in the flat file storage system
Bug fixes found by unit tests are also included.

* Get rid of event iterators and add separate folders for batching (#5829)

* Store events and implement unit tests (#5790)

Resolves many TODOs in the flat file storage system
Bug fixes found by unit tests are also included.

* Change to explicit nil comparison

* Address test comments

- Makes all APIs async
- Add thread-safety test
- Comments on where files are stored
- Broke up the library data get/set API into two blocks

* Implement events and batch expirations (#5842)

The upload coordinator will signal an expiration check whenever the app foregrounds.

* Remove prioritizers, upload packages, data futures, and backwards compatibility (#5874)

* Remove prioritizers, upload packages, data futures, and backwards compatibility

* Update a comment and console logging

* Reset the storage for every unit test, move event generation out of core init

* Return a nil request if data is nil.

* Fix prioritizer reference in watchOS test app

* Remove unnecessary check for termination

* Give more leeway for eventID generation

Change the transport API to a GDTCORTarget
Don't continue a long test after a failure

* Return nil if an eventID couldn't be created

Fix some swift in the watchOS test app

* Fix all rawValue GDTCORTarget usage

* Update CHANGELOGs

* Remove initializer annotations (#5878)

* Remove initializer designation from GDTCOREvent

Change the target property to the target type

* Add a couple more types

* Make implicit retain explicit

Nothing I do seems to get this warning to trigger locally...

* Address loss of precision issue

* Re-enable a disabled test

Don't delete while enumerating

* Additional test tweaks and hasEvents is now on the queue and async

* Adjust GDTCCT to the new API

* Always attempt to create a path, get rid of implicit self refs

* Change all waiting for be DISPATCH_TIME_FOREVER for debugging purposes

I suspect the travis machines are just overwhelmed, since the test errors don't make any sense.

* Try using a different separator

* Enable debug logging

* Enable verbose pod lib lint output

* Handle cases where the mappingID contains path components

* Revert "Enable debug logging"

This reverts commit b3e4a6ecc5.

* Revert "Enable verbose pod lib lint output"

This reverts commit a5b54460f9.

* Revert "Change all waiting for be DISPATCH_TIME_FOREVER for debugging purposes"

This reverts commit 55e4fff5dc.

* Attempt to create a folder before writing

* Temporarily enable verbose pod lib lint, again

* Temporarily add debug logging back in

* Don't double store events

Also print out errors if there are any

* Remove debug logging

* Put debug logs back in for Crashlytics test

* Update mock APIs to match and use a real target

* remove debug logging

* Apply nullability specifiers to Crashlytics mock class
2020-06-23 16:04:18 -07:00
Michael Haney 93eb6d8c48 Revert GDTCORStorage changes for now (#5778)
* Revert "Refactor event saving and searching (#5773)"

This reverts commit a61786cbf1.

* Revert "Expand the storage protocol and provide an event iterator (#5742)"

This reverts commit 641d8531a3.

* Revert "Implement helpers for storing and finding events (#5738)"

This reverts commit 9a0aed81b0.
2020-06-09 13:55:15 -07:00
Michael Haney a61786cbf1 Refactor event saving and searching (#5773)
* Refactor event saving and searching

No more symlinks to try and speed searching. Just get a directory's contents and check the filename, which will contain metadata about the event.

* Change one string formatter

* Fix the type of some params, remove unused code

* Minor update

* Continue running after failing to process a file
2020-06-09 11:19:38 -07:00
Michael Haney 641d8531a3 Expand the storage protocol and provide an event iterator (#5742)
* Expand the storage protocol and provide an event iterator

* Remove unused variable

* Remove unused variable

* Make implicit self retain explicit

* Just check for NSFileTypeRegular instead

And fix a comment

* Missed a damned implicit self retain

* Implement some tests for the new API

* Fix yet another implicit self retain
2020-06-05 12:16:33 -07:00
Michael Haney 9a0aed81b0 Implement helpers for storing and finding events (#5738)
* Implement helpers for storing and finding events

* Fix missing imports

* Add clarifying comments and remove unneeded nullability

Also remove a test and fix the behavior of searchPathsWithEventSelector

* Added missing docs and properly implemented +reset
2020-06-03 14:10:08 -07:00
Doudou Nan 68fb6624f4 Expand watchOS support for GoogleDataTransport (#5443)
* Add watchOS support for GoogleDataTransport with specifying notification center and WKExtensionDelegate for watchOS application, handling background tasks and network reachability flags on watchOS.

* Add independent watchOS test app for GoogleDataTransport and GoogleDataTransportCCTSupport.

* Add companion watchOS test app and notification service extension for GoogleDataTransportCCTSupport.
2020-04-24 15:19:43 -07:00
Michael Haney 7b78ac7866 Don't leak upload packages or deadlock on lifecycle events (#5382)
* Don't leak upload packages or deadlock on lifecycle events

GDTCORUploadPackages would call targetToStorage in -initWithTarget. When this method was running during app terminate, this would cause a deadlock because calling targetToStorage is re-entrant while GDTCORRegistrar is sending lifecycle events to all the instances it's tracking.

NSTimer's invalidate method is also called whenever a package is completed, retried, or expired, without regard to the presence of a package handler.

A unit test is added to ensure that GDTCORUploadPackage's are not leaked by a retain

TSAN caught no issues

* Don't mark GDTCORUploadPackage -init as unavailable
2020-04-15 13:03:18 -07:00
Michael Haney 6c14c21bc9 Merge storage refactor back to master (#5322)
* Make mph-gdtsqlite run travis (#3861)

* Implement a SQLite helper (#4075)

* Add support for in-memory and other types of stores (#4158)

* Add support for in-memory and other types of stores

* Temporarily add -x to know what's causing check.sh to fail

* Remove -x

* DO NOT MERGE TO MASTER: Remove commit math from check.sh temporarily

* Initial implementation of GDTCORDatabase (#4180)

* Initial implementation of GDTCORDatabase

* Style

* Check result of running a statement

* Simplify init

Also support >1 :memory: db

* Write test docs and fix one tsan issue (#4211)

* wip moving storage to sqlite

* Revert "wip moving storage to sqlite"

This reverts commit ee1895f1bd.

* Address tsan issue and document tests

* Address implicit self retains

Don't know why these aren't surfaced locally, I have the warning on.

* Implement bulk sql execution (#4528)

* Change custom prioritization dictionary to bytes (#4534)

* Change custom prioritization dictionary to bytes

* Fix a missed change for customBytes

* Refactor GDT's storage to be an interface (#5022)

* Refactor GDT's storage to be an interface

- Events are now removed by eventID
  - Events now generate an ID when they are alloc/inited
  - Copying events also copies their eventID
  - Event hashing takes into account the eventID
  - A static method in GDTCOREvent creates  event IDs as the concat of <session counter><event counter during session>
  - Session counters are incremented and stored in user defaults at app start
- Storage instances register themselves with GDTCORRegistrar like prioritizers and uploaders
- Storage fakes are refactored to be registered with GDTCORRegistrar
- Added some missing docs and fixed some typos

* Remove an unintended change from assert macro

* Use a variable

* Additional docs and test fixes

* Move saving the counter value to a file

* Update docs

* Rename GDTCORStorage to GDTCORFlatFileStorage (#5079)

* Rename GDTCORStorage to GDTCORFlatFileStorage

* Fix the NSCoding keys

* Fixes to get building and unit tests passing

* Address lack of thread safety affecting lifecycle tests

* Style

* Transition customPrioritizationDict to customBytes

Adds a category  on GDTCOREvent to make checking network connection info easier.

* Remove all sqlite and database stufff or the purposes of merging back to maste (#5330)

* Update .travis.yml
2020-04-09 13:14:15 -07:00
Michael Haney aeb4def61d Libraries should save state if a completion block is requested (#5147)
* Libraries should save state if a completion block is requested

Currently, the onComplete block of sendXEvent:onComplete: is invoked when the event bytes are written to disk. However, if the app crashes before a lifecycle event occurs (trigger GDTCORStorage to serialize to disk), that event if effectively lost because the metadata has all been lost. This change triggers the saving of state of GDTCORStorage and prioritizers (if they implement it) if an onComplete block has been given.

* Remove flaky check

* Update GDTCORStorage.m
2020-03-18 15:00:37 -07:00
Michael Haney 27847eea38 Refactor event fileURLs to be derived (#5137)
* Refactor event fileURLs to be derived

Apparently, since iOS 8, it's possible for the app's bundle to change directories as often as every launch. This is a mechanism that's hopefully slightly more robust against that, as data is supposed to carry over, the UUID component of the app's absolute directory might just change.

* Convert a couple of tests, actually use a non-absolute URL in GDTCOREvent

* Make sure to use the event fileURL property as the source of truth.

Add some additional logs and improve the integration test

* Fix KVC not working for the removed property
2020-03-18 08:09:18 -07:00
Michael Haney 1cca0bcd8c Make GDTCORReachability a public API for support libraries (#4969) 2020-02-25 11:39:27 -08:00
Doudou Nan d69c642277 Delete GDTCORStoredEvent, modify apis and tests (#4880)
* Delete GDTCORStoredEvent, modify apis and tests

* styled

* Delete GDTCORDataFuture

* header changes

* add writeToURL in COREvent

* Remove dataObjectTransportBytes property in COREvent

* Set fileURL in COREvent nullable

* Change method writeToURL private and modify comments

* Modify encoding keys in GDTCORStorage and GDTCORUploadPackage

* comments and headers changes

* testMigration passed without fileURL

* Regression test passed with retrieving fileURL

* styled and added fileURL NOTNIL assert in test

* Modify backwords compatibility of GDTCORStoredEvent decoding

* Comments changes

* Add GDTCOREvent translation mapping in UploadPackage
2020-02-18 16:09:07 -08:00
Michael Haney f138a75a33 Add a storage completion handler to the event transport pipeline (#4794) 2020-02-03 08:15:31 -08:00
Chen Liang e4456c011b Enables Firebase Messaging push notification function on watch only and independent watch app (#4016) 2019-12-23 16:45:19 -08:00
Ryan Wilson 03ecafd06d Refactor GDT Backgrounding (#3896)
* Fix missing `runningInBackground` flags.

* Attempt at fixing background issues with GDT.

* Add missing deprecated implementation.

* Transfer extra call to background call.

* Style

* Further review comments.

* Fix testing.

* Refactored FLLIntegrationTest.

This does a few things - split the tests into a specific multithreaded
test and a recursive run without crashing.

It also adds a notification for the upload to post how many events were
uploaded. This should be a bit more reliable and helps the overall
validation during the "run without crashing" test.

* Removed extra log.

* Fix flag for Catalyst.

* Disable currently non-functional test.

* Further review comments.

* Migrate CCT tests to match FLL.

* Revert accidental partial change.

* Clarify comment for starting the background tasks.

* Review feedback, changelog additions and bug numbers.
2019-10-02 10:40:45 -04:00
Michael Haney 18e2721c1d Rename GDT core libraries to GDTCOR (#3808) 2019-09-09 15:21:29 -07:00