Commit Graph
37 Commits
Author SHA1 Message Date
Maksym Malyhin bde8e07844 GDT: move some headers from public to internal (#6643)
* Move extra public headers to Internal and fix imports

* Cleanup

* Use relative paths in internal headers

* Fix imports in internal headers

* ./scripts/style.sh

* Fix GDTTestApp

* Fix test app

* Use real uploader for watchOS test app

* comma
2020-10-05 14:00:49 -04:00
Maksym Malyhin babd8e63bd GDT: Move GDTCORAssert.h to internal headers (#6638)
* Move GDTCORAssert.h to internal headers

* Fix import
2020-10-01 20:56:26 -04:00
Maksym Malyhin 38be2cc8c6 GDTCORDirectorySizeTracker: fix NSURL constructor (#6580)
* GDTCORDirectorySizeTracker: replace optional NSURL constructor with non-optional.

* GDTCORDirectorySizeTracker tests for the issue

* ./scripts/style.sh

* changelog
2020-09-28 11:01:32 -04:00
Maksym Malyhin 8779479510 GDTCORFlatFileStorageTest size limit tests: use timeout proportional to the number of events (#6537) 2020-09-23 13:44:20 -04:00
Maksym Malyhin cd844903ab GDT tests: Xcode 12 fixes (#6469)
* GDT tests: Xcode 12 fixes

* Silent unused warnings for weakSelf vars (for Xcode 12)
2020-09-16 10:45:31 -04:00
Maksym Malyhin 57101b28a9 GDTTransformer: fix background tasks handling (#6457)
* GDTCORTransformer: background task tests

* GDTCORTransformer: background tasks handling fix

* Copyright

* Changelog

* Comments

* rand() -> arc4random()

* GDTCORTransformer tests for default initializer

* Fix default initializer

* ./scripts/style.sh

* Pass weak application reference to the blocks
2020-09-15 11:02:48 -04:00
Maksym Malyhin a59febae5a GDT: storage size limit (#6365)
* GDTCORFlatFileStorageTest: test storage size limit

* Size limit slow implementation.

* GDTCORFlatFileStorage: prefetch file size

* GDTCORDirectorySizeCalculator introduced

* GDTCORDirectorySizeCalculator integration with GDTCORFlatFileStorage

* Storage size calculation tests

* Cleanup

* Cleanup

* API docs

* Fix type

* Tests

* Tests - don't hardcode event size

* Changelog

* Fix tests on slower CI.

* API docs

* Tests comments

* GDTCORFlatFileStorageTest failure debug

* GDTCORFlatFileStorageTest debug: make sure all generated events stored.

* GDTCORFlatFileStorageTest debug

* GDTCORFlatFileStorageTest debug: dynamically generate events

* Cleanup

* Renaming GDTCORDirectorySizeCalculator->GDTCORDirectorySizeTracker

* Size tracker lazy initialization

* Remove old code

* Size tracker methods renaming

* Cleanup

* Additional tests
2020-09-11 10:42:30 -04:00
Maksym Malyhin 97fb24e09c GDTCORClockTest: decrease expected timestamp accuracy to fix test flakes (#6330) 2020-08-25 16:19:20 -04:00
Maksym Malyhin 2683e9da53 GDTCORClock: use NSTimeZone to get a timezone with respect of daylight (#6246)
* GDTCORClock: use NSTimeZone to get a timezone with respect of daylight

* Changelog
2020-08-10 17:00:26 -04:00
Paul Beusterien 85e3354346 GDT - new public header structure (#6177) 2020-07-31 08:12:23 -07:00
Maksym Malyhin fa26512015 GDT: fix time units for the request and events uptime (#6102)
* GDTCORClock: fix uptime and kernelBootTime units

* GDTCCTNanopbHelpers: fix time units for the batch and events snapshot uptime

* Formatting

* GDTCORClock: properties rename and deprecation.

* Changelog updates

* Fix renaming issues

* Cleanup

* GDTCORClockTest: reduce timestamp comparison accuracy
2020-07-22 11:49:55 -04:00
Maksym Malyhin 744fd1865b GDTCORFlatFileStorage: keep not expired events when expired batch removed (#6010)
* GDTCORFlatFileStorage: expiration clean up tests

* GDTCORFlatFileStorage: keep not expired events when expired batch removed

* Cleanup

* Cleanup

* Comments

* Comments

* Remove additional delays in tests

* style

* Changelog
2020-07-09 17:12:54 -04:00
Maksym Malyhin 4dd48bdcd9 GDT: Retry uploading batches failed to upload before the app termination (#5942)
* GDTCCTUploaderTest: test a lost batch upload retry

* GDTCCTUploader: refactoring to prepare for lost batches uploading

* GDTCCTUploader: Upload previously batched events first

* Fix GDTCCTTestStorage

* Refactor GDTCCTUploaderTest

* GDTCORFlatFileStorageTest: batch tests

* GDTCORFlatFileStorage: refactor removeBatchWithID to prepare for eventsForbatchID

* eventsInBatchWithID implementation

* Fix tests

* Uploader fix for empty batch state

* GDTCORUploadCoordinator: high priority targets

* style

* GDTCCTUploaderTest

* GDTCCTUploaderTest: cleanup

* Cleanup and API docs.

* GDTCORUploadCoordinator: revert high priority targets

* GDTCCTUploader: make sure GDTCOREventQoSFast is part of a regular batch as well.

* Cleanup

* GDTCCTUploader: don't report ready for upload until there are events.

* GDTCCTUploaderTest: move failed batch back to the queue

* GDTCORFlatFileStorage: removeBatchWithID with deleteEvents==NO

* GDTCORFlatFileStorage: remove batch directory

* style

* GDTCORFlatFileStorage: comments and debug logs

* formatting

* GDTCCTUploaderTest: recover tests

* GDTCCTTestServer: support for async request handler

* GDTCCTUploaderTest: ongoing request tests

* GDTCORStorageProtocol: remove redundant `eventsInBatchWithID` method.

* GDTCORFlatFileStorageTest: `removeBatchWithID` tests

* Cleanup

* GDTCORFlatFileStorage: cleanup

* GDTCORFlatFileStorageTest: better coverage and cleanup

* Comments and improvements

* GDTCCTUploaderTest: reduce code duplication

* GDTCCTUploaderTest: conditions tests

* GDTCCTUploader: fix next request wait time

* run ./scripts/style.sh
2020-07-07 09:10:22 -04:00
Paul Beusterien d653c9584c Update copyright presubmit and fix 2020 copyrights (#5944) 2020-06-29 13:16:19 -07:00
Maksym Malyhin 527ce9049f Use UUID as GDTCOREvent.eventID (#5907)
* Remove unused header from FIRCoreDiagnosticsTest

* Use UUID as GDTCOREvent

* Fix analyzer issue

* run ./scripts/style.sh

* Code review comments
2020-06-25 09:46:01 -04:00
Paul Beusterien 5a5914da7e GDT Repo-relative imports and SPM (#5909) 2020-06-24 17:50:15 -07:00
Michael Haney a00703a543 Completely reimplement storage in GDT (#5877)
* Revert "Revert GDTCORStorage changes for now (#5778)"

This reverts commit 93eb6d8c48.

* Store events and implement unit tests (#5790)

Resolves many TODOs in the flat file storage system
Bug fixes found by unit tests are also included.

* Get rid of event iterators and add separate folders for batching (#5829)

* Store events and implement unit tests (#5790)

Resolves many TODOs in the flat file storage system
Bug fixes found by unit tests are also included.

* Change to explicit nil comparison

* Address test comments

- Makes all APIs async
- Add thread-safety test
- Comments on where files are stored
- Broke up the library data get/set API into two blocks

* Implement events and batch expirations (#5842)

The upload coordinator will signal an expiration check whenever the app foregrounds.

* Remove prioritizers, upload packages, data futures, and backwards compatibility (#5874)

* Remove prioritizers, upload packages, data futures, and backwards compatibility

* Update a comment and console logging

* Reset the storage for every unit test, move event generation out of core init

* Return a nil request if data is nil.

* Fix prioritizer reference in watchOS test app

* Remove unnecessary check for termination

* Give more leeway for eventID generation

Change the transport API to a GDTCORTarget
Don't continue a long test after a failure

* Return nil if an eventID couldn't be created

Fix some swift in the watchOS test app

* Fix all rawValue GDTCORTarget usage

* Update CHANGELOGs

* Remove initializer annotations (#5878)

* Remove initializer designation from GDTCOREvent

Change the target property to the target type

* Add a couple more types

* Make implicit retain explicit

Nothing I do seems to get this warning to trigger locally...

* Address loss of precision issue

* Re-enable a disabled test

Don't delete while enumerating

* Additional test tweaks and hasEvents is now on the queue and async

* Adjust GDTCCT to the new API

* Always attempt to create a path, get rid of implicit self refs

* Change all waiting for be DISPATCH_TIME_FOREVER for debugging purposes

I suspect the travis machines are just overwhelmed, since the test errors don't make any sense.

* Try using a different separator

* Enable debug logging

* Enable verbose pod lib lint output

* Handle cases where the mappingID contains path components

* Revert "Enable debug logging"

This reverts commit b3e4a6ecc5.

* Revert "Enable verbose pod lib lint output"

This reverts commit a5b54460f9.

* Revert "Change all waiting for be DISPATCH_TIME_FOREVER for debugging purposes"

This reverts commit 55e4fff5dc.

* Attempt to create a folder before writing

* Temporarily enable verbose pod lib lint, again

* Temporarily add debug logging back in

* Don't double store events

Also print out errors if there are any

* Remove debug logging

* Put debug logs back in for Crashlytics test

* Update mock APIs to match and use a real target

* remove debug logging

* Apply nullability specifiers to Crashlytics mock class
2020-06-23 16:04:18 -07:00
Michael Haney 93eb6d8c48 Revert GDTCORStorage changes for now (#5778)
* Revert "Refactor event saving and searching (#5773)"

This reverts commit a61786cbf1.

* Revert "Expand the storage protocol and provide an event iterator (#5742)"

This reverts commit 641d8531a3.

* Revert "Implement helpers for storing and finding events (#5738)"

This reverts commit 9a0aed81b0.
2020-06-09 13:55:15 -07:00
Michael Haney a61786cbf1 Refactor event saving and searching (#5773)
* Refactor event saving and searching

No more symlinks to try and speed searching. Just get a directory's contents and check the filename, which will contain metadata about the event.

* Change one string formatter

* Fix the type of some params, remove unused code

* Minor update

* Continue running after failing to process a file
2020-06-09 11:19:38 -07:00
Michael Haney 641d8531a3 Expand the storage protocol and provide an event iterator (#5742)
* Expand the storage protocol and provide an event iterator

* Remove unused variable

* Remove unused variable

* Make implicit self retain explicit

* Just check for NSFileTypeRegular instead

And fix a comment

* Missed a damned implicit self retain

* Implement some tests for the new API

* Fix yet another implicit self retain
2020-06-05 12:16:33 -07:00
Michael Haney 9a0aed81b0 Implement helpers for storing and finding events (#5738)
* Implement helpers for storing and finding events

* Fix missing imports

* Add clarifying comments and remove unneeded nullability

Also remove a test and fix the behavior of searchPathsWithEventSelector

* Added missing docs and properly implemented +reset
2020-06-03 14:10:08 -07:00
Michael Haney 1be5de4899 Provide an arbitrary data persistence mechanism on storage instances (#5637)
* Provide a library data mechanism on storage instances

This could allow most singletons to no longer need NSCoding, they just store the individual fields they need with an appropriately unique key

* Additional  tests

And no more implicit removal

* Implement read error capture

* Make test file data keys more unique

Also, there _should_ be an error reading from a non-existent file.
2020-05-19 15:29:05 -07:00
Michael Haney 7b78ac7866 Don't leak upload packages or deadlock on lifecycle events (#5382)
* Don't leak upload packages or deadlock on lifecycle events

GDTCORUploadPackages would call targetToStorage in -initWithTarget. When this method was running during app terminate, this would cause a deadlock because calling targetToStorage is re-entrant while GDTCORRegistrar is sending lifecycle events to all the instances it's tracking.

NSTimer's invalidate method is also called whenever a package is completed, retried, or expired, without regard to the presence of a package handler.

A unit test is added to ensure that GDTCORUploadPackage's are not leaked by a retain

TSAN caught no issues

* Don't mark GDTCORUploadPackage -init as unavailable
2020-04-15 13:03:18 -07:00
Michael Haney 78ee63dab3 Handle events that fail to write or are missing and add logging levels (#5357)
* Change removeItemAtURL to removeItemAtPath

It seems to work better, for reasons that aren't clear. It may have something to do with forming URLs being more strict.

* Add logging levels, fix failed write events

* Update comment
2020-04-14 12:30:29 -07:00
Michael Haney 6c14c21bc9 Merge storage refactor back to master (#5322)
* Make mph-gdtsqlite run travis (#3861)

* Implement a SQLite helper (#4075)

* Add support for in-memory and other types of stores (#4158)

* Add support for in-memory and other types of stores

* Temporarily add -x to know what's causing check.sh to fail

* Remove -x

* DO NOT MERGE TO MASTER: Remove commit math from check.sh temporarily

* Initial implementation of GDTCORDatabase (#4180)

* Initial implementation of GDTCORDatabase

* Style

* Check result of running a statement

* Simplify init

Also support >1 :memory: db

* Write test docs and fix one tsan issue (#4211)

* wip moving storage to sqlite

* Revert "wip moving storage to sqlite"

This reverts commit ee1895f1bd.

* Address tsan issue and document tests

* Address implicit self retains

Don't know why these aren't surfaced locally, I have the warning on.

* Implement bulk sql execution (#4528)

* Change custom prioritization dictionary to bytes (#4534)

* Change custom prioritization dictionary to bytes

* Fix a missed change for customBytes

* Refactor GDT's storage to be an interface (#5022)

* Refactor GDT's storage to be an interface

- Events are now removed by eventID
  - Events now generate an ID when they are alloc/inited
  - Copying events also copies their eventID
  - Event hashing takes into account the eventID
  - A static method in GDTCOREvent creates  event IDs as the concat of <session counter><event counter during session>
  - Session counters are incremented and stored in user defaults at app start
- Storage instances register themselves with GDTCORRegistrar like prioritizers and uploaders
- Storage fakes are refactored to be registered with GDTCORRegistrar
- Added some missing docs and fixed some typos

* Remove an unintended change from assert macro

* Use a variable

* Additional docs and test fixes

* Move saving the counter value to a file

* Update docs

* Rename GDTCORStorage to GDTCORFlatFileStorage (#5079)

* Rename GDTCORStorage to GDTCORFlatFileStorage

* Fix the NSCoding keys

* Fixes to get building and unit tests passing

* Address lack of thread safety affecting lifecycle tests

* Style

* Transition customPrioritizationDict to customBytes

Adds a category  on GDTCOREvent to make checking network connection info easier.

* Remove all sqlite and database stufff or the purposes of merging back to maste (#5330)

* Update .travis.yml
2020-04-09 13:14:15 -07:00
Michael Haney ea3521d263 Only delete events when package was successfully delivered (#5172)
* Only delete events when package was successfully delivered

* Add unit test to check that retrying a package delivery doesn't delete an event file
2020-03-23 14:11:40 -07:00
Michael Haney 27847eea38 Refactor event fileURLs to be derived (#5137)
* Refactor event fileURLs to be derived

Apparently, since iOS 8, it's possible for the app's bundle to change directories as often as every launch. This is a mechanism that's hopefully slightly more robust against that, as data is supposed to carry over, the UUID component of the app's absolute directory might just change.

* Convert a couple of tests, actually use a non-absolute URL in GDTCOREvent

* Make sure to use the event fileURL property as the source of truth.

Add some additional logs and improve the integration test

* Fix KVC not working for the removed property
2020-03-18 08:09:18 -07:00
Michael Haney 4ee3288747 Fix deprecation warnings by consolidating coding calls (#5131)
* Fix deprecation warnings by consolidating coding calls

Fixes #5086

* Fix all NSSecureCoding references

* Fix changed API usage
2020-03-17 19:06:05 -07:00
Paul Beusterien df21e99b7a Add Catalyst testing for GHA (#5115) 2020-03-17 07:54:34 -07:00
Michael Haney eae70b5845 Fix assertions not printing correctlty (#5026) 2020-03-06 09:10:46 -08:00
Doudou Nan dd242dc243 Add tests for GDTCORPlatform (#4972)
* add GDTCORPlatformTest

* styled

* tests changes

* add endbackgroundtest assert
2020-02-25 16:39:05 -08:00
Doudou Nan d69c642277 Delete GDTCORStoredEvent, modify apis and tests (#4880)
* Delete GDTCORStoredEvent, modify apis and tests

* styled

* Delete GDTCORDataFuture

* header changes

* add writeToURL in COREvent

* Remove dataObjectTransportBytes property in COREvent

* Set fileURL in COREvent nullable

* Change method writeToURL private and modify comments

* Modify encoding keys in GDTCORStorage and GDTCORUploadPackage

* comments and headers changes

* testMigration passed without fileURL

* Regression test passed with retrieving fileURL

* styled and added fileURL NOTNIL assert in test

* Modify backwords compatibility of GDTCORStoredEvent decoding

* Comments changes

* Add GDTCOREvent translation mapping in UploadPackage
2020-02-18 16:09:07 -08:00
Michael Haney 2e9bc30b98 Fixes appWillTerminate not running on GDTStorage's queue (#4879)
* Fixes appWillTerminate not running on GDTStorage's queue

Fixes #4546

Thanks @plu for the test case!

* Use the correct lifecycle event.

* Additional comments.
2020-02-10 13:28:27 -08:00
Michael Haney f138a75a33 Add a storage completion handler to the event transport pipeline (#4794) 2020-02-03 08:15:31 -08:00
Michael Haney ade283450d Change GDTCORUploader and GDTCORPrioritizer APIs (#4766)
from readyToUploaderWithConditions: to readyToUploadTarget:conditions: in order to allow the same object to act as an uploader for multiple targets.

similarly, uploadPackageWithConditions: to uploadPackageWithTarget:conditions:

Deletes support and tests for the FLL backend,  since it's being rolled into the CCT uploader and prioritizer objects.
2020-01-28 17:59:42 -08:00
Paul Beusterien 4a8ee3de60 Fix Catalyst define checks (#3958) 2019-09-30 16:37:21 -07:00
Michael Haney 18e2721c1d Rename GDT core libraries to GDTCOR (#3808) 2019-09-09 15:21:29 -07:00