Summary: 714 instances, 655 unique Text Count // TODO not really in use as of now. may need in some failure cases. Revisit and remove if no plausible use 1 // TODO: Remove the constraint SERVER_KNOBS->DESIRED_TEAMS_PER_SERVER == 3 to ensure that 1 // TODO: Replace the heartbeat mechanism with FDB failure monitoring mechanism 1 // FIXME: don't fetch tag localities, all tags, and history tags if tss. Just fetch pair's tag 1 // FIXME: enable 1 // TODO: should we throw the cold_cache_server() error here instead? 1 // FIXME: SevWarnAlways trace if "close" to out of memory 1 // TODO: Consider gracefully abort a restore and restart. 1 //TODO Everything in this class before here is in the doc. 1 // TODO REMOVE with V2 of protocol 2 // TODO: A Standalone key type isn't ideal because on lookups a ref will be made Standalone just for the search 1 when(wait(dbInfoChange)) { // FIXME: does this actually happen? 1 wait(delay(0, TaskPriority::Low) || delay(SERVER_KNOBS->COMMIT_SLEEP_TIME)); // FIXME: Is this still right? 1 false); // FIXME: This seems impossible and doesn't seem to be covered by testing 1 // TODO: Remove this once memroy radix tree works as a log engine 1 ** TODO: How should dropping a table be handled? How should renaming a 1 // FIXME: these shards are being inserted into the map sequentially, it would be much more CPU 1 // TODO: `issues` is right now only updated by `monitorTraceLogIssues` and thus is being `set` on every update. 1 // TODO : find why this not work 1 // FIXME: prevent from calling this from another main thread! 1 // FIXME: doesn't work once we add granule merging, could be multiple ranges and/or multiple parents 1 // TODO: check transaction size here 1 // TODO get rid of this and do Reference>? 1 // FIXME: could binary search? 1 // TODO: Many functions copied from bindings/flow/Tuple.cpp. Merge at some point. 1 // FIXME: The logic to pick a satellite in a remote region is not 1 // TODO: rename this struct 1 // TODO find out more 1 // TODO: Adapt the simulation framework to not advance time quickly when background reads/writes are occurring. 1 ** TODO: This function allocates a single block of memory to load 1 removals[i]); // FIXME: Make PTreeImpl::insert do this automatically (see also VersionedMap.h FIXME) 1 // TODO: NEELAM: what exactly does it do? Writing some mutations to log. Do we need it for caches? 1 // TODO: inject read error 1 if (maxTLogQueueInfo.first > 1e5) // FIXME: Should be zero? 1 // FIXME: test unset dcID 1 // TODO: inject write error 1 populateAtomicOpMap(); // FIXME: NOOOOOO! 1 // TODO These five methods are definitely in the recipe book 1 // FIXME: remove when we no longer support upgrades from 5.X 2 // TODO change in V2 1 // FIXME: TLog recruitment process does not guarantee the best fit is not worsened. 1 // TODO: double check the following block of code!! 1 return ConfigurationResult::INCOMPLETE_CONFIGURATION; // FIXME: correct return type 1 // TODO double check if it's okay to let go of the the insert version of the "right half" 1 // FIXME: does not support cancellation 1 // TODO: caching disabled for this merge 1 ** TODO: collapse this trunk into a neighbor if searchList? 1 // FIXME: log replicas must be more than storage replicas because otherwise better master exists will not 1 // TODO: do we need to check changeCounter here? 1 tokencmp(tokens[0], "getrangekeys")) { // FIXME: support byte limits, and reverse range reads 1 // TODO The next four elements are in the recipe book: 1 // TODO Note that, we are currently not handling the case when cached key-ranges move out 1 // TODO: why is this seemingly random delay here? 1 // TODO storageCache server shares quite a bit of storageServer functionality, although simplified 1 // TODO : add limitation on set operation 1 // TODO: This needs to fiddle with g_currentDeliveryPeerAddress 1 // TODO: map is not supported in FakeFDBTransaction yet. 1 // TODO: Decide if it is okay to update if the subtree boundaries are expanded. It can result in 1 // TODO add comments + documentation 2 // FIXME: this will peeking way too many cursors when satellites exist, and does not need to peek 1 // TODO: Remove this once Redwood works as a log engine 1 /* TODO: Check if all of these are really required. */ 1 // TODO REMOVE sanity checks! 1 sampleBatch.push_back_deep(sampleBatch.arena(), samples[i]); // TODO: may not need deep copy 1 // TODO: create a execution context abstraction that allows independent flow instances within a process. 1 // FIXME: Add the knownCommittedVersion to this message and change ratekeeper to use that version. 3 // TODO: Only mapped keys are returned, which are not supposed to be in the range. 1 // TODO: remove dependency on fdbrpc. 1 // TODO This is the equivalent of the "with" block in the 1 // FIXME: limits are not implemented yet, and this code has not be tested with reverse=true 1 // FIXME: More efficient 1 // TODO: Use toml to config 1 // FIXME: enable when debugKeyRange is active 1 // TODO: If it turns out this struct only has appliers in the final version, we will rename it to a more specific name, 1 // TODO: Don't do this hacky investigation-specific thing 1 // TODO: Validate that the range version map has exactly the restored ranges in it. This means that for any 1 // TODO: Do something to read and initialize the database in case 1 // TODO: Make this an ordered container some day. 1 // FIXME: The commented code below could be adapted to find the actual stack size using pthreads 1 // FIXME: ensure this can only read data from the current version 1 // FIXME: instead of using random chance, send wrong_shard_server when the call in from 1 /* TODO: look at how perl does stat (non-sloppy), unlink (ro-files), utime, link */ 1 // TODO Revisit during failure handling. Might we loose some private mutations? 1 // FIXME: Error handling 1 printf("TODO: Check if restore can succeed if dirty restore is performed first\n"); 1 // TODO probably should add things like limit/bytelimit at some point? 1 # FIXME: Should we read unpacked after getting the result of the packed data? If we do, then we can more accurately limit the number 1 oldLogSystem->getEnd()); //< FIXME: (1) the log adapter should do this automatically after recovery; (2) if we 1 // FIXME: This will break for DynamicEventMetric instances that are reused, such as use cases outside 1 // FIXME: g_simulator.datacenterDead could return false positives. Relaxing checks until it is fixed. 1 // TODO: Load balance 4 // FIXME: we can remove this wait once we are able to have log routers which can ship data to the remote 1 /* FIXME: this needs to be true for better tracking memory use in our 1 // FIXME: this code will not work for caching a direct serialization from ObjectWriter, because it adds an ErrorOr, 1 /* TODO: update */ 1 // TODO: Enable this function in correctness test 1 // FIXME: Return a fixed point Time class 1 std::map> server_and_tss_info; // TODO could replace this with an efficient way to do a read-only concatenation of 2 data structures? 1 } else if (mode == "remote_three_data_hall") { // FIXME: not tested in simulation 1 // FIXME: buggify byte limits on internal functions that use them, instead of globally 2 } while (endKey() == originalEnd); // TODO: pointer only comparison; maintain equality of pointers to keys 1 // FIXME: this does all logic in one transaction. Adding a giant range to an existing database to blobify would 1 // TODO: Sanity check restoreRequests' key ranges do not overlap 1 // TODO: Get rid of this by constexpr or other template magic in getRange 1 // FIXME: Start workers with tester class instead, at least sometimes run tests with the testers-only flag 1 // TODO These two methods--the get and set methods--are in the recipes. 1 // FIXME: Initializing here causes simulation issues, these must be fixed 1 // TODO do we need otherError here? 1 state std::vector> fReplies(numRoles, Never()); // TODO: Reserve memory for this vector 1 // FIXME: this is a tacky workaround for a potential uninitialized use in trackTLogQueueInfo (copied from 1 // TODO: How should this cache be sized - not really a cache. it should hold all extentIDs? 1 // FIXME: run the actual FDBCLI command instead of copy/pasting its implementation 1 // TODO REMOVE in api V2 1 // TODO double check 1 // TODO: getHistory? 1 // TODO: re-enable the following logging or remove them. 1 // FIXME: is this the correct set of errors? 1 // FIXME: with low probability, could immediately revoke it from the new assignment and move it back 1 // FIXME: if we're still reading from old change feed, we should probably compact if we're making a 1 // TODO maybe this won't be true eventually, but right now the only time the blob worker reports back is 1 // TODO could skip this whole thing if tss if request counter is zero? 1 // TODO: When n is large, split into multiple transactions. 1 // FIXME: this could throw connection_failed and we could handle catch this the same as the failure 1 // TODO: NEELAM: what's this for? 1 // TODO look into making this per-version compaction. (We could keep track of updated nodes at each version for example) 1 // TODO this is all duplicated from above, simplify? 1 // FIXME: do skip pointers with single back-pointer and neighbor pointers 1 // TODO: Check to see if there is a leak in the FutureBucket since an invalid task (validation key fails) will 1 // TODO: To support restore even after we attempt dirty restore. Not implemented in the 1st version 1 // FIXME: defer sending the message until we know the connection is established 1 keyAfter(result[result.size() - 1].key))); // TODO : the keyAfter will just return if key == \xff\xff 1 // TODO: Version Tracking. If the bug is in handling a version rather than a key, then it'd be good to be able to log 1 // TODO: Should only call ingest when deleting from memory 1 // FIXME: typedef this pair type and/or chunk list 1 // TODO what flowlock to acquire during compaction? 1 // FIXME: do trivial key bounds here if key range is not fully contained in request key range 1 // FIXME: something like BinaryReader(ar) >> arena >> *(T*)this; to guarantee standalone arena??? 1 // FIXME: use the configuration to calculate a more precise minimum recovery version. 1 // TODO: Should this just truncate? 1 // TODO: check whether this is reasonable to do in the fast path 1 // TODO: insert recs in a random order to avoid new subtree being entirely right child links 1 FDB_DECLARE_BOOLEAN_PARAM(DstOnly); // TODO: More descriptive name? 1 // TODO REMOVE sanity check 1 // FIXME: handle errors here (forward errors) 1 // FIXME: Sanity checking 1 ** TODO: Add a test to make sure that the same column is not named 1 // TODO: Allow empty files in version batch; Filter out them here. 1 int8_t popLocality) { // FIXME: do not need to pop all generations of old logs 1 // TODO: how to handle this? 1 // TODO: this could read from FDB instead if it knew there was a large range clear at the end or 1 // TODO: Also fetch other linked subtrees of meta keys 1 // TODO: Make hint-based seek() use the hint logic in this, which is better and actually improves seek times, 1 // TODO: change the following back to `private` once blob obj access is refactored 1 // TODO: Use a lower level API may be better? Or tweak priorities? 1 // FIXME: It's not straightforward to rollback certain versions from the VersionedMap. 1 // TODO could add team check logic in if we added synchronous way to turn this into a fixed getRange request 1 // TODO The recipe book definitely includes the following things: 1 // FIXME: What if we are using way too much memory for buffer? 1 // - TODO: More specializations for common policies 1 // TODO probably knobs here? This should always be pretty small though 1 // TODO is this something we want? 1 // FIXME: load balance holds pointers to individual members of the interface, and this assignment will 1 // FIXME: not true for merges 2 // TODO: bias for immediately re-reading to catch rollback cases 1 // FIXME: better way to convert here? 1 // TODO: Could optimize this 1 // TODO: Check Tlog 1 // TODO: Load balance to avoid always hitting the 1 // TODO These three methods, xyToZ, zToXy, and setLocation, are allin the recipe book. 1 } // TODO: else trace a warning 1 // FIXME: remove when we no longer need to test upgrades from 4.X releases 3 self->quiescentWaitTimeout)); // FIXME: should be zero? 1 // FIXME: better handling to disable row limits 1 // TODO: refactor to not duplicate IBackupContainer::openContainer. It's the exact same 1 Params.beginVersion().set(task, 1); // FIXME: remove in 6.X, only needed for 5.2 backward compatibility 1 // TODO: If better performance is needed, make AsyncFileReadAheadCache work and be enabled by SQLITE_OPEN_READAHEAD 1 when(wait(delay(5.0))) {} // TODO REMOVE this once empty version logic is fully implemented 1 // FIXME: push shouldn't return anything. We should call getNextPushLocation() again. 2 // TODO: Consider clearrange traffic in write traffic control 1 // TODO: Support addPrefix and removePrefix 1 result.more = limit == 0 || *pLimitBytes <= 0; // FIXME: Does this have to be exact? 2 // TODO: Add support for longer maps if necessary. 1 // TODO: Get actual max MetaKey size limit from Pager 1 // FIXME: add counter for granule rollbacks and rollbacks skipped? 1 // TODO: Once seekLessThanOrEqual() with a hint is as fast as seekLessThanOrEqualOld, remove it. 1 // FIXME: could submit multiple chunks to start_load_f in parallel? 1 // TODO: Is there a more efficient way to deal with invalidation? 1 wait(delay(0.1)); // TODO KNOB? 1 // FIXME: let the caller pass in where to write the data. 1 /* FIXME: issue with large values because of result saturation */ 1 // FIXME: also have these be async, have each pop change feed wait on the prior one, wait on them before 1 // TODO: refactor this to separate out the "deal with blob store" stuff from the backup business logic 1 // TODO use flatbuffers depending on version 1 // TODO: magic so this works even if const-ness doesn not match. 1 // TODO: When n is large, split into multiple transactions. 2 // TODO : replace using ExclusionInProgressRangeImpl in special key space 1 // TODO: in the future, run a check based on estimated data size like the existing storage servers' 1 // TODO this should never happen? 1 (defined(__s390x__) && defined(__VEC__)) && defined(__GNUC__) /* TODO: IBM XL */ 1 // FIXME: this conflict range is too large, and should be updated continously as results are returned 1 // TODO unit test this assignment, particularly out-of-order insertions! 1 ** (WARNING/TODO: If P1 is a pseudo-cursor and P2 is dynamically 1 // FIXME: This code is relying for liveness on an undocumented property of the log system implementation: that after 1 // FIXME: can this safely avoid another main thread jump? 1 try { // FIXME: This field was added in a patch release, the try-catch can be removed for the 7.0 release 1 // TODO: Not resolving symbolic links does not yet behave well on Windows because of drive letters 1 // FIXME: we only need log routers for the same locality as the master 1 // TODO: disable completion 1 // FIXME: does not take into account fallback satellite policies 1 // FIXME: This implementation isn't actually asynchronous - it just does operations synchronously! 1 // TODO change to 100e8 after figuring out no-disk redwood mode 1 // TODO These four lines are the equivalent of the initialization 1 // FIXME: match order table with inventory table 1 // TODO: iterator construction, other constructors 1 // TODO might be more efficient to mark a boolean on DiskStore in getDiskStores, but that kind of breaks 1 // TODO double check if the insert version of the previous clear needs to be preserved for the "left half", 1 // FIXME: More efficient? Eliminate? 1 e.key); // FIXME: Make PTreeImpl::insert do this automatically (see also VersionedMap.h FIXME) 1 // TODO: Move this to a flow header once it is mature. 1 // FIXME: enable disk stalls 1 // TODO: Use SplitStringRef (unless it ends up being slower) 1 // FIXME: refactor out the parts of this that are just for interacting with blob stores from the backup business 1 // TODO Everything in this class up until the few methods at the end 1 Error e = internal_error(); // TODO: Something better? 1 // TODO: We need to wait for all the responses to come in 1 // throw platform_error(); // FIXME: How to recover from this situation? 1 // TODO: Add SFINAE overloads for functors returning void or a non-Future type. 1 // FIXME: determine when to fail the cluster controller when a primaryDC has not been set 1 // FIXME: currently, neither of bytesPerKSecond or iosPerKSecond are actually used in DataDistribution calculations. 1 // TODO: update this map upon file writes. 1 // FIXME: Maintain a heap of tags ordered by version to make this O(1) instead of O(n). 2 // FIXME: Where is this intended to be used? 1 // TODO: moving_data["rate_bytes"] = makeCounter(hz, c, r); 1 // FIXME: Trim this down 1 } // FIXME: check that this is triggered when the range is moved to a different 1 // FIXME: Merge this function with IAsyncFileSystem::incrementalDeleteFile(). 1 // FIXME: Should we suppress this if there are cluster messages implying that the database has no 1 init( BG_URL, "" ); // TODO: store in system key space, eventually 1 // TODO These three methods (setUser, getUser, and getUserIDsInRegion) 1 // TODO: Move the log file and range file format encoding/decoding stuff to this file and behind interfaces. 1 // TODO change back from SevError? 1 // TODO: Sanity check each asset has been received exactly once! 1 .castTo()) { // FIXME: Make this check more specific, here or by reading 1 // FIXME: this will not catch if the secondary address of the process was excluded 1 // TODO: Read from coordinators for more up-to-date config database status? 1 // FIXME: Optimistically assume that this will not run into the system keys, and only reissue if the result 2 // TODO: This assertion can cause flaky behaviour since sometimes a watch can take longer to fire 1 // TODO change it to a deque, already contains mutations in version order 1 // TODO is it desired that not all records are committed? This could commit again to ensure any records set() since 1 // TODO: Implement this 1 // FIXME: metadata in queue? 4 // TODO: Allow loading both range and log files in parallel 1 // FIXME: do not add data in flight to servers that were already in the src. 1 // TODO: If cursor hints are available, use (cursor, 1) 1 // FIXME: binary search instead of linear search, especially when file count is large 1 wait(delay(1.0)); // FIXME: WHY!!! //wait for machines to boot 1 exitCode = 2181; // FIXME: WHAT? 1 // TODO: Document further. 1 // TODO could improve this check by also ensuring DD is currently recruiting a TSS by using quietdb? 1 // FIXME: maybe just want to insert here if there are racing queries for the same worker or something? 1 // TODO is this explicit conflit range necessary with the above read? 1 // FIXME: once we have empty versions, only include up to startState.changeFeedStartVersion in the read 1 /* TODO: What if the cursor is in CURSOR_REQUIRESEEK but all table entries 1 // TODO: make an easy to use Future fileExists, and port lots of code over to it. 1 if (maxThrottleDelay > 0.0 && !canRecheck) { // TODO: allow delaying? 1 // TODO: Is it contractually okay to always to read at the latest version? 1 // TODO: Look at the last LogMessageVersion, if it set or clear, we can ignore the rest of versions. 1 // TODO: Defining these here is just asking for ODR violations. 1 // FIXME: This seems pretty horrible. Now a Database can't die until all of its watches do... 1 // FIXME: Limit to approximately DESIRED_TOTATL_BYTES somehow. 2 // FIXME: update tlogMetrics to include new information, or possibly only have one copy for the shared instance 4 // TODO: Figure out what is broken with the RocksDB engine in simulation. 1 // FIXME: probably want to chunk this up with yields to avoid slow task for blob worker re-snapshotting by calling the 1 /* TODO: replace above code with this once blob manager recovery is handled 1 // TODO: Use std::get_time implementation for all platforms once supported 1 // TODO there is probably a better way to do this 1 // FIXME: does not use testers which are recruited on workers 1 // TODO: this is a temporary fix, getting the number of available coordinators should move to 1 // TODO: Possibly limit size of remap queue since it must be recovered on cold start 1 // TODO for large amount of chunks, this should probably have some sort of buffer limit like ReplyPromiseStream. 1 // TODO: After 6.3, review FR knobs, remove unneeded ones and change default value 1 // TODO not really in use as of now. may need in some failure cases. Revisit and remove if no plausible use 1 it = parent->entries.lastLessOrEqual(Entry(key, key, VectorRef())); // TODO: finger query? 1 // TODO could refactor the file reading code from here and the delta file function into another actor, 1 // TODO: time_in_recovery: 0.5 1 // FIXME: Is this the right behavior? dataMovementComplete will never be sent! 1 // TODO: More descriptive error? 1 // TODO: Update the function to get rid of the Database after refactoring 1 // TODO: Add a knob for the block cache size. (Default is 8 MB) 1 // TODO: Compare all fields for loadingParam 1 // TODO: should we throw the cold_cache_server() error here instead? 1 // FIXME: this should invalidate the location cache for cacheServers 1 // TODO would it be better to put all of them in primary DC? 1 // FIXME: if range is assign, have some sort of semaphore for outstanding assignments so we don't assign 1 // FIXME: enable when debugKeyRange is active 2 // TODO: Share arena 2 // TODO double check which ones we need for storageCache servers 2 // TODO This method in recipe book. 1 // TODO The next six methods are all in the recipe book: 1 // TODO revisit- we are not splitting the cached ranges based on shards as of now. 1 // TODO: Use IPAddress::parse 1 else if (mutation.type != MutationRef::ClearRange) { // TODO NEELAM: ClearRange mutations are ignored (why do we 1 kevent(list, &ev, 1, nullptr, 0, nullptr); // FIXME: check? 2 // FIXME: Get worker ID here 1 // TODO: Stop monitoring and close the connection with no onDisconnect requests outstanding 1 int TOO_MANY; // FIXME: this should really be split up so we can control these more specifically 1 // TODO: this needs to be more clever in the future 1 ** buffer using memcpy(). TODO: Add a way to transfer control of the 1 // TODO: Change to my restore agent code 1 int killType = deterministicRandom()->randomInt(0, 3); // FIXME: enable disk stalls 1 // FIXME: Should log this if the recruit count stays the same but the other numbers update? 1 FIXME: Use an actual platform abstraction for this stuff! Is there any reason we can't use underlying net2 for 1 // FIXME: add above as details 1 // FIXME: is the merge case needed 1 // Delegate everything else. TODO: Make a base class NetworkWrapper for delegating everything in INetwork 1 // TODO understand this, should we add delay here? 1 // TODO: mimic what StorageServerInterface does with sequential endpoint IDs 1 // TODO should we store the logProtocol? 1 float32x2_t a02 = vset_lane_f32(a0, a22, 1); /* TODO: use vzip ?*/ 1 // FIXME: forward errors somewhere from here 1 deterministicRandom()->random01() < 0.25) { // FIXME: why doesnt this work when we are changing machines? 1 // TODO: Remove later, this is just to force the statics to be initialized 1 init( MAX_TL_SS_VERSION_DIFFERENCE, 1e99 ); // if( randomize && BUGGIFY ) MAX_TL_SS_VERSION_DIFFERENCE = std::max(1.0, 0.25 * VERSIONS_PER_SECOND); // spring starts at half this value //FIXME: this knob causes ratekeeper to clamp on idle cluster in simulation that have a large number of logs 1 // FIXME: killDataCenter breaks simulation if forceKill=false, since some processes can survive and 1 // TODO probably other things here eventually 1 // FIXME: eventually we probably want to cache this and invalidate similarly to storage servers. 1 state Future ccMonitor = monitorServerDBInfo(cc, LocalityData(), dbInfo); // FIXME: locality 1 // TODO: Do this more efficiently, as the range file list for a snapshot could potentially be hundreds of 1 // FIXME: This check fails for us, probably it is a (performance) bug! 1 // TODO: Somehow check / report if the restorable range is not or may not be contiguous. 1 // TODO: Change AsyncTrigger to a Reference 1 // TODO: Sanity check the log file's range is overlapped with the restored version range 1 // TODO: Reenable the sanity check after TooManyFiles error is resolved 1 // TODO: If the file exists but appears to never have been successfully committed is this an error or 1 // TODO : fix knobs for row_unlimited 1 // TODO: filter if pipelined with large commit 1 // FIXME: don't do the delay in setup, as that delays the start of all workloads 1 * FIXME: Clang's output is still _much_ faster -- On an AMD Ryzen 3600, 1 // FIXME: Remove this ASSERT when popping into uncommitted pages is fixed 1 // FIXME: snapshot restore does not support multi-region restore, hence restore it as single region always 1 cursor->clear(a.range); // TODO: at most one 1 // FIXME: Check that eventCount < MAXIMUM_WAIT_OBJECTS, here and/or in loadConf 1 // TODO: register watch first and then check if the key exist 1 // FIXME: this is hiding the fact that we can recruit a new storage server on a location the has 1 // FIXME: both versions of erase are broken, because they do not remove items in the queue with will subtract a 1 /* TODO: This definition is just included so other modules compile. It 2 // FIXME: Is setting lastCommittedSeq to -1 instead of 0 necessary? 1 // FIXME: check if there are tagged logs. for each tag, there is no version gap. 1 popActors.add(popFromLog(this, log, tag, 1.0)); //< FIXME: knob 1 // TODO: Check why this may happen?! 1 writer->addref(); // FIXME: We are not shutting down the writer thread at all, because the ThreadPool 1 // TODO: Verify reads, do more writes and reads to make this a real pager validator 1 // TODO This loop in recipe book. 1 // FIXME: lookup both the src and dest shards as of the pop version to ensure all locations are popped 1 // FIXME: use actual 16 bytes of UID instead of converting it to 32 character string and then that 1 // TODO: Currently backup and restore related simulation tests are failing when run with rocksDB storage engine 1 u.skipLen = 0; // TODO: set this 1 // FIXME: this is a tacky workaround for a potential uninitialized use in trackStorageServerQueueInfo 1 // TODO unused, remove? 1 // TODO cache servers should write the LogProtocolMessage when they are created 1 // TODO: Set FASTRESTORE_WAIT_FOR_MEMORY_LATENCY to a large value. It should be able to avoided 1 .arena()); // FIXME: Not sure what this line is for, is it necessary? 1 ASSERT(g_network->isSimulated()); // TODO: Enforce elsewhere 1 // FIXME: getNextReadLocation should ASSERT( initialized ), but the memory storage engine needs 1 // TODO: Enable this when Azure backups are ready 1 // TODO change this for merge 1 // TODO: Support using this option to determine atomicity 2 // TODO: In the event that the client times out just before the request completes (so the client is unaware) then 2 self->extentUsedList.numEntries); // TODO this is overreserving. is that a problem? 1 // FIXME: configure url with database configuration instead of knob eventually 1 // TODO: We have not implemented the code commented out in this case 1 // FIXME: could change this to O(N) instead of O(NLogN) by doing a sorted merge instead of requesting the 1 // TODO: Merge this RestoreConfig with the original RestoreConfig in FileBackupAgent.actor.cpp 1 // TODO : give this function a more descriptive name 1 // TODO: There should be an easier way to identify the difference between flow_ref and non-flow_ref types. 1 // TODO: improve the interface of this function so that it doesn't need 1 // TODO These next five methods (setEdge, deleteEdge, getOutNeighbors, getInNeighbors, 1 // TODO revisit the following block of code 1 // FIXME I think there's something with nextReadLocation we can do here when initialized && !recovered. 1 // FIXME: logAntiQuorum not supported, remove it later, 1 // FIXME: use streaming range read 1 // FIXME: original Epoch/Seqno is now not necessary with originalReq 1 writing.send(Void()); // FIXME: This could happen before the wait if AsyncFileKAIO dealt properly with 1 // TODO: fast load balancing? 1 // TODO: Register restore performance data to restoreStatus key 1 // TODO: Reenable this when running the ConfigIncrement workload with reboot=false 1 // TODO could do a reply promise stream of file mutations to bound memory requirements? 1 // FIXME: THIS IS FALSE!! delta can commit by getting committed version out of band, without 1 // FIXME: Possibly use SetFileInformationByHandle( file.native_handle(), FileEndOfFileInfo, ... ) instead 1 // TODO: Support FDB_API_ADDED in generate_asm.py and then this can be replaced with fdb_api_ptr_unimpl. 1 memset(static_cast(&p), 0, sizeof(Page)); // FIXME: unnecessary? 1 // FIXME: make sure what's persisted here is not affected by innocent changes elsewhere 2 // TODO Not sure if it's okay to do this here!! 1 // TODO filter old mutations won't be necessary, SS does it already 1 ** database. TODO: For xConnect(), it could verify that said tables exist. 1 // TODO: Is the right behavior? 1 // TODO: This probably is no longer possible as there is no ability to merge json objects with an 1 // FIXME: This code logic can be simplified. 1 // TODO: Better data structure 1 UNREACHABLE(); // FIXME: is this right? 1 // FIXME: expire from map after a delay when granule is revoked and the history is no longer needed 1 // FIXME: this will get more complicated with merges where this could potentially include partial 1 // TODO this is probably not correct for all KeySelector instances--we'll want to match with real behavior 1 // FIXME: probably want a dedicated exception like blob_range_too_old or something instead 1 // TODO revisit this check. there could be nodes in PTree that were inserted, but never updated. their 1 // FIXME: As part of reverting snapshot V1, make ignorePopUid a UID instead of string 3 // TODO: Is it fine to keep TransactionLineage::Operation::GetKeyValues here? 1 // FIXME: enable when debugMutation is active 1 // TODO rename wrong_shard_server error to wrong_cache_server 1 // TODO: Could a dispatched read try to write to page after it has been destroyed if this actor is cancelled? 2 // TODO: don't hardcode this path 1 // FIXME: Factor address parsing from coordinators, include, exclude 1 * TODO: A possible optimization here could be to only use existing btree leaf page boundaries as keys, 1 /* TODO: Is it safe to use Pager.dbFileSize here? */ 1 // TODO: Add SevError here 1 // FIXME: Actual locality! 1 // FIXME: create a load balance function for this code so future users of reply streams do not have 1 // FIXME: this code is pretty much identical to LoadBalance.h 1 // TODO: resize is zero-initializing the space we're about to overwrite, so do something else, which probably 1 init( MIN_SHARD_BYTES, 50000000 ); if( buggifySmallShards ) MIN_SHARD_BYTES = 40000; if (simulationMediumShards) MIN_SHARD_BYTES = 200000; //FIXME: data distribution tracker (specifically StorageMetrics) relies on this number being larger than the maximum size of a key value pair 1 // TODO: this is getting kind of large. Should try to split out this actor if it continues to grow? 1 // FIXME: this is re-implemented on the commit proxy to include a yield, any changes to this function should also 1 // TODO It's a synchronous function call as of now. Should it asynch? 1 // FIXME: This policy for calculating the cache pop version could end up popping recent data in the remote DC after 3 // FIXME: name and description will be the same unless the message is 'incorrect_cluster_file_contents', which is 1 // TODO: Implement someday 2 // TODO: error handling 1 // TODO may enable these later 1 // FIXME: Sometimes, connections to/from this process will explicitly close 1 // TODO: Batch to avoid large clear ranges? 1 // TODO why construct a new one and not just copy the old one and change proxies + id? 1 // FIXME: create a load balance function for this code so future users of reply streams do not have 1 // TODO might want to separate this out for valid values for range assignments vs read requests. Assignment conflict 1 // FIXME: flip order of {filetype, version} 1 // TODO: Client should add mapped keys to conflict ranges. 1 // TODO: Pop old requests whose version batch <= finishedBatch.get() 1 // TODO: ServerKnob should be removed ones all tooling scripts and usage is identified and updated accordingly 1 // TODO: Needs to keep this monitor's future. May use actorCollection 1 //FIXME: Low priority reads are disabled by assigning very high knob values, reduce knobs for 7.0 1 // TODO: Not Implemented yet 1 // TODO test that fails if we don't wait on tx->pendingReads() 1 BinaryWriter::toValue(1, Unversioned()); // FIXME: remove in 6.X, only needed for 5.2 backward compatibility 1 // FIXME: replace or augment this with logging on the proxy which tracks 1 // TODO: GetKeyValuesWithFlatMapRequest doesn't respect limit yet. 1 // version().subsequence will be zero) < FIXME: Can we lose this property? 1 // FIXME: One should be able to use SFINAE to choose between serialize and serialize_unversioned. 1 if (!client->failed || runOnFailedClients) { // TODO: Should we ignore some failures? 1 // FIXME: probably some threshold of a small percentage of the data is actually changed, where it makes sense to 1 // TODO do we need to add throttling for cache servers? Probably not 1 if (e.code() != error_code_broken_promise && // FIXME: Get rid of these broken_promise errors every time we are 1 // TODO could filter out delta files that don't intersect the key range being requested? 1 // TODO change back from SevError? 1 // TODO: add debugging info for when this is a failure 1 // FIXME: enable when debugKeyRange is active 1 // FIXME: MoveFileEx(..., MOVEFILE_WRITE_THROUGH) in thread? 1 // FIXME: filtering on key range != change feed range doesn't work 1 // TODO some sort of directory structure would be useful maybe? 1 // FIXME: improvement would be to add history of failed workers to assignment so it can try other ones first 1 // FIXME: iteration 1 // TODO: Revisit if we should have (self->inflightSendingReqs > 0 && self->inflightLoadingReqs > 0) 1 return 0; // TODO: Should this throw since the input isn't really valid? 1 DATABASE_UNREACHABLE, // FIXME: not detected 1 // FIXME: Better way to ensure that all client profile data has been flushed to the database 1 // FIXME: Is 'fileChangeHandle' signalling each time around the loop? 1 it.skip(key.getKey()); // TODO: or precondition? 1 // FIXME: This hack to bump satellite fault tolerance, is to make it consistent 1 // TODO: If this actor is used to send large files then combine the summing and packetization into a loop with a 1 // TODO : workaround to write this two together to make the code compact 1 // TODO: Catch error_code_process_behind and retry with 1 // FIXME: change all BlobManagerData* to Reference to avoid segfaults if core loop gets error 1 k); // FIXME: Make PTreeImpl::insert do this automatically (see also WriteMap.h FIXME) 1 // FIXME: start role for every tlog instance, rather that just for the shared actor, also use a 1 // TODO later 1 // TODO: If there was more to be fetched and we hit the limit before - possibly a case where data 1 min_of_n == nullptr ? minimum(n) : min_of_n; // minimum(n); //FIXME: reuse from before if already taken 1 // TODO This is the code for the context manager that appears at 1 ** TODO: Strangely, it is not possible to associate a column specifier 1 // FIXME: better way to do this than timer? 1 // TODO: revisit all const_cast usages 1 // FIXME: re-enable test! 1 // TODO refactor the "find snapshot file" logic to GranuleFiles? 1 // TODO: This should fully spill, but currently doing so will cause us to no longer update poppedVersion 3 key.setKey(maxKey); // maxKey is a KeyRef, but points to a LiteralStringRef. TODO: how can we ASSERT this? 1 // FIXME: use skip pointers here 1 // FIXME: enable when debugMutation is active 1 // FIXME: need for upgrades from 5.X to 6.0, remove once this upgrade path is no longer needed 1 // TODO : Rewrite related code using C api 1 // TODO: Avoid copying page bytes, but this is not trivial due to how pager checksums are currently handled. 1 // TODO: Support [[maybe_unused]] attribute for actors 2 // FIXME: This logic will fallback unnecessarily when usable dcs > 1 because it does not check all combinations of 1 // TODO (IPv6) Use IPv6? 1 // TODO: pick a reasonable byte limit instead of just piggy-backing 1 // FIXME: We cannot inject faults after renaming the file, because we could end up with two asyncFileNonDurable 1 // FIXME: remove when we no longer support upgrades from 5.X 1 // FIXME: implement 2 // returns (Optional(),0,0) //< FIXME: The returned rgen is not zero, but >= gen2! (Specification bug) 1 // FIXME: stop if one of the above phases throws an exception 1 // TODO: Add support for longer strings if necessary. 1 // TODO more efficient data structure besides std::map? PTree is unecessary since this isn't versioned, but some other 1 // FIXME: if best set is exhausted, do not peek remote servers 1 // TODO this should really be renamed "TSSComparison.cpp" 1 // FIXME: this really belongs in a header somewhere since it is actually used. 1 // TODO: use streaming range read 1 // FIXME: add this back in when load balancing works with local requests 1 // TODO: Should this be serial? 1 // FIXME: Do FlushFileBuffers in a worker thread (using g_network->createThreadPool)? 1 // TODO: Add support for longer vectors if necessary. 1 // FIXME: If range.begin=="" && !forward, return success? 2 bandwidthSample; // FIXME: iops and bandwidth calculations are not effectively tested, since they aren't 1 // TODO: Currently it is possible that extra configuration classes may be returned, we 1 // TODO knob for limit? 1 // FIXME: Randomize request order as in LegacyLogSystem? 1 // FIXME: We should be able to allocate larger magazine sizes here if we 1 // FIXME: range checking for length, here and in other serialize code 1 // TODO: Get rid of all of these confusing boolean flags 1 // FIXME: check that the set range is currently inactive, and that a revoked range is currently its own 1 // TODO: Also send information about highest seen version 1 u.skipLen = 0; // TODO: set this 1 // FIXME: temporarily code to test storage cache 1 // FIXME: Find a way to check that contentsSize is not greater than a certain limit. 1 // TODO add unit tests for assign/revoke range, especially version ordering 1 // FIXME: implement 2 // FIXME: enable when DEBUG_MUTATION is active 1 // TODO is it better to just pass empty metrics to estimated? 1 // TODO: This is abstraction breaking. This should probably be stored as a member, calculated once on construction 1 // TODO some sort of directory structure would be useful? 1 // TODO define the shuttingDown state of cache server 1 // FIXME: this whole thing 1 // TODO: could eventually add other types of metrics to report back to the manager here 1 // TODO These two methods (addHelp and getLong) are used by the methods 1 // TODO: Is this cancellation logic actually needed? 1 .error(e); // FIXME: consistency check does not retry in this case 1 // FIXME: we need spare capacity even when we're just going to be cancelling work via TEAM_HEALTHY 1 // shard->transferredVersion = batch->changes[0].version; //< FIXME: This obeys the documented properties, and 1 # TODO: This doesn't seem to be good style, but I couldn't find a better way so far 1 reportErrors(monitorServerDBInfo(cc, LocalityData(), db), "MonitorServerDBInfo")); // FIXME: Locality 1 // FIXME: better way to convert? 1 // TODO: trace error 1 int removedElements = 0; // deterministicRandom()->randomInt(0, numElements); // FIXME: the overhead factor 3 /* TODO: add other return types as we need them */ 1 // FIXME: This will not work correctly in simulation as all workers would share the same roles map 1 // FIXME: This copy is technically an asymptotic problem, definitely a waste of memory (copy of keyAfter 2 m.iosPerKSecond = 1; //FIXME: this should be 1/r.data.size(), but we cannot do that because it is an int 2 // FIXME: for a write-hot shard, we could potentially batch these and only pop the largest one after several 1 // FIXME: add readGuard again 1 unlimited, // TODO: rename to workload? 1 // TODO: For backward compatibility because request_timeout was renamed to request_timeout_min 1 // TODO: This is really slow! 1 // TODO: Detect if process is being restarted and only throw injected if so? 1 // FIXME: should this be other than 0? Should we care if this fails? 1 // TODO: double check. 1 // TODO: caching disabled for this merge 1 // TODO: replace with toml_parse_error(); 1 // TODO: is DefaultPromiseEndpoint the best priority for this? 2 // FIXME: Other stuff not filled in; why isn't this constructed here and passed down to the other 1 // TODO: read to begin, read through end flags for result 1 // TODO: Concurrency control? 1 // TODO: Ideally it should not be necessary to create a template object to parse knobs 1 // TODO: This is broken, it's not faster than not using a hint. See Make thisUnfortunately in a microbenchmark 1 // FIXME: This is conservative - not everything in system keyspace is necessarily processed by 1 // TODO: Test background error in simulation. RocksDB doesn't use flow IO in simulation, which limits our ability to 1 // FIXME: test that this uses the keyRange arena and doesn't create another one 2 // FIXME: Filtering after reading from disk is very inefficient 1 Version version; // FIXME: ? 1 // TODO how do we yield here? This may not be enough, because compact() does the heavy lifting 1 // FIXME: Is this really right? Weaken this and see if there is a problem; 1 m.iosPerKSecond = 1; //FIXME: this should be 1/r.data.size(), but we cannot do that because it is an 1 // TODO confirm: using reliable delivery this should only trigger if the worker is marked as failed, right? 1 // TODO: Initialize controllerData and all loaders and appliers' data for each restore request! 1 // FIXME: Reinitialize in order to update dependent knobs? 1 // TODO probably new exception type instead 1 // FIXME: could avoid max uid for granule ids here 1 // FIXME: currently validation works on numSnap = 1 1 int nDummy; /* TODO: Remove this */ 1 // FIXME: Simplify this function, such as removing unnecessary printf 1 // TODO check if the following is correct 1 // TODO: make this delay configurable 2 if (targetRateRatio < .75) //< FIXME: KNOB for 2.0 1 // TODO: Use a lower level API may be better? Or tweak priorities? 1 } // FIXME: Return a 'HumanReadableDescription' instead of std::string, make TraceEvent::detail accept that (for 1 // TODO what's this? 1 // TODO This method is in the recipe book. 2 // TODO Add cache metrics, such as available memory/in-use memory etc to help dat adistributor assign cached ranges 1 // TODO: What about when peerReference == -1? 1 // TODO: Add a counter to warn when too many requests are waiting on the actor 1 // TODO REMOVE sanity check eventually 1 // FIXME: implement actual proper file format for this 1 // TODO: Load balance over quorum. Also need to catch 1 // FIXME: The client cache does not notice when servers are added to a team. To read from a local storage server 1 # TODO: Make this fdb.tuple.subspace() or similar? 1 /* TODO: Could use temporary registers here. Also could attempt to 1 // TODO: Remove the need for this by changing usages to steal this's content 1 // FIXME: optimize to avoid creating new netNotifiedQueue for each message 1 // FIXME: wait for known committed version on the storage server before replying, 1 // TODO: Make a knob for this as well. 1 if (self->configuration.usableRegions > 1) { // FIXME: add support for generating random teams across DCs 1 // TODO: Add minPopped locality and minPoppedId, similar as tLog Metrics 1 // TODO since client request passes version, we don't need to include the version of each mutation in the response if we 1 // TODO: Could we use EmitNativeLoop() here? 1 // TODO: use notify to only send a heartbeat once per interval 1 // FIXME: This field was added in a patch release, for the 7.0 release move this to above loop 1 # FIXME: Unpack the overlapping packed block and try again 1 // FIXME: technically we can free up deltaArena here to reduce memory 1 // TODO: inject read error 1 // FIXME: Doesn't work as written with auto-reset 1 // FIXME: collectTags is needed to support upgrades from 5.X to 6.0. Remove this code when we no longer support 1 Key key; // TODO: Maybe not needed? 1 // TODO: track worker's reads/writes eventually 1 // FIXME: this is re-implementing the serialize function of MutationListRef in order to have a yield 1 // FIXME: A bunch of these can block with or without the following delay 0. 2 NOT_COORDINATORS, // FIXME: not detected 1 /* TODO: Maybe it shouldn't be possible to change the ReadUncommitted 1 // TODO These next two methods contain code that is referenced 1 // TODO: Create a TraceEventGroup that forwards all calls to each element of a std::vector, 1 // FIXME: retry on error if dispose fails? 1 // TODO: pSent is very hackish, do something better. 1 // TODO: We should make this dependent on the CPU. Maybe cmake 1 // FIXME: move this function to a common location 1 #include "fdbclient/BlobWorkerInterface.h" // TODO move the functions that depend on this out of here and into BlobWorkerInterface.h to remove this depdendency 1 // TODO: Shall we throw exception if the key doesn't exist or the range is empty? 1 // TODO: do we need this kind of wait? we are not going to make anything durable and hence no fear of wrong 1 // FIXME: temporarily code to test storage cache 1 // FIXME: optimize to avoid creating new netNotifiedQueueWithAcknowledgements for each stream duplication 1 // TODO is this easy to read as a flatbuffer from reader? Need to be sure about this data format 1 // FIXME: add remove support to localitySet so we do not have to recreate it 1 req, self, cx)); // TODO: Check how FDB uses TaskPriority for ACTORS. We may need to add 1 free); // TODO: we could potentially do better in the available field by accounting for the 1 // TODO: Is there a more efficient way to deal with invalidation? 2 // TODO: async in case it needs to read from other servers. 2 // TODO These two lines are the code in the recipe book to handle 1 // FIXME: Rename to InitializeMasterRequest, etc 1 // TODO: The timeout is a hack to get the test to pass before rollforward and 1 // FIXME: Above comment is not what this code block does, it actually just looks for a specific message 1 // FIXME: Shouldn't if logData.first isValid && !isReady, shouldn't we 1 // TODO: Support more complex patterns? 1 // FIXME: add the log_group in addition to the network address 1 // TODO revisit if we change the data structure of the VersionedMap 1 // TODO: Add more error types? 1 ** TODO: Technically the following is an error because it assumes that 1 //delete[] x; // TODO: Free large pages 1 // FIXME: log error 1 // FIXME: we cannot use one satellite replication with more than one satellite per region because 1 // TODO: Remove the number of invocations of operator<, and replace with something closer to memcmp. 1 // TODO: Improve this: 2 // TODO we should probably change this for cache server 1 // TODO what about otherError 2 // TODO: Combine with bindings/flow/Tuple.*. This code is copied from there. 1 // TODO: Do I need to read data to warm the cache of the keySystem like ReadWrite.actor.cpp (line 465)? 1 // FIXME: If remote DC is used as satellite then this logic only ensures that required number of remote 1 // FIXME: pass the deque back rather than copy the data 2 // FIXME: do this merge on one of the logs in the other data center to avoid sending multiple copies 2 // TODO: Make the actor priority lower than sendMutation priority. (Unsure it will help performance though) 1 // FIXME: Answer why this can happen? 1 // // TODO: Explain contract required for deltaSize to be used to predict final 1 // FIXME: the primary and remote can both be considered dead because excludes are not handled properly by the 1