Sort the buffer beforehand instead of sorting it in the goroutine used for
writing the buffer to disk. The writeToDisk goroutines are throttled and
making it expensive causes other goroutines to block.

This change significantly improves the restore map phase.

This commit brings in sroar with some more optimizations like
cleanup of the bitmap, it also implements histogram in the sbs
tool, which will help in better comparison.

We store the groupId and userId in a predicate named dgraph.xid.There was a subtle bug where if a we create the group with same name as that of a user, then the user is not able to log in. This happens because we were not applying a filter by type.

This PR fixes that.

(cherry picked from commit 35538556)

Print final result only after all workers are done

Add sbs comparison tool, this tool can be used to validate an alpha server
against other alpha in terms of the data they contain and the query responses.

Bring the latest sroar to master.

No need to execute filter subgraph if there are no source UIDs.

(cherry picked from commit 849b587e)

* feat(restore): Introduce incremental restore (#7942)

This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.

(cherry picked from commit dfa5daec)

* fix: fixing graphql schema update when the data is restored + skipping /probe/graphql from audit (#7925)

* fix: fixing grapgql schema update when the data is restored

* making audit to skip /probe/graphql endpoint as this is health endpoint for kube

(cherry picked from commit b85036ef)

Co-authored-by: aman bansal <bansalaman2905@gmail.com>

We see frequent snapshots in an HA cluster because of the leadership re-election because of leader crash/shutdown. The local variable that we use for snapshot frequency is not synchronized across alphas. So each alpha thinks differently about the time the last snapshot was taken.

(cherry picked from commit 43055bf0)

If a Zero follower receives a JoinCluster request, it won't process the proposal because it can't forward the request to leader. Raft doesn't send an error back in this case. In the current code, it would retry indefinitely. With this change, we now use the request context to give up at some point.

Write rolled up keys at (max ts of the deltas + 1) because if we write
the rolled-up keys at the same ts as that of the delta, then in case of
WAL replay the rolled-up key would get over-written by the delta which
can bring DB to an invalid state.

Related to dgraph-io/dgraph-lambda#22

This PR changes the way lambda is handled by Dgraph. Earlier, the user had to start the server with a predefined lambda script. This does not scale well in a multi-tenant environment. As we will need N servers (or endpoints) to handle the load. Also, each time the script is changed, the lambda server needs to be restarted.

With this change, instead of loading the script from the predefined file, we will be sending the script in the request body to the lambda-server.

The details are mentioned in the discuss post. To summarize:

Storage
We will store the lambda script within dgraph.graphql.schema predicate. The value will be JSON marshal of

type GQL struct {
   Script:     String
   Schema: String
}
This change has been done in a backward-compatible manner.

New resolvers
This adds 2 new admin resolvers:

UpdateLambdaScript → accepts base64 encoded lambda script
GetLambdaScript → Returns base64 encoded script
Export/Import:
The GraphQL schema is exported in a separate file in the following JSON format:
[ExportedGQL] where ExportedGQL type is {Namespace: Int, Schema: String}. We will extend it to {Namespace: Int, Schema: String, Script: String} and hence the bulk loader can handle it properly.

JoinCluster loop was getting the connection from pool upfront, and then looping over it. This opened up a bug because in https://github.com/dgraph-io/dgraph/pull/7918 , we close the connection in case it becomes unhealthy.

This PR gets the latest connection available in the loop. This was the only place in the codebase where I found this issue.

For big datasets, we're seeing a big slowdown due to loading schema and types serially using a single iterator. Using the Stream framework, makes this metadata loading step much faster, resulting in a much faster Alpha initialization.

(cherry picked from commit d03d5ad1)

Co-authored-by: Manish R Jain <manish@dgraph.io>

The kv version should be set to restore timestamp for rolled-up keys
and schema keys as well.

The ForceFull parameter was not being passed in the backup request queue, 
causing all the backups to be incremental despite the request with `forceFull=True`.
This commit fixes this issue.

Earlier we were showing the opposite status: unbanned namespaces were shown as
banned. This change fixes that.

(cherry picked from commit 43a1bd0b09f157144a075ab872c7faae4b28abb4)

Co-authored-by: Daniel Mai <daniel@dgraph.io>

Add --only-summary flag in dgraph debug to show LSM tree and namespace sizes.

(cherry picked from commit cc871539)

Earlier, whenever the alpha starts(or restarts), we were upserting guardian and groot for all the namespaces. This is not actually needed.
The change was made in the PR #7759 to fix a bulk loader edge case.
This PR fixes that by generating the required RDFs in the bulk loader itself. Essentially, it inserts the ACL RDFs when force loading into non-Galaxy namespace.

In case the heartbeats in a connection pool stop, try to re-establish the connection via a redial. That's a more robust way than waiting for current connection to become usable again and allows much faster recovery from network partitions.

When streaming raft messages in k8s cluster, we don't seem to get an
error if the send didn't succeed. The packets get queued up, but don't
fail and don't get sent. This causes a long re-election process.

This PR periodically tries to send a message to the destination node via
IsPeer, so it has another way to test the connection. If that fails, the
streaming fails too, and the node is marked as unreachable.

We've been tracking this metric but didn't actually expose it in the metrics
view, so Prometheus metrics in /metrics never showed it.

This change updates sets the view properly so it shows up in /metrics.

We should consider the state of UIdMatrix (whether it was in the form of SortedUids
or bitmap) in order to set the result after pagination. Pagination with cascade is
handled separately, where it is possible to have no order.

Use roaring bitmaps in pb.List by default. We fall back to using sorted_uids 
when we need the list to be ordered. The APIs defined in the codec acts as
an abstraction layer that decides if sorted_uids or bitmap should be used. 
sorted_uids are used if it is present, otherwise, the bitmap is used.

Co-authored-by: Manish R Jain <manish@dgraph.io>

Fixes GQLSAAS-1236.
This PR optimizes `eq` filter GraphQL queries.
For ex:
The below DQL query:
```
query {
  queryPerson(filter: {nick: {eq: "Dgraph"}}){
    id
    name
    nick
  }
}
```
will now be written into:
```
query {
  queryPerson(func: eq(Person.nick, "Dgraph")) @filter(type(Person)) {
    Person.id : uid
    Person.name : Person.name
    Person.nick : Person.nick
  }
}
```
which was earlier written into:-
```
query {
  queryPerson(func: type(Person)) @filter(eq(Person.nick, "Dgraph")){
    Person.id : uid
    Person.name : Person.name
    Person.nick : Person.nick
  }
}
```

Fixes DGRAPH-3370.
This PR reverts the change related to `cascade with sort` which is currently causing panic on the master.

When restoring from backup taken on version <v21.03, we should append the galaxy namespace to the typename.

- fixing query timeouts for GraphQL queries too
- fixing online restore test

(cherry picked from commit bb0358e6)

compose.go changes

* fixing query timeouts for graphql queries too

* fixing online restore test

(cherry picked from commit bb0358e6)