Qri
Datastores
persisted data structures
Qri makes extensive use of the datastore pattern. isolating subsystems from storage concerns. Ideally, no business logic should exist within a datastore, it should only consist of data reading and writing. However, it's perfectly normal for datastore to handle bulk data operations at once (such as looking up a list of elements at once) for efficiency's sake.
Core Data Stores
name | primary data structure | tag |
---|---|---|
collection | VersionInfo | datasets |
workflowStore | Workflow | automation |
runStore | Run | automation |
keyStore | Key | authentication |
tokenStore | JWT (UCAN) | authentication |
profileStore | Profile | identity |
opStore | Oplog | sync |
config | Config | configuration |
qfs | files, folders, "JSON" | filesystem |
Cloud Data Stores
Names in bold have no corresponding store within core.
name | tag |
---|---|
PreviewDatastore | datasets |
AdminDatastore | configuration |
IssueDatastore | meta |
NotificationDatastore | meta |
MetricsDatastore | ops |
StatDatastore | datasets |
RefDatastore | datasets |
LogDatastore | datasets |
CollectionDatastore | datasets |
WorkflowDatastore | automation |
RunDatastore | automation |
UserDatastore | identity |
ProfileDatastore | identity |
APIClientDatastore | authentication |
OAuthDatastore | authentication |
KeyDatastore | authentication |
qfs | filesystem |
// Datastore abstracts the storage layer of cloud backend, isolating subsystems
// from storage concerns. Ideally, no business logic should exist within the
// datastore, it should only consist of data reading and writing. However, it's
// perfectly normal for datastore to handle bulk data operations at once (such
// as looking up a list of elements at once) for efficiency's sake.
type Datastore interface {
// PreviewDatastore is the substore for dataset previews
PreviewDatastore() PreviewDatastore
// StatDatastore is the substore for stats about datasets, like pull counts and views
StatDatastore() StatDatastore
// RefDatastore is the substore for reference storage and resolution
RefDatastore() RefDatastore
// LogDatasotre instance
LogDatastore() LogDatastore
// AdminDatasotre instance
AdminDatastore() AdminDatastore
// IssueDatastore instance
IssueDatastore() IssueDatastore
// NotificationDatastore instance
NotificationDatastore() NotificationDatastore
// MetricsDatastore instance
MetricsDatastore() MetricsDatastore
// APIClientDatastore instance
APIClientDatastore() APIClientDatastore
// OAuthDatastore instance
OAuthDatastore() OAuthDatastore
// UserDatastore instance
UserDatastore() UserDatastore
// CollectionDatastore instance
CollectionDatastore() CollectionDatastore
// WorkflowDatastore instance
WorkflowDatastore() WorkflowDatastore
// RunDatastore instance
RunDatastore() RunDatastore
// KeyDatastore instance
KeyDatastore() KeyDatastore
// ProfileDatastore instance
ProfileDatastore() ProfileDatastore
// ... additional methods elided
}
Frontend State tree
Instead of datastores, frontend data is structured into a single state tree that is mutated by calling a series of mutually-exclusive reducers that each update part of the root tree. Whenever it's practical we should aim to structure the state tree as a sparse selection that selectively populates from backend backend stores. While some elements of the state tree will be specific to frontend needs, much of the work of state tree maintenance is getting data from a backend datastore into the state tree. In those cases, we should aim to align types, API endpoints, and storage structures to re-use insights across codebases.
field | primary data structure | tag |
---|---|---|
app | ui | |
scroller | ui | |
websocket | ui | |
collection | datasets | |
commits | datasets | |
dataset | datasets | |
dsPreview | datasets | |
activityFeed | LogItem | automation |
deploy | automation | |
workflow | automation | |
userProfile | identity | |
session | authentication | |
search | meta | |
transfers | meta |
The following is an expanded view of the root state tree & associated types:
export interface RootState {
app: {
modal: Modal,
navExpanded: Boolean
},
scroller: {
scrollerPos: number
scrollAnchorID: string
}
websocket: {
status: WSConnectionStatus,
reconnectTime?: Date, // when disconnected, the time to wait until reconnecting
reconnectAttemptsRemaining?: number
}
activityFeed: {
datasetLogs: Record<string,LogItem[]>
},
collection: {
// collection is a record of all the workflow infos
collection: Record<string, VersionInfo>
// running contains the ids of the currently running workflows in reverse chronological
// order based on latestRunTime
running: string[]
collectionLoading: boolean
runningLoading: boolean
},
commits: {
commits: Record<string,LogItem[]>
loading: boolean
loadError: string
}
dataset: {
dataset: Dataset
loading: boolean
}
dsPreview: {
preview: Dataset
hasWorkflow: boolean
loading: boolean
}
deploy: {
status: Record<string,DeployStatus>
}
search: {
results: SearchResult[]
pageInfo: PageInfo
loading: boolean
}
session: {
token: string
refreshToken: string
}
transfers: Record<string, RemoteEvent>
userProfile: {
results: SearchResult[]
pageInfo: PageInfo
loading: boolean
}
workflow: {
runMode: RunMode
workflow: Workflow
// working dataset for editing transform steps, setting dataset name, etc
dataset: Dataset
// stores the "clean" state of triggers, steps, and hooks, used to compare
// with working state to determine isDirty
workflowBase: WorkflowBase
isDirty: boolean
lastDryRunID?: string
lastRunID?: string
events: EventLogLine[]
// state for the async request to /apply
applyStatus: ApplyStatus
}
edits: DatasetEditsState {
dataset: Dataset
hasEdits: boolean
// latest version
head?: Dataset
loading: boolean
}
}
One-of-many pattern
It's common for the frontend UI to display a single item from a homogenous backend datastore of many items. We call this the "one of many" pattern. By design, the frontend can only retain a single element that should be expired whenever the user navigates away from viewing the single element. The one-of-many pattern works well when:
- The frontend uses the singular form to describe a stored type
- The backend uses plural form to describe a store of many of the same type
- An API endpoint exists to select one element from the many store without prefetching anything other than a user token.