Qri

Visit qri.io

Datastores

persisted data structures


Qri makes extensive use of the datastore pattern. isolating subsystems from storage concerns. Ideally, no business logic should exist within a datastore, it should only consist of data reading and writing. However, it's perfectly normal for datastore to handle bulk data operations at once (such as looking up a list of elements at once) for efficiency's sake.

Core Data Stores

nameprimary data structuretag
collectionVersionInfodatasets
workflowStoreWorkflowautomation
runStoreRunautomation
keyStoreKeyauthentication
tokenStoreJWT (UCAN)authentication
profileStoreProfileidentity
opStoreOplogsync
configConfigconfiguration
qfsfiles, folders, "JSON"filesystem

Cloud Data Stores

Names in bold have no corresponding store within core.

nametag
PreviewDatastoredatasets
AdminDatastoreconfiguration
IssueDatastoremeta
NotificationDatastoremeta
MetricsDatastoreops
StatDatastoredatasets
RefDatastoredatasets
LogDatastoredatasets
CollectionDatastoredatasets
WorkflowDatastoreautomation
RunDatastoreautomation
UserDatastoreidentity
ProfileDatastoreidentity
APIClientDatastoreauthentication
OAuthDatastoreauthentication
KeyDatastoreauthentication
qfsfilesystem
cloud_datastore.go
// Datastore abstracts the storage layer of cloud backend, isolating subsystems
// from storage concerns. Ideally, no business logic should exist within the
// datastore, it should only consist of data reading and writing. However, it's
// perfectly normal for datastore to handle bulk data operations at once (such
// as looking up a list of elements at once) for efficiency's sake.
type Datastore interface {
	// PreviewDatastore is the substore for dataset previews
	PreviewDatastore() PreviewDatastore
	// StatDatastore is the substore for stats about datasets, like pull counts and views
	StatDatastore() StatDatastore
	// RefDatastore is the substore for reference storage and resolution
	RefDatastore() RefDatastore
	// LogDatasotre instance
	LogDatastore() LogDatastore
	// AdminDatasotre instance
	AdminDatastore() AdminDatastore
	// IssueDatastore instance
	IssueDatastore() IssueDatastore
	// NotificationDatastore instance
	NotificationDatastore() NotificationDatastore
	// MetricsDatastore instance
	MetricsDatastore() MetricsDatastore
	// APIClientDatastore instance
	APIClientDatastore() APIClientDatastore
	// OAuthDatastore instance
	OAuthDatastore() OAuthDatastore
	// UserDatastore instance
	UserDatastore() UserDatastore
	// CollectionDatastore instance
	CollectionDatastore() CollectionDatastore
	// WorkflowDatastore instance
	WorkflowDatastore() WorkflowDatastore
	// RunDatastore instance
	RunDatastore() RunDatastore
	// KeyDatastore instance
	KeyDatastore() KeyDatastore
	// ProfileDatastore instance
	ProfileDatastore() ProfileDatastore

    // ... additional methods elided
}

Frontend State tree

Instead of datastores, frontend data is structured into a single state tree that is mutated by calling a series of mutually-exclusive reducers that each update part of the root tree. Whenever it's practical we should aim to structure the state tree as a sparse selection that selectively populates from backend backend stores. While some elements of the state tree will be specific to frontend needs, much of the work of state tree maintenance is getting data from a backend datastore into the state tree. In those cases, we should aim to align types, API endpoints, and storage structures to re-use insights across codebases.

fieldprimary data structuretag
appui
scrollerui
websocketui
collectiondatasets
commitsdatasets
datasetdatasets
dsPreviewdatasets
activityFeedLogItemautomation
deployautomation
workflowautomation
userProfileidentity
sessionauthentication
searchmeta
transfersmeta

The following is an expanded view of the root state tree & associated types:

root.ts
export interface RootState {
  app: {
    modal: Modal,
    navExpanded: Boolean
  },
  scroller: {
    scrollerPos: number
    scrollAnchorID: string
  }
  websocket: {
    status: WSConnectionStatus,
    reconnectTime?: Date, // when disconnected, the time to wait until reconnecting
    reconnectAttemptsRemaining?: number
  }

  activityFeed: {
    datasetLogs: Record<string,LogItem[]>
  },
  collection: {
    // collection is a record of all the workflow infos
    collection: Record<string, VersionInfo>
    // running contains the ids of the currently running workflows in reverse chronological
    // order based on latestRunTime
    running: string[]
    collectionLoading: boolean
    runningLoading: boolean
  },
  commits: {
    commits: Record<string,LogItem[]>
    loading: boolean
    loadError: string
  }
  dataset: {
    dataset: Dataset
    loading: boolean
  }
  dsPreview:  {
    preview: Dataset
    hasWorkflow: boolean
    loading: boolean
  }
  deploy:  {
    status: Record<string,DeployStatus>
  }
  search: {
    results: SearchResult[]
    pageInfo: PageInfo
    loading: boolean
  }
  session: {
    token: string
    refreshToken: string
  }
  transfers: Record<string, RemoteEvent>
  userProfile: {
    results: SearchResult[]
    pageInfo: PageInfo
    loading: boolean
  }
  workflow: {
    runMode: RunMode
    workflow: Workflow
    // working dataset for editing transform steps, setting dataset name, etc
    dataset: Dataset
    // stores the "clean" state of triggers, steps, and hooks, used to compare
    // with working state to determine isDirty
    workflowBase: WorkflowBase
    isDirty: boolean
    lastDryRunID?: string
    lastRunID?: string
    events: EventLogLine[]
    // state for the async request to /apply
    applyStatus: ApplyStatus
  }
  edits: DatasetEditsState {
    dataset: Dataset
    hasEdits: boolean
    // latest version
    head?: Dataset
    loading: boolean
  }
}

One-of-many pattern

It's common for the frontend UI to display a single item from a homogenous backend datastore of many items. We call this the "one of many" pattern. By design, the frontend can only retain a single element that should be expired whenever the user navigates away from viewing the single element. The one-of-many pattern works well when:

  1. The frontend uses the singular form to describe a stored type
  2. The backend uses plural form to describe a store of many of the same type
  3. An API endpoint exists to select one element from the many store without prefetching anything other than a user token.