Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 0 additions & 22 deletions .github/workflows/node.js.yml

This file was deleted.

38 changes: 38 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Test and lint
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true

on:
push:
branches: [main]
pull_request:
branches: ["**"]

jobs:
check:
name: Test and lint
runs-on: ubuntu-latest
timeout-minutes: 30

steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5

- name: Node setup
uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5
with:
cache-dependency-path: package.json
node-version: "20.x"
cache: "npm"

- name: Install and build
run: |
npm i
npm run build
- name: Publish package for testing branch
run: npx pkg-pr-new publish || echo "Have you set up pkg-pr-new for this repo?"
- name: Test
run: |
npm run test
npm run typecheck
npm run lint
6 changes: 1 addition & 5 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,6 @@ dist-ssr
explorations
node_modules
.eslintcache
# components are libraries!
package-lock.json

# this is a package-json-redirect stub dir, see https://github.com/andrewbranch/example-subpath-exports-ts-compat?tab=readme-ov-file
react/package.json
# npm pack output
*.tgz
*.tsbuildinfo
3 changes: 0 additions & 3 deletions .prettierrc

This file was deleted.

4 changes: 4 additions & 0 deletions .prettierrc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"trailingComma": "all",
"proseWrap": "always"
}
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Changelog

## 0.2.0

- Adds /test and /\_generated/component.js entrypoints
- Drops commonjs support
- Improves source mapping for generated files
- Changes to a statically generated component API
40 changes: 40 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Developing guide

## Running locally

```sh
npm i
npm run dev
```

## Testing

```sh
npm run clean
npm run build
npm run typecheck
npm run lint
npm run test
```

## Deploying

### Building a one-off package

```sh
npm run clean
npm ci
npm pack
```

### Deploying a new version

```sh
npm run release
```

or for alpha release:

```sh
npm run alpha
```
78 changes: 38 additions & 40 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@

<!-- START: Include on https://convex.dev/components -->

This component adds counters to Convex. It acts as a key-value store from
string to number, with sharding to increase throughput when updating values.
This component adds counters to Convex. It acts as a key-value store from string
to number, with sharding to increase throughput when updating values.

Since it's built on Convex, everything is automatically consistent, reactive,
and cached. Since it's built with [Components](https://convex.dev/components),
Expand All @@ -18,8 +18,8 @@ For example, if you want to display
count the checkboxes in real-time while allowing a lot of the boxes to change in
parallel.

More generally, whenever you have a counter that is changing frequently, you
can use this component to keep track of it efficiently.
More generally, whenever you have a counter that is changing frequently, you can
use this component to keep track of it efficiently.

```ts
export const checkBox = mutation({
Expand All @@ -46,16 +46,17 @@ export const getCount = query({
```

This relies on the assumption that you need to frequently modify the counter,
but only need to read its value from a query, or infrequently in a mutation.
If you read the count every time you modify it, you lose the sharding benefit.
but only need to read its value from a query, or infrequently in a mutation. If
you read the count every time you modify it, you lose the sharding benefit.

## Pre-requisite: Convex

You'll need an existing Convex project to use the component.
Convex is a hosted backend platform, including a database, serverless functions,
and a ton more you can learn about [here](https://docs.convex.dev/get-started).
You'll need an existing Convex project to use the component. Convex is a hosted
backend platform, including a database, serverless functions, and a ton more you
can learn about [here](https://docs.convex.dev/get-started).

Run `npm create convex` or follow any of the [quickstarts](https://docs.convex.dev/home) to set one up.
Run `npm create convex` or follow any of the
[quickstarts](https://docs.convex.dev/home) to set one up.

## Installation

Expand All @@ -65,22 +66,22 @@ First, install the component package:
npm install @convex-dev/sharded-counter
```

Then, create a `convex.config.ts` file in your app's `convex/` folder and install the
component by calling `use`:
Then, create a `convex.config.ts` file in your app's `convex/` folder and
install the component by calling `use`:

```ts
// convex/convex.config.ts
import { defineApp } from "convex/server";
import shardedCounter from "@convex-dev/sharded-counter/convex.config";
import shardedCounter from "@convex-dev/sharded-counter/convex.config.js";

const app = defineApp();
app.use(shardedCounter);

export default app;
```

Finally, create a new `ShardedCounter` within your `convex/` folder, and point it to
the installed component.
Finally, create a new `ShardedCounter` within your `convex/` folder, and point
it to the installed component.

```ts
import { components } from "./_generated/api";
Expand Down Expand Up @@ -175,8 +176,8 @@ to get an accurate count. This takes a read dependency on all shard documents.
- In a mutation, that means any modification to the counter causes an
[OCC](https://docs.convex.dev/error#1) conflict.

You can reduce contention by estimating the count: read from a smaller number
of shards and extrapolate based on the total number of shards.
You can reduce contention by estimating the count: read from a smaller number of
shards and extrapolate based on the total number of shards.

```ts
const estimatedCheckboxCount = await counter.estimateCount(ctx, "checkboxes");
Expand All @@ -190,9 +191,9 @@ from more shards, at the cost of more contention:
const estimateFromThree = await counter.estimateCount(ctx, "checkboxes", 3);
```

If the counter was accumulated from many
small `counter.inc` and `counter.dec` calls, then they should be uniformly
distributed across the shards, so estimated counts will be accurate.
If the counter was accumulated from many small `counter.inc` and `counter.dec`
calls, then they should be uniformly distributed across the shards, so estimated
counts will be accurate.

In some cases the counter will not be evenly distributed:

Expand All @@ -201,8 +202,8 @@ In some cases the counter will not be evenly distributed:
values, because each operation only changes a single shard
- If the number of shards changed

In these cases, the count might not be evenly distributed across the shards.
To repair such cases, you can call:
In these cases, the count might not be evenly distributed across the shards. To
repair such cases, you can call:

```ts
await counter.rebalance(ctx, "checkboxes");
Expand All @@ -211,23 +212,21 @@ await counter.rebalance(ctx, "checkboxes");
Which will even out the count across shards.

You may change the number of shards for a key, by changing the second argument
to the `ShardedCounter` constructor. If you decrease the number of shards,
you will be left with extra shards that won't be written to but are still
read when computing `count`.
In this case, you should call `counter.rebalance` to delete
to the `ShardedCounter` constructor. If you decrease the number of shards, you
will be left with extra shards that won't be written to but are still read when
computing `count`. In this case, you should call `counter.rebalance` to delete
the extraneous shards.

NOTE: `counter.rebalance` reads and writes all shards, so it could cause
more OCCs, and it's recommended you call it sparingly, from the Convex dashboard
or from an infrequent cron.
NOTE: `counter.rebalance` reads and writes all shards, so it could cause more
OCCs, and it's recommended you call it sparingly, from the Convex dashboard or
from an infrequent cron.

NOTE: counts are floats, and floating point arithmetic isn't infinitely
precise. Even if you always add and subtract integers, you may get a fractional
counts, especially if you use `estimateCount` or `rebalance`.
Values distributed across shards may be added in different combinations, and
NOTE: counts are floats, and floating point arithmetic isn't infinitely precise.
Even if you always add and subtract integers, you may get a fractional counts,
especially if you use `estimateCount` or `rebalance`. Values distributed across
shards may be added in different combinations, and
[floating point arithmetic isn't associative](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html).
You can use `Math.round` to ensure your final count is an integer, if
desired.
You can use `Math.round` to ensure your final count is an integer, if desired.

## Counting documents in a table

Expand Down Expand Up @@ -262,8 +261,7 @@ async function insertUser(ctx, user) {
```

3. Register a [Trigger](https://www.npmjs.com/package/convex-helpers#triggers),
which automatically runs code when a mutation changes the
data in a table.
which automatically runs code when a mutation changes the data in a table.

```ts
// Triggers hook up writes to the table to the ShardedCounter.
Expand Down Expand Up @@ -312,9 +310,9 @@ Walkthrough of steps:
2. Create a new document in this table, with fields
`{ creationTime: 0, id: "", isDone: false }`
3. Wherever you want to update a counter based on a document changing, wrap the
update in a conditional, so it only gets updated if the backfill has processed
that document. In the example, you would be changing `insertUserBeforeBackfill`
to be implemented as `insertUserDuringBackfill`.
update in a conditional, so it only gets updated if the backfill has
processed that document. In the example, you would be changing
`insertUserBeforeBackfill` to be implemented as `insertUserDuringBackfill`.
4. Define backfill functions similar to `backfillUsers` and `backfillUsersBatch`
5. Call `backfillUsersBatch` from the dashboard.
6. Remove the conditional when updating counters. In the example, you would be
Expand Down
8 changes: 0 additions & 8 deletions commonjs.json

This file was deleted.

7 changes: 7 additions & 0 deletions convex.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"$schema": "./node_modules/convex/schemas/convex.schema.json",
"functions": "example/convex",
"codegen": {
"legacyComponentApi": false
}
}
Loading
Loading