-
Notifications
You must be signed in to change notification settings - Fork 26
perf: db unbundled blocks query optimization #204
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 99 commits
ce1370f
5af6fbc
dd1c081
116868a
70bb95e
7354012
2684357
5f7f6fb
e74403b
70ba25e
8ae9199
291c06c
abe9c6f
0c7142c
8762dcc
e45ba16
b3604a5
7c20526
a2b4fbf
e2fc9f0
2fd8d60
86ce42c
9c374e4
aca924d
d9a2434
9263c7d
9e2931f
dc4a837
dc12cb4
c0b379e
0a9d3a8
cc9b19f
b19955e
43fb7c5
4ac6b07
c0dd24e
bf7203d
6f4a71d
deb541d
9a41c53
327bd41
70ca57d
09eb541
65a30a3
0fc3990
3825c27
a851998
2769edc
2980c0d
e453f97
cfb6d23
c16f32e
be4e902
ea99414
bdb9544
c101d5c
d8c61ea
115e63d
d0e91cc
ee983a8
7aea795
d05c5e6
4470b25
45c0a1a
62a34ad
ced1880
3101549
9869f3a
d339a05
b427bb0
af06d79
87cd2a6
dcca551
ef2fa35
04cd76b
a7ae5cd
deb5ead
52d1700
1fa5f54
b9c017c
ca81705
a41b586
4686eb9
9196010
317dd2e
6698128
406e643
e7667d4
26f627b
e7c0b51
bb8da83
80ada8c
7a28ae3
2467def
3c5ab90
a88fbf4
ce20491
02efed7
db95a59
b117ad1
92176ce
6ffff3a
50b893e
c53ca9f
4c6abd3
85415bf
c06e4eb
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
This file was deleted.
This file was deleted.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -49,7 +49,7 @@ spec: | |
| httpGet: | ||
| path: /health | ||
| port: http | ||
| initialDelaySeconds: 10 | ||
| initialDelaySeconds: 60 | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Had to bump it because the migration is expected to take around 60s to complete.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am curios is there a way to know how long will be migration on the real database from devnet/testnet/mainnet?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I ran something similar to this: SELECT COUNT(DISTINCT fb.height)
FROM fuel_blocks fb
JOIN bundles b ON fb.height BETWEEN b.start_height AND b.end_height;on all of the databases. Longest execution time was around 50s. We have a pruning period set to 8w currently so the amount of entries in the dbs doesn't change. |
||
| periodSeconds: 5 | ||
| timeoutSeconds: 10 | ||
| resources: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| BEGIN; | ||
|
|
||
| -- 1. Add the column without enforcing NOT NULL initially | ||
| ALTER TABLE fuel_blocks | ||
| ADD COLUMN IF NOT EXISTS is_bundled BOOLEAN; | ||
|
||
|
|
||
| -- 2. Set is_bundled to true for blocks that fall within any bundle's range. | ||
| UPDATE fuel_blocks fb | ||
| SET is_bundled = true | ||
| FROM bundles b | ||
| WHERE fb.height BETWEEN b.start_height AND b.end_height; | ||
|
|
||
| -- 3. For blocks not updated above, set is_bundled to false | ||
| UPDATE fuel_blocks | ||
| SET is_bundled = false | ||
| WHERE is_bundled IS NULL; | ||
|
|
||
| -- 4. Make the column NOT NULL and set the default for future inserts. | ||
| ALTER TABLE fuel_blocks | ||
| ALTER COLUMN is_bundled SET NOT NULL, | ||
| ALTER COLUMN is_bundled SET DEFAULT false; | ||
|
|
||
| -- Create the composite index. | ||
| CREATE INDEX IF NOT EXISTS idx_fuel_blocks_is_bundled_height | ||
| ON fuel_blocks(is_bundled, height); | ||
|
|
||
| COMMIT; | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nit] Something like:
Could be more readable=D
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
c06e4eb