Skip to content

Conversation

@lunny
Copy link
Member

@lunny lunny commented Jun 8, 2025

For git version 2.36, git cat-file --batch-command was introduced which can replace git cat-file --batch and git cat-file --batch-check.

This PR implements an abstract layer for the batch commands so that both git 2.36 and lower version git can work.
If git version is lower than 2.36, it will start two subprocesses git cat-file --batch and git cat-file --batch-check.
If git version is greater than 2.36, only git cat-file --batch-command will be started.

This reduced half of child processes of git catfiles.

@lunny lunny added the type/refactoring Existing code has been cleaned up. There should be no new functionality. label Jun 8, 2025
@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jun 8, 2025
@github-actions github-actions bot added the modifies/go Pull requests that update Go code label Jun 8, 2025
@lunny lunny added this to the 1.25.0 milestone Jun 8, 2025
@lunny lunny closed this Jun 12, 2025
@GiteaBot GiteaBot removed this from the 1.25.0 milestone Jun 12, 2025
@lunny lunny reopened this Jun 12, 2025
@lunny lunny force-pushed the lunny/catfile_batch_refactor branch from f0b6480 to 334eb9e Compare June 18, 2025 17:15
@lunny lunny added this to the 1.25.0 milestone Jun 18, 2025
@wxiaoguang wxiaoguang removed this from the 1.25.0 milestone Aug 29, 2025
@lunny lunny added this to the 1.26.0 milestone Sep 29, 2025
@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Oct 20, 2025
@wxiaoguang wxiaoguang marked this pull request as draft October 20, 2025 23:33
@lunny lunny marked this pull request as ready for review October 25, 2025 17:10
@lunny
Copy link
Member Author

lunny commented Oct 25, 2025

It's ready to review again.

wxiaoguang
wxiaoguang previously approved these changes Oct 27, 2025
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Oct 27, 2025
@wxiaoguang wxiaoguang enabled auto-merge (squash) October 27, 2025 00:20
@wxiaoguang wxiaoguang disabled auto-merge October 27, 2025 00:58
@wxiaoguang
Copy link
Contributor

wxiaoguang commented Oct 27, 2025

Saw many flaky tests:


--- FAIL: TestPullCreate (0.71s)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed GET /user/login for test-mock:12345, 200 OK in 7.8ms @ auth/auth.go:184(auth.SignIn)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed POST /user/login for test-mock:12345, 303 See Other in 9.8ms @ auth/auth.go:197(auth.SignInPost)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed GET /user1/repo1 for test-mock:12345, 404 Not Found in 3.4ms @ context/repo.go:417(context.RepoAssignment)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed GET /user2/repo1 for test-mock:12345, 200 OK in 86.0ms @ repo/view_home.go:367(repo.Home)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed GET /user2/repo1/fork for test-mock:12345, 200 OK in 27.3ms @ repo/fork.go:126(repo.Fork)
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed POST /user2/repo1/fork for test-mock:12345, 500 Internal Server Error in 22.5ms @ repo/fork.go:137(repo.ForkPost)
    repo_fork_test.go:54: Response length:  2439
    repo_fork_test.go:54: 
        	Error Trace:	/home/runner/work/gitea/gitea/tests/integration/integration_test.go:354
        	            				/home/runner/work/gitea/gitea/tests/integration/integration_test.go:160
        	            				/home/runner/work/gitea/gitea/tests/integration/repo_fork_test.go:54
        	            				/home/runner/work/gitea/gitea/tests/integration/pull_create_test.go:138
        	            				/home/runner/work/gitea/gitea/tests/integration/git_helper_for_declarative_test.go:88
        	            				/home/runner/work/gitea/gitea/tests/integration/pull_create_test.go:136
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 500
        	Test:       	TestPullCreate
        	Messages:   	Request: POST /user2/repo1/fork
    testlogger.go:134: Flushing queues failed with error context canceled
=== TestPullCreate_TitleEscape (tests/integration/pull_create_test.go:164)
--- FAIL: TestPullCreate_TitleEscape (0.64s)
    git_helper_for_declarative_test.go:62: 
        	Error Trace:	/home/runner/work/gitea/gitea/tests/test_utils.go:236
        	            				/home/runner/work/gitea/gitea/tests/integration/git_helper_for_declarative_test.go:62
        	            				/home/runner/work/gitea/gitea/tests/integration/pull_create_test.go:164
        	Error:      	Received unexpected error:
        	            	failed to generate sequence update: context canceled
        	Test:       	TestPullCreate_TitleEscape
    testlogger.go:61: 2025/10/27 00:39:18 HTTPRequest [I] router: completed GET /user/login for test-mock:12345, 200 OK in 8.0ms @ auth/auth.go:184(auth.SignIn)
    testlogger.go:61: 2025/10/27 00:39:19 HTTPRequest [I] router: completed POST /user/login for test-mock:12345, 303 See Other in 10.1ms @ auth/auth.go:197(auth.SignInPost)
    testlogger.go:61: 2025/10/27 00:39:19 HTTPRequest [I] router: completed GET /user1/repo1 for test-mock:12345, 404 Not Found in 4.0ms @ context/repo.go:417(context.RepoAssignment)
    testlogger.go:61: 2025/10/27 00:39:19 HTTPRequest [I] router: completed GET /user2/repo1 for test-mock:12345, 200 OK in 86.5ms @ repo/view_home.go:367(repo.Home)
    testlogger.go:61: 2025/10/27 00:39:19 HTTPRequest [I] router: completed GET /user2/repo1/fork for test-mock:12345, 200 OK in 31.8ms @ repo/fork.go:126(repo.Fork)
    testlogger.go:61: 2025/10/27 00:39:19 HTTPRequest [I] router: completed POST /user2/repo1/fork for test-mock:12345, 500 Internal Server Error in 27.8ms @ repo/fork.go:137(repo.ForkPost)
    repo_fork_test.go:54: Response length:  2439
    repo_fork_test.go:54: 
        	Error Trace:	/home/runner/work/gitea/gitea/tests/integration/integration_test.go:354
        	            				/home/runner/work/gitea/gitea/tests/integration/integration_test.go:160
        	            				/home/runner/work/gitea/gitea/tests/integration/repo_fork_test.go:54
        	            				/home/runner/work/gitea/gitea/tests/integration/pull_create_test.go:166
        	            				/home/runner/work/gitea/gitea/tests/integration/git_helper_for_declarative_test.go:88
        	            				/home/runner/work/gitea/gitea/tests/integration/pull_create_test.go:164
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 500
        	Test:       	TestPullCreate_TitleEscape
        	Messages:   	Request: POST /user2/repo1/fork
    testlogger.go:134: Flushing queues failed with error context canceled
=== TestPullBranchDelete (tests/integration/pull_create_test.go:226)

=== TestAPIChangeFiles (tests/integration/api_repo_files_change_test.go:63)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:142 +0x447
goroutine 692 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc002960620)
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/util/buffer_pool.go:206 +0xc8
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 691
	/home/runner/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/util/buffer_pool.go:237 +0x17a
goroutine 1179514 [sync.WaitGroup.Wait, 10 minutes]:
sync.runtime_SemacquireWaitGroup(0xc0047158c0?, 0x0?)
	/opt/hostedtoolcache/go/1.25.3/x64/src/runtime/sema.go:114 +0x2e
sync.(*WaitGroup).Wait(0xc001f30320)
	/opt/hostedtoolcache/go/1.25.3/x64/src/sync/waitgroup.go:206 +0x85
code.gitea.io/gitea/tests/integration.TestRepoCommitsStatusParallel(0xc0085fddc0)
	/home/runner/work/gitea/gitea/tests/integration/repo_commits_test.go:208 +0x2aa
testing.tRunner(0xc0085fddc0, 0x40e0668)
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1934 +0xea
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1997 +0x465
goroutine 653866 [select, 16 minutes]:
code.gitea.io/gitea/modules/log.(*EventWriterBaseImpl).Run(0xc00806b860, {0x457a4f8, 0xc0040b94c0})
	/home/runner/work/gitea/gitea/modules/log/event_writer_base.go:81 +0x279
code.gitea.io/gitea/modules/log.eventWriterStartGo.func1()
	/home/runner/work/gitea/gitea/modules/log/event_writer_base.go:157 +0xa5
created by code.gitea.io/gitea/modules/log.eventWriterStartGo in goroutine 653864
	/home/runner/work/gitea/gitea/modules/log/event_writer_base.go:153 +0x191

@wxiaoguang wxiaoguang marked this pull request as draft October 27, 2025 01:26
@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch from dc7576a to d318098 Compare October 29, 2025 13:08
@wxiaoguang
Copy link
Contributor

No easy fix unless rewrite more legacy code

Do you have the determination to continue.

// FIXME: for debugging purpose only
// At the moment, the old logic (debugTestAlwaysNewBatch=false: reuse the existing batch if not in use)
// causes random test failures: it makes the `t.Context()` occasionally canceled with unknown reasons.
// In theory, the `t.Context()` should never be affected by testing code and can never be canceled, but it does happen.
// The stranger thing is that the failure tests are almost around TestAPIPullUpdateByRebase,
// it almost are during MSSQL testing, sometimes PGSQL, never others.

@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch from 095d622 to a686188 Compare October 29, 2025 16:15
@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch from e73929b to 751ccd0 Compare October 30, 2025 04:44
@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch from 751ccd0 to 533687c Compare October 30, 2025 04:45
@github-actions github-actions bot added the modifies/api This PR adds API routes or modifies them label Oct 30, 2025
@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch from 44243e5 to e7e9a04 Compare October 30, 2025 05:46
@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch 6 times, most recently from 731c2a4 to 9b89692 Compare October 30, 2025 08:24
@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch from 9b89692 to 2bd1873 Compare October 30, 2025 08:47
@wxiaoguang wxiaoguang force-pushed the lunny/catfile_batch_refactor branch from f266926 to 6394d98 Compare October 30, 2025 10:06
@wxiaoguang
Copy link
Contributor

Give up. Old branch is saved at https://github.com/lunny/gitea/tree/lunny/catfile_batch_refactor-old , but it can't be merged because there are unclear random CI failures.

Even I have rewritten from scratch by a simpler approach, the unclear random failure still exists.

I have traced down the problem to GetRefCommitID. If we force it to always use a new --batch process (but not reuse the previous created one), then CI can always succeed. However:

  1. Keep creating new process for GetRefCommitID is not right.
  2. Don't know the root cause, no clue what happens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm/need 1 This PR needs approval from one additional maintainer to be merged. modifies/api This PR adds API routes or modifies them modifies/go Pull requests that update Go code modifies/internal performance/cpu type/refactoring Existing code has been cleaned up. There should be no new functionality.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants