Ruby dependency license scanning support via Gemfile.lock. #205

pboling · 2025-09-11T00:05:03Z

Implements [INFRA/eyes] Support for Ruby/Rails files skywalking#7744
Library projects (with a *.gemspec in the same directory as Gemfile.lock) ignore development dependencies and include runtime dependencies and their transitives.
App projects (no *.gemspec) include both runtime and development dependencies from Gemfile.lock.
Will only work if Gemfile.lock is committed to version control, but this is the official recommendation of RubyGems / Bundler:
- https://bundler.io/guides/faq.html#using-gemfiles-inside-gems
License resolution honors user overrides/exclusions and may query the RubyGems API when necessary, with proper support for handling of various status codes.
Documentation updated (README.md)
- Ruby setup and GitHub Actions example are in
  tag to reduce noise
- You can preview what the README will look like here: https://github.com/pboling/skywalking-eyes/blob/ruby-support/README.md

- Implements apache/skywalking#7744 - Library projects (with a *.gemspec in the same directory as Gemfile.lock) ignore development dependencies and include runtime dependencies and their transitives. - App projects (no *.gemspec) include both runtime and development dependencies from Gemfile.lock. - Will only work if Gemfile.lock is committed to version control, but this is the official recommendation of RubyGems: - https://bundler.io/guides/faq.html#using-gemfiles-inside-gems - License resolution honors user overrides/exclusions and may query the RubyGems API when necessary, with proper support for handling of various status codes. - Documentation updated (README.md) - Ruby setup and GitHub Actions example are in <details> tag to reduce noise

- Error: pkg/deps/ruby.go:289:15: httpNoBody: http.NoBody should be preferred to the nil request body (gocritic req, err := http.NewRequest(http.MethodGet, url, nil)

- Error: pkg/deps/ruby.go:284:1: cyclomatic complexity 24 of func `fetchRubyGemsLicenseFrom` is high (> 20) (gocyclo) func fetchRubyGemsLicenseFrom(url string) (string, error) { ^

kezhenxu94 · 2025-09-11T01:16:06Z

Thank you @pboling ! Looks like some files are missing license header themselves, causing the CI to fail. Can you add license headers to the newly added files ?

pboling · 2025-09-11T05:28:43Z

@kezhenxu94 Only one additional file in this PR was able to have a license header. .lock files do not have a format that allows license headers. I'm not sure if the remaining files need to be configured as excluded.

wu-sheng · 2025-09-11T05:30:12Z

If the header is not allowed, please add them to exclude list. Otherwise, CI will fail still.

- Error: pkg/deps/ruby.go:341:1: unnamedResult: consider giving a name to these results (gocritic) func handleRubyGemsResponse(resp *http.Response) (string, time.Duration, bool, error) {

pboling · 2025-09-11T06:08:53Z

I think it is done @wu-sheng @kezhenxu94

kezhenxu94 · 2025-09-11T06:16:15Z

pkg/deps/ruby.go

+	url := fmt.Sprintf("https://rubygems.org/api/v2/rubygems/%s/versions/%s.json", name, version)
+	licenseID, err := fetchRubyGemsLicenseFrom(url)
+	if err == nil && licenseID != "" {
+		return licenseID, nil
+	}
+	// Fallback to latest info
+	url = fmt.Sprintf("https://rubygems.org/api/v1/gems/%s.json", name)


Does gem package manager downloads the files to a local location like other package manager, if so we can read from the local file system.

Also, do we need to change the url to the one set in the Gemfile.lock? If users want to use they own registry?

remote: https://rubygems.org/

Does gem package manager downloads the files to a local location like other package manager, if so we can read from the local file system.

Yes, if the GHA workflow is setup with Ruby, and runs the bundle install to install all the dependencies. We could check them that way in most scenarios, but:

On some systems the gem package is split when it is installed, e.g., the gemspec may not exist at the project root, and with relevance to this use case, the split would make the installed dependencies more difficult to parse/walk through than the project's raw Gemfile.lock.

Downloading entire dependency packages, and doing the complete version solving routine, which requires downloading the entire gem index, may actually be significantly heavier load than API calls for each dependency.

This would also require that the project be in a good working state. Parsing agnostically like this doesn't require the project to be in any kind of serviceable state.

Also, do we need to change the url to the one set in the Gemfile.lock? If users want to use they own registry?

It is possible that they can use other registries... but

if they do, they are usually private

accessing private ones is far more complex

often involves an API token set in the ENV

which should never be in version control, and has no uniform naming scheme

these are edge cases

I would suggest that support for them be handled in further PRs by people who need that functionality (big corps can pay for things they need to use!)

So, https://rubygems.org is the canonical default, and we could read it from the Gemfile, but it is a lot of complexity for a tiny fraction of users who are not paying me anything.

I'm just trying to provide better than 80/20 functionality for the "rest of us", the open source maintainers.

@kezhenxu94 does that resolve the concern? Do you need more details?

kezhenxu94 · 2025-09-11T10:02:00Z

Thanks @pboling

pboling · 2025-09-13T05:45:20Z

@kezhenxu94 @wu-sheng @CodePrometheus I've tried setting this up, against main, in the oauth2 repo:
ruby-oauth/oauth2#676

I don't understand the output. There basically isn't any, and perhaps it exits with success because it hasn't found any dependency to check?

make: Leaving directory '/home/runner/work/_actions/apache/skywalking-eyes/main'
Run license-eye -v info -c .licenserc.yaml dependency check 
  license-eye -v info -c .licenserc.yaml dependency check 
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    GITHUB_TOKEN: 
INFO Loading configuration from file: .licenserc.yaml

And that's it. It exits successfully. The problem is, this library does have runtime dependencies that should have been found. I did initially have debugging / logging code in the PR, but it was flagged by the linter / compiler as bad, so I removed it.

What would be an ideal approach to debugging this? My preference would be that it die hard/loud on a failure to parse.

Another strange bit is in the Post Check there is a warning about a primary key not being generated:

Post job cleanup.
Post job cleanup.
/opt/hostedtoolcache/go/1.23.12/x64/bin/go env GOMODCACHE
/opt/hostedtoolcache/go/1.23.12/x64/bin/go env GOCACHE
/home/runner/go/pkg/mod
/home/runner/.cache/go-build
Primary key was not generated. Please check the log messages above for more errors or information

This is my first time doing anything with Go on GHA.

kezhenxu94 · 2025-09-13T06:30:39Z

@pboling looking into your PR, I found that the git dependency is lack of the spec and the version (after adding some logs).

~/workspace/oauth2 ❯ ~/workspace/skywalking-eyes/bin/darwin/license-eye -v debug d c
INFO Loading configuration from file: .licenserc.yaml 
DEBUG License header is: copyright copyright 2025 permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'software'), to deal in the software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the software, and to permit persons to whom the software is furnished to do so, subject to the following conditions: the above copyright notice and this permission notice shall be included in all copies or substantial portions of the software. the software is provided 'as is', without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. in no event shall the copyright holder or contributors be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the software. 
DEBUG Base name: Gemfile.lock                      
DEBUG Base name: Gemfile.lock                      
DEBUG Base name: Gemfile.lock                      
DEBUG Base name: Gemfile.lock                      
DEBUG Ruby: processing gem faraday-net_http: &{Name:faraday-net_http Version:3.4.1 Deps:[net-http]} 
DEBUG Ruby: processing gem snaky_hash: &{Name:snaky_hash Version:2.0.3 Deps:[hashie version_gem]} 
DEBUG Ruby: processing gem version_gem: &{Name:version_gem Version:1.1.9 Deps:[]} 
DEBUG Ruby: processing gem multi_xml: &{Name:multi_xml Version:0.7.2 Deps:[bigdecimal]} 
DEBUG Ruby: processing gem bigdecimal: &{Name:bigdecimal Version:3.2.3 Deps:[]} 
DEBUG Ruby: processing gem rack: &{Name:rack Version:3.2.1 Deps:[]} 
DEBUG Ruby: processing gem faraday: &{Name:faraday Version:2.13.4 Deps:[faraday-net_http json logger]} 
DEBUG Ruby: processing gem net-http: &{Name:net-http Version:0.6.0 Deps:[uri]} 
DEBUG Ruby: processing gem json: &{Name:json Version:2.13.2 Deps:[]} 
DEBUG Ruby: processing gem hashie: &{Name:hashie Version:5.0.0 Deps:[]} 
DEBUG Ruby: processing gem git: <nil>              
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x39d0be9]

goroutine 1 [running]:
github.com/apache/skywalking-eyes/pkg/deps.(*GemfileLockResolver).Resolve(0x0?, {0xc0001221e0, 0x2f}, 0xc0004d0cb0, 0xc0005920f0)
        /Users/kezhenxu94/workspace/skywalking-eyes/pkg/deps/ruby.go:98 +0x329
github.com/apache/skywalking-eyes/pkg/deps.Resolve(0xc0004d0cb0, 0xc0005920f0)
        /Users/kezhenxu94/workspace/skywalking-eyes/pkg/deps/resolve.go:45 +0xa7
github.com/apache/skywalking-eyes/pkg/deps.Check({0xc00011e740, 0x3}, 0xc0004d0cb0, 0x0)
        /Users/kezhenxu94/workspace/skywalking-eyes/pkg/deps/check.go:72 +0xa5
github.com/apache/skywalking-eyes/commands.init.func1(0xc0004d0a00?, {0x39e831d?, 0x4?, 0x39e8321?})
        /Users/kezhenxu94/workspace/skywalking-eyes/commands/deps_check.go:46 +0xbc
github.com/spf13/cobra.(*Command).execute(0x4815380, {0xc00032bd20, 0x2, 0x2})
        /Users/kezhenxu94/go/pkg/mod/github.com/spf13/[email protected]/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0x4815f00)
        /Users/kezhenxu94/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/kezhenxu94/go/pkg/mod/github.com/spf13/[email protected]/command.go:968
github.com/apache/skywalking-eyes/commands.Execute()
        /Users/kezhenxu94/workspace/skywalking-eyes/commands/root.go:62 +0x151
main.main()
        /Users/kezhenxu94/workspace/skywalking-eyes/cmd/license-eye/main.go:28 +0x13

pboling · 2025-09-13T06:35:23Z

@kezhenxu94 Yes, I'm seeing that failure now. It should handle that gracefully, right?

kezhenxu94 · 2025-09-13T06:38:29Z

@kezhenxu94 Yes, I'm seeing that failure now. It should handle that gracefully, right?

Yes, depends on how we want to deal with the dependencies that are missing version/license info, typically we will add it to the list of unknown, so users can check it manually, and set it in the dependency.licenses, like https://github.com/apache/skywalking/blame/79860ca5c76a77bbd93e76ce4861b24707dd5ee3/.licenserc.yaml#L86

kezhenxu94 · 2025-09-13T06:38:45Z

@pboling can you open a PR to address that?

pboling · 2025-09-13T06:40:55Z

My last comment wasn't clear - what I mean is I think the error should be trapped, and it should track the list of dependencies with missing licenses, and if the list of missing licenses, or incompatible licenses is non-empty, it should fail.

However, thinking about this a bit more - the git gem is an optional, development-only dependency, and it was my intent that it not be relevant to a check like this, since oauth2 is a library. So I messed up the logic somewhere.

pboling · 2025-09-13T07:29:56Z

@kezhenxu94 First fix: #207

pboling · 2025-09-13T08:08:38Z

Second fix #208

wu-sheng requested review from CodePrometheus and kezhenxu94 September 11, 2025 00:14

pboling added 2 commits September 10, 2025 18:19

Address lint error

d74e85e

- Error: pkg/deps/ruby.go:289:15: httpNoBody: http.NoBody should be preferred to the nil request body (gocritic req, err := http.NewRequest(http.MethodGet, url, nil)

Address lint error

02855a0

- Error: pkg/deps/ruby.go:284:1: cyclomatic complexity 24 of func `fetchRubyGemsLicenseFrom` is high (> 20) (gocyclo) func fetchRubyGemsLicenseFrom(url string) (string, error) { ^

License in file header

76db177

pboling added 3 commits September 10, 2025 23:35

Fix markdown HTML formatting

96af87f

Address lint error

7be2d58

- Error: pkg/deps/ruby.go:341:1: unnamedResult: consider giving a name to these results (gocritic) func handleRubyGemsResponse(resp *http.Response) (string, time.Duration, bool, error) {

Ignore gemfile.lock files for license headers

d992b25

CodePrometheus approved these changes Sep 11, 2025

View reviewed changes

kezhenxu94 reviewed Sep 11, 2025

View reviewed changes

kezhenxu94 merged commit 3655e78 into apache:main Sep 11, 2025
1 check passed

wu-sheng added this to the 0.8.0 milestone Sep 11, 2025

wu-sheng added the feature label Sep 11, 2025

pboling deleted the ruby-support branch September 12, 2025 00:14

Ruby dependency license scanning support via Gemfile.lock. #205

Ruby dependency license scanning support via Gemfile.lock. #205

Uh oh!

Conversation

pboling commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kezhenxu94 commented Sep 11, 2025

Uh oh!

pboling commented Sep 11, 2025

Uh oh!

wu-sheng commented Sep 11, 2025

Uh oh!

pboling commented Sep 11, 2025

Uh oh!

kezhenxu94 Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

pboling Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pboling Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kezhenxu94 commented Sep 11, 2025

Uh oh!

pboling commented Sep 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kezhenxu94 commented Sep 13, 2025

Uh oh!

pboling commented Sep 13, 2025

Uh oh!

kezhenxu94 commented Sep 13, 2025

Uh oh!

kezhenxu94 commented Sep 13, 2025

Uh oh!

pboling commented Sep 13, 2025

Uh oh!

pboling commented Sep 13, 2025

Uh oh!

pboling commented Sep 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pboling commented Sep 11, 2025 •

edited

Loading

pboling Sep 11, 2025 •

edited

Loading

pboling commented Sep 13, 2025 •

edited

Loading